Formally verify software at large scale with Claude
Chapter 01: Introduction. Task: Chapter 01: Introduction PublishedOct 22, 2026 Published Oct 22, 2026 TeamFrontier Red Team Team Frontier Red Team TopicsCybersecurityCodingOpen sourcePartnershipsSee more... Topics Cybersecurity Coding Open source Partnerships Frontier Red TeamOct 20, 2026Using Claude to reduce the attack surface of code Frontier Red Team Oct 20, 2026 Keane Lucas, Aengus Lynch, Luna Tong, Luke Champine, Zygi Straznickas, Dino Distefano, Harshikaa AgrawalEvyatar Ben Asher, Ava Howell, Sami Jawhar, Chris Rohlf, Nickolai ZeldovichNicholas Carlini, Rajashree Agrawal, Jason Gross Most software is riddled with bugs, which AI can now easily find and exploit. The standard defenses against cyberattacks are no longer sufficient: we need ways to prove that code is free of bugs. Formal software verification could provide this guarantee, but it has historically required years of expert effort, making it impractical for most real-world projects. In a collaboration with Theorem, however, we show this is no longer the case: LLMs can now produce formal proofs at large scale, enabling new software development workflows that include proven security guarantees. Using Claude, we proved security properties at the assembly code level (RISC-V) in over 100 real-world open-source software libraries. To our knowledge, this represents the first large-scale effort to use AI to apply formal verification to production software at the assembly code level. While writing the proofs, Claude also discovered bugs that had eluded detection for years, despite trillions of test runs. As model capabilities continue to improve, we expect formal verification to become an essential part of secure software development. Introduction Even mature, heavily tested software contains bugs, and LLMs are increasingly adept at finding them. The Linux kernel—one of the most well-tested pieces of software today, deployed on billions of devices worldwide—recently had over 1,000 vulnerabilities disclosed. With frontier models becoming more capable of chaining seemingly benign bugs into critical exploits, we need techniques that can detect and fix these bugs before they can be exploited in the wild. Ideally, such techniques could find and patch all bugs, and then provide some proof that there are no more to be found. Formal verification is one way to do this. In formal verification, developers translate their code into a mathematical model, then try to prove that it always follows some defined set of security rules. If they fail to prove this, that means there could be a bug in the code that would violate a rule, and the nature of the failure implies what the bug might be. Repeating this process (after eliminating the cause of failure each time) will ultimately yield proof that the code is devoid of any bug that would violate those rules. However, formal verification isn’t easy. Proofs can be many times larger than the original code, and writing them requires significant effort and a niche, highly specialized skill set. For instance, it took two person-years to implement the seL4 operating system kernel, but 20 person-years to formally verify it. Accordingly, formal verification has been practical only in truly mission-critical applications, such as Airbus’s avionics code. But LLMs are making it far easier. Frontier models can now produce correctness proofs for real-world software fully autonomously. To demonstrate this, we asked Claude to prove useful security properties across 123 open-source libraries, including showing that some of their executable artifacts can never crash under certain conditions. Specifically, Claude proved that these programs will always exit normally without triggering memory-safety errors or failing assertions, for every input less than 64 MB and every modeled Linux operating system (OS) response and memory layout, under certain assumptions (namely that the program runs as a single-threaded static binary with a fixed memory budget on a RISC-V1 CPU using Linux OS). This is still far away from fixing all possible bugs, but it does get us much closer than before and provides a roadmap toward that end. We’re publishing a subset of these proofs so that anyone can verify and reuse them, along with the RISC-V and OS semantics. Theorem, our collaborator on this project, is also offering to run the same pipeline for other open-source projects, and is publishing a Claude skill to help developers get started with verifying their own code.; tools: Claude. Human role: The original post does not say how a person stayed involved.; limitations: Formal verification proofs can consume hundreds of millions of tokens, making cost a significant barrier. The current approach assumes single-threaded, static RISC-V binaries, which differs from typical dynamic x86/ARM deployments. 'Formally verified' does not mean bug-free; it only guarantees compliance with the specific model and specification, leaving gaps if the model diverges from real-world hardware or OS behavior..
Tools
Published on Agent Work Atlas:
Overview
Chapter 01: Introduction
Workflow
Chapter 01: Introduction PublishedOct 22, 2026 Published Oct 22, 2026 TeamFrontier Red Team Team Frontier Red Team TopicsCybersecurityCodingOpen sourcePartnershipsSee more... Topics Cybersecurity Coding Open source Partnerships Frontier Red TeamOct 20, 2026Using Claude to reduce the attack surface of code Frontier Red Team Oct 20, 2026 Keane Lucas, Aengus Lynch, Luna Tong, Luke Champine, Zygi Straznickas, Dino Distefano, Harshikaa AgrawalEvyatar Ben Asher, Ava Howell, Sami Jawhar, Chris Rohlf, Nickolai ZeldovichNicholas Carlini, Rajashree Agrawal, Jason Gross Most software is riddled with bugs, which AI can now easily find and exploit. The standard defenses against cyberattacks are no longer sufficient: we need ways to prove that code is free of bugs. Formal software verification could provide this guarantee, but it has historically required years of expert effort, making it impractical for most real-world projects. In a collaboration with Theorem, however, we show this is no longer the case: LLMs can now produce formal proofs at large scale, enabling new software development workflows that include proven security guarantees. Using Claude, we proved security properties at the assembly code level (RISC-V) in over 100 real-world open-source software libraries. To our knowledge, this represents the first large-scale effort to use AI to apply formal verification to production software at the assembly code level. While writing the proofs, Claude also discovered bugs that had eluded detection for years, despite trillions of test runs. As model capabilities continue to improve, we expect formal verification to become an essential part of secure software development. Introduction Even mature, heavily tested software contains bugs, and LLMs are increasingly adept at finding them. The Linux kernel—one of the most well-tested pieces of software today, deployed on billions of devices worldwide—recently had over 1,000 vulnerabilities disclosed. With frontier models becoming more capable of chaining seemingly benign bugs into critical exploits, we need techniques that can detect and fix these bugs before they can be exploited in the wild. Ideally, such techniques could find and patch all bugs, and then provide some proof that there are no more to be found. Formal verification is one way to do this. In formal verification, developers translate their code into a mathematical model, then try to prove that it always follows some defined set of security rules. If they fail to prove this, that means there could be a bug in the code that would violate a rule, and the nature of the failure implies what the bug might be. Repeating this process (after eliminating the cause of failure each time) will ultimately yield proof that the code is devoid of any bug that would violate those rules. However, formal verification isn’t easy. Proofs can be many times larger than the original code, and writing them requires significant effort and a niche, highly specialized skill set. For instance, it took two person-years to implement the seL4 operating system kernel, but 20 person-years to formally verify it. Accordingly, formal verification has been practical only in truly mission-critical applications, such as Airbus’s avionics code. But LLMs are making it far easier. Frontier models can now produce correctness proofs for real-world software fully autonomously. To demonstrate this, we asked Claude to prove useful security properties across 123 open-source libraries, including showing that some of their executable artifacts can never crash under certain conditions. Specifically, Claude proved that these programs will always exit normally without triggering memory-safety errors or failing assertions, for every input less than 64 MB and every modeled Linux operating system (OS) response and memory layout, under certain assumptions (namely that the program runs as a single-threaded static binary with a fixed memory budget on a RISC-V1 CPU using Linux OS). This is still far away from fixing all possible bugs, but it does get us much closer than before and provides a roadmap toward that end. We’re publishing a subset of these proofs so that anyone can verify and reuse them, along with the RISC-V and OS semantics. Theorem, our collaborator on this project, is also offering to run the same pipeline for other open-source projects, and is publishing a Claude skill to help developers get started with verifying their own code.
Reported result
Claude proved no-crash properties for 239 entry points across 123 libraries, discovering hundreds of bugs including long-standing vulnerabilities. In a separate experiment, an AI-written Python interpreter matched CPython behavior in 99.95% of 1.7 million tests. Newer models like Opus 5.5 required 40% fewer tokens than Opus 5, and shared workspaces reduced token usage by 17% compared to solo attempts.
Human role
The original post does not say how a person stayed involved.
Limitations
Formal verification proofs can consume hundreds of millions of tokens, making cost a significant barrier. The current approach assumes single-threaded, static RISC-V binaries, which differs from typical dynamic x86/ARM deployments. 'Formally verified' does not mean bug-free; it only guarantees compliance with the specific model and specification, leaving gaps if the model diverges from real-world hardware or OS behavior.
Try it
I want to replicate the formal verification workflow for a small open-source C library. First, select a library with existing fuzz tests and compile it to RISC-V assembly using a static musl libc link. Use Claude to convert the fuzz test entry points into formal specifications asserting no crashes for inputs under 64 MB. Instruct Claude to generate Rocq proofs for these specs. If proofs fail, have Claude identify the bug, patch the source code, and regenerate the spec. Run a dry check on one entry point to verify the proof checker accepts the output before scaling to more functions. Ensure I review the trusted components (RISC-V semantics and proof checker) before accepting any security guarantees.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.