LLMs automate formal verification for open-source libraries at scale

Software developers have long struggled with the high cost of formal verification, a method that mathematically proves code is free of specific bugs. A recent collaboration between an AI research team and Theorem shows that large language models can now automate this process at scale. By using Claude to analyze compiled binaries, the team successfully proved security properties for over 100 real-world open-source libraries, marking a significant shift in how software correctness can be guaranteed.

What happened

The project focused on applying formal verification to production software at the assembly code level, specifically targeting the RISC-V architecture. The team selected 123 open-source libraries from Google’s OSS-Fuzz initiative, which already contains thousands of developer-authored entry points designed to crash when bugs are found. Instead of relying on brute-force testing, the team asked Claude to prove that these programs would never crash for any input under 64 MB, assuming a single-threaded static binary running on Linux.

Over a period of a few weeks, Claude generated proofs for 239 entry points across these libraries. The resulting proofs totaled approximately 180 million lines of Rocq code, with individual proofs averaging around 600,000 lines. During this process, the model not only generated the proofs but also identified and patched bugs that had persisted despite trillions of previous test runs. This effort represents the first large-scale application of AI to formally verify production software at the assembly level.

To make these findings accessible, the team is publishing a subset of the proofs along with the RISC-V and operating system semantics used in the analysis. Theorem is also offering to run this verification pipeline for other open-source projects and has released a Claude skill to help developers begin verifying their own code. This approach moves formal verification from a niche practice reserved for mission-critical systems like avionics to a more broadly applicable tool for general software development.

How it works

Formal verification requires translating code into a mathematical model and proving it adheres to a set of security rules. The primary challenge has always been defining what "correct" means for complex software. In this project, the team bypassed the need for manual specifications by converting existing fuzz tests into formal specs. Fuzz tests generate random inputs to find crashes, but they cannot guarantee every path is tested. The new method defines the spec as "this test suite always passes," applied to the compiled RISC-V instructions rather than the source code.

Claude operates by analyzing the executable artifacts and the formal semantics of the RISC-V instruction set and Linux process behavior provided by Theorem. The model generates massive proofs that demonstrate no memory-safety errors or assertion failures can occur under the defined conditions. These proofs are then validated by the Rocq proof checker, a trusted kernel that ensures the mathematical logic holds. Because the checker validates the proof automatically, developers do not need to manually review the millions of lines of generated code, trusting only the underlying models and the checker itself.

Key details

• The project verified 239 entry points across 123 open-source libraries from the OSS-Fuzz initiative.
• Claude generated approximately 180 million lines of Rocq proof code, with typical proofs reaching 600,000 lines.
• The verification targeted RISC-V assembly code, proving no crashes for inputs under 64 MB on Linux.
• The model discovered and patched bugs that had survived trillions of previous fuzz test runs.
• Trust is placed in the RISC-V and Linux formal models, the theorem statements, and the Rocq proof checker.
• Theorem is releasing a Claude skill to assist developers in applying similar verification workflows to their own projects.

Why it matters

For software engineers, this development lowers the barrier to entry for formal verification, which has historically required years of specialized expert effort. By automating the creation of proofs, LLMs make it feasible to apply rigorous security guarantees to everyday libraries and applications. This shift allows teams to move beyond reactive bug fixing and proactive fuzzing toward a model where critical components are mathematically proven to be safe from entire classes of vulnerabilities, such as memory safety errors.

The ability to verify compiled binaries rather than just source code is particularly significant. It means that security properties can be guaranteed for the actual executable artifacts deployed in production, accounting for compiler behaviors and low-level interactions. As model capabilities improve and semantics for more architectures become available, this approach could become a standard part of the software development lifecycle, reducing the risk of exploits in widely used open-source dependencies.

What you can do

• Explore the published subset of proofs and RISC-V semantics to understand the structure of AI-generated formal verification.
• Try the new Claude skill released by Theorem to start experimenting with verification workflows on your own codebases.
• Review your project’s existing fuzz tests, as these can serve as the basis for formal specifications in future verification efforts.
• Consider integrating formal verification steps for critical libraries in your dependency chain, especially those handling untrusted input.
• Stay updated on developments in RISC-V and Linux formal semantics, as expanding these models will broaden the scope of verifiable software.
• Evaluate whether single-threaded static binary assumptions apply to your current projects as a starting point for verification.