Meta's JiTTests use LLMs to catch bugs before code lands

Meta has moved its Just-in-Time Tests (JiTTests) from research into full production infrastructure. This system uses large language models to generate executable tests that identify bugs in code changes before they are merged. The deployment aims to catch severe regressions earlier while reducing the manual review burden on engineers.

What happened

Over the last year, Meta researchers have explored using large language models to catch software bugs on the fly. They introduced JiTTests as a novel approach where AI generates tests to detect issues that traditional testing might miss. In the five months since unveiling this work, the team focused on determining if the approach could scale across Meta’s massive codebase without slowing down development velocity.

The results show that JiTTests are now an integral part of Meta’s engineering workflow. The system helps engineers catch more bugs with less computational load and a lower rate of false positives compared to previous prototypes. By shifting testing left, the platform identifies risks proactively, allowing teams to respond effectively before code impacts users.

This transition marks a shift from experimental AI tools to reliable infrastructure. The team published a paper titled “Just-in-Time Catching Test Generation at Meta,” which details the initial deployment. The goal was to handle hundreds of millions of lines of code constantly evolving, ensuring that traditional testing methods could keep pace with the speed of modern software delivery.

How it works

JiTTests differ fundamentally from standard AI code reviewers. While typical agentic AI reviews code based on prior experience similar to a human reviewer, JiTTests generate executable tests. These tests demonstrate concrete behavioral differences introduced by a code change. This method requires more computational steps but catches specific classes of bugs that semantic reviews often overlook.

The system operates through two tandem workflows. The "intent-aware" workflow infers the purpose of a code change and identifies potential risks. It then constructs mutants, which are hypothetical bugs, and generates tests to kill them. The "dodgy diff" workflow treats every change as if it were a bug, generating tests to expose behavioral differences without reasoning about intent. Both workflows produce "weak catches," which are potential failures that require further validation.

To filter these weak catches, Meta uses an ensemble called "LLM-as-Judge." Multiple state-of-the-art LLMs review the test code, execution traces, and the diff itself to classify results as true or false positives. This is complemented by RubFake, a rule-based assessor that provides deterministic judgments. The combination ensures high precision before any human engineer is involved.

Key details

• JiTTests generated and ran tests on diffs deemed risky using a Diff Risk Score mechanism.
• The system caught 74 serious privacy bugs in the first half of 2026 before they reached production.
• Engineers reported a 60% true positive rate for the findings surfaced by JiTTests.
• The approach reduced the overall occurrence of privacy severity incidents by 20% during the same period.
• 589 engineers, representing 64% of those contacted, engaged with the JiTTests results.
• The workflow uses a human-on-the-loop model, where engineers review aggregate outcomes rather than individual test cases.

Why it matters

For teams building software at scale, the speed of deployment often outpaces the capacity for thorough manual testing. JiTTests demonstrate that AI can provide an additional safety layer capable of catching regressions that static analysis or traditional unit tests might miss. By automating the generation of executable proofs of failure, organizations can reduce organizational risk without sacrificing developer velocity.

The human-on-the-loop design is particularly significant for engineering leads. Instead of forcing developers to review every AI-generated test, the system filters noise and only escalates high-confidence findings. This reduces friction and ensures that human expertise is applied to confirming behavioral changes rather than debugging false alarms. It makes AI-assisted testing socially and technologically scalable.

Furthermore, the success of JiTTests highlights the value of combining probabilistic LLM judgments with deterministic rule-based systems. Purely AI-driven approaches can suffer from non-determinism, making them hard to trust in critical pipelines. By integrating RubFake and LLM-as-Judge, Meta created a hybrid system that balances semantic understanding with verifiable logic, offering a blueprint for robust AI integration in CI/CD pipelines.

What you can do

• Evaluate your current CI pipeline to identify gaps where behavioral regression tests are missing.
• Experiment with LLMs to generate unit tests for complex functions, focusing on edge cases derived from code intent.
• Implement a filtering layer for AI-generated tests using simple static rules before involving human reviewers.
• Define clear criteria for what constitutes a "risky" change in your codebase to prioritize automated testing efforts.
• Adopt a human-on-the-loop mindset by aggregating AI findings for batch review rather than interrupting developers per item.
• Read the paper “Just-in-Time Catching Test Generation at Meta” to understand the architectural details of the dodgy diff and intent-aware workflows.