Reviewing AI code by verifying intent instead of reading diffs

Engineers at Aviator have restructured their code review process to handle a fourfold increase in AI-generated pull requests. By shifting focus from reading every line of code to verifying intent, acceptance criteria, and system invariants, the team reduced median merge time from 2.1 days to 3.4 hours. This approach maintains human oversight while automating routine checks, allowing nine engineers to manage over 400 merged PRs per week.

What happened

As Aviator’s development workflow shifted from human-written code with AI assistance to AI-generated code with human guidance, the volume of changes surged. Weekly merged pull requests jumped from approximately 100 to more than 400, with median PR sizes reaching 640 lines. This exceeded the 400-line threshold where bug detection typically declines. Traditional review methods failed to scale: adding an AI reviewer buried critical design decisions under trivial nitpicks, while imposing a four-hour review SLA led to empty approvals as engineers skimmed to meet deadlines.

The team realized that catching bugs was never the primary goal of code review. Citing a 2013 Microsoft study, they noted that only 14% of review comments address defects. The core value of review is knowledge sharing and maintaining a current mental model of the system. With AI agents writing most of the code, this knowledge-transfer function was at risk. Instead of trying to read the diff better, which shows what changed but not why, the team stopped reviewing the diff entirely.

How it works

The new workflow splits the jobs of code review into distinct stages handled by different mechanisms. First, review agents cross-examine each other before a PR exists, arguing over implementation details and recording disagreements. Only unresolved issues, such as naming conventions, are escalated to humans. Second, the system captures intent and acceptance criteria directly from the coding session. Every time an engineer corrects an agent or answers its questions, those interactions become verifiable criteria. Intent explains why a change is made, while acceptance criteria define how it should behave.

Third, the team codified repeated review comments into system invariants. By mining a year of review history, they distilled 11,400 comments into 64 approved invariants, such as requiring audit-log entries for config changes. These invariants are compiled into deterministic rules using tools like ast-grep, which run in seconds alongside probabilistic model checks. Finally, every acceptance criterion is proven with evidence. Agents spin up preview environments to drive user-like scenarios, recording screenshots and logs to verify behavior. This provides concrete proof that a feature works as intended, rather than just passing a CI check.

Key details

• Throughput increase: The same nine engineers now merge 4x more pull requests per week, rising from ~100 to >400.
• Speed improvement: Median time to merge dropped from 2.1 days to 3.4 hours.
• Quality metric: Reverts per 100 PRs decreased by 40%, indicating higher initial code quality.
• Review efficiency: Human comment threads per PR fell from 6.2 to 1.3, focusing only on high-level decisions.
• Invariant creation: 11,400 historical comments were analyzed to create 64 approved invariants and 212 candidate rules.
• Verification speed: Nineteen key invariants run deterministically in 0.3 seconds, providing fast feedback without model latency.

Why it matters

For engineering teams adopting AI coding agents, this case study demonstrates that traditional review bottlenecks can be resolved by changing what humans evaluate. As PR sizes grow beyond human cognitive limits for detailed inspection, relying on diffs becomes ineffective. By automating the verification of rules and behavior, engineers can focus on architectural decisions and system understanding. This shift protects the knowledge-sharing aspect of review, which is critical for long-term maintainability, while letting machines handle the repetitive validation of constraints.

The approach also highlights the importance of capturing intent early in the development cycle. When acceptance criteria are derived from real-time interactions between engineers and AI agents, they reflect the actual requirements more accurately than post-hoc test writing. This ensures that the verification process tests what matters to the business logic, not just what is easy to automate. Teams building AI-native workflows should consider separating rule enforcement from decision-making to maintain velocity without sacrificing quality.

What you can do

• Audit your review comments: Categorize past PR comments into rules, decisions, and one-off fixes to identify candidates for automation.
• Define system invariants: Extract recurring constraints from your codebase and encode them as deterministic checks using tools like ast-grep.
• Capture intent during coding: Save the dialogue between engineers and AI agents to generate acceptance criteria automatically.
• Implement evidence-based verification: Use automated agents to run user-like scenarios in preview environments and record logs as proof.
• Shift human review focus: Train reviewers to skip line-by-line diffs and instead evaluate intent, evidence, and unresolved design decisions.
• Pre-PR agent debate: Configure AI agents to argue over implementation details before creating a PR, resolving trivial issues automatically.