How Cloudflare cut AI review costs by 68% through prompt caching

Cloudflare engineers successfully reduced the average cost of their AI-powered code reviews by 68 percent, dropping from $2.46 to just $0.77 per review. This optimization campaign, executed between mid-September and early October 2026, also slashed median review time from over four minutes to under two minutes without sacrificing detection quality.

What happened

Earlier in the year, Cloudflare’s AI code reviewer saw its costs spiral upward despite stable architecture. In April, an average review cost $1.19, but by late August, that figure had more than doubled to roughly $2.46. The team identified two primary drivers for this increase: the growing complexity of code changes submitted by engineers, which pushed more work into expensive full-review tiers, and a strategic switch to Anthropic models that offered better performance but at a higher price point. Specifically, moving to Claude Opus 4.7 for internal standards checking raised the cost of that specific component from $0.25 to $0.88 per review.

In mid-September, the team launched an aggressive optimization initiative. They analyzed token usage patterns and discovered that while cache hit rates were high, the sheer volume of tokens being processed was driving up bills. By early October, after implementing a series of structural and model-level changes, the average cost fell to $0.77. This represented a 68 percent reduction from the peak spending weeks. Simultaneously, the median review duration dropped from 4 minutes and 9 seconds to 1 minute and 32 seconds, significantly improving developer experience alongside financial efficiency.

How it works

The core technical breakthrough involved rethinking how the system utilized Anthropic’s prompt caching. Prompt caching allows large language models to store computed prefixes of a conversation, enabling cheaper reuse for subsequent requests. However, Cloudflare found that their initial implementation failed to share cached data across different repositories. The system prompt included repository-specific details like working directories and dates before the cache breakpoint, meaning every new session started with a cache miss. By splitting the system prompt and placing a cache marker after the static agent instructions, they ensured that the shared 11k to 21k token prefix could be reused across all jobs, turning frequent writes into cheap reads.

Further efficiency gains came from reducing the number of LLM calls per session. Previously, sub-reviewers would attempt to use tools they did not have permission to access, resulting in wasted calls and error recovery loops. The team removed these inaccessible tools and parallelized file reading, allowing reviewers to ingest context in a single turn rather than multiple sequential steps. They also implemented logic to skip unnecessary reviewers during re-reviews and canceled jobs that continued running in the background after being marked as complete. These changes collectively reduced the average number of LLM calls and the total token count per review.

Key details

• Average review cost dropped from $2.46 to $0.77, a 68 percent reduction.
• Median review time improved from 4m 09s to 1m 32s.
• Cache hit rates for the shared prefix increased significantly after restructuring the system prompt.
• Switching from Sonnet 5 to Sonnet 5.5 reduced costs by nearly half for some reviewer types due to lower token usage.
• Code changes accounted for $0.65 of the savings, while model and effort adjustments accounted for $1.06.
• The 99th percentile review time is now faster than 75 percent of reviews were in April.

Why it matters

For engineering teams building with large language models, this case study highlights that architectural stability does not guarantee cost stability. As models become more capable, users tend to submit more complex inputs, which can inadvertently drive up token consumption. Cloudflare’s experience demonstrates that high cache hit rates are not enough if the cached data is not structured for maximum reuse. Developers must actively manage prompt structure to ensure that static, shared content is isolated from dynamic, variable content to leverage caching effectively.

Additionally, the shift in cost drivers from infrastructure to model selection and effort levels suggests that ongoing tuning is required. Simply picking a powerful model is not a set-and-forget decision. Teams need to continuously evaluate whether higher-effort settings yield proportional value in findings. Cloudflare found that for some tasks, medium effort was sufficient, while for others, high effort was worth the marginal cost increase. This granular approach to configuration prevents over-provisioning compute resources for tasks that do not require deep reasoning.

What you can do

• Audit your prompt structure to ensure static prefixes are separated from dynamic variables for better cache utilization.
• Monitor the number of LLM calls per session and identify wasted turns caused by tool errors or unnecessary summaries.
• Test newer model versions against current baselines, as efficiency improvements can sometimes offset higher per-token prices.
• Implement parallel processing for context retrieval to reduce sequential latency and call counts.
• Review effort settings for different task tiers to ensure you are not over-computing for simple checks.
• Ensure cancelled jobs are truly terminated at the script level to prevent background resource waste.