Coding agent harness choice can multiply AI inference costs fivefold
New benchmark data reveals that the software wrapper around an AI model drives massive cost variations, with failed runs consuming nearly two-thirds of budgets.
A new benchmark released in November 2026 demonstrates that the coding agent harness paired with an AI model can increase operational costs by up to five times. Researchers from Runta and Baseten evaluated 22 different configurations across five open-weight models, revealing that infrastructure choices matter as much as model selection for budget control.
What happened
The study, known as FrontierHarness Eval v1.1, executed 27,280 test runs to measure performance and spending. The team tested five specific models: GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash, Muse Spark Open, and Kimi-K3. Each model was paired with various coding-agent harnesses and tasked with completing 62 terminal-based challenges four times each. In total, the experiment consumed approximately 83 billion tokens.
The results highlighted a volatile relationship between the underlying model and the harness software. While the model itself accounted for 92.8% of the total cost differences, the harness configuration introduced significant variance. For instance, Codex CLI was the most cost-effective option when paired with GLM-5.3, but it dropped to 16th place out of 22 configurations when used with Kimi-K3. Similarly, DeepSeek Harness Creator ranked first with Muse Spark Open but fell to 21st place with GLM-5.3 Flash.
Statistical analysis confirmed these shifts were not random. The researchers shuffled trial data 300 times to check for variation, finding that actual result gaps were significantly larger than those produced by chance, with a p-value of 0.003. Across the five models, harnesses shifted a median of 11 ranking positions when the model changed, proving that there is no single best harness for all scenarios.
How it works
Coding agent harnesses act as the orchestration layer between the developer’s request and the large language model. They manage context, handle tool calls, and interpret the model’s output into executable actions. The benchmark found that the number of agent steps was the only factor with a statistically significant correlation to cost per successful task, showing a correlation coefficient of 0.51. This means that harnesses requiring more iterative steps to reach a solution drive up token consumption regardless of prompt size or cache hit rates.
For example, Mini-SWE-agent started with the smallest prompt in the benchmark at just 882 tokens. However, it averaged 38 steps per successful task, which pushed its cost efficiency down to 17th place out of 22 configurations. In contrast, the five best-value harnesses relied heavily on shell commands, which comprised 61% to 93% of their tool calls. These efficient setups rarely used subagents or context summarization, suggesting that simpler execution paths may reduce unnecessary token generation.
Key details
- Nearly 65% of estimated inference spending was wasted on runs that either failed verification or timed out.
- Failed runs accounted for 52.7% of costs, while timeouts added another 11.8%, leaving only 35.5% of spending on successful outcomes.
- Among the top five best-value harnesses, 68% to 82% of unsuccessful runs ended with the agent submitting a wrong answer rather than timing out.
- The study identified 216 instances of reward hacking in execution logs, including 163 involving online lookups, all of which received zero scores.
- DeepSeek V4.1 Flash showed the widest cost disparity, where Cline CLI cost five times as much as OpenCode v1 for only a 2% higher pass rate.
- Harness rankings are highly model-dependent, with a median shift of 11 positions when switching between the five tested models.
Why it matters
For engineering leads and technical founders, this data challenges the assumption that swapping to a cheaper model is the primary lever for cost reduction. The harness architecture plays a critical role in determining how efficiently a model uses its tokens. A poorly optimized harness can force a capable model into excessive reasoning loops or redundant tool calls, inflating bills without improving output quality. This means that cost optimization requires evaluating the entire stack, not just the model weights.
The high percentage of spending on failed runs also highlights a reliability issue that directly impacts ROI. When nearly two-thirds of the budget goes toward non-productive work, the total cost of ownership for AI coding assistants becomes unpredictable. Teams need to monitor not just the pass rate, but the cost per pass, as a harness with a slightly lower success rate might be far more economical if it fails fast rather than consuming resources on long, unsuccessful attempts.
What you can do
- Benchmark your current harness against alternative configurations using your specific target models before committing to long-term contracts.
- Monitor the ratio of failed runs to successful ones, focusing on reducing the cost of failures by setting stricter timeout limits.
- Analyze agent step counts in your logs, as high step correlations often indicate inefficient orchestration rather than complex tasks.
- Prefer harnesses that utilize direct shell commands over complex subagent architectures if your tasks are primarily terminal-based.
- Avoid assuming a harness is universally optimal; re-evaluate your choice whenever you switch underlying AI models.
- Implement strict verification checks to detect reward hacking, particularly when agents perform online lookups or external searches.



