Benchmarking AI agent costs across JavaScript frameworks

In October 2026, developer Nolan Lawson published a data-driven comparison of how modern JavaScript frameworks perform when built by AI agents. The study challenges the prevailing "vibes-based" decision-making in frontend development by measuring actual cost, time, and quality metrics. Using Claude Opus 5.5, Lawson tasked an AI agent with constructing the same Pokémon application across twelve different technology stacks to determine which tools truly optimize for agentic workflows.

What happened

Lawson designed a benchmark to test the hypothesis that certain frameworks are inherently better for AI-assisted coding. He selected a non-trivial but well-defined project: a Pokémon directory app using the PokéAPI. The application required listing Pokémon, displaying their types and moves, and implementing search and filter functionality. To ensure fairness, each framework was paired with its official Model Context Protocol (MCP) server or agent skills where available, preventing disadvantages due to training data gaps.

The experiment involved running the agent five times for each framework to account for the stochastic nature of large language models. The only exception was Angular, which failed to produce a measurable app in two instances, resulting in only three valid iterations for some metrics. The agent operated in a sandbox environment without knowledge that it would be evaluated on Lighthouse performance scores, though it was provided with a conformance test suite to prevent cheating. The goal was to measure real-world outcomes rather than theoretical developer experience.

The results revealed substantial variance in both cost and efficiency. Hono combined with htmx emerged as the most cost-effective option, with a median cost of $2.67 per run. In contrast, Angular proved the most expensive at $4.97, while SvelteKit and Gracile plus Lit also ranked high in cost. Interestingly, vanilla JavaScript with Vite underperformed several frameworks, suggesting that structured abstractions can aid AI agents more than raw infrastructure. The study highlights that claims about framework superiority for AI often lack empirical backing, and this benchmark provides a concrete baseline for evaluation.

How it works

The benchmark relies on a strict specification and automated evaluation pipeline. Lawson instructed the AI agent, Claude Opus 5.5, to build the app using high-effort settings. The agent was not given specific architectural mandates, such as choosing between server-side rendering or client-side rendering, allowing it to make decisions based on the framework's conventions. This approach tests how well the agent navigates the ecosystem without heavy-handed human guidance.

Evaluation focused on four key dimensions: development time, token consumption, lines of code, and quality metrics. Quality was assessed using Lighthouse for performance and accessibility scores, alongside a custom conformance suite that checks the accessibility tree rather than the raw DOM. This method accommodates frameworks like Lit that use shadow DOM, ensuring that structural differences do not penalize the score. By measuring median values across multiple runs, the study mitigates the impact of random outliers inherent in generative AI outputs.

Key details

• Hono+htmx was the cheapest framework to build with, costing a median of $2.67 per run.
• Angular was the most expensive at $4.97 median cost and took the longest time at 17.2 minutes.
• Vanilla JavaScript with Vite cost $2.98, underperforming several structured frameworks in efficiency.
• SvelteKit agents frequently read the v3 migration guide, increasing token usage due to its novelty in training data.
• Agent usage of MCP servers and skills was inconsistent, with Nuxt’s MCP never used and Next.js skills rarely accessed.
• Lighthouse metrics favored multi-page application frameworks due to inherent biases against single-page applications.

Why it matters

For technical leads and founders, this study underscores the financial implications of framework choice in an agentic workflow. As AI agents take on more coding tasks, the cost per feature becomes a tangible metric rather than an abstract concern. Frameworks that require fewer tokens and less time to generate correct code directly reduce operational expenses. The variance observed, such as Angular’s higher failure rate and cost, suggests that some ecosystems are not yet optimized for AI interaction, potentially leading to budget overruns in large-scale projects.

Furthermore, the results challenge the notion that "vanilla" code is always the most efficient path for AI. The data shows that frameworks provide guardrails and conventions that help agents avoid common pitfalls, reducing the need for extensive context loading. This insight is crucial for teams deciding whether to adopt new meta-frameworks or stick with established libraries. It suggests that the right abstraction can lower the cognitive load on the AI, leading to faster and more reliable output.

What you can do

• Benchmark your current framework by having an AI agent build a small, standardized component and measure token usage.
• Ensure your team’s AI tools are configured to use official MCP servers or skills to maximize context relevance.
• Consider the total cost of ownership, including AI token consumption, when evaluating new JavaScript frameworks.
• Run multiple iterations of AI-generated code to account for stochastic variance before making architectural decisions.
• Use accessibility tree-based testing rather than DOM inspection to fairly evaluate frameworks using shadow DOM.
• Monitor agent behavior to see if it effectively utilizes available documentation and skill files during development.