Building with AI

Airbnb uses cost per task to manage coding agents on Amazon Bedrock

Airbnb evaluates coding agent models using cost per task metrics and CloudWatch signals to set defaults and monitor production performance on Amazon Bedrock.

A dashboard showing cost and token metrics above code files.
Illustration generated for this article

Airbnb has detailed its strategy for operating internal coding agents at scale, focusing on a metric called cost per task to guide model selection. The engineering team runs these agents through an internal tool called AirChat, which connects to Amazon Bedrock for large language model inference. By combining pre-deployment evaluations with real-time production monitoring, the company manages latency, capacity, and spending as new models are released.

What happened

Thousands of engineers and engineering-adjacent staff at Airbnb use AirChat every week to assist with software development tasks. This internal platform allows users to run coding agents either on their personal cloud development machines, known as AirDev workspaces, or via AirChat Remote, which executes tasks in separate cloud environments. All model requests from these agents pass through a central large language model gateway that routes traffic to Amazon Bedrock. This setup ensures that prompts, code, and model responses remain within the AWS infrastructure, leveraging its access controls and monitoring capabilities.

The landscape of available foundation models changes rapidly. In a recent five-week period, eight new models were released on Amazon Bedrock. By mid-October, approximately two-thirds of the requests sent by engineers through coding agents were directed to these newly released models. Each new model release introduces variability in output token counts, costs, latency, and capacity limits. Benchmark charts provided by model publishers often fail to reflect how a model performs on specific internal engineering workflows. To address this gap, Airbnb runs parallel systems: one for evaluating models before rollout and another for observing behavior in production.

Before deploying a new model as a default, the team runs evaluations on a fixed set of representative tasks. These evals measure both the quality score and the cost per task for each model and reasoning effort level. After rollout, observability tools track actual usage patterns. If production data diverges from evaluation results, engineers investigate the discrepancy to understand why. This continuous feedback loop allows the team to adjust model defaults and effort ceilings based on real-world performance rather than static benchmarks.

How it works

The core of Airbnb’s strategy is the cost per task metric. A coding agent typically requires multiple requests to complete a single task, and the model re-reads the accumulated context with each request. Therefore, pricing individual requests is insufficient. Instead, the team calculates the total list-price cost of all tokens used to finish one task. Amazon Bedrock reports four types of tokens for each request: uncached input, cache reads, cache writes, and output. Each type has a different price point. Cache reads are cheaper than uncached input, while cache writes can be more expensive depending on the model. A lower cache hit rate can significantly increase the cost per request, so tracking these components is essential.

To evaluate models, Airbnb uses a framework called Harbor, an open-source tool for running agent evaluations in sandboxes. Each evaluation task is defined as a folder containing the instruction, environment configuration, tests, and a grader. The grader is a program or model that scores the agent’s work against a reference solution. For example, a machine learning task might ask an agent to compare two candidate models and decide whether to roll out a new one. The grader recomputes the correct answer from frozen data and awards a reward only if all checks pass. This approach ensures that evaluations are consistent and reproducible.

Observability in production relies on Amazon CloudWatch metrics within the AWS/Bedrock namespace. Engineers monitor latency using InvocationLatency and TimeToFirstToken, which measures the wait before generation starts. They also track throttling via InvocationThrottles and quota consumption using EstimatedTPMQuotaUsage. This latter metric includes cache write tokens and output burndown multipliers, allowing teams to set proactive alarms before hitting rate limits. Cost monitoring involves tracking InputTokenCount, CacheReadInputTokenCount, CacheWriteInputTokenCount, and OutputTokenCount. By comparing these production signals with evaluation scores, the team can determine when to change model defaults or adjust effort ceilings.

Key details

  • Cost per task definition: The total list-price cost of all tokens an agent uses to complete one task, including uncached input, cache reads, cache writes, and output.
  • Evaluation speed: New models can be evaluated across all task suites within hours, with full results available by the next morning, enabling rapid deployment decisions.
  • Reasoning effort impact: Increasing reasoning effort can improve scores but significantly raises costs; for one model, moving from low to high effort increased cost per task by 2.5 times.
  • Production metrics: Key CloudWatch metrics include TimeToFirstToken, OutputTokenCount, and derived output tokens per second (OTPS), calculated as OutputTokenCount / (InvocationLatency - TimeToFirstToken) * 1000.
  • Task structure: Evaluation tasks include instructions, environment definitions, graders, and reference solutions, ensuring consistent scoring across different models and efforts.
  • Quota management: EstimatedTPMQuotaUsage helps track token consumption across all inference APIs, allowing teams to request quota increases before rate limiting occurs.

Why it matters

For teams building software with AI, relying solely on public benchmarks can lead to unexpected costs and performance issues. Public charts do not account for specific workflow characteristics, such as context length, caching behavior, or the complexity of internal codebases. By adopting a cost per task metric, organizations can make more informed decisions about which models to use for specific types of work. This approach shifts the focus from raw token prices to the actual economic efficiency of completing engineering tasks. It also highlights the importance of caching strategies, as cache miss rates can dramatically affect overall spend.

Furthermore, the integration of evaluation and observability creates a robust operational framework. As new models are released frequently, having a standardized process for testing and monitoring allows teams to adopt improvements quickly while mitigating risks. Setting effort ceilings based on evaluation data prevents runaway costs from overly verbose or inefficient reasoning processes. This discipline ensures that AI tools remain sustainable and valuable assets for engineering teams, rather than becoming unmanaged expenses. The ability to derive metrics like OTPS from basic CloudWatch data also empowers teams to diagnose latency issues without waiting for vendor support.

What you can do

  • Define a set of representative engineering tasks with clear instructions, environment configurations, and automated graders to establish a baseline for model evaluation.
  • Implement a gateway or proxy in front of your LLM provider to record model, client, team, and user data for each request, enabling aggregate cost and capacity planning.
  • Activate model invocation logging in all relevant AWS regions and restrict access to logs to protect sensitive prompt and response data.
  • Use CloudWatch metrics to monitor TimeToFirstToken and OutputTokenCount, and derive output tokens per second to diagnose latency issues in production.
  • Calculate cost per task by summing the costs of all token types (uncached input, cache reads, cache writes, output) for each completed agent task.
  • Set proactive alarms using EstimatedTPMQuotaUsage to track token consumption and request quota increases before reaching rate limits.
  • #amazon bedrock
  • #ai agents
  • #cost optimization
  • #llm ops
  • #airbnb

Tools from the Bytechap store

$89

DocBento

Self-hosted document management that reads every scan and answers with page citations.

Live demo

Keep reading

All stories