{
  "id": 20662617,
  "title": "How Airbnb uses cost per task to operate coding agents on Amazon Bedrock",
  "url": "https://urgent.news/2026/11/18/how-airbnb-uses-cost-per-task-to-operate-coding-agents-on-amazon",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-18T21:48:19.000Z",
  "source": {
    "name": "AWS Machine Learning",
    "slug": "aws-machine-learning",
    "url": "https://aws.amazon.com/blogs/machine-learning/how-airbnb-uses-cost-per-task-to-operate-coding-agents-on-amazon-bedrock/"
  },
  "original_language": "en",
  "account": "Amazon Bedrock enables access to foundation models from multiple providers via a unified API, with AWS managing the model infrastructure. The platform integrates with AWS access controls and monitoring, processing prompts, code, and model responses within AWS. As coding agents transition into production, alterations in models and usage can influence latency, capacity, and cost. Amazon Bedrock runtime metrics in Amazon CloudWatch assist teams in investigating latency increases, throttling, and variations in token consumption. Assessments, or evals, function as a complement to operational metrics by gauging the quality of an agent's output on a predefined set of representative tasks. In this post, we explore how Airbnb evaluates models for coding agents and manages their utilization on Amazon Bedrock, elucidating how cost per task informs model selection and how production data enables the reassessment of these choices as usage evolves. We also delve into the runtime indicators employed to analyze latency, throttling, and spending.\n\nAt Airbnb, a vast number of engineers and engineering-related personnel utilize AirChat, our internal tool for running coding agents, on a weekly basis. The agents transmit their model requests through our large language model (LLM) gateway, which operates on Amazon Bedrock. All of Airbnb's code, prompts, and model responses traverse AWS. In the span of five weeks, eight new models emerged. By mid-October, approximately two-thirds of the requests from Airbnb's engineers directed to the coding agents were routed to these fresh models. Each release can alter output tokens, cost, latency, and capacity concurrently, and benchmark charts presented with model launches do not reflect how a model performs under actual engineering workloads. Consequently, we operate two systems concurrently. Prior to a rollout, we conduct evaluations on our own tasks to ascertain each model's score and cost per task. Cost per task is calculated as the list-price cost of all tokens an agent employs to complete a single task. Following a rollout, observability reveals what has changed in production. When production usage diverges from eval results, we seek to understand the reasons behind this discrepancy.",
  "summary": "Airbnb runs evals on real engineering tasks and uses Amazon Bedrock observability to choose models for its coding agents and control production costs. This post shows how cost per task guides model selection, why production spending can differ from eval results, and how to monitor latency, manage quotas, and attribute spend.",
  "key_points": [
    "Airbnb uses cost per task to select coding agents on Amazon Bedrock.",
    "Agents transmit model requests through Airbnb's large language model gateway on Bedrock.",
    "Production data informs reassessment of model choices as usage evolves."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}