{
  "id": 2533417,
  "title": "AWS Releases Aws-Bench to Evaluate Agents on Cloud Tasks",
  "url": "https://urgent.news/2026/08/22/aws-releases-aws-bench-to-evaluate-agents-on-cloud-tasks",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-22T08:00:00.000Z",
  "source": {
    "name": "InfoQ",
    "slug": "infoq",
    "url": "https://www.infoq.com/news/2026/08/aws-bench-agent-evaluation/"
  },
  "original_language": "en",
  "account": "Amazon Web Services (AWS) has unveiled a new open-source benchmark tool called aws-bench, designed to assess the proficiency of AI agents in executing real-world tasks within the cloud environment. Unlike conventional benchmarks, aws-bench evaluates agents by conducting actual AWS tasks, such as identifying misconfigurations, provisioning infrastructure, and managing live cloud resources. This approach aims to provide a more realistic representation of the work developers and teams typically perform on AWS.\n\nEach benchmark scenario is deployed within isolated AWS accounts, consisting of a collection of AWS resources defined in Cloud Development Kit (CDK) stacks. When a scenario is available, the AI agent under evaluation runs the task in a sandboxed container using scoped credentials. Following completion, the result is scored using an automated verifier, which can be either an artificial intelligence system (such as a large language model, or LLM) or a programmatic check against the live AWS state.\n\nThe benchmark comes with a set of predefined datasets encompassing various use cases, including observability, compute and data, databases and storage, EC2 multi-region, serverless, streaming and IoT, reference architectures, and multi-service troubleshooting. While the primary audience for aws-bench includes AI researchers and model providers, engineering teams are also encouraged to experiment with the benchmark, expanding the available scenarios and tasks to align with their specific use cases.\n\nThe project is constructed upon Harbor, an open-source platform for evaluating AI agents that has been adapted to offer enhanced features like AWS resource provisioning, scenario development, and verification mechanisms. Built-in adapters for the tool include popular agents and models like Claude Code, Codex, Kiro CLI, Mini-SWE-Agent, Gemini CLI, and OpenCode, as well as any agent compatible with Harbor, such as Gemini CLI and OpenCode.\n\nTo execute the benchmark, users require more than a local installation. They must possess credentials with access to their organization's management account and the necessary permissions to manage member accounts and organizational units. The current setup is limited to the us-east-1 region, but it may create persistent resources that could incur costs even if not utilized.\n\nAt present, aws-bench does not provide any metrics or a standardized leaderboard, with these aspects expected to be addressed in future releases. The benchmark and its associated datasets are hosted on GitHub under the Apache-2.0 license. The introduction of aws-bench arrives amidst increasing doubts about the reliability of current agent evaluation benchmarks. A team of researchers from the Center for Responsible, Decentralized Intelligence at UC Berkeley recently published an article detailing how they developed an agent capable of attaining near-perfect scores on prominent benchmarks, including Terminal-Bench and SWE-Bench, without actually solving any tasks. These occurrences are indicative of a broader issue: the benchmarks used to gauge AI capabilities are susceptible to exploitation, mirroring the very capabilities they are intended to measure. As adoption of aws-bench increases, its resistance to such exploitation will likely come under increased scrutiny, particularly given that most tasks in the benchmark are evaluated using an LLM judge, with AWS acknowledging the potential for leftover state to lead to undesired passes or random failures.",
  "summary": "AWS has released aws-bench, an open-source benchmark for evaluating AI agents on real AWS tasks such as misconfigurations and infrastructure provisioning. Unlike traditional benchmarks, it uses real resources in disposable AWS accounts, scoring agent performance through automated verifiers. By Gianmarco Nalin",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}