AWS Releases Aws-Bench to Evaluate Agents on Cloud Tasks
AWS has released aws-bench, an open-source benchmark for evaluating AI agents on real AWS tasks such as misconfigurations and infrastructure provisioning. Unlike traditional benchmarks, it uses real resources in disposable AWS accounts, scoring agent performance through automated verifiers. By Gianmarco Nalin
Amazon Web Services (AWS) has unveiled a new open-source benchmark tool called aws-bench, designed to assess the proficiency of AI agents in executing real-world tasks within the cloud environment. Unlike conventional benchmarks, aws-bench evaluates agents by conducting actual AWS tasks, such as identifying misconfigurations, provisioning infrastructure, and managing live cloud resources. This approach aims to provide a more realistic representation of the work developers and teams typically perform on AWS.
Each benchmark scenario is deployed within isolated AWS accounts, consisting of a collection of AWS resources defined in Cloud Development Kit (CDK) stacks. When a scenario is available, the AI agent under evaluation runs the task in a sandboxed container using scoped credentials. Following completion, the result is scored using an automated verifier, which can be either an artificial intelligence system (such as a large language model, or LLM) or a programmatic check against the live AWS state.
The benchmark comes with a set of predefined datasets encompassing various use cases, including observability, compute and data, databases and storage, EC2 multi-region, serverless, streaming and IoT, reference architectures, and multi-service troubleshooting. While the primary audience for aws-bench includes AI researchers and model providers, engineering teams are also encouraged to experiment with the benchmark, expanding the available scenarios and tasks to align with their specific use cases.
The project is constructed upon Harbor, an open-source platform for evaluating AI agents that has been adapted to offer enhanced features like AWS resource provisioning, scenario development, and verification mechanisms. Built-in adapters for the tool include popular agents and models like Claude Code, Codex, Kiro CLI, Mini-SWE-Agent, Gemini CLI, and OpenCode, as well as any agent compatible with Harbor, such as Gemini CLI and OpenCode.
To execute the benchmark, users require more than a local installation. They must possess credentials with access to their organization's management account and the necessary permissions to manage member accounts and organizational units. The current setup is limited to the us-east-1 region, but it may create persistent resources that could incur costs even if not utilized.
At present, aws-bench does not provide any metrics or a standardized leaderboard, with these aspects expected to be addressed in future releases. The benchmark and its associated datasets are hosted on GitHub under the Apache-2.0 license. The introduction of aws-bench arrives amidst increasing doubts about the reliability of current agent evaluation benchmarks.
A team of researchers from the Center for Responsible, Decentralized Intelligence at UC Berkeley recently published an article detailing how they developed an agent capable of attaining near-perfect scores on prominent benchmarks, including Terminal-Bench and SWE-Bench, without actually solving any tasks. These occurrences are indicative of a broader issue: the benchmarks used to gauge AI capabilities are susceptible to exploitation, mirroring the very capabilities they are intended to measure.
As adoption of aws-bench increases, its resistance to such exploitation will likely come under increased scrutiny, particularly given that most tasks in the benchmark are evaluated using an LLM judge, with AWS acknowledging the potential for leftover state to lead to undesired passes or random failures.
Written by urgent.news from InfoQ's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
