WebArena, GAIA and Agentic Benchmarks

An agentic benchmark differs from traditional benchmarks, such as MMLU or HumanEval, in that it evaluates a system's performance through a series of actions that modify an environment's state, and grades the final outcome rather than a single output. This approach acknowledges that multiple correct paths to a goal exist, and grading the final state is the only practical option.

However, this method also means that agentic benchmarks cannot distinguish between an efficient and an inefficient solution, as it cannot differentiate between a successful outcome achieved through smart problem-solving and one that merely happened to work by chance.

The independence of steps in an agentic benchmark is a significant factor in its scoring. For instance, if a task consists of 20 independent steps with a success rate of 95% per step, the overall success rate drops to 35.8%. This compounding effect highlights the importance of error recovery, as a model that seems reliable step-by-step might still produce low task scores due to its inability to correct mistakes.

Agentic benchmarks encompass various environments, each with its unique characteristics and challenges. WebArena, for example, consists of self-hostable, real-world web applications and tasks with programmatic reward functions. This set of benchmarks is reproducible because users can host them. GAIA, on the other hand, includes several hundred real-world questions requiring web browsing, tool use, file handling, and multi-modal input.

The difficulty of the questions is tiered based on the number of steps and tools required. Most of the answer set is held back and evaluated upon submission to minimize overfitting.

SWE-bench, another agentic benchmark, uses real GitHub issues from Python repositories paired with the pull requests that resolve them. The agent must generate a patch, and grading is based on running the repository's test suite. This requires the previously failing tests to pass and the previously passing tests to remain passing. A human-validated subset is present to acknowledge that some instances are unsolvable or poorly specified.

Operating system and desktop environments present another type of agentic benchmark. These tasks are conducted within a real operating system with real applications, and grading is performed using execution scripts that inspect files and application state. This approach is both the most realistic and the most fragile, as an application update can change the definition of a correct trajectory.

Live-web evaluations further illustrate the dynamic nature of agentic benchmarks, as agents are assessed against the actual internet, often graded based on a model's assessment of screenshots.