How to tell if your AI agent actually got better

Determining if an AI agent has improved is a common question. The answers to this query are simple but may not seem so at first. The agent’s output of "You're booked for Thursday" reveals nothing about its performance. Anthropic’s guidelines on agent evaluations indicate that the focus should be on the state the agent leaves behind, rather than the language it uses.

The crucial questions to ask are whether the target row was inserted into the database, and whether the appropriate tool calls were executed with the right parameters. Demos often appear impressive because they showcase the agent’s replies, but an evaluation truly monitors the system’s behavior. While language models (LLMs) can help in some cases, they introduce their own biases.

A study of 21 LLM judges demonstrated that their verdicts varied significantly, sometimes flipping their decision up to 38% of the time. Observing replies alone does not measure the agent's capability; it merely reflects its politeness. To accurately assess an agent's performance, it is necessary to calibrate the judge against human labels.

For evaluation purposes, choosing between a 1 to 10 scale or a binary pass/fail system is recommended. Starting with 20 to 50 tasks, ideally drawn from real failures, proves more effective than relying on synthetic cases. Evaluating an LLM’s capabilities can sometimes precede actual use, as it establishes clear expectations. However, when addressing failures, the approach should be reverse: ship the product, analyze the traces, and then formulate an evaluation tailored to the specific failure encountered.