How to Design AI Evaluations You Can Actually Trust

Google has released a collection of AI Agent Skills for its products on GitHub, aimed at assisting AI agents in interacting with their technologies. However, testing these skills and ensuring they work as expected is crucial. Developer Relations at Google emphasizes the importance of having reliable performance signals to improve AI agents over time.

Just as you wouldn't deploy a production API without unit tests, you should apply the same standard to AI agents. Instead of relying on vague, terminal-based testing, establish a structured, automated evaluation pipeline to benchmark your integration. The evaluations, or "evals," are the actions you ask the agent to perform, graded using scorers, also known as rubrics, which determine if the agent succeeded.

This article will focus on evaluations and discuss tips for creating scoring rubrics in a subsequent post. However, AI evaluations consume real tokens, so it is essential to use them efficiently to gain valuable insights for building better tools. To ensure that every token spent yields useful metrics, follow these five key rules for designing better evaluations:

1. Understand your evaluation environment: Before crafting evaluations, familiarize yourself with the chosen framework's setup and limitations. Determine if the system uses an ephemeral sandbox, what tools are available, and how outputs are captured. Tailor graders accordingly to handle the specific environment, considering limitations and access to real resources. For example, if access to real resources is required, create isolated credentials or use mock tools instead.

2. Avoid the ceiling effect: High baseline accuracy without the agent's tool in place may not demonstrate its value. Design harder prompts that require multi-step reasoning and reflect complex, real-world use cases where the tool can differentiate itself from the pre-trained model.

3. Prevent the prompt-grader mismatch: Your evaluation prompts and graders should be complementary, only testing for things explicitly included in the prompt. Avoid scope creep by being explicit about the knowledge or implementation details you want to evaluate. Grade the final output, not the journey, as agents may use inherent model knowledge to arrive at the correct answer without utilizing your custom tools.

4. Curate your evaluation dataset: A strong evaluation suite should test diverse, real-world use cases. However, repetitive testing of the same capability can lead to overfitting and noisy metrics. Use real-world examples, including sanitized sample data, to ground evaluations. Maximize signal by ensuring each prompt tests a distinct concept or capability, and remove overlap by consolidating redundant prompts. This approach provides clearer metrics, prevents overfitting, and conserves tokens.