Your Agent Eval Set Is Rotting: Build a Failure-Mining Loop for Google ADK
An agent evaluation set starts healthy. It contains the obvious intents, a few tool failures, and the happy paths used during development. Six months later, production has changed. New tools exist. Users phrase requests differently. A fallback introduced last quarter now handles 30% of traffic. Yet CI still runs the same twelve examples and reports green. The problem is not only stale prompts.…
An agent evaluation set can become outdated as production changes. New tools, user phrasing, and fallbacks affect the effectiveness of the evaluation. The evaluation set itself needs to evolve. To address this, create a controlled path from observed failures back into tests. Begin by defining a FailureCandidate record to triage failures, without including raw customer content.
Mine failure candidates across various dimensions, such as intent and failure role, to avoid selecting the loudest incidents as representatives. Create a signature for deduplication and convert failures only after review. The promotion checklist should confirm real failures, identify reproducible inputs, remove sensitive data, label expected behavior, and assign owners.
Evaluate not just the final answer but also tool-use quality, multi-turn tool use, trajectory quality, task success, and final-response quality. Use deterministic checks for structural violations and separate semantic graders for judgment questions. Maintain AgentInspect for local traces and deterministic trajectory checks. Retire or revise cases when necessary, tracking suite composition by intent and failure role.
Monitor the promotion funnel for changes in cases detected, reviewed, accepted, rejected, and retired. This approach ensures the evaluation suite remains relevant and effective.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
