Your evals pass. That doesn't mean they work.
Last week I wrote about tracelint , a linter that catches structural bugs in agent traces — the classic one being an agent that calls charge_card , gets a failure back, and just... keeps going and tells the customer their order shipped. The response was better than I expected, and one question kept coming up in different forms: "Okay — but wouldn't my eval suite have caught that anyway?" Fair…
This article explores the limitations of evaluation suites in software systems. It describes a tool called muteval that can identify gaps in eval suites by deliberately injecting errors into the system and measuring which tests fail to detect them. The key insight is that a green eval run only confirms that the system passed the current tests, but does not guarantee the tests would catch future regressions.
The article demonstrates how combining muteval with a structural trace linting tool can uncover missing eval checks, such as detecting when a system incorrectly reports a declined payment as successful. The author emphasizes that mutation testing and deterministic trace checking are complementary approaches that should be used together to improve eval suite coverage.
However, the article also cautions that mutation coverage percentages alone are not sufficient to validate the correctness of a system, and points out limitations in the current approach, such as the need for a re-runnable system and the distinction between sensitivity and validity.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
