Your eval suite passes. I built the tool that checks whether it checks anything.

Three weeks ago, the reporter questioned the effectiveness of their own LLM regression suite by asking whether a model could silently get worse in an area they cared about without triggering any checks. Rather than reasoning about it, they created a tool to answer the question mechanically, similar to how mutation testing works for ordinary code.

The tool, named evalmut, injects a known defect into the system being tested, runs the evaluation suite, and reports which checks remained green. A surviving mutation represents a weakness in the evaluation process.

Evalmut is now publicly available for installation via pip. The tool includes 18 mutation operators, each backed by a real, documented defect found in production failures or issue trackers. The operators were carefully selected and are never added for the purpose of artificially increasing coverage. Evaluations using evalmut are deterministic and reproduceable, with no LLM involved in making judgments. The results are presented as either red or green indicators, ensuring reproducibility.

The development of evalmut went through eight rounds of adversarial cold-critique before the reporter trusted it. In the initial stages, the tool itself was found to have false positives. However, by the sixth round, false positives on well-formed suites dropped to zero, maintaining that level of accuracy. When an empty suite is encountered, evalmut purposely exits with a non-zero status code, indicating that no holes were found in the evaluation process.

The reporter has made evalmut available on GitHub (MIT licensed, version 0.1-paper) along with a short paper detailing the methodology used. They encourage anyone running a trusted evaluation suite to run evalmut on it before placing increased trust in the results. The reporter is open to receiving feedback on what evalmut uncovers in their suite.