When Your Judge Can't Decide
Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI . It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, MCP server, 7 export formats, and a bundled git rule pack. The field test report evaluated 4 models across 394 trajectories and is the source for every number in this article. Release…
The article details the findings of a field test evaluating four AI models across 394 trajectories. Over half of the results were inconclusive, meaning the testing tool could not decide whether the AI model's rule was good or bad. The strongest cloud model, Meta-LLama's Llama-3.1-8b-instruct, had the lowest inconclusive rate at 42.9%, while the paid cloud model GPT-4o-mini had the highest at 62.9%.
This suggests that the AI models produced candidates that the replay engine could not evaluate, indicating different types of failures that require different fixes. The issue seems to stem from the replay engine's matcher, which uses simple substring and token-overlap heuristics to decide if a rule is good or bad. This heuristic can either over or underestimate the rule's effectiveness, leading to inconclusive results.
The test also revealed that raw corpora, especially those from sibling repositories, produced a higher percentage of inconclusive results, up to 100% in some cases. This suggests that the issues with the replay engine's matcher are more pronounced when dealing with raw data, highlighting the need for better instrumentation and evaluation methods.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
