My Extraction Score Was 0.08 and the Model Was Innocent: Rebuilding the Ruler
Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI . It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.1 field test report evaluated 2 cloud models across 40 corpora and 4,742 trajectory-runs and is the…
CauterRule, an open-source tool that learns standing rules from repeated agent failures, has been released on GitHub and PyPI. It evaluates the performance of cloud models across 40 corpora and 4,742 trajectory-runs. The v0.3.1 release introduced a direct extraction-accuracy metric to address the question of when a correct but reworded rule deserves credit.
The initial reading of the metric was 0.08, which was puzzling until the team looked deeper. The metric was found to be penalizing the model for rephrasing the rule, rather than rewarding it for extracting the correct trigger. The fix involved adjusting the agreement calculation to focus only on the trigger portion of the rule. This change led to a more accurate representation of the model's performance, with agreement scores ranging from 0.74 to 0.93.
Additionally, several smaller fixes were made to ensure the metric accurately reflects the data and the state of the corpus.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
