{
  "id": 7196859,
  "title": "My Extraction Score Was 0.08 and the Model Was Innocent: Rebuilding the Ruler",
  "url": "https://urgent.news/2026/09/13/my-extraction-score-was-0-08-and-the-model-was-innocent-rebuilding",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-13T23:08:45.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/debashish_ghosal/my-extraction-score-was-008-and-the-model-was-innocent-rebuilding-the-ruler-2fc1"
  },
  "original_language": "en",
  "account": "CauterRule, an open-source tool that learns standing rules from repeated agent failures, has been released on GitHub and PyPI. It evaluates the performance of cloud models across 40 corpora and 4,742 trajectory-runs. The v0.3.1 release introduced a direct extraction-accuracy metric to address the question of when a correct but reworded rule deserves credit. The initial reading of the metric was 0.08, which was puzzling until the team looked deeper. The metric was found to be penalizing the model for rephrasing the rule, rather than rewarding it for extracting the correct trigger. The fix involved adjusting the agreement calculation to focus only on the trigger portion of the rule. This change led to a more accurate representation of the model's performance, with agreement scores ranging from 0.74 to 0.93. Additionally, several smaller fixes were made to ensure the metric accurately reflects the data and the state of the corpus.",
  "summary": "Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI . It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.1 field test report evaluated 2 cloud models across 40 corpora and 4,742 trajectory-runs and is the…",
  "key_points": [
    "CauterRule tool evaluates cloud models across 40 corpora and 4,742 trajectory-runs",
    "Extraction-accuracy metric initially 0.08, later adjusted to focus on trigger portion",
    "Adjusted metric shows agreement scores from 0.74 to 0.93, accurate performance representation"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}