{
  "id": 7001887,
  "title": "The Model Wrote the Right Rule and My Replay Rejected It: The Extraction-vs-Replay Split",
  "url": "https://urgent.news/2026/09/12/the-model-wrote-the-right-rule-and-my-replay-rejected-it-the",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-12T22:04:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/debashish_ghosal/the-model-wrote-the-right-rule-and-my-replay-rejected-it-the-extraction-vs-replay-split-4304"
  },
  "original_language": "en",
  "account": "The Model Wrote the Right Rule, Yet My Replay Rejected It: The Extraction-vs-Replay Split\n\nA new tool, CauterRule, has been released. This open-source sidecar learns standing rules from repeated agent failures, extracting lessons from trajectories, replay-testing them, and attempting to separate useful guidance from noisy overgeneralization. The tool is the result of two releases, during which it was treated as a single problem, but it is now clear that the issue is actually two problems fused together.\n\nThe pipeline of CauterRule has two main parts that answer two different questions: Extraction and Replay/Evaluation. Extraction asks whether a model produces the correct rule given a failure, while Replay/Evaluation checks if a rule can be verified against history. However, these two parts were measured using only one metric, leading to confusion and misattribution of blame.\n\nTo illustrate the issue, consider the example trajectory F-001. The expected rule was \"when git push fails with non-fast-forward, pull latest changes before pushing.\" The model successfully extracted this rule, but during replay, it scored the rule as \"inconclusive.\" This was because three unrelated successes were counted as \"broken\" by the git-push rule due to a shared token \"git,\" despite the rule being correct. The model wrote the right rule, but the grader rejected it based on superficial wording similarity.\n\nThe problem lies in assuming that the replay verdict tells us whether the extracted rule is good. In reality, it only tells us whether the extracted trigger's surface form resembles stored reference surface forms. A correct rule phrased differently may score poorly, while an incorrect rule that happens to share vocabulary can score high. The metric used is a text-similarity proxy, which is the gate for the replay process. This assumption led to the undercounting of extraction accuracy and the underestimation of replay fidelity.\n\nThe two halves of the pipeline need to be separated to avoid misattribution of blame. Previously, low pass rates were seen as a symptom and the model's fault. However, the true variables are extraction accuracy and replay fidelity. By separating these two, the model can no longer be blamed for the grader's poor evaluation. The ground truth for extraction accuracy is already present in the corpus and does not require additional labeling.\n\nThe release of CauterRule is a significant step forward, as it addresses the issue of misattribution and provides a clearer picture of the model's performance. F-001, with its correct rule and inconclusive replay verdict, serves as a prime example of the problem and the solution. By recognizing that the replay metric masquerades as validation, we can move towards a more accurate assessment of the model's performance.",
  "summary": "Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI . It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.0 field test report evaluated 2 cloud models across 40 corpora and 4,768 trajectory-runs and is the…",
  "key_points": [
    "CauterRule tool extracts rules from agent failures and tests them via replay.",
    "Extraction and Replay/Evaluation are two distinct pipeline parts.",
    "Replay verdict only checks surface form similarity, not rule correctness."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}