The Model Wrote the Right Rule and My Replay Rejected It: The Extraction-vs-Replay Split

The Model Wrote the Right Rule, Yet My Replay Rejected It: The Extraction-vs-Replay Split

A new tool, CauterRule, has been released. This open-source sidecar learns standing rules from repeated agent failures, extracting lessons from trajectories, replay-testing them, and attempting to separate useful guidance from noisy overgeneralization. The tool is the result of two releases, during which it was treated as a single problem, but it is now clear that the issue is actually two problems fused together.

The pipeline of CauterRule has two main parts that answer two different questions: Extraction and Replay/Evaluation. Extraction asks whether a model produces the correct rule given a failure, while Replay/Evaluation checks if a rule can be verified against history. However, these two parts were measured using only one metric, leading to confusion and misattribution of blame.

To illustrate the issue, consider the example trajectory F-001. The expected rule was "when git push fails with non-fast-forward, pull latest changes before pushing." The model successfully extracted this rule, but during replay, it scored the rule as "inconclusive." This was because three unrelated successes were counted as "broken" by the git-push rule due to a shared token "git," despite the rule being correct. The model wrote the right rule, but the grader rejected it based on superficial wording similarity.

The problem lies in assuming that the replay verdict tells us whether the extracted rule is good. In reality, it only tells us whether the extracted trigger's surface form resembles stored reference surface forms. A correct rule phrased differently may score poorly, while an incorrect rule that happens to share vocabulary can score high.

The metric used is a text-similarity proxy, which is the gate for the replay process. This assumption led to the undercounting of extraction accuracy and the underestimation of replay fidelity.

The two halves of the pipeline need to be separated to avoid misattribution of blame. Previously, low pass rates were seen as a symptom and the model's fault. However, the true variables are extraction accuracy and replay fidelity. By separating these two, the model can no longer be blamed for the grader's poor evaluation. The ground truth for extraction accuracy is already present in the corpus and does not require additional labeling.

The release of CauterRule is a significant step forward, as it addresses the issue of misattribution and provides a clearer picture of the model's performance. F-001, with its correct rule and inconclusive replay verdict, serves as a prime example of the problem and the solution. By recognizing that the replay metric masquerades as validation, we can move towards a more accurate assessment of the model's performance.