A Rule Can Be Specific and Still Be Too Broad

A recent open-source tool called CauterRule aims to turn repeated agent failures into permanent standing rules by learning from them. The tool extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization. The developers evaluated the tool using four models across 745 trajectories and released a field test report.

The specificity scorer of CauterRule is designed to identify specific and concrete triggers, such as "when git push fails with non-fast-forward," while also flagging generic triggers. However, the tool has revealed that specific triggers can sometimes still be too broad, leading to unintended consequences when the trigger matches both failed and successful trajectories.

For example, a trigger like "when git push fails with non-fast-forward" is specific about the tool, the action, and the error. However, it may also match clean git push success trajectories in the reference corpus, causing the trigger to fire unnecessarily. This demonstrates that specificity does not necessarily mean safety. The trigger is specific about the failure but is broad about when it fires, which can lead to breaking clean successes.

The v0.2.0 release introduces a broad-trigger penalty to address this issue. The penalty distinguishes between three cases: dangerous (dangerously broad), broad but fixable, and safe. The specific triggers that are too broad are classified as "broken but ≤ prevented," meaning they prevent real failures but also break some successes.

These triggers are not dangerously broad but are still not safe to promote. The broad-trigger penalty helps ensure that the triggers are both specific enough to match the failure and broad enough not to break clean successes.