Every Rule We Added Had a Bug. So We Stopped Adding Rules.
We wrote a handful of small rules on November 12. Each one was a line or two. Each one was obviously right. A read-only review found a real bug in nearly every one of them before they shipped. That is not a story about bad rules. I think it is a story about the wrong kind of question being answered with rules at all. I build Since, which tracks the packages, AI models, APIs and runtimes your code…
On November 12, a handful of small rules were written for a tool called Since. Since tracks packages, AI models, APIs and runtimes that code depends on, and informs developers about changes and fixes. The team added rules for prefixes, keywords and dotted versions of model IDs, but each rule had a bug. A read-only review found a real bug in nearly every rule before they shipped.
The issue was that rules were being used to answer questions that a person would answer by reading the code. For example, determining which provider a model ID goes to requires understanding the context and how the code uses the model. Instead of adding more rules, the team decided to rely on code analysis and human reviewers to determine the correct answer.
They drew a line: code only settles what it can prove. Rules are only used for questions that cannot be answered by the existing code review process. The team also discovered that their existing code for checking pasted snippets had its own smaller matcher, which was not integrated with the main rule set. This caused additional bugs and inconsistencies.
To fix the problem, the team decided to route the snippet check through the shared reader, delete the copy of rules, and write the agreement as a test. They found that rules had been the source of problems before, and it was time to stop being overly strict and allow for more flexibility in code analysis.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.