My code checker was wrong. How I turned its false positives into tests
Disclosure: I'm the author of rebar , the open-source checker this post is about. It is alpha. rebar fails a CI job when a repository breaks a rule. Examples: formatting nobody checks, a hook git will never run, a credential in a tracked file. A checker like that can fail in one way that matters more than any other: accusing a repository that is doing the right thing. After a couple of those,…
This story recounts the author's experience with a code checker tool called rebar, and how they discovered that some of its false positives could be turned into proper tests. The author is the creator of rebar, an open-source checker with several rules, such as formatting, git hooks, and hardcoded secrets. False positives caused rebar to flag valid code, leading people to disengage with the checker.
The author tested rebar against 28 public repositories, reading the checker's accusations and identifying three cases where the accusations were incorrect. Each case followed the same structure: the accusation, evidence proving the accusation was wrong, the fix to address the false positive, and the test to ensure the fix remains effective.
In the first case, rebar falsely accused the repository mastra of having a non-executable Git hook. The author demonstrated that the issue was due to husky 9 managing the hooks differently, and fixed the problem by exempting husky-managed hooks from the rule. The tests confirmed that the fix worked both ways, failing on the old code and catching the real violation.
The second case involved rebar flagging the AWS credential in dotenv's documentation example as a hardcoded secret. The author explained that deleting the line from history was insufficient, as the credential was still present in the repository. The fix involved rotating the key and adding tests to ensure the fix would not inadvertently introduce new issues. The author then expanded the test to include a broader search rule, and found that 151 findings were removed without any added false positives.
Finally, the third case concerned rebar flagging formatting issues in e2b's repository, which led to the job failing when a file was out of format. The author identified that e2b's lint workflow ran formatting and then checked for any changes in the status. The fix recognized this pattern and adjusted rebar's rule accordingly. The tests confirmed that the fix worked in both scenarios and that the other repositories remained unaffected.
Overall, the author concluded that turning false positives into tests proved effective in catching real violations, improving the reliability and trustworthiness of rebar.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
