My detector was blind across 1,029 of 6,309 lines and its three tests stayed green
The guard worked. I had written it so that no session open in one project could touch another project's files, it had three tests, and all three were green. They had been green for months. It was blind across 1,029 of the 6,309 lines it was watching. Eight hundred and twenty-four of them executable. And the three tests kept passing, because all three looked at paths that were already clean. The…
The guard successfully ensured individual project files could not be accessed from other projects, with all three tests remaining green for months. However, it was blind to 1,029 out of 6,309 monitored lines, including 824 executable lines. The issue did not lie in the detector but in how it was tested. Test cases should be derived from the system's healthy state, not just negative examples.
A bench composed solely of negative cases comes out green regardless of the detector's functionality. The detector's true test should include both positive and negative cases, with the positive case failing through the actual code path. The detection mechanism should also be tested against synthetic input, including files that both contain and do not contain the violation.
Running the new pair of tests against the previous version of the rule can reveal if the fix discriminates between the correct and flawed rule. In this case, a false positive was identified, which led to the discovery of a silent false negative. A guard that emits false positives must be inspected for potential false negatives, which can often be found by examining discipline and rigor in the testing process.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
