A Detector That Only Ever Says "Clean" Proves Nothing

A self-testing detector that consistently reports "clean" is not sufficient to guarantee its reliability. One script in the author's project incorrectly labeled its control as "NEGATIVE CONTROL" due to a lowercase regex flag, resulting in 13 out of 13 tests appearing clean. This highlights the potential for detectors to miss issues if not tested properly.

The issue is amplified by the proliferation of detectors such as pre-commit hooks, custom lint rules, and audit scripts. These detectors run on every commit and often produce no output, which can create a false sense of security. A detector that finds no problems and another that cannot detect any problems output the same "clean" result, making it impossible to distinguish between a functioning detector and a broken one based solely on its output.

Mutation testing, which perturbs production code and re-runs tests, offers a potential solution. However, it is typically applied to core application code, not the smaller detector scripts written by developers. These smaller scripts often lack a mechanism to verify their own functionality, making them particularly vulnerable to false positives and false negatives.

To address this, the author proposes adding a '--self-test' flag to every detector, paired with positive and negative controls. The positive control should be a case that the detector is designed to catch, while the negative control should be an invented case that resembles a real one but is not. If any control fails, the detector should not print a verdict at all, but rather indicate that it is unsound.

By explicitly asserting the reliability of a detector through self-testing, developers can more confidently rely on the output of their quality assurance machinery.