I built a tool to catch empty scans read as passes, and shipped it with an empty scan read as a pass
The tool has a self-test. It plants two positive fixtures that must be found and four negative fixtures that must stay quiet. Then it prints a control line, and you read the real numbers only after that line is green. It printed the control line. All four negative fixtures were clean. Both positives were reported as seen. The skip list contained fixtures . Zero files were scanned. Every fixture…
A developer created a tool to detect empty scans being mistakenly interpreted as successful scans. The tool included a self-test with positive and negative fixtures, a control line, and mutation controls. However, the self-test failed when scanning an empty set, reading silence as agreement, and not noticing the issue until later.
The tool's developers had previously written down the principle of not folding unmeasurable measurements into failures or successes, but they broke their own rule when writing the self-test. The mutation controls only covered two of the five signals, leaving three untested. The developers did not run the self-test on another machine or consider all possible failure cases.
A bug was discovered that the fixtures did not cover, as a person manually reading a file found the defect when the machine reported zero. The self-test was run only on a Windows-shaped machine, and the skip list interacted with path separators. The repository with runnable reproductions of the defects can be found at https://github.com/mahirhir/unanswered-approval.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
