Ten Packages, One Rule: A Check Must Be Able to Fail

Between August 20th and August 31st, ten small packages were published, with the majority being released on the final day. These packages are verification and measurement tools written in Python and JavaScript, installable via PyPI and npm. Each package demonstrates one way a green check can indicate nothing about the quality of the code it represents. The failure mode behind these packages is not the measured subject, but the instrument reporting success without proper validation.

The research codebase for these packages, like the ones mentioned in the previous posts, is not publicly available. The failure pattern exhibited by these packages can be observed across various programming contexts. For instance, running `go test ./...` on a repository with no test files will print "[no test files]" and exit with a status code of 0, indicating a false sense of success. Similarly, a Pytest suite where every test is skipped will report "2 skipped" and also exit with a status code of 0.

Another common failure pattern involves JUnit reports, where a run with 50 tests marked as skipped will display "tests= 50 skipped= 50". This appears to be a green run of nothing wearing the total. Lint rules disabled through a configuration merge are also known to report as green, regardless of the actual code quality. In each of these cases, a check has stopped checking, and the repository does not exhibit any visible signs of issues.

The underlying principle governing all ten packages is encapsulated in the README of the didrun package: a check must independently answer whether it ran, whether it failed, and whether the failure is the correct one. If any of these three aspects are not separately addressed, defects in this family of packages can arise. The corollary to this principle is the title of this account: every one of the ten commits must contain at least one test that would report the package's own premise as false.

This test is not about the tool's functionality, but rather about the tool's ability to detect flaws in its own implementation.

Two recurring themes in these packages are the use of a ladder and a witness. A ladder is a predetermined list of probe inputs that a tool walks through sequentially, while a witness is an input that produces different results under different conditions. The assay-checks package exemplifies both of these concepts, as it serves as a control for evaluating the correctness of other tools.

Three key controls are worth highlighting due to their unique contributions to the testing framework. Test_in_process_repetition_would_have_missed_it verifies that calling a function 20 times within a single interpreter does not reveal variations, while the same function executed in fresh processes does expose those variations. This test ensures that the nondeterministic behavior is accurately detected.

Test_SIGTERM_leaves_a_try_finally_harness_broken spawns a child process running a try/finally harness, kills it, and asserts that the mutated file remains altered. This test is crucial because try/finally statements are not guaranteed to execute on SIGTERM signals.

Test_without_the_gate_the_same_function_is_pinned_and_the_pin_is_flaky demonstrates the importance of maintaining determinism in code execution. By disabling the determinism gate, pinning a function whose output depends on hash order, and then changing the underlying tree without any visible modifications, this test reveals whether the pinning tool accurately detects changes. If the tool fails to report a change, it indicates a flaw in its ability to identify defects in the codebase.

These ten packages collectively serve as a diagnostic toolkit, providing various methods to uncover hidden defects and ensure the reliability of code. By following the rule that every check must answer separately whether it ran, failed, and whether the failure is the correct one, developers can maintain a high level of code quality and prevent the propagation of unnoticed errors.