Three checks in our codebase that could not fail

Three checks within our codebase proved to be structurally incapable of failing, despite being designed to catch potential issues. This discovery was made through a thorough examination of our site, which requires every factual claim to be verified and displayed. The three checks shared a common trait, a shape that seems to be prevalent in many test suites.

The first check relied on a verifier that contained a line-for-line copy of the rule it was supposed to enforce. Instead of checking the actual implementation, the verifier compared its own expectations against the copy, resulting in a perfect agreement despite both inheriting the same bug. The check never received the data it was meant to verify, creating a false sense of security.

The second check was a pinned result that asserted its own arithmetic. The page displaying this check claimed a PASS with 109/109 assertions, while the actual verifier reported a FAIL with 106/109 assertions. The issue persisted for two weeks before being corrected by a single line of code. The corrected version ensures that the failures are read before the assertion is made, resulting in a fixed point.

The third check was an estimator used in a page about Zipf's law. It fit a power law to word frequencies and printed the exponent, citing a continuous maximum-likelihood estimator. However, the estimator had been wrong for two years, and all the checks tolerated this error because a plausible number is not a valid check. The estimator's right-hand side was based on a range wide enough to contain any answer the estimator could produce.

These findings highlight the importance of designing assertions that are based on the thing being tested, rather than a function of the thing itself. A check whose right-hand side moves with the code cannot fail, as it confirms a derivation. Additionally, introducing a defect to test the check's effectiveness is a crucial step in ensuring its proper functioning.

With these lessons learned, we can now focus on improving our test suite by planting known answers, reproducing published numbers, and intentionally breaking the system to confirm the checks' effectiveness.