{
  "id": 436975,
  "title": "Three checks in our codebase that could not fail",
  "url": "https://urgent.news/2026/08/10/three-checks-in-our-codebase-that-could-not-fail",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-10T04:40:30.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/artificial_wasteland/three-checks-in-our-codebase-that-could-not-fail-4e3l"
  },
  "original_language": "en",
  "account": "Three checks within our codebase proved to be structurally incapable of failing, despite being designed to catch potential issues. This discovery was made through a thorough examination of our site, which requires every factual claim to be verified and displayed. The three checks shared a common trait, a shape that seems to be prevalent in many test suites.\n\nThe first check relied on a verifier that contained a line-for-line copy of the rule it was supposed to enforce. Instead of checking the actual implementation, the verifier compared its own expectations against the copy, resulting in a perfect agreement despite both inheriting the same bug. The check never received the data it was meant to verify, creating a false sense of security.\n\nThe second check was a pinned result that asserted its own arithmetic. The page displaying this check claimed a PASS with 109/109 assertions, while the actual verifier reported a FAIL with 106/109 assertions. The issue persisted for two weeks before being corrected by a single line of code. The corrected version ensures that the failures are read before the assertion is made, resulting in a fixed point.\n\nThe third check was an estimator used in a page about Zipf's law. It fit a power law to word frequencies and printed the exponent, citing a continuous maximum-likelihood estimator. However, the estimator had been wrong for two years, and all the checks tolerated this error because a plausible number is not a valid check. The estimator's right-hand side was based on a range wide enough to contain any answer the estimator could produce.\n\nThese findings highlight the importance of designing assertions that are based on the thing being tested, rather than a function of the thing itself. A check whose right-hand side moves with the code cannot fail, as it confirms a derivation. Additionally, introducing a defect to test the check's effectiveness is a crucial step in ensuring its proper functioning. With these lessons learned, we can now focus on improving our test suite by planting known answers, reproducing published numbers, and intentionally breaking the system to confirm the checks' effectiveness.",
  "summary": "Last night I found three checks in our own codebase that could not fail. Not checks that were failing to catch things. Checks that were structurally incapable of ever going red, while reporting green forever. We run a site where every factual claim has to be checked and the check shown. That is the whole product. So this was worth understanding properly, and the three had the same shape…",
  "key_points": [
    "Three code checks in the system are structurally incapable of failing",
    "First check uses verifier with line-for-line copy of the rule it enforces",
    "Second check asserts its own arithmetic, reporting PASS with mismatched assertions"
  ],
  "editors_take": "The discovery of three structurally infallible checks in the codebase underscores the need for designing assertions based on the thing being tested, not the thing itself, to ensure effective validation.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}