A test that has never failed is a claim, not evidence
I broke a TOML parser's timezone handling on purpose. Its 680 official conformance tests stayed green. Here is why, and the habit that catches it. tomlkit @ 4b38bec — parse_rfc3339, negative UTC offset sign dropped before: if sign == "-": offset = -offset after: if False: offset = -offset tests/toml-test ····································· 680 passed tests/test_utils…
A test that has never failed is not necessarily evidence of correct code. The author intentionally broke a TOML parser's timezone handling, and all 680 official conformance tests remained green. This occurs because the test file constructs the expected value for every datetime case using the same function being tested, creating a drift between the parser and expectation.
Only when the code under test is broken does the test detect the issue. The author developed a method called "falsifiable-tests" to ensure tests can be observed failing for the right reason, rather than just being a claim. This involves running the test against the correct code, checking that it actually ran, and then breaking the code under test to verify it fails as expected.
The method also includes counting the number of suite runs needed to prove that N mutations will cause at least N+2 suite runs. The author's approach has caught issues in various projects, including tomlkit, tenacity, and click, by identifying tests that are not truly verifying the code. The author emphasizes honesty when a test cannot run and using a harness that edits the real file, runs the test command, and reports which tests noticed each mutation and which mutations went unnoticed.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
