{
  "id": 8119468,
  "title": "69 Tests. All Passing. Zero Bugs Caught.",
  "url": "https://urgent.news/2026/09/17/69-tests-all-passing-zero-bugs-caught",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-17T22:19:31.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/marvinoka4/69-tests-all-passing-zero-bugs-caught-27k5"
  },
  "original_language": "en",
  "account": "A Python module was subjected to 69 tests generated by an AI model, all of which passed. However, the model had intentionally inserted eleven bugs into the module, and none of the tests detected any of them. A second test setup, focused on the specific bugs, required 17 attempts to catch ten of the eleven bugs. This contrast highlights the purpose of the project: to understand the limitations of existing test suites in detecting faults.\n\nThe AI model created a mutation-testing harness that alters the source code in small ways, such as flipping a comparison or altering a constant, then runs the existing test suite and records which changes cause the suite to fail. A change that goes unnoticed by the test suite indicates a fault that the tests are unable to detect. While coverage measures which lines of code are executed, mutation testing reveals whether those executed lines would have failed if the code contained errors.\n\nThe project tested twelve widely used Python libraries, generating 455 mutations. Out of these, 53 mutations survived the existing tests and were located on lines that the tests actually executed. The AI model successfully caught one prompt, leading to the creation of 556 tests: 9 tests per function call, with no targeting; 53 tests targeting specific faults; and 44 tests aimed at the specific faults with a pass/fail gate. Hand-auditing the nine tests that were missed revealed that seven of them were provably unkillable, and six of those were due to type annotations inside TYPE_CHECKING blocks that never execute at runtime.\n\nThe findings from this project emphasize three important points: most undetected faults are due to unreached code, not weak assertions; only 53 of 133 surviving mutations sit on lines executed by the tests; and the gate, which checks if a test passes or fails, turns out to have only one job in practice: to ensure that tests detect faults they were shown. Despite the promising results, only one-third of the 44 tests that caught faults generalize to other functions, and 36 of these tests catch only one mutation each. This highlights the limitations of automated testing and the importance of thoroughly evaluating evaluation instruments used in testing processes.",
  "summary": "An AI model wrote 69 tests for a Python module. Every one passed. Together they caught zero of the eleven bugs I had deliberately planted in that module. A second setup, pointed at the specific bugs rather than at the module, used 17 attempts and caught ten. That contrast is the whole project. What I built A mutation-testing harness. It changes source code in small ways, flips a comparison,…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}