{
  "id": 20057304,
  "title": "Do your tests really catch bugs? / Seus testes pegam erro de verdade?",
  "url": "https://urgent.news/2026/11/14/do-your-tests-really-catch-bugs-seus-testes-pegam-erro-de-verdade",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-14T21:54:34.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/arthur2122/do-your-tests-really-catch-bugs-seus-testes-pegam-erro-de-verdade-4560"
  },
  "original_language": "en",
  "account": "I measured this alongside Claude, conducting research and tests to create a digital garden exploring whether tests truly catch bugs. The concept is straightforward: a test may pass yet remain inadequate. To determine the extent of this inadequacy, I deliberately introduced flaws into the code and counted the number of bugs that managed to slip through the tests. Subsequently, I replicated this process for each stage of my AI workflow, from the specification outlining the desired functionality to the final commit. In the digital garden, each stage is assigned a unique number, and notes detailing ideas that did not prove fruitful are also included. The garden also features a note addressing uncertainties, given the study's limited scope, its origin from a few projects, and the absence of external verification. Additionally, a step-by-step guide is provided for replicating the measurement in your own project. A digital garden is an interconnected collection of notes maintained by the author over time, accommodating unfinished notes and displaying them with a badge indicating the level of provenance for each. To view the digital garden, visit Digital-Garden.",
  "summary": "I measured this together with Claude, ran research and tests, and built a garden about whether tests really catch bugs. The idea is simple: a test can pass and still be weak. To find out how weak, I broke the code on purpose and counted how many bugs the tests let through. Then I did the same for each step of my AI workflow, from the spec (the text that describes what the code should do) to the…",
  "key_points": [
    "Tests can pass without catching all bugs",
    "Conducted research to measure test adequacy",
    "Guide provided for replicating the study"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}