Do your tests really catch bugs? / Seus testes pegam erro de verdade?

I measured this alongside Claude, conducting research and tests to create a digital garden exploring whether tests truly catch bugs. The concept is straightforward: a test may pass yet remain inadequate. To determine the extent of this inadequacy, I deliberately introduced flaws into the code and counted the number of bugs that managed to slip through the tests.

Subsequently, I replicated this process for each stage of my AI workflow, from the specification outlining the desired functionality to the final commit. In the digital garden, each stage is assigned a unique number, and notes detailing ideas that did not prove fruitful are also included. The garden also features a note addressing uncertainties, given the study's limited scope, its origin from a few projects, and the absence of external verification.

Additionally, a step-by-step guide is provided for replicating the measurement in your own project. A digital garden is an interconnected collection of notes maintained by the author over time, accommodating unfinished notes and displaying them with a badge indicating the level of provenance for each. To view the digital garden, visit Digital-Garden.