I Built an Agent to Fix Bad Tests. I Found Eight Bugs in My Own Ruler.
Here is a Python function and a test for it. def withdraw ( balance , amount ): if amount <= 0 : raise ValueError ( " amount must be positive " ) if amount > balance : raise ValueError ( " insufficient funds " ) return balance - amount def test_withdraw (): assert withdraw ( 100 , 30 ) == 70 That test gives you 47% line coverage. It gives you a 9.5% mutation kill score. The harness generates 21…
The author built an agent to automatically generate tests for fixing bad code, specifically focusing on mutation testing. They discovered that many existing tests in popular Python libraries were either too weak or not executing the relevant code at all. This led the author to reframe their project as a measuring instrument rather than a test generator.
The author identified eight bugs in their own measuring instrument, which would have resulted in inaccurate results if not caught. These bugs included incorrect handling of editable installs, silent failures on certain package layouts, concurrency issues, and incorrect test file selection. The author suggests that similar measurement bugs may be common in published results, as better numbers are more likely to be published while flaws go unnoticed.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
