48-Hour Field Notes: My Sandbox Said PASS, and Zero Tests Had Run
I spent 48 hours letting a free model generate fixes I never applied, and the most useful thing I learned was that my harness was the bug. The sandbox told me PASS while zero tests had run, then I hashed the wrong artifact, and later I blamed a patch for a timezone problem I had written myself. Here is the loop I ended up with, the three false signals I had to remove, and the escalation rule I…
In a 48-hour experiment, a reporter discovered that a free model generated fixes which were never applied, leading to a false PASS result. The issue stemmed from a bug in the harness, and the reporter blamed a patch for a timezone problem. The reporter learned the importance of generating multiple candidates, running them in a disposable environment, and maintaining a receipt for each attempt.
The process involved three stages: generating a text output, replaying each candidate in a disposable run, and triaging a machine-readable receipt to determine what requires attention. The reporter realized that relying solely on exit codes from tests was misleading, as a zero exit code did not necessarily indicate successful tests.
To address this, the reporter introduced a new verdict, "EMPTY," for runs with zero collected tests, regardless of the exit code. This change helped the reporter accurately identify and reclassify certain successes as harness problems rather than candidate issues.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
