{
  "id": 14835025,
  "title": "48-Hour Field Notes: My Sandbox Said PASS, and Zero Tests Had Run",
  "url": "https://urgent.news/2026/10/16/48-hour-field-notes-my-sandbox-said-pass-and-zero-tests-had-run",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-16T01:11:55.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/codepy_1473/48-hour-field-notes-my-sandbox-said-pass-and-zero-tests-had-run-5hll"
  },
  "original_language": "en",
  "account": "In a 48-hour experiment, a reporter discovered that a free model generated fixes which were never applied, leading to a false PASS result. The issue stemmed from a bug in the harness, and the reporter blamed a patch for a timezone problem. The reporter learned the importance of generating multiple candidates, running them in a disposable environment, and maintaining a receipt for each attempt. The process involved three stages: generating a text output, replaying each candidate in a disposable run, and triaging a machine-readable receipt to determine what requires attention. The reporter realized that relying solely on exit codes from tests was misleading, as a zero exit code did not necessarily indicate successful tests. To address this, the reporter introduced a new verdict, \"EMPTY,\" for runs with zero collected tests, regardless of the exit code. This change helped the reporter accurately identify and reclassify certain successes as harness problems rather than candidate issues.",
  "summary": "I spent 48 hours letting a free model generate fixes I never applied, and the most useful thing I learned was that my harness was the bug. The sandbox told me PASS while zero tests had run, then I hashed the wrong artifact, and later I blamed a patch for a timezone problem I had written myself. Here is the loop I ended up with, the three false signals I had to remove, and the escalation rule I…",
  "key_points": [
    "Reporter found free model generated fixes never applied, leading to false PASS",
    "Bug in harness caused issue, blamed patch for timezone problem",
    "Introduced new verdict \"EMPTY\" for runs with zero collected tests"
  ],
  "editors_take": "Relying on simple exit codes creates a dangerous illusion of success that requires more rigorous verification through disposable environments and machine-readable receipts to distinguish real fixes from harness failures.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}