{
  "id": 2664889,
  "title": "\"Trust this file over the exit status\": the note our test suite left itself — that nothing ever read",
  "url": "https://urgent.news/2026/08/22/trust-this-file-over-the-exit-status-the-note-our-test-suite-left",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-22T21:58:12.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/bryanw/trust-this-file-over-the-exit-status-the-note-our-test-suite-left-itself-that-nothing-ever-read-4gej"
  },
  "original_language": "en",
  "account": "This submission is part of DEV's Summer Bug Smash: Smash Stories powered by Sentry. The story revolves around a bug that was incorrectly decided not to be fixed, while a second hidden bug was missed despite the solution already existing. The system in question is an AI agent harness with various test batteries, including security tests, memory-integrity tests, and security fuzzing. If any test battery fails, the chain refuses to let changes land. The goal is for a verdict to be trusted, as no verdict is worse than an untrusted one.\n\nThe crash occurred due to a runtime race in Node v25.9.0 on Windows. Four variants were tested to fix the issue, but the targeted fix crashed 3/6 runs, while doing nothing crashed 0/6 in the same time window. The environment proved more significant than any code change tested.\n\nInstead of fixing the code, the CI was modified to report the crash state distinct from failure. This decision concealed the real bug for a week. A week later, the same battery kept failing, only to pass the next run. The classifier's logic seemed sound, parsing the battery's stdout for passed and failed counts. If the process crashed but the counts looked green, it was classified as CRASH; if it crashed and there were no counts, it was classified as FAIL. The problem arose when the abort and the final console.log were in a race, determined by flush timing, which could not be controlled.\n\nThe real bug was a runtime race, while the apparent defect was a verdict channel that only worked when losing the race. The fix was to make the sidecar a contract, with the runner handing every battery a path and the battery writing its counts there synchronously before exiting. This approach ensures that the verdict never rides on a flush race.",
  "summary": "This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry . This is the story of a bug with two layers: a crash we correctly decided not to fix — and the verdict-flipping second bug hiding behind it that we missed for a week, even though the fix for it already existed, sitting unread on disk with a note that literally said \"trust this file over the exit status.\" The setup I…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}