{
  "id": 10112058,
  "title": "Your check says PASS. It looked at zero files.",
  "url": "https://urgent.news/2026/09/27/your-check-says-pass-it-looked-at-zero-files",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-27T02:02:23.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/el_e_3dc94dfc336bfb025ef2/your-check-says-pass-it-looked-at-zero-files-4dji"
  },
  "original_language": "en",
  "account": "A verification step that yields no results appears identical to one that uncovers no issues. Both print PASS, and both exit with 0. One of them performed actual work, while the other did not. The author experienced this bug three times within the same codebase before recognizing its nature. The issue centers around a scanner that scans a directory and reports any leaked secrets. The code iterates through the provided paths, scans each file, and accumulates any findings. If there are findings, the script exits with a non-zero status. Otherwise, it prints PASS and exits successfully. This behavior proved effective for two months, with the scanner reporting PASS before every release. However, the author discovered that passing a list of files instead of a directory resulted in zero files being scanned, zero findings, and a misleading PASS exit. This caused every release to be \"verified\" by a scanner that examined nothing at all, without any errors or warnings. The log line, which indicated \"files scanned: 0,\" went unnoticed as the accompanying word was PASS. The author realized that this type of bug is particularly insidious because it presents itself as a successful verification, despite having done nothing. It survives due to three key factors: the output being indistinguishable from a successful scan, the guard failing to block, and accumulating green runs creating false confidence. To address this issue, the author implemented a single-line policy: if scanned == 0: print(\"BLOCK: scanner matched zero files — refusing to report PASS\") sys.exit(3) This exit code (3) clearly distinguishes the \"could not do my job\" scenario from a genuine problem (exit code 1), or a missing path (exit code 2). The author also added a line to explicitly state what the scanner looked at, including the number of files scanned, to prevent relying on the reassuring word \"PASS\" alone. This bug is not unique to this particular case, and the author has observed it manifest in various contexts: silent-zero failures in other checks, pattern matches in tests, ignored files in linters, empty manifests in backup verifications, and greppy guards in CI pipelines. Each of these scenarios can lead to the same deceptive success, where a check reports PASS without providing any information about what it examined. The author emphasizes the importance of asking what a check examined when it reports success. If the check cannot provide a number, it should be considered invalid evidence. In practice, this means that any check that gates critical actions, such as releases, deployments, or payments, should print the denominator (the total number of items examined) instead of just the verdict. The exit code should be distinct for \"found a problem\" versus \"could not run.\" Additionally, a regression test should be written to feed the checker an input that matches nothing and assert that it fails, which is often overlooked. The author has packaged this concept into a simple tool called zero_match_guard, which is a four-line Python script that blocks when a glob matches nothing and exits with a distinct code. The tool is available on GitHub and can be used in various projects to ensure that checks provide meaningful feedback.",
  "summary": "A verification step that finds nothing looks exactly like a verification step that finds nothing wrong. Both print PASS . Both exit 0. Only one of them actually did anything. I shipped this bug three times in the same codebase before I understood the shape of it. The bug A scanner walked a directory and reported leaked secrets: def main ( paths ): findings = [] for root in paths : for f in glob .…",
  "key_points": [
    "Scanner reports PASS even when scanning zero files",
    "Misleading PASS exit code causes false confidence",
    "New policy requires printing scanned file count"
  ],
  "editors_take": "A silent failure to scan any files masquerades as success when a check reports PASS without specifying what it examined, highlighting the need for checks to provide detailed feedback.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}