{
  "id": 17136,
  "title": "The Hardest Bug to Find Was a False Positive.",
  "url": "https://urgent.news/2026/08/02/the-hardest-bug-to-find-was-a-false-positive",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-02T01:00:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/mohi_uddin/the-hardest-bug-to-find-was-a-false-positive-48cc"
  },
  "original_language": "en",
  "account": "On Tuesday, the company released Proactive Scan, a tool that evaluates risky files and analyses them contextually before any crashes occur. To evaluate its effectiveness, the company published a detailed benchmark. The benchmark encompassed all the data, methodology, and the instances where the initial version of the scan incorrectly flagged code (false positives).\n\nThe benchmark comprised 28 test cases, including 20 files with intentionally embedded defects and 8 files serving as controls. The categories measured were single-file defects, cross-file defects, and false alarms on clean files. For single-file defects, the tool scored perfectly, correctly identifying 8 out of 8 planted defects. Cross-file defects were also accurately detected by the tool, with all 12 expected defects flagged correctly. Most impressively, the tool did not produce any false alarms on the clean files, achieving a score of 0/8.\n\nThe benchmark was run on July 2, 2026, against the company's staging API, which operates on the same pipeline as the production environment. The API utilizes Voyage embeddings for indexing the codebase and Claude Haiku 4.5 for analysis.\n\nIn response to concerns about the presence of competitors in the benchmark, the company explained that they deliberately wrote all the cases themselves to ensure that the models hadn't been trained on any public sources containing similar bugs. This decision aimed to assess the tool's ability to analyze code independently without prior exposure to known issues.\n\nThe 28 cases were meticulously crafted in late June, each representing a unique failure class but not copied from any existing public repositories. The goal was to create original and realistic scenarios that would challenge the scan's capabilities.\n\nOne illustrative example is XF-05, which involves two files. In the first file, the value of `WINDOW_SLOTS` is set to 0, while in the second file, `rate.py`, the code attempts to divide by `WINDOW_SLOTS`. When the scan is run without the context of `limits.py`, it does not flag this as an issue. However, with the cross-file context, it correctly identifies the `ZeroDivisionError`.\n\nScoring is based on predefined signals for each test case, which dictate the expected keywords and issue classes that must be mentioned in the findings. For instance, XF-05 requires the scan to report either \"division by zero\" or the name of the offending constant. The grading is lenient, focusing on whether the scan detects the problem, irrespective of the phrasing used. The company published the benchmark, expected signals, and runner for transparency, allowing others to verify their scoring.\n\nThe benchmark revealed a significant challenge: false positives. Despite the tool's success in detecting bugs, it also flagged clean code, leading to numerous instances of false alarms. The company iterated quickly to address this issue through several versions of the scan. Initially, the first version (scan-v3) flagged not only actual bugs but also style preferences in clean files, resulting in a significant number of false positives. The second version (scan-v4) addressed this by refining the prompts to ensure that findings were tied to concrete failure scenarios with identifiable inputs. The final version (scan-v5) took the most time to perfect, ensuring that the tool would remain silent for 8 clean files, acknowledging that sometimes, \"I found nothing\" is the appropriate response.",
  "summary": "The Benchmark Behind Proactive Scan On Tuesday we shipped Proactive Scan: one command that ranks your riskiest files and reads each with cross-file context, before anything crashes. Today we're publishing the benchmark behind it. All of it: the numbers, the methodology, the part where our first version cried wolf, and the kinds of bugs it still misses. The Result 28 test cases, written fresh for…",
  "key_points": [
    "Proactive Scan tool evaluates risky files contextually before crashes",
    "Benchmark includes 28 test cases with 20 files containing intentional defects",
    "Tool achieved 0 false alarms on clean files, scoring 0/8"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}