The Hardest Bug to Find Was a False Positive.

On Tuesday, the company released Proactive Scan, a tool that evaluates risky files and analyses them contextually before any crashes occur. To evaluate its effectiveness, the company published a detailed benchmark. The benchmark encompassed all the data, methodology, and the instances where the initial version of the scan incorrectly flagged code (false positives).

The benchmark comprised 28 test cases, including 20 files with intentionally embedded defects and 8 files serving as controls. The categories measured were single-file defects, cross-file defects, and false alarms on clean files. For single-file defects, the tool scored perfectly, correctly identifying 8 out of 8 planted defects.

Cross-file defects were also accurately detected by the tool, with all 12 expected defects flagged correctly. Most impressively, the tool did not produce any false alarms on the clean files, achieving a score of 0/8.

The benchmark was run on July 2, 2026, against the company's staging API, which operates on the same pipeline as the production environment. The API utilizes Voyage embeddings for indexing the codebase and Claude Haiku 4.5 for analysis.

In response to concerns about the presence of competitors in the benchmark, the company explained that they deliberately wrote all the cases themselves to ensure that the models hadn't been trained on any public sources containing similar bugs. This decision aimed to assess the tool's ability to analyze code independently without prior exposure to known issues.

The 28 cases were meticulously crafted in late June, each representing a unique failure class but not copied from any existing public repositories. The goal was to create original and realistic scenarios that would challenge the scan's capabilities.

One illustrative example is XF-05, which involves two files. In the first file, the value of `WINDOW_SLOTS` is set to 0, while in the second file, `rate.py`, the code attempts to divide by `WINDOW_SLOTS`. When the scan is run without the context of `limits.py`, it does not flag this as an issue. However, with the cross-file context, it correctly identifies the `ZeroDivisionError`.

Scoring is based on predefined signals for each test case, which dictate the expected keywords and issue classes that must be mentioned in the findings. For instance, XF-05 requires the scan to report either "division by zero" or the name of the offending constant. The grading is lenient, focusing on whether the scan detects the problem, irrespective of the phrasing used. The company published the benchmark, expected signals, and runner for transparency, allowing others to verify their scoring.

The benchmark revealed a significant challenge: false positives. Despite the tool's success in detecting bugs, it also flagged clean code, leading to numerous instances of false alarms. The company iterated quickly to address this issue through several versions of the scan. Initially, the first version (scan-v3) flagged not only actual bugs but also style preferences in clean files, resulting in a significant number of false positives.

The second version (scan-v4) addressed this by refining the prompts to ensure that findings were tied to concrete failure scenarios with identifiable inputs. The final version (scan-v5) took the most time to perfect, ensuring that the tool would remain silent for 8 clean files, acknowledging that sometimes, "I found nothing" is the appropriate response.