1Password's AI patching benchmark is misleading

1Password's AI patching benchmark report, released on August 6, 2026, presents a misleading depiction of AI patching capabilities. The headline claims that the models successfully created clean fixes in only 26% of cases. However, this figure is misleading, as it encompasses experiments where agents were explicitly instructed to apply incorrect fixes, as well as instances where agents were unable to compile or test their patches.

Disregarding the headline's implications, the report reveals that when considering only the trials where agents could execute code and were not prompted to produce the incorrect solution, 86% of the patches generated by 1Password's models effectively blocked the supplied exploit. This finding demonstrates that the models possess a degree of patching proficiency under reasonable working conditions, which the headline fails to convey.

The study also uncovers several issues related to the grading process and the sample selection. Grading errors may erroneously penalize valid fixes or permit flawed patches to pass. Furthermore, the report's sample was carefully curated, and the experiments were designed to produce undesirable outcomes. Consequently, the headline lacks a solid foundation for assessing AI patching abilities.

To gain a more accurate understanding of agent failures, it is essential to consider the frequency with which developers submit incomplete fixes. Drawing from the company's security consulting experience, the authors analyzed the first proposed fixes submitted for 2,265 vulnerabilities across 236 Trail of Bits security assessments conducted between 2024 and 2026. In these circumstances, 283 first fixes failed to fully resolve the reported issue, representing a 12.5% failure rate (one in eight).

Additionally, the authors note that developers frequently revise their initial attempts before deeming the issue resolved. This factor further complicates the assessment of AI patching abilities based solely on the headline figure. The authors also exclude cases where the available records do not establish whether the fix was effective, limiting the direct comparison between agents and human patchers.

In contrast, the authors present data from their own Patch the Planet initiative, a collaboration with OpenAI. They examined 186 pull requests merged or closed by September 14, 2026, in Patch the Planet submissions. With an acceptance rate of 67.7%, the authors found that maintainers accepted 126 out of 186 pull requests, with 72.2% of the accepted submissions mirroring the original proposed fix.

However, this does not guarantee the correctness of every patch, as maintainers may close other submissions without merging them, citing various reasons such as policy, process, scope, or maintenance issues.