AI struggles to patch vulns without adult supervision

A recent study conducted by researchers at 1Password's Off-by-1 Labs has revealed concerning findings about the effectiveness of AI-powered security patches. When tested, two frontier models - ChatGPT 5.5 and Claude Opus 4.8 - produced autonomous patches that only correctly fixed vulnerabilities approximately 26% of the time. The remaining 74% of patches either failed to fully resolve the vulnerability or introduced new issues.

Keith Hoodlet, the director of security research at 1Password, emphasized the need for human review in the process, stating that across six recently disclosed CVEs, the average success rate for generating a fully resolved patch was a mere 26.0%. Of the AI-generated patches, 20.1% altered application behavior, while 2.3% introduced new security issues.

In total, 49.3% of patches failed to fix at least one existing exploit path, and 2.2% both failed to fix the vulnerability and introduced a new exploit path. The researchers coined the acronym FLAWED to describe these automated LLM patches, which stand for Fix-Like Artifacts With Embedded Defects. The study suggests that the cost-effectiveness of AI-generated patches may be misleading, and that human supervision is crucial for ensuring the safety and effectiveness of the patches.