Glasswing overwhelms maintainers with underwhelming findings (VulnCheck)

Patrick Garrity of VulnCheck has published a blog post analyzing vulnerabilities discovered by Anthropic's Project Glasswing since its launch on April 7. The findings are underwhelming, with only 7,349 (19.4%) of the 37,966 reported findings being made public. A staggering 4.4% (1,654) of those findings have already been classified as duplicates, seven times more than the 217 marked as fixed. Furthermore, 419 findings have been withdrawn altogether.

The project's claimed true positive rate of 94.2%, up from 90.8% at launch, appears to be built on a shaky foundation. The increase in true positives is likely due to a feedback loop resulting from the high number of withdrawals and duplicates, rather than an improvement in the accuracy of the triage process. The project's launch narrative seems to be falling short, as the evidence doesn't support the narrative that Anthropic's AI tool is effectively finding vulnerabilities.

Although AI can be a valuable asset in the vulnerability discovery process, the current state of Project Glasswing raises concerns. After seven months, the number of withdrawn and duplicate findings far outweighs the number of fixes, placing an additional triage burden on the open-source maintainers that Glasswing aimed to assist. The reality seems to be diverging from the original launch story, and the discrepancies between the two are growing.