We ran the control a reader asked for. It also turned out we had benchmarked the wrong model.
Two people left long, technical comments on the posts I published this week. One of them asked for an experiment I should have run before publishing anything at all. The original number was 10/10: take ten real CVE fix commits, revert each one, and ask our security agent to find what was reintroduced. It found all ten. I wrote a whole post about why that number is weak, and listed four reasons. A…
A reader requested a specific control experiment for a post published earlier this week. The original plan was to take ten real CVE fix commits, revert each one, and ask the security agent to find what was reintroduced. However, it turned out that the reviewer wasn't flagging every reverted commit. Seven out of ten ordinary reverts went unnoticed, while all ten CVE reverts were flagged.
The experiment used a control design where a reverted fix was compared to a commit with no security relevance, but the reviewer detected deletions rather than vulnerabilities. The controls showed that a reverted fix was flagged as a vulnerability, but the security framing was invented. The post discusses the three comments produced by the reviewer, which accurately described the reverted changes but promoted ordinary robustness bugs as security vulnerabilities.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.