{
  "id": 20984896,
  "title": "We ran the control a reader asked for. It also turned out we had benchmarked the wrong model.",
  "url": "https://urgent.news/2026/11/21/we-ran-the-control-a-reader-asked-for-it-also-turned-out-we-had",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-21T02:43:28.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/revhive/we-ran-the-control-a-reader-asked-for-it-also-turned-out-we-had-benchmarked-the-wrong-model-13bb"
  },
  "original_language": "en",
  "account": "A reader requested a specific control experiment for a post published earlier this week. The original plan was to take ten real CVE fix commits, revert each one, and ask the security agent to find what was reintroduced. However, it turned out that the reviewer wasn't flagging every reverted commit. Seven out of ten ordinary reverts went unnoticed, while all ten CVE reverts were flagged. The experiment used a control design where a reverted fix was compared to a commit with no security relevance, but the reviewer detected deletions rather than vulnerabilities. The controls showed that a reverted fix was flagged as a vulnerability, but the security framing was invented. The post discusses the three comments produced by the reviewer, which accurately described the reverted changes but promoted ordinary robustness bugs as security vulnerabilities.",
  "summary": "Two people left long, technical comments on the posts I published this week. One of them asked for an experiment I should have run before publishing anything at all. The original number was 10/10: take ten real CVE fix commits, revert each one, and ask our security agent to find what was reintroduced. It found all ten. I wrote a whole post about why that number is weak, and listed four reasons. A…",
  "key_points": [
    "Reader requested control experiment for post",
    "Seven out of ten reverts went unnoticed",
    "Reviewer promoted ordinary bugs as security vulnerabilities"
  ],
  "editors_take": "The failure to distinguish between routine code reverts and genuine security vulnerabilities reveals that the model relies on superficial pattern recognition rather than a true understanding of security risks.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}