{
  "id": 7663673,
  "title": "Counting bugs is the hard part of comparing AI review tools",
  "url": "https://urgent.news/2026/09/16/counting-bugs-is-the-hard-part-of-comparing-ai-review-tools",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-16T00:15:10.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/tessainsley/counting-bugs-is-the-hard-part-of-comparing-ai-review-tools-1cm1"
  },
  "original_language": "en",
  "account": "When comparing AI review tools, the most critical aspect that often goes unaddressed is the verification process behind counting bugs. The benchmark published by Entelligence compares GPT-5.6 Luna and GPT-6 Astra on 50 public pull requests, revealing a stark difference in precision despite similar model choices. Luna reported 69 verified bugs with a precision of 74%, while Astra found 92 bugs with a higher precision of 96%. However, the reason behind this discrepancy lies in the verification process.\n\nEntelligence employed a rigorous method, where two judges scored every finding and deemed a bug valid only when both judges agreed. This approach yielded 91% agreement, with a bug only counted when both judges validated it. This method of labeling and verification is rare in the category, making Entelligence's benchmark stand out. Most comparisons merely quote recall figures without disclosing the labeling process, the involvement of the vendor's own staff, or the rate of disagreements between judges. These missing details make the published precision numbers unreliable.\n\nThe practical implication for teams is significant. When reading tool comparisons, it's crucial to seek out three key pieces of information: the label protocol, the involvement of the vendor's own judge, and the rate of disagreements between judges. These factors influence the precision significantly. For instance, Keycloak's auth code showed Luna's precision dropping to 50%, indicating that in complex cases, a review tool's decision might amount to a coin flip. The solution is not to chase better benchmarks but to implement a verification step that involves human judgment.\n\nThus, when confronted with recall figures from vendors, treat them as unverified until you know the labeling process. Vendor benchmarks are scarce, but those that publish their methods are valuable and warrant careful examination. Until vendors start disclosing these crucial details, the reliability of comparison figures remains questionable.",
  "summary": "Model benchmarks for code review usually report the same two numbers: how many bugs the tool found, and how many of its comments were actually right. The part nobody can verify is the counting. Who decided a finding was a real bug? That single choice moves precision by more than the model choice does. The Entelligence benchmark comparing GPT-5.6 Luna and GPT-6 Astra on review is a useful case,…",
  "key_points": [
    "Verification process crucial in comparing AI review tools",
    "Entelligence's benchmark uses double-judge validation method",
    "Lack of disclosed labeling process makes precision numbers unreliable"
  ],
  "editors_take": "The reliability of AI review tool comparisons hinges on disclosure of their verification processes, specifically label protocols, vendor involvement, and judge agreement rates, which significantly impact reported precision numbers.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}