Counting bugs is the hard part of comparing AI review tools
Model benchmarks for code review usually report the same two numbers: how many bugs the tool found, and how many of its comments were actually right. The part nobody can verify is the counting. Who decided a finding was a real bug? That single choice moves precision by more than the model choice does. The Entelligence benchmark comparing GPT-5.6 Luna and GPT-6 Astra on review is a useful case,…
When comparing AI review tools, the most critical aspect that often goes unaddressed is the verification process behind counting bugs. The benchmark published by Entelligence compares GPT-5.6 Luna and GPT-6 Astra on 50 public pull requests, revealing a stark difference in precision despite similar model choices. Luna reported 69 verified bugs with a precision of 74%, while Astra found 92 bugs with a higher precision of 96%. However, the reason behind this discrepancy lies in the verification process.
Entelligence employed a rigorous method, where two judges scored every finding and deemed a bug valid only when both judges agreed. This approach yielded 91% agreement, with a bug only counted when both judges validated it. This method of labeling and verification is rare in the category, making Entelligence's benchmark stand out.
Most comparisons merely quote recall figures without disclosing the labeling process, the involvement of the vendor's own staff, or the rate of disagreements between judges. These missing details make the published precision numbers unreliable.
The practical implication for teams is significant. When reading tool comparisons, it's crucial to seek out three key pieces of information: the label protocol, the involvement of the vendor's own judge, and the rate of disagreements between judges. These factors influence the precision significantly. For instance, Keycloak's auth code showed Luna's precision dropping to 50%, indicating that in complex cases, a review tool's decision might amount to a coin flip.
The solution is not to chase better benchmarks but to implement a verification step that involves human judgment.
Thus, when confronted with recall figures from vendors, treat them as unverified until you know the labeling process. Vendor benchmarks are scarce, but those that publish their methods are valuable and warrant careful examination. Until vendors start disclosing these crucial details, the reliability of comparison figures remains questionable.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
