Comparing AI review tools: what the numbers actually tell you
Most "best AI code review tool" comparisons start and end with a benchmark score. That score hides almost everything a team needs to decide with. The useful move is to read the method behind the number, because the number only means what the method lets it mean. Entelligence published a comparison last week between two models on the same pull requests: GPT-5.6 Luna at $0.20 per million input…
Comparing AI code review tools often centers on a benchmark score, which doesn't reveal the full picture. Entelligence's recent comparison of GPT-5.6 Luna and GPT-6 Astra on the same pull requests highlighted a 28x price difference. Luna cost $0.0041 per million input tokens, while Astra cost $0.113, revealing the potential cost gap.
However, the benchmark also showed Luna found 69 verified bugs compared to Astra's 92, with precision of 74% and 96%, respectively. These numbers are often highlighted, but the comparison needs more context for usefulness.
Delving deeper, the tool's performance varied by codebase. In Sentry, Discourse, and Grafana, Luna and Astra were within two verified bugs of each other. However, in Cal.com, the gap widened to 21 versus 30, and in Keycloak, an identity and access management server, Luna found only 6 verified bugs to Astra's 14, with only half of Luna's findings validating. Astra's precision was notably higher at 93%.
Specifically for security bugs, Luna caught 9 of 24, while Astra caught 19 of 24. This discrepancy indicates that the cheap model isn't universally sufficient. The bugs Luna missed on Keycloak, such as a permission model oversight and federated recovery codes, highlight the tool's limitations in handling complex authorization logic. This reveals the critical role of reasoning and understanding the underlying code structure, which a model trained on older code might not fully grasp.
The authors are transparent about the benchmark's limitations, such as excluding defect-added-for-benchmark purposes and the models' training cutoff dates, which were before 2013. This omission is common in public code review benchmarks, potentially overstating the models' effectiveness on unseen codebases. Moreover, running the same PRs multiple times reveals varying results, underscoring that a model's performance can fluctuate, and a single run might miss bugs on subsequent runs.
Hence, relying solely on aggregate scores can be misleading, as it doesn't capture the tool's variability and the nuances of different codebases. The practical takeaway is that comparisons should reveal both the tool's strengths and weaknesses, especially when measured against the actual code you write.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
