A code review benchmark that isn't the vendor ranking itself

In the world of AI code review tools, various rankings and benchmarks are published by different sources, each with their own interests and methods. The problem with these rankings is that they often come from the vendors themselves, making the results potentially biased. In response to this issue, a neutral party named Martian has created an open and reproducible AI code review benchmark called Code Review Bench.

The benchmark runs on real open source pull requests on GitHub where AI review bots actively participate. It reconstructs each review timeline, from bot suggestion to developer response to code change. Each tool is scored based on precision, recall, and F1 score, and the whole process can be reproduced since the code is MIT-licensed on GitHub.

When compared to other rankings, the Martian benchmark shows that the top tools have a narrower spread in F1 scores. For example, Cubic Dev AI, GitHub Copilot, and Claude all have F1 scores around 65-64 percent. However, the key factor to consider is the precision and recall values rather than just the F1 score. Greptile has the highest precision, meaning developers act on its suggestions most of the time, but it has the lowest recall, leaving many issues unreported.

On the other hand, CodeAnt AI has a high precision but a recall of only 38%, meaning it misses a significant number of issues that a reviewer would flag.

The benchmark scores 16,017 pull requests, and the top 14 tools show a wide range of precision and recall values. No single tool dominates, and the gap between the top and bottom is around 15 points. This means there is no single "best" AI code review tool, and vendors often hide the other 13 tools in their rankings.

To make a decision based on this benchmark, you should take the tools with precision and recall values that fit your needs and test them on a slice of your real pull requests for a couple of weeks. This will give you a better understanding of how well each tool performs on your specific codebase, compliance rules, and review culture. The benchmark provides a floor for the conversation about AI code review tools, removing the vendor's influence on the evidence and helping you make a more informed decision.