Our AI code reviewer problem wasn't that it was wrong. It was that it wouldn't shut up
I've been thinking about a problem with AI code review that I don't see discussed enough: a reviewer can be technically correct and still make the whole review experience worse. Here's the PR that made this click for me. Small change, maybe 200 lines, nothing exotic. New validation path, a couple helper functions. The kind of thing that used to get a comment or two and an approve. The AI reviewer…
AI code review systems can sometimes provide too much information, making the reviewer experience worse despite their technical correctness. Consider a recent code change involving only around 200 lines of code with a new validation path and a few helper functions. The AI reviewer generated dozens of comments, some suggesting naming improvements, redundant null checks, potential race conditions, and extracting functions from lengthy ones. Most of these suggestions were accurate, but the majority were not particularly important.
The reviewer found themselves dismissing many comments without reading them, leading to a lack of focus on the genuinely significant issues. This highlights a fundamental distinction between technical correctness and the practicality of addressing each comment. Review tools often only answer the first question, failing to address the crucial second question: is this worth interrupting someone for?
This issue-centric approach, driven by the assumption that more found issues are better, leads to an overwhelming number of comments that diminish the overall value of the review process.
The frustration triggered the development of Codzee, aiming to address not just issue detection but also signal-to-noise and trust problems. The goal is to determine what genuinely deserves a developer's attention rather than treating a PR as a checklist to maximize. While the project is still in its early stages, the team is curious about how others perceive the balance between helpfulness and annoyance in AI-generated review comments.
The question arises: how many AI review comments are acceptable before the experience becomes annoying? At what point does the reviewer tune out AI-generated comments due to previous negative experiences? Developers must decide between tolerating potential low-confidence warnings or missing them entirely. Establishing trust in automated reviewers involves discerning useful warnings from mere noise.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
