Your Team Added an AI Reviewer and Now Nobody Reads the Human Comments

Three months ago, your team introduced an AI reviewer for every pull request. Initially, it proved to be impressive, detecting missing null checks and unclosed file handles in the first week. However, the AI now leaves fifteen or twenty comments on each pull request, most of which are related to naming, small refactors, or issues already addressed elsewhere.

Concerns arose when team members began resolving comments in bulk without thoroughly reading them. Last week, a colleague provided a careful comment about a race condition, which was marked as resolved alongside twelve other comments within a minute. This scenario highlights the real cost of noisy feedback – each comment, regardless of its importance, trains people to disengage and apply the same treatment to valuable comments.

Address the issue with the team as a matter of attention rather than a criticism of the tool. Present real examples and roughly calculate the proportion of the AI reviewer's comments that actually resulted in changes over the past month. This should settle the discussion on its own. Subsequently, agree on a set of rules. Most AI tools can be customized, so turn off the categories that never lead to changes, such as style points that your formatter already handles.

Determine what constitutes a necessary response from the author to an AI comment. It could be a reply or a reaction, but a fix should only be required if the author agrees with the comment. Define what an AI comment requires from the author, perhaps a reply or a reaction, but a fix is only necessary if the author endorses it. Determine what a human comment requires, which should be more substantial.

Make human feedback easily identifiable, such as by asking reviewers to start blocking comments with a clear label or by providing a brief summary at the beginning of the review, outlining what they checked and their concerns. This distinction helps separate human judgment from the general noise. Finally, maintain human review as it remains essential.

While the AI can serve as a useful initial pass, it lacks understanding of the change's context, dependencies, and past team experiences. Approving a change solely because the AI found no issues is not a genuine review; it's a guess with a green check mark.