Same 8 drafts, one reviewer said revise 2, the other revise 7: calibrating rubrics for AI-on-AI review
We handed the same eight reply drafts, the same scoring rubric, and the same instructions to two independent AI reviewers. One returned revise 2 of 8 . The other returned revise 7 of 8 . If your first instinct is "one of them is broken," it was ours too. It's also wrong, and the actual explanation reshaped how we write rubrics for any AI-on-AI review — code review, tone review, product QA, all of…
Two AI reviewers were given the same eight reply drafts, rubric, and instructions. One reviewer found two drafts needing revision, while the other found seven. Initially, it was unclear whether one reviewer was flawed. However, this discrepancy led to a deeper understanding of how to calibrate rubrics for AI-on-AI review. The key causes of disagreement were unstated tolerance, batch-level rules being sensitive to batch size, and a poisoned premise.
To address these issues, rubric items should now explicitly state tolerance, counting rules should be defined at the batch level, and premises should be labeled by verification recency. By treating disagreement as a signal, the focus should be on calibrating the rubric rather than the reviewers.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
