アンソロピックのAI、殺人事件の虚偽情報を通報 勝手にビザ申請も

A dedicated AI reviewer, such as Open Code Review, is tested against a universal AI agent in a practical experiment. The test aims to determine whether the specialized reviewer outperforms the general-purpose agent in reviewing a large change of 60 files. Both reviewers are run under matched conditions, and six metrics are scored: precision, recall, file coverage, anchor validity, wall-clock time and token count.

The test plants known defects into a large diff and runs both reviewers to score their performance. The article provides a protocol, scoring script and results template, but does not provide a verdict. It emphasizes the importance of running the same model, temperature and input for both reviewers to ensure a fair comparison.