I benchmarked 8 models on our own code reviews. The first bug was ours.
Claude Opus 5.5 failed the same review three runs out of three. Same pull request, same reviewer role, same result: the stage output "failed to parse", the review was marked errored, and Opus dropped to the bottom of my table. When I finally pulled the raw response, Opus had done exactly what it was asked. It wrote two short paragraphs explaining its findings and then gave a well-formed JSON…
Three models were evaluated on the company's code review process. Claude Opus 5.5 failed the reviews three times out of three. The issue was with how Opus parsed the review response, mistaking a phrase in the explanation for JSON data and throwing an error. Vals AI ranked Kimi K3.1 Pro as the top open-weights model, while Plotly's data-analysis exam scored it 42 out of 43. Despite strong community benchmarks, the company's in-house evaluation found Opus and Sonnet to have issues with parsing and tool usage in code reviews.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
