{
  "id": 18502112,
  "title": "I benchmarked 8 models on our own code reviews. The first bug was ours.",
  "url": "https://urgent.news/2026/11/06/i-benchmarked-8-models-on-our-own-code-reviews-the-first-bug-was-ours",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-06T13:05:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/grumbash/i-benchmarked-8-models-on-our-own-code-reviews-the-first-bug-was-ours-2c5p"
  },
  "original_language": "en",
  "account": "Three models were evaluated on the company's code review process. Claude Opus 5.5 failed the reviews three times out of three. The issue was with how Opus parsed the review response, mistaking a phrase in the explanation for JSON data and throwing an error. Vals AI ranked Kimi K3.1 Pro as the top open-weights model, while Plotly's data-analysis exam scored it 42 out of 43. Despite strong community benchmarks, the company's in-house evaluation found Opus and Sonnet to have issues with parsing and tool usage in code reviews.",
  "summary": "Claude Opus 5.5 failed the same review three runs out of three. Same pull request, same reviewer role, same result: the stage output \"failed to parse\", the review was marked errored, and Opus dropped to the bottom of my table. When I finally pulled the raw response, Opus had done exactly what it was asked. It wrote two short paragraphs explaining its findings and then gave a well-formed JSON…",
  "key_points": [
    "Claude Opus 5.5 failed three code reviews",
    "Kimi K3.1 Pro ranked top open-weights model",
    "Opus and Sonnet had parsing and tool issues"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}