{
  "id": 2376312,
  "title": "Most coding agent benchmarks skip large-scale refactoring. Not this one.",
  "url": "https://urgent.news/2026/08/21/most-coding-agent-benchmarks-skip-large-scale-refactoring-not-this-one",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-21T11:54:00.000Z",
  "source": {
    "name": "The New Stack",
    "slug": "the-new-stack",
    "url": "https://thenewstack.io/ai-agents-refactoring-benchmarks/"
  },
  "original_language": "en",
  "account": "An AI coding benchmark developed by researchers at Shanghai Jiao Tong University, Peking University, Douyin Group and others, named SWE-Bench ProMax, has highlighted the struggles of AI coding agents with large-scale refactoring. The benchmark features 170 instances from real code commits across seven programming languages, all going through multi-stage curation to improve quality. The creators aimed to address issues found in other benchmarks, such as imprecise issue descriptions, overly narrow or broad tests, and tasks lacking sufficient complexity or cross-file scope.\n\nRefactoring poses significant challenges to AI coding agents as it requires zero errors, no behavior changes, and complete reversibility. Shane Warden, principal architect at ActiveState, argues that token proximity does not guarantee structural understanding, and engineers who treat LLMs as text-processing engines may not grasp the complexities of large systems. Another challenge is attention, where highly optimized algorithms are necessary for large codebases but can lead to missing important observations when dealing with complex code. Time is also a factor, as race condition errors, loss of idempotency, loss of atomicity, and incorrect retry handling can occur when parallel tasks execute in different orders.\n\nThese complexities explain why many benchmarks overlook large-scale refactoring; however, SWE-Bench ProMax is designed to test these areas where current AI coding agents fall short. The benchmark aims to identify gaps in agent performance and drive improvements in LLMs' ability to handle complex, cross-file refactoring tasks.",
  "summary": "AI coding agents still struggle with large-scale refactoring, with the best model achieving only a 41.2% resolve rate on a The post Most coding agent benchmarks skip large-scale refactoring. Not this one. appeared first on The New Stack .",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}