Most coding agent benchmarks skip large-scale refactoring. Not this one.
AI coding agents still struggle with large-scale refactoring, with the best model achieving only a 41.2% resolve rate on a The post Most coding agent benchmarks skip large-scale refactoring. Not this one. appeared first on The New Stack .
An AI coding benchmark developed by researchers at Shanghai Jiao Tong University, Peking University, Douyin Group and others, named SWE-Bench ProMax, has highlighted the struggles of AI coding agents with large-scale refactoring. The benchmark features 170 instances from real code commits across seven programming languages, all going through multi-stage curation to improve quality.
The creators aimed to address issues found in other benchmarks, such as imprecise issue descriptions, overly narrow or broad tests, and tasks lacking sufficient complexity or cross-file scope.
Refactoring poses significant challenges to AI coding agents as it requires zero errors, no behavior changes, and complete reversibility. Shane Warden, principal architect at ActiveState, argues that token proximity does not guarantee structural understanding, and engineers who treat LLMs as text-processing engines may not grasp the complexities of large systems.
Another challenge is attention, where highly optimized algorithms are necessary for large codebases but can lead to missing important observations when dealing with complex code. Time is also a factor, as race condition errors, loss of idempotency, loss of atomicity, and incorrect retry handling can occur when parallel tasks execute in different orders.
These complexities explain why many benchmarks overlook large-scale refactoring; however, SWE-Bench ProMax is designed to test these areas where current AI coding agents fall short. The benchmark aims to identify gaps in agent performance and drive improvements in LLMs' ability to handle complex, cross-file refactoring tasks.
Written by urgent.news from The New Stack's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
