SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation
Coding agents are typically evaluated using one-shot, well-formulated prompts. Benchmarks such as SWE-bench use existing GitHub issues as input for a coding agent and evaluate whether the agent successfully solves the task. However, in real-world use cases, prompts are often not as well-formulated or structured as existing GitHub issues. Additionally, a coding agent session is not limited to a…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
