Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark
Large language models (LLMs) are increasingly used in coding tasks, but their ability to reason about code execution remains unclear. Existing repository-level QA benchmarks mainly evaluate static code understanding and often rely on LLM-based evaluation, while execution-reasoning benchmarks are mostly limited to snippets or functions. We introduce SWE-Flux, a repository-level benchmark for…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
