QuoteBench: How Matched Scores Can Hide Command-Path Failures
LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
