Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models
Tool-using agents can fail twice: a required tool can fail, and the agent can then report success without the evidence needed to justify it. Existing benchmarks often entangle this reporting failure with tool selection, recovery, and environment dynamics. We introduce Failure-Transparent Agents (FTA), a controlled benchmark that fixes the failed observation and required evidence state before…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
