ORCA-bench: How Ready Are Language Model Agents for Oncall?
Large language models can write, patch, and search code, but oncall root cause analysis (RCA) demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began. We introduce ORCA-bench, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting. ORCA-bench…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
