UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training
Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user. This simulated user controls what information the agent receives and when, yet current benchmarks score only the agent and do not directly measure whether the user correctly executed its assigned role. We introduce UserProxyBench, an evaluation layer over the…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
