Overwhelmed by Choice: Studying LLM Decision Making at Scale
Multiple-choice and candidate-selection evaluations are widely used to assess LLM reasoning and decision-making, yet most benchmarks contain relatively small candidate sets. It remains unclear whether conclusions drawn from these settings remain valid as the candidate space scales. We systematically evaluate LLMs as the number of competing candidates increases and find substantial accuracy…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
