Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

The wire material reports on a browser game where players act as human-in-the-loop for an AI coding agent, approving or denying its commands under time pressure. The data from over 40,000 runs and 409,000 individual approve/deny decisions reveals that humans missed one-third of potential threats. While destructive commands were caught more reliably, those that exfiltrate credentials were missed three times more often.

The most commonly missed command, npm run analyze, was approved 64.7% of the time despite typically being innocent. The game shows that players often don't closely examine the history log above the permission prompt. Three commands were missed 52.5% of the time, doubling their success rate when hidden behind familiar script names.

The model of approving specific commands is criticized as ineffective, as actions perceived as innocent can conceal dangerous intent. The game also highlights the issue of permission fatigue, where users become less diligent in their supervision as the number of approvals increases. This is especially problematic when commands appear benign but have been modified with malicious intent.

While some commands that are harmless in reality were routinely blocked, this added noise could lead users to become more lax in their security practices. The article discusses the move by Anthropic to implement Claude's Auto mode as the default, citing internal studies showing the 'ask the user for everything' approach causes fatigue and is not effective at catching threats.