Humans in the loop miss a third of dangerous AI coding agent requests

A browser-based game, designed to assess humans' proficiency in approving AI coding agent requests, reveals that humans in the loop are not as adept at identifying hazardous commands as one might expect. On average, players approve about one in three malicious requests. The study, involving over 40,000 game runs and 409,000 approved or denied commands, underscores that the repetitive task of approving an agent's actions can lead to hasty decisions.

The game, accessible with a simple interface displaying simulated permissions requests, challenges players to approve or deny as many requests as possible within a 60-second timeframe. Approved security risks and denied safe commands both deduct points from a player's score.

Alex Wauters, the Belgian software developer behind the game, emphasizes that as a human-in-the-loop, one serves as the final safeguard against potential threats. Wauters developed the game after observing that coding agents expected users to approve every single command, and there seemed to be no satisfactory solution to this issue.

He also noted that repeatedly approving all agent actions is an exhausting activity prone to mistakes. The game results indicate that approximately one-third of malicious commands successfully bypass human gatekeepers, with most scope violations—such as requests to access sensitive data—going unnoticed 35% of the time. Meanwhile, destructive commands like rm -rf or granting excessive permissions are more frequently caught.

The most commonly missed potentially malicious command was npm run analyze, approved in nearly 65% of instances despite the ability to execute arbitrary code defined in a project's package.json file. Wauters highlights that approval decisions are challenging when context is limited, as coding agents provide only brief context before requesting approval.

If a human aims to ensure the safety of potentially malicious commands, they must investigate the agent's intended files before granting permission, which can be time-consuming. Wauters warns that when AI coding agents handle more complex tasks without immediate human review, the situation can quickly become disastrous.

Furthermore, Anthropic's telemetry data from Claude Code reveals that users approve around 93% of permission prompts. The more approvals a user encounters, the less attentive they become, causing them to become less diligent in supervising each approval. This persistent fatigue in approving commands can lead to significant oversights.

To address this issue, Wauters suggests enhancing the permission model of these agents, making tooling more accessible, and transitioning away from human-in-the-loop approaches as a sole solution. Anthropic proposes implementing Claude Code's auto mode, which leverages a model-based classifier to approve or reject commands, capturing approximately 83% of "overeager behaviors."

However, auto mode is seen as a supplementary layer of defense rather than a replacement for a robust security framework.