LM Studio built a judge for AI commands. Then the judge started agreeing with the defendant.

LM Studio developed an AI coding agent called Bionic, equipped with a command judge to prevent dangerous actions. This judge, called the Shell Judge, evaluates commands differently based on their structure rather than just searching for dangerous strings. It uses language models like mvdan/sh for Bash, Zsh and SH, and PowerShell's own AST support.

The judge keeps track of values passed from one command to another to assess the potential impact. Despite having 11,651 test cases to cover various command quirks and tool-specific behaviors, the judge can sometimes approve risky actions, leading to problems. When the judge fails to identify a threat, the command is sent to a separate reviewer, the Shell Reviewer.

The reviewer assesses the command based on risk, authorization, and correctness without knowing the passing criteria. However, the review process has its own limitations as it relies on the conversation history, making it vulnerable to prompt injection attacks. Additionally, the judge assumes that executables like git are not compromised, which could be a blind spot if the system has been compromised. As coding agents gain more autonomy, these vulnerabilities become more significant.