An Agent on a Leash, or why my AI agent doesn't make business decisions

This article is part of a series exploring the creation of a trustable LLM-powered support agent that can be relied upon to make decisions. The author explains that an LLM's confidence is not a measure of reliability, but rather a correlation metric. They emphasize that handing an LLM the keys to the kingdom is problematic, hence the need for boundaries and risk tiers.

The article outlines the pipeline of the agent, where user input is first classified to understand intent. Deterministic validation software then assesses whether the action is low, medium or high risk. Low-risk actions allow the AI to assist in executing the decision, medium-risk actions require human confirmation, and high-risk actions are approved with an audit trail before a human executes them. Very high-risk actions are never executed by the system, as the AI can only propose them.

The author introduces the concept of an autonomy budget, where the AI has a limited amount of agency and once it's spent, the human takes over. They provide examples of low, medium, and high-risk actions, emphasizing that money or critical data involvement elevates risks. The author notes that while the higher-risk gates add latency and engineering effort, they are intentional to ensure safety.

The text concludes that the design's trade-offs are honest and real, with potential future evaluation of AI's judgment accuracy. The author uses an example of an insomniac customer waiting for a refund, which is eventually executed by a human after the AI classifies the refund as an eligible action.