Day 19/90: When Should an AI Agent Ask for Clarification?
An AI agent can execute a valid tool call and still perform the wrong action. Consider the request: “Cancel my order.” If the customer has two active orders, the intended action is clear, but the target is unresolved . Selecting the most recent order introduces an assumption that the user has not confirmed . The challenge is to distinguish requests that require clarification from those that can…
AI agents can still perform incorrect actions even after executing a valid tool call. In the case of "Cancel my order," the AI must determine which of the two active orders should be canceled. Selecting the most recent order introduces an assumption about the user's intent, which may not be correct. The key is distinguishing requests that need clarification from those that can be handled using the existing context.
Asking too infrequently leads to incorrect actions, while asking too often creates unnecessary friction. Research on ambiguity detection, clarification question generation, and answering after clarification is separate but related. The CLAM framework proposes detecting ambiguous questions, requesting clarification, and answering using the additional information.
CAmbigNQ is a dataset with 5,653 ambiguous questions and relevant passages. CLAMBER evaluates ambiguity handling across different models and taxonomies. These studies suggest evaluating the entire clarification process. While recognizing an ambiguous request is important, it does not guarantee the AI will ask the right question or use the answer correctly.
A practical decision rule is to ask whether unresolved information would change the answer or action significantly. This involves considering the request, conversation history, and application state. Examples such as "Cancel my order," "Track order ORD-1042," "Make it shorter," "Compare these models," and "Explain Python decorators" illustrate how to handle various scenarios.
Identifying the missing preferences does not always stop a useful response. The key is to ask a clear question that identifies the exact information needed, like "Which order should be canceled: ORD-1042 or ORD-1087?" A useful clarification question should specify the necessary information, not leave the user to guess. Before asking, the AI should check if the needed information is already available through conversation context, application state, or lookup.
The goal is to resolve uncertainty with minimal user effort. A clear question can still be unanswerable if the relevant policy is unavailable. In such cases, the AI should retrieve the policy or explain the missing information instead of asking for clarification. For RAG systems, unsuccessful retrieval should be distinguished from uncertainty about user intent.
Evaluation should include requests requiring clarification, answerable requests, and ambiguous requests resolved by context. Measures like clarification recall, unnecessary clarification rate, question usefulness, completion after clarification, and premature action rate help assess AI behavior. When essential information is missing, the AI should preserve the task, request the missing detail, update the context, and verify resolution before proceeding.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
