Urgent.News

What's breaking now, across thousands of outlets.

AI

Your AI agent needs a way to say "I don't know"

TL;DR: Agents built with only two exits, answer or error, decide anyway on the cases they should not. The third exit, abstention, has to be designed in code, tested in the eval set and recorded in the decision log. The model's confidence is not the exit The tempting shortcut is to ask the model how sure it is and set a threshold. What comes back is a well-formed sentence, or a number the model…

When incorporating AI agents into systems, it is essential to account for scenarios where the model cannot provide a definitive answer. Traditionally, agents are designed with only two possible outcomes: answering the query or returning an error. However, this binary approach fails to handle situations where the model's confidence is insufficient to provide a reliable response.

To address this limitation, a third exit, known as "abstention," must be explicitly implemented in the code. This exit point should be designed, tested, and recorded in the decision log, just like any other decision made by the agent. The key is to determine when the agent should abstain and route those cases to a queue with an assigned owner for further evaluation.

Relying solely on the model's confidence level as the trigger for abstention is not sufficient. The temptation to ask the model how certain it is and set a threshold can lead to inaccurate results, as the model may generate well-formed sentences or numbers rather than calibrated probabilities. The decision to abstain must be based on predefined conditions that are checked both before and after the model's execution.

These conditions can include missing required fields, validation failures, tool results outside the expected range, or values that contradict master data. By incorporating explicit stop conditions, the abstention exit becomes a deterministic and explainable part of the system. The output schema should include a "cannot decide" value alongside the normal outcomes, accompanied by a reason code from a predefined list. This allows for clear routing of abstention cases to the designated queue.

Recording abstentions in the decision log is crucial for auditing and risk assessment purposes. The log should capture input references, versions, reason codes, the person responsible for resolving the case, and how it was resolved. This ensures that the abstention process is transparent and traceable, enabling auditors to evaluate the system's behavior under uncertainty.

To measure the effectiveness of the abstention mechanism, an evaluation set should be expanded to include cases where the correct output is "I don't know." These cases can represent incomplete inputs, conflicting rules, or documents outside the scope of the system. By tracking the frequency of abstentions over time and comparing it to the cases that should have been abstained from but were not, valuable insights can be gained regarding the workload sent to human operators and the potential risk associated with silent errors.

The balance between tightening the abstention conditions to increase the queue size and loosening them to allow silent errors is a business decision that should be documented and communicated to the process owner. By incorporating the third exit into the design document, output contract, and acceptance tests, the system can be validated before deployment, ensuring that the agent is tested on hard cases and not simply relying on its tendency to abstain.

Ultimately, a system that never abstains is not accurate and has not been thoroughly tested on challenging scenarios. It is essential to incorporate the abstention exit in the agent's design, output contract, and acceptance tests from the outset. By doing so, the system can provide more reliable and trustworthy responses, even when faced with uncertain or ambiguous inputs.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in AI

More from Tuesday 17 November →