Vercel AI Gateway confidence fallbacks: measure before rollout

Vercel AI Gateway introduces a confidence-based decision fallback mechanism that allows developers to run a secondary model when a primary decision meets a specific confidence threshold. This feature is currently in beta, and when triggered, it runs a second decision stage, incurring an additional cost and latency. The primary value of this feature is to evaluate whether using a second model improves the accuracy of specific decisions enough to justify the added resources.

The traditional fallback lists address cases where a model fails to deliver a response, whereas the confidence condition targets situations where the primary model's response, even if successful, does not meet the desired confidence level. This new condition can be combined with other criteria, enabling routing decisions based on multiple signals.

For Choice and Score questions, confidenceBelow can be used to trigger the fallback when the confidence level falls below a specified threshold. This threshold represents the concentration of the answer's probability distribution, not the probability of the answer being correct.

If the question field is omitted from the conditional condition, Vercel's gateway applies the condition to all Choice or Score questions of the matching type, triggering the fallback if any of them meet the criteria. This broader scope may be easily overlooked when a single decide call includes multiple questions. The conditional model object must be the first entry in the providerOptions.gateway.models array and can contain only one conditional model. Following the conditional model, plain model-name entries can specify an execution-error fallback.

Requests without a conditional object will continue to function as before. This feature is explicitly marked as beta, so it is crucial to test the request shape and observed behavior in your own environment before relying on it in a production setting. To begin, select a decision with a reviewable answer and monitor its performance.

Avoid enabling the second model for all AI requests solely because a confidence field exists. Instead, identify specific decisions where an incorrect low-confidence answer could have significant consequences. For example, routing a billing issue to the wrong support team could be a suitable candidate. Evaluating free-form assistant responses is more challenging, as it is difficult to determine consistently whether the second stage improved the response.

To implement this feature, create a small, representative evaluation set derived from actual request shapes, ensuring that personal data and secrets are removed before use. For each example, record the input, the expected choice or score, and the potential consequences of an incorrect answer. Include both ordinary cases and difficult boundary cases that your current system may struggle with.

Keep this evaluation set separate from the data used to tune the prompt or threshold, and verify that the tuning did not merely memorize the initial set. Run the primary model alone first, recording its output, confidence, correctness, and request duration to establish a baseline. This baseline will help you understand how often the primary model produces incorrect answers, the distribution of these errors across confidence values, and the categories most affected by the errors.

When setting the confidence threshold, ensure it effectively separates cases where the fallback can improve the decision from those where it cannot. For Boolean decisions, use the probabilityBetween syntax, which specifies an inclusive range for the probability of the answer being true. If the condition matches, Vercel's AI Gateway will rerun the entire decision with the original state and all original questions, returning the fallback result as a complete response.

If the conditional model fails, the request will also fail, and this failure scenario should be considered in the application design, particularly for decisions that affect money, account access, or other consequential actions. Native decision models report Choice and Score confidence, while language-model fallbacks do not return confidence distributions.