Stop Asking One Model to Do Everything: A Four-Layer Pattern for AI Workflows

OpenAI released its Decisions API this week, marking a shift in how AI tasks can be structured. The key takeaway is that not all AI work requires the same general-purpose model. Instead, an AI workflow can be divided into four distinct layers, each optimized for specific types of inference.

The first layer focuses on deterministic rules for hard constraints. Decisions that must always produce the same result, such as permissions, spending limits, and security policies, should be handled by conventional application logic rather than probabilistic inference. While a model can help identify relevant context, it should not override these non-negotiable rules. If a condition must consistently yield the same outcome, it's best to implement it using standard code.

The second layer deals with bounded models for repetitive judgments. This includes tasks like classification, prioritization, routing, and scoring, which differ from open-ended generation. The output from these bounded decision layers—such as probabilities, selections from a predefined set, or scores against a rubric—is easier to validate and consume by other parts of the application.

However, validation is still necessary. Probabilities aren't inherently well-calibrated, so confidence thresholds should be derived from representative application data rather than set arbitrarily.

The third layer is dedicated to generative models for open-ended work. This is where generative models are most effective, such as drafting explanations, synthesizing documents, writing code, or planning multi-step tasks. The aim isn't to avoid larger models altogether, but rather to reserve them for tasks where their broader capabilities add value.

A streamlined workflow might involve checking if a request violates a hard policy (in which case it's blocked or escalated), classifying and routing the decision, or using a selected model for the task when the decision is low-confidence or high-impact.

Lastly, there's a human-review path built into the architecture. This is crucial for decisions with significant consequences, such as irreversible actions, those affecting others, accessing sensitive data, or those outside the scope of test evaluations. Human review ensures accountability and accuracy in such critical scenarios.

When evaluating this architectural approach, it's important to consider more than just the raw model accuracy. Key metrics include decision calibration, end-to-end latency, cost per successfully completed workflow, escalation and fallback rates, performance drift over time, and the ability to audit each decision. The overarching lesson is that the choice of model should be guided by the specific type of inference required at each step, rather than by a one-size-fits-all approach.