Models retire faster than operating systems

Model providers often retire their models, but the transition is usually smoother than changing an operating system's API. When an API deprecates, developers receive a year's notice, a migration guide, and even a conference talk. However, when a model provider deprecates a model, the process is less formal, typically just a blog post and a date for when the model will no longer be available.

Our AI features are built on models that are constantly being swapped out. While we haven't had to make an emergency switch in production, we do experiment with many models. Each experiment becomes a rehearsal, as the day a retirement notice arrives, the move is already anticipated.

A model is simply a catalog line in our generation pipeline. It's not a choice hardwired into the code, but an entry in a catalog that specifies which provider serves it, what key unlocks it, and what it costs. The catalog includes models from various major providers like OpenAI, Anthropic, and Google. Production runs a trimmed selection of these models.

Each step of the pipeline is mapped to a specific model. The planning step doesn't rely on the model that writes the output, and the step that repairs a single field doesn't need the model that generates a whole component. This means that swapping a model has a limited impact, as only the step using the new model changes. The prompt faces the model, and the evaluations face us.

When trying a new family of models, adapting the prompt is real work, but it's not a major overhaul. Each family has its own dialect in terms of constraints, structure, and repetition. The contract, which outlines what must come out of the model, remains unchanged during the swap.

The evaluation suite is where we encountered a challenge. It tries to anticipate what a given model will answer, effectively doing the prompt's job, which proved not to be effective. Our evaluations validate the output against our standard, which applies to all models and includes checks for parsing, schema compliance, and conversational accuracy.

The test fixtures are derived from real production failures, and the thresholds are clearly defined. Importantly, these thresholds do not mention the model by name, which is the key to making model swaps routine instead of a crisis.

The customer holds the same dial when it comes to model selection. On our Retrieval-Augmented Generation (RAG) chatbot, the customer chooses the tier of model their chatbot runs on, depending on their volume and the need for nuance. This dial is set by the customer and cannot be altered without their consent, as the model is essentially the staffing for the feature. Other aspects like retrieval, access rules, and grounding in their content define the feature. The model is simply the staff that carries out this function.

When a model is retired, the plan is simple: add a catalog line, adapt the dialect, run the harness, and run the evaluation suite. This unassuming approach ensures that the swap is not a crisis but a routine change. The model is treated as a vendor under contract, and we don't upgrade models; we re-certify them.