Your LLM app has an expiry date: pin the model and keep a migration eval ready
Wednesday AI Opinion. Here's my hot take: "we use the latest model" is not a strategy. It's a dependency you haven't pinned. Every LLM feature you ship is built on a model that will either change under you or get switched off. Treat the model like any other production dependency: pin it, watch it, and keep a migration eval ready before the retirement email shows up. The "same" model doesn't stay…
Calling a specific model version isn't a good strategy; it's a dependency you haven't pinned. Every LLM feature you deploy will rely on a model that might change or be switched off. Tackle the model like any other production dependency: pin it, monitor it, and prepare for migration before it's retired. The same model doesn't stay the same.
A study comparing ChatGPT's behavior showed significant changes in accuracy and code generation capabilities between March and June 2023 versions of GPT-3.5 and GPT-4. Both models made more formatting mistakes in code generation in June than in March. This highlights the need for continuous monitoring of LLM behavior. Pinning is easy now.
Use it. Providers give you stable identifiers for their models. Anthropic's models overview lists every Claude model ID as a pinned snapshot, including dateless IDs used since the 4.6 generation. OpenAI also provides pinning options, but with different deprecation policies. In production, always send an ID that names exactly one model.
Aliases and -latest names are fine for testing but not for production use. However, pinning alone isn't enough. A pinned model doesn't live forever. Anthropic issues retirement dates, and requests to models past the retirement date will fail, giving customers at least 60 days notice. OpenAI provides at least 6 months notice for generally available models and 3 months for specialized variants.
Real-life examples show that model deprecations can happen with short notice, turning a planned migration into a weekend incident. The key is your evaluation set. Both providers stress testing your applications with newer models well before the retirement date. Create a file of real inputs from production, redacted, with expected properties or rubrics.
Run representative requests against both models, comparing outputs, latency, and cost before switching production traffic to the replacement. If you don't have an eval set, a forced model swap means manually checking a handful of prompts and hoping. The process should be like upgrading any other dependency: pin the model ID in config, not scattered across code, log the model ID with every request, keep a migration eval set, watch for shutdowns programmatically, and roll out like a dependency upgrade with canary testing. This approach ensures model upgrades are manageable and doesn't catch you off guard.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.