Urgent.News

What's breaking now, across thousands of outlets.

AI

Your LLM app has an expiry date: pin the model and keep a migration eval ready

Wednesday AI Opinion. Here's my hot take: "we use the latest model" is not a strategy. It's a dependency you haven't pinned. Every LLM feature you ship is built on a model that will either change under you or get switched off. Treat the model like any other production dependency: pin it, watch it, and keep a migration eval ready before the retirement email shows up. The "same" model doesn't stay…

Calling a specific model version isn't a good strategy; it's a dependency you haven't pinned. Every LLM feature you deploy will rely on a model that might change or be switched off. Tackle the model like any other production dependency: pin it, monitor it, and prepare for migration before it's retired. The same model doesn't stay the same.

A study comparing ChatGPT's behavior showed significant changes in accuracy and code generation capabilities between March and June 2023 versions of GPT-3.5 and GPT-4. Both models made more formatting mistakes in code generation in June than in March. This highlights the need for continuous monitoring of LLM behavior. Pinning is easy now.

Use it. Providers give you stable identifiers for their models. Anthropic's models overview lists every Claude model ID as a pinned snapshot, including dateless IDs used since the 4.6 generation. OpenAI also provides pinning options, but with different deprecation policies. In production, always send an ID that names exactly one model.

Aliases and -latest names are fine for testing but not for production use. However, pinning alone isn't enough. A pinned model doesn't live forever. Anthropic issues retirement dates, and requests to models past the retirement date will fail, giving customers at least 60 days notice. OpenAI provides at least 6 months notice for generally available models and 3 months for specialized variants.

Real-life examples show that model deprecations can happen with short notice, turning a planned migration into a weekend incident. The key is your evaluation set. Both providers stress testing your applications with newer models well before the retirement date. Create a file of real inputs from production, redacted, with expected properties or rubrics.

Run representative requests against both models, comparing outputs, latency, and cost before switching production traffic to the replacement. If you don't have an eval set, a forced model swap means manually checking a handful of prompts and hoping. The process should be like upgrading any other dependency: pin the model ID in config, not scattered across code, log the model ID with every request, keep a migration eval set, watch for shutdowns programmatically, and roll out like a dependency upgrade with canary testing. This approach ensures model upgrades are manageable and doesn't catch you off guard.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in AI

AI Is Starting to Help Prove Code Correct, Not Just Find Bugs.

Anthropic recently worked with Theorem to use Claude to formally verify specific security properties across 123 open-source libraries, covering 239 test entry points.

  • Anthropic and Theorem use Claude to verify security properties in 123 open-source libraries.
  • Claude generates proofs showing programs won't crash with inputs under 64 MB.
  • Formal verification historically required 20 person-years, Claude completed in weeks.

More from Wednesday 18 November →