Treat a Model Upgrade Like a Schema Migration
The most common LLMOps incident I've seen is not a prompt bug or a jailbreak. It's a one-line diff. Someone bumps gpt-4o to gpt-4o-2024-08-06 (or claude-3-5-sonnet-latest to a newer snapshot), CI passes because nothing asserts on model output shape, and three days later a downstream consumer crashes on priority: null where it used to get "normal" . Nobody changed the prompt. Nobody changed the…
We haven't written up this one. Dev.to has the full story — the link below goes straight to it.