What Aggregate Scores Miss: Measuring Item-Level Regressions in Commercial LLM API Migrations
Context: Software systems that depend on commercial large language model APIs must migrate to successor versions when vendors deprecate older models. Migration decisions typically rely on aggregate benchmark scores, which compress heterogeneous item-level behaviour into a single net figure. Objective: We measure what that compression conceals. Method: On three pairwise upgrades in the GPT-5.4 to…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
