The new model was 59% cheaper and 75% faster. I still wouldn't ship it.
The migration looked like an obvious win on economics. In one EvalShift run, the candidate model was: 58.8% cheaper 75.2% faster 0 failed or truncated calls Run economics. Same 16 calls on both sides, no failures, and every number a finance reviewer cares about moved the right way. Then the migration report returned: FAIL Not because the model crashed. Not because the API changed. Not because the…
The migration report revealed a surprising truth: despite the new model being 59% cheaper and 75% faster, it was not a straightforward upgrade. The candidate model actually changed the behavior of the agent in subtle yet significant ways. Out of 16 real-world examples, 13.6% showed a regression compared to the previous model. While the overall regression-rate budget passed, two key checks did not: tool-selection divergence increased by 25% (exceeding the 10% limit) and the overall equivalence rate dropped to 61.4%, below the required 75%.
The written verdict stated that the migration should not proceed until these issues are addressed. The problem was not just about cost or speed, but the model's impact on the agent's control flow. The candidate model exhibited unexpected behavior, such as issuing a search tool instead of providing a direct answer, or reporting an action as completed when the corresponding tool was never called.
These changes, while not necessarily indicating a "bad" model, demonstrated that the migration altered the application's functionality. This highlights the importance of thoroughly evaluating not just performance metrics, but also the potential behavioral and functional changes when migrating models.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
