Model Routing Is Simple. Until It Isn’t.

At first glance, creating a routing system for your agent might seem straightforward. You send simple requests to cheaper models, reserve more powerful ones for complex tasks, and use a classifier or heuristic to make the decision. This leads to cost savings and maintains performance. However, this is not the full story. Most routing systems assume that model selection is a simple classification problem.

But in practice, it quickly becomes a much more complex systems optimization problem. Three key factors contribute to this complexity. First, they expected GPT-4.1 to be cheaper than Claude Sonnet 4.6. However, in their experience, Sonnet actually cost less - $79 total ($0.19/task) compared to $155 for GPT-4.1 ($0.37/task) across 417 tasks on the AppWorld Test Challenge.

This seems counterintuitive at first, but can be explained by cache behavior. Agent workloads often reuse large chunks of context across steps, and when the cache is hit frequently, the effective input costs drop significantly. Sonnet's lower cache-read pricing benefits disproportionately from this pattern, making it more cost-effective despite having a higher base price and longer processing time.

The second factor is that difficulty is often difficult to predict at the moment of routing. A request like "summarize this contract" might seem simple, but could involve retrieval, compliance checks, tool use, and multiple rounds of refinement. Meanwhile, a highly technical prompt might be efficiently handled by a smaller specialized model.

Even if you could perfectly estimate difficulty, it's only one piece of the puzzle. In production routing, you need to balance cost, latency, model specialization, and reliability simultaneously. Enterprise deployments add more considerations like compliance requirements, data residency rules, privacy constraints, and approved model lists.

A task that would ideally be handled by one model might need to be routed elsewhere due to governance policies. The third factor is routing latency. While bigger models are generally slower, the user experience also depends on other factors such as routing overhead, hardware serving the model, cache warmness, and endpoint load. A theoretically faster model can still result in a slower experience if the serving conditions aren't ideal.

The routing system itself, along with infrastructure factors and the number of decision points, can also introduce latency. The takeaway is that routers should not be viewed as classification problems but as optimization problems. Instead of asking "which model is best for this task?", the algorithm should optimize across cost, quality, and latency simultaneously.

This approach allows for more flexibility and can lead to significant savings while maintaining good performance. By treating routing as an optimization problem, you can explore a range of operating points to choose from depending on whether you prioritize cost, latency, or accuracy.