Routing by task difficulty: the numbers that changed how our AI company spends on models

Until recently, companies spent on language models like most teams do - pick the strongest model, make it the default, and move on to the next task. However, this approach changed when the company started instrumenting production traffic and analyzing where the money went. They discovered that a single frontier model, gpt-4o, was handling 77% of production calls and 97% of model spend. This prompted a reevaluation of their spending strategy.

The team applied a routing heuristic to the AI agents that ran their company, finding that 77% of production traffic went to gpt-4o, while only 3% of the spend occurred on cheaper models. This cost gap revealed that moving the mispriced majority of frontier calls down a tier would significantly reduce the bill. They defined a routing policy based on task difficulty: use cheaper or mid-tier models for tasks with clear targets, specs, or written descriptions; use the frontier model for file edits, debugging, refactoring, architecture decisions, research, security-sensitive changes, content drafting, and ambiguous requirements.

If a cheap model session struggles, it should be escalated mid-task to avoid losing money on incorrect responses.

The team also applied this same rule to their internal AI agents, which consume model capacity throughout the day for writing code, drafting documents, doing research, and handling routine operations. By measuring the cost-per-task rather than cost-per-call, they could better assess the impact of their routing decisions. The company built a small free tool, an LLM cost calculator, to estimate token counts and compare costs across models, helping them weigh routing options before committing.