We measured a week of inference. Routing by task difficulty cuts our cost per call roughly 48x — and flips which users are profitable.
We did the thing everyone building on LLMs does. We defaulted to a strong frontier model, because the demo has to be good and nobody gets fired for picking the strongest model. Then we measured a week of production traffic, and the numbers were embarrassing enough to write down. One frontier model was carrying 77 percent of our calls and 97 percent of our inference spend. Not because 77 percent…
A week-long measurement of inference revealed a stark disparity in cost efficiency when routing calls based on task difficulty. Initially, a strong frontier model dominated 77% of calls and consumed 97% of inference spend, despite being unsuitable for the actual workload. The default choice was driven by convenience, leading to a 48x difference in cost per call between the frontier and a capable open-weight model, which was only $0.00011 per call.
The margin between a profitable and a loss-making user is determined by the engagement level. At $20/month, a plan with 5,000 messages incurs a loss of $0.55, while the same cost at 500 messages yields a 99.7% margin. This highlights the critical need to route calls according to task difficulty rather than defaulting to the strongest model.
However, the route chosen must also consider the cost per successful workflow, accounting for retries, escalations, and corrections. A cheap route that results in failed tasks is not cost-effective. It is essential to verify the quality of the model before routing and disclose any data usage terms to avoid privacy issues. Conducting this analysis will help identify the usage level where spending goes underwater, allowing for informed pricing and limit decisions.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
