The Hidden Math: Why "Cheapest Model" Doesn't Mean Cheapest Execution

Contrary to intuition, selecting the cheapest model for an AI agent does not guarantee the lowest overall execution costs. While it may seem logical to simply pick the least expensive model for each task, the reality is more complex. When an agent runs code that involves multiple model calls, the total cost can be much higher than anticipated. This discrepancy arises from the hidden complexities of token costs, output lengths, and routing logic.

Token costs do not compound uniformly across different models. For instance, consider an agent summarizing a customer support transcript consisting of 2,000 input tokens. Using Haiku for this task incurs a very low input cost of $0.0016 (2,000 tokens × $0.80 per million tokens). However, if Haiku encounters a rate limit and the code falls back to Sonnet, the same 2,000 tokens now cost $0.006 (2,000 tokens × $3 per million tokens) for input alone. This represents a nearly fourfold increase in cost due to the fallback decision.

Moreover, output lengths are not transparent. An agent might produce a brief response with only 500 output tokens or a more detailed reasoning with 5,000 tokens. With Sonnet's output pricing at $15 per million tokens, the cost difference between short and long outputs can be significant. For example, the cost could range from $0.0075 to $0.075 per run, depending on the output length. Over a 1,000 daily executions, this could translate to an overbudget of $37 without any indication of why.

The situation becomes even more complicated when routing across multiple providers with several fallback models. In such scenarios, it is challenging to reason about costs at a glance. It becomes unclear whether a model executed successfully or if it failed and was substituted by another model. Furthermore, the output generated by each model may vary greatly in length, further obscuring the true costs of the execution.

To address these issues, a practical approach is to log and measure every model call, capturing input and output token counts. By logging the actual token counts from response metadata, you replace assumptions about costs with real data. This enables you to group executions by the model used and the routing path taken. You can then measure fallback rates, overall costs, and the frequency of expensive operations.

Additionally, computing costs inline using per-model rates allows you to estimate expenses in real-time during execution. By grouping executions by model and routing path, you can identify which models are being utilized most frequently and whether fallbacks are occurring more often than expected.

Finally, setting up alerts for cost outliers can help detect anomalies promptly. If a single execution's cost deviates significantly from the average, it may indicate an unexpected output length, a fallback that was not anticipated, or an excessive number of retries. This level of visibility ensures that you are not flying blind when it comes to the arithmetic behind your AI agent's execution costs.

By measuring and accounting for these factors, you can move from an intuitive, but potentially inaccurate, understanding of costs to a data-driven approach that ensures budgeting based on factual information rather than assumptions.