Same token price is not the same bill

Claude Sonnet 5.5 and GPT-6.1 Sol were released a day apart and priced identically at $2 per million input tokens and $10 per million output. However, the pricing sheet fails to reveal the most crucial difference between the two models. To illustrate this, the author conducted a comprehensive comparison of 132 agent runs using a real open source repository, TinyDB, through a customized harness.

Although the initial results were a 30/30 tie, the author then increased the difficulty of the tasks and ran them at two different effort levels. Surprisingly, the scores remained tied after this adjustment.

Despite the models performing equally well in terms of correctness, there were significant differences in cost, speed, tool use, and code-review tradeoffs. The underlying reason for this disparity lies in the fact that a single token price does not account for the varying number of tokens a model consumes to complete a task. The tokenization process does not consistently count the same text in the same manner across different models, further complicating the comparison.

Moreover, agents exhibit varying levels of tool calls and turns required to achieve a passing result.

The author emphasizes that choosing a coding model based solely on pricing information from the pricing page is misleading, as it compares the wrong unit. To accurately assess the cost effectiveness of a model, one should measure the cost per graded task on their own repository. The author concludes that while the pricing per token may appear identical, the underlying habits and efficiencies of the models differ significantly, leading to different outcomes in real-world applications.