Cheapest per token is not cheapest per finished job: what 2,400 runs show

A recent benchmark of 10 models revealed that a model with a 33% success rate was the cheapest per finished job. This might seem like a free lunch, but there are important factors to consider. The metric used to calculate cost per successful task includes every attempt, whether it was successful, failed, retried, or timed out. This metric does not overlook retries, malformed outputs, or unfinished runs.

To demonstrate, Arize and Fireworks ran 10 models on 40 real command-line tasks using Terminal-Bench. The models included gpt-oss-120b, gemini-3.1-flash-lite, GPT-5.5, Kimi K3, Claude Sonnet 5, and gemini-3.5-flash. The results showed that while higher-cost models like GPT-5.5 and Kimi K3 had higher pass rates, the cheapest model, gpt-oss-120b, was still 12 times cheaper per finished task than GPT-5.5.

The benchmark highlights the importance of considering retries and other unsuccessful attempts when evaluating a model's cost. A cheap model may not always result in a cheaper finished product. Additionally, coverage is a crucial factor to consider, as it refers to the percentage of tasks a model solved reliably. In this benchmark, gpt-oss-120b covered 8 out of 40 tasks, while GPT-5.5 covered 25 and Kimi K3 covered 26.

Furthermore, the benchmark revealed that price tier did not necessarily predict success across different difficulty levels. For instance, GPT-5.5 had a 69% pass rate on easy tasks, Kimi K3 achieved 100% on tasks requiring frontier-priced resources, and gpt-oss-120b achieved 65% on harder tasks, despite being significantly cheaper per attempt. This suggests that while price may be a factor, it should not be the sole consideration in selecting a model for a specific task.

In conclusion, the key takeaway from this benchmark is that the cost per successful task should be the primary metric when evaluating models for a specific task. This approach ensures that the chosen model is not only cost-effective but also reliable and capable of handling the task's requirements. By running this metric on your own prompts and comparing the costs and performance of various models, you can make an informed decision on which model is the best fit for your needs.