{
  "id": 18215949,
  "title": "Cheapest per token is not cheapest per finished job: what 2,400 runs show",
  "url": "https://urgent.news/2026/11/05/cheapest-per-token-is-not-cheapest-per-finished-job-what-2-400-runs",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-05T00:47:31.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/grain64/cheapest-per-token-is-not-cheapest-per-finished-job-what-2400-runs-show-1484"
  },
  "original_language": "en",
  "account": "A recent benchmark of 10 models revealed that a model with a 33% success rate was the cheapest per finished job. This might seem like a free lunch, but there are important factors to consider. The metric used to calculate cost per successful task includes every attempt, whether it was successful, failed, retried, or timed out. This metric does not overlook retries, malformed outputs, or unfinished runs.\n\nTo demonstrate, Arize and Fireworks ran 10 models on 40 real command-line tasks using Terminal-Bench. The models included gpt-oss-120b, gemini-3.1-flash-lite, GPT-5.5, Kimi K3, Claude Sonnet 5, and gemini-3.5-flash. The results showed that while higher-cost models like GPT-5.5 and Kimi K3 had higher pass rates, the cheapest model, gpt-oss-120b, was still 12 times cheaper per finished task than GPT-5.5.\n\nThe benchmark highlights the importance of considering retries and other unsuccessful attempts when evaluating a model's cost. A cheap model may not always result in a cheaper finished product. Additionally, coverage is a crucial factor to consider, as it refers to the percentage of tasks a model solved reliably. In this benchmark, gpt-oss-120b covered 8 out of 40 tasks, while GPT-5.5 covered 25 and Kimi K3 covered 26.\n\nFurthermore, the benchmark revealed that price tier did not necessarily predict success across different difficulty levels. For instance, GPT-5.5 had a 69% pass rate on easy tasks, Kimi K3 achieved 100% on tasks requiring frontier-priced resources, and gpt-oss-120b achieved 65% on harder tasks, despite being significantly cheaper per attempt. This suggests that while price may be a factor, it should not be the sole consideration in selecting a model for a specific task.\n\nIn conclusion, the key takeaway from this benchmark is that the cost per successful task should be the primary metric when evaluating models for a specific task. This approach ensures that the chosen model is not only cost-effective but also reliable and capable of handling the task's requirements. By running this metric on your own prompts and comparing the costs and performance of various models, you can make an informed decision on which model is the best fit for your needs.",
  "summary": "A model with a 33% pass rate turned out to be the cheapest way to finish a task in a recent 10-model benchmark. The same benchmark shows why that result is less of a free lunch than it looks. Here is the metric behind it and how to compute it on your own prompts. The metric Cost per successful task = everything the model spent across every attempt (failed, retried, timed out) divided by the…",
  "key_points": [
    "Cheapest model gpt-oss-120b 12 times cheaper per finished task than GPT-5.5",
    "Coverage crucial factor: gpt-oss-120b solved 8/40 tasks, GPT-5.5 solved 25/40",
    "Price tier not always predicts success: GPT-5.5 69% pass rate on easy tasks"
  ],
  "editors_take": "Prioritizing token pricing over successful task completion ignores the hidden expenses of retries and failures, making reliability a more critical factor for true cost-effectiveness in model selection.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}