{
  "id": 18524564,
  "title": "What I got wrong estimating LLM costs before the request",
  "url": "https://urgent.news/2026/11/06/what-i-got-wrong-estimating-llm-costs-before-the-request",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-06T16:12:22.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/elson_pang_69c87e7fb31893/what-i-got-wrong-estimating-llm-costs-before-the-request-1fla"
  },
  "original_language": "en",
  "account": "Ever since artificial intelligence took over the internet, the author has been pondering how to optimize the full potential of AI in our lives. To achieve this, they built a small gateway that sits between your code and the OpenAI, Anthropic, and Gemini APIs. The gateway's sole job is to prevent a request from being sent out if it would exceed a daily spending limit set by the user. The reasoning behind this is quite simple: an agent getting stuck in a loop, repeatedly retrying, each time sending a longer conversation, can lead to unnoticed bills. The gateway aims to say no before the bill arrives.\n\nThe author's initial idea was straightforward. Before a request is sent, they estimate the cost based on the output price and the maximum number of tokens the request could generate. However, they soon discovered several inaccuracies in their cost estimation.\n\nFirstly, it's impossible to predict the output accurately, so the only honest estimate is the maximum possible cost: max_tokens multiplied by the output price per million. The author tested this by setting a daily cap of $0.001 and sending a prompt with 10,000 max_tokens to GPT-6-Luna. The gateway refused the request, stating a 402 error code. Surprisingly, this meant that the max_tokens parameter stopped being a length setting and became a budget setting.\n\nSecondly, reasoning models can consume the entire budget while processing, even if the actual output is minimal. In a test with GPT-6-Luna, a short prompt produced an empty response but consumed all 20 output tokens for reasoning. The author noted that under small caps, such models would get refused, but the empty reply could still cause issues for the other party.\n\nThirdly, the author underestimated the number of input tokens, especially when using the o200k tokenizer. In a test, they estimated 1 input token, but OpenAI billed 7. This discrepancy was mainly due to formatting added by the chat format, which wasn't accounted for. The author had not fixed this issue yet, as it grows with each message in long agent conversations.\n\nFourthly, the author's initial approach of pricing unknown models using the most expensive one they knew wasn't reliable. Different models have different pricing, and models can be retired. The author now strictly checks if a model is in their price table before allowing a request. If it's not, the request is rejected with a 400 error code and a note to use the model name without a date.\n\nFifthly, handling images was a guessing game initially. The author estimated each image as 4,000 input tokens, ignoring the text. This approach undercounted the cost for image-heavy requests. The author now counts the text normally and reserves each image based on the provider's worst-case token limit, which can be up to 36,000 tokens for some newer OpenAI models. This approach is safe but can lead to reservations taking up space before the real cost is incurred, especially for small caps.\n\nLastly, caching can mislead the cost estimation. Cached input is charged at a lower rate, but the author's initial estimate charges all input at full price. In a long-running agent scenario, 95.4% of the tokens were cached re-reads. The author's check would hit the cap early, while the actual bill would be significantly lower. To address this, the author suggests estimating based on the recent cache-hit rate, but this is still an ongoing work in progress.",
  "summary": "Ever since AI has taken over the internet, I've been wondering how we can optimize the full ability of AI in our life. Therefore I began building a small gateway that sits between your code and the OpenAI, Anthropic and Gemini APIs. It has one job: if a request would push today's spend past a daily cap you set, refuse it before it goes out. Why bother? Because the stories are always the same. An…",
  "key_points": [
    "Initial cost estimation method was inaccurate due to inability to predict output accurately.",
    "Underestimation of input tokens, especially with o200k tokenizer, led to billing discrepancies."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}