Billing LLM usage per token: the pitfalls nobody warns you about
I run a multi-provider LLM gateway in production (OpenAI, Anthropic, Google, DeepSeek and a dozen others behind one endpoint) with prepaid, per-token billing. Getting the metering correct took more iterations than the entire proxy itself. Here is what I wish someone had told me. One request is never one price A single chat request produces up to six differently priced components: text input,…
Billing usage per token for multilingual language models (LLMs) can be a complex endeavor. In production, managing prepaid, per-token billing for providers like OpenAI, Anthropic, Google, DeepSeek, and more, each with distinct pricing components, can be challenging. A single chat request may involve up to six different priced components: text input, cached input, cache writes, output, reasoning tokens, and tool units like web searches or image generations.
To avoid shortcuts, it's crucial to charge each component individually with its own rate and sum them per request trace. Streaming usage, however, is tricky. If you rely solely on the final SSE chunk for usage, you may undercount. OpenAI provides usage in the last chunk when requested via stream_options, while Anthropic reports input and cache tokens in message_start and the rest in message_delta.
To ensure accuracy, accumulate every usage payload seen anywhere in the stream, keeping cached and uncached input separate, and only price the total at the very end.
Client aborts can also lead to unexpected charges. When a user hits stop or closes the tab, the provider still bills you, but you might not see the final usage frame, resulting in zero recorded usage. To capture usage, continuously drain the provider stream after the client disconnects. It's essential to test scenarios where the user presses stop and closes the tab separately, as they may behave differently.
A preflight check is necessary to estimate the worst-case scenario before initiating a request. Calculate the worst-case usage by considering the prompt tokens and the maximum tokens per output rate. If the balance cannot cover the estimated usage, refuse the request. After the request is completed, settle against the real usage.
Two crucial details to remember: globally cap the maximum tokens to prevent any single request from defeating any sane check, and treat agent recursion limits as a cost brake, as one rate-limited tool can initiate an agent loop that burns real money before the limit is reached.
When dealing with multiple currencies, store the FX rate used at grant time and derive display costs from the USD source of truth. Ensure that display fields are divided by the rate instead of multiplied, as a bug in the past demonstrated. Implement reconciliation jobs that assert the difference between granted and consumed costs equals the balance per batch to catch any discrepancies.
All of these considerations are implemented in production at kral.ai, a managed LibreChat platform with integrated billing. For more details, including the full write-up on SSO, teams, and per-token billing, visit LibreChat for the enterprise. Should you have any questions about the failure modes discussed, feel free to ask in the comments.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
