Cutting LLM inference cost and latency without making the feature worse
The first bill for an LLM feature is usually a surprise in one of two directions. Either it is tiny and nobody thinks about it again until usage grows, or it is larger than expected and someone asks for the cheapest model by Friday. Both reactions skip the useful step, which is finding out where the tokens and the seconds actually go. These are the levers I reach for, in roughly the order I pull…
The cost of using large language models (LLMs) can be a surprise, either being lower than expected or higher than anticipated. To address this, it's crucial to understand where the tokens and time are being consumed. The levers to reduce costs are presented in order of their impact. The simplest levers are almost free, while the more complex ones may require trade-offs.
First, log the cost per request to understand the baseline. Every decision afterward depends on this information. The code uses the Anthropic TypeScript SDK and records the route, model, token counts, time to first token, total time, and cost. By analyzing the logs, one can identify the most expensive features and tasks.
Lever one involves prompt caching. Most product prompts share a common structure: instructions, tool definitions, and reference material. By caching the common prefix, the cost of processing it is incurred only once, while subsequent requests only incur a fraction of the cost. This technique also reduces the time to first token, as the model skips reprocessing the cached content.
The cache matches a prefix byte for byte, ensuring the request is read in a fixed order: tools, then the system prompt, and finally the changing message blocks.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
