Prompt caching explained: what a cache hit actually saves on your LLM bill
An agent loop sends the same long prefix on every call: the system prompt, the tool definitions, the reference documents pinned at the top, the conversation so far. On a long session that prefix can dwarf the actual question, and you pay input price for all of it again and again. When a provider matches a repeated prefix, it reuses the earlier work and bills those tokens at a lower cache read…
When using large language models (LLMs) for tasks such as summarizing documents, the same long prefix often appears in every request. This prefix includes the system prompt, tool definitions, reference documents, and previous conversation history. Sending the same prefix multiple times can be costly, as you are paying the full input price each time, even if the actual question varies.
A cache helps mitigate these costs by storing the stable parts of the prefix so they can be reused. When a provider recognizes a repeated prefix, it reuses the earlier work and bills the tokens at a lower cache read rate. The ratio between the cache-read price and the input price determines whether using caching is beneficial. Models with a higher cache-read price relative to the input price make caching more valuable.
To maximize caching benefits, structure your prompt by placing the stable parts first (system prompt, tool definitions, reference material) and the volatile parts last (latest user message, timestamps, request IDs). Keep the prefix byte-identical between calls to ensure the cache hit is recognized. Frequent follow-up requests soon after the first can also help maintain cache hits.
Cache expiration varies by provider, and staying on a single model for a conversation helps maintain the cache. Checking if caching is working can be done by examining response logs, which report the tokens served from cache. For example, OpenAI's API response format includes `usage.prompt_tokens_details.cached_tokens`, while Anthropic uses `usage.cache_read_input_tokens`, and DeepSeek includes `prompt_cache_hit_tokens`.
If the cached token count is zero on a repeated prefix, something in that prefix is changing between calls.
Running the process twice with the same long prompt typically shows most of the prefix being cached on most providers. Caching benefits are not limited to a single provider, as ModelSite's Anthropic-compatible endpoint also reports cache hits. However, not all models report cache hits, and some providers price cache writes separately. The model page on each provider's website provides the full breakdown of pricing.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
