Prompt Caching: Why cache_control Writes But Never Reads

The article discusses a common issue with prompt caching in AI agent loops. The author enabled prompt caching for a system prompt that is resent on every turn, expecting a performance boost. However, the caching mechanism only wrote to the cache and never read from it, resulting in a 40% increase in input tokens while still delivering the same result.

The key issue is that the cache_control breakpoint is placed after content that changes with each request, such as timestamps or user-specific information. This causes the cache to create a new entry for every request, resulting in no cached reads. The cache key is based on the rendered prompt up to each cache_control breakpoint, and any change at or after a breakpoint invalidates the cache entry.

The author explains that prompt caching works as a prefix match, where the cache key is the exact bytes of the rendered prompt up to the breakpoint. If a single byte changes at or after a breakpoint, the entire cache entry is invalidated. Therefore, if the cache_control breakpoint is placed after content that changes each request, the cache will always write a new entry and never read from the cache.

The article also highlights two silent killers that cause byte-identical payloads to still miss the cache: a 20-position lookback window in long tool turns and parallel fan-out, where an entry is only readable after the first response starts streaming. These issues can lead to the cache writing unnecessarily and never being read.

To ensure prompt caching is working correctly, the author suggests checking three fields in the usage object: input_tokens (uncached remainder), cache_creation_input_tokens (cost of writing to the cache), and cache_read_input_tokens (cost of reading from the cache). The sum of these fields should match the total prompt tokens used. If cache_read_input_tokens is consistently zero for repeated requests with identical prompts, something upstream is likely rewriting the prefix.

The article concludes with a top-level cache_control placement rule: it should be at the end of the shared portion of the prompt, not at the end of the whole prompt. This ensures that the breakpoint lands at the end of the fixed preamble followed by the unique per-request question, allowing the cache to effectively store and retrieve the unique parts of each request.