The cheapest LLM call is the one you don't make: a caching layer that actually pays off
The cheapest LLM call is the one you don't make: a caching layer that actually pays off In the last post I wrote about routing across providers to cut our bill ~40%. Caching was the second lever — and honestly the more underrated one. Here's what we learned shipping it. Routing gets most of the attention because it's sexy: traffic dancing across providers, failover kicking in, dashboards lighting…
The most economical way to access language models is by not making unnecessary calls, according to the author. Caching emerged as the second critical strategy after optimizing routing across providers, which had previously cut their bill by approximately 40%. The author discovered that a significant portion of their traffic consisted of repeated requests with near-identical inputs, prompting a rethink of their approach to model calls.
Three caching strategies proved highly effective: the first being an exact cache, which stored full request hashes and served pre-stored completions when identical requests were encountered. Despite initial skepticism, this straightforward method significantly reduced costs on high-traffic endpoints. The second strategy involved a semantic cache that compared user inputs using embeddings in a vector index, reusing completions for similar prompts.
However, this approach is limited to deterministic tasks like classifications and extractions, as it risks serving outdated information for creative generation. The third cache strategy focused on deterministic preprocessing steps, transforming them into pure functions that could be cached without involving the model.
The author recommends adjusting cache expiration times based on the volatility of the data, with stable reference answers benefiting from longer TTLs and fast-moving data requiring shorter or no expiration. Additionally, token costs associated with cache lookups should be minimized, as they can still incur expenses. The author emphasizes the importance of measuring cache hit rates rather than solely focusing on savings, as the hit rate can indicate when caching measures are no longer effective due to changes in prompts or use cases.
Implementing caching alongside routing and circuit breaker techniques can lead to substantial reductions in costs while also improving response times. The author emphasizes that these caching strategies are not novel, but rather an extension of traditional caching practices applied to model calls. They invite further discussion on the subject, offering to share experiences and insights in a more casual setting.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
