Why an old caching trick is your secret to lower LLM costs
An LLM can answer the same question a thousand times and charge you each time. Before paying for another answer, The post Why an old caching trick is your secret to lower LLM costs appeared first on The New Stack .
LLMs can answer the same question multiple times and charge for each instance. Before paying for another answer, inspect the request, its context, the model settings, and the underlying data to see if any of those have changed. Create an exact-match cache key by fingerprinting those inputs. If the key points to a valid and safe answer, return it without invoking the model. This saves costs by avoiding unnecessary computations.
In production data pipelines, a nightly job often recalculates aggregations that haven't changed since the previous run. It successfully moves the results to production while consuming compute that could have been allocated elsewhere. This waste goes unnoticed until a cost review reveals a significant portion of upstream compute is redundantly answering a question whose inputs remain unchanged.
The solution lies in change detection: hash the upstream inputs that might alter between runs, fingerprint the job's dependencies, and avoid recomputation when the fingerprints match.
The same principle applies to LLM workloads. Repeated requests lead to repeated charges since billing is based on tokens. Many APIs treat duplicate requests as new ones. To mitigate this, normalize the model request body, hash it with a cryptographic algorithm like SHA-256, and store the hash in an in-memory store like Redis. If a match is found, return the answer without waiting for model inference. This approach works well when model requests are predictable and bounded.
For many workloads, exact match caching is sufficient. However, when a query is close but not identical, a semantic match approach is needed. Transform the user's query into an embedding using an embedding model and store the vector in a vector database. When a new query arrives, run it through the same embedding model and search for close matches using cosine similarity.
A common threshold range is between 0.90 and 0.95, but this should be tuned based on the embedding model and data. Be cautious with loose thresholds as they increase the risk of incorrect matches. Tier 3 combines both exact-match and semantic search approaches, prioritizing cheap exact matches for repeat traffic.
To ensure cache entries are accurate, key on more than just the query text. Include the context and documents in the prompt, the model and its settings, the version of any retrieved source, and the caller's access scope. Two identical questions asked with different documents or by users with varying permissions should not share a cache entry.
The provided pseudocode demonstrates the full flow, incorporating normalization, hash generation, Redis cache lookup, embedding, vector database search, similarity comparison, and TTL management.
Written by urgent.news from The New Stack's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
