Semantic Cache in AI Tokenomics
Introduction A few weeks ago I was looking at logs for a support bot that answers the same handful of questions all day long. "How do I reset my password", "how do I reset my login", "I forgot my password, help", "can't log in, need to reset it". Four different customers, four different sentences, one single question underneath all of them. The bot had a cache. It just never hit. Every single one…
In a recent incident involving a support bot, it was observed that the bot was generating a new model call for each variation of a single question, despite the questions being semantically similar. For instance, four different customers asked the same question in four distinct ways, but the bot treated each as a unique query, resulting in redundant computations. This issue arises because traditional caching techniques rely on exact string matching, which fails to recognize semantically equivalent queries.
The solution lies in implementing a semantic cache, which focuses on meaning rather than exact text. A semantic cache utilizes three key components: an embedding, a similarity score, and a threshold. Embeddings are numerical representations of sentence meanings, generated by embedding models provided by AI providers. The similarity score, ranging from 0 to 1, quantifies the closeness of two embeddings using cosine similarity.
A threshold, such as 0.92, determines when a cached answer is considered sufficiently similar to a new query, warranting its reuse.
To illustrate, consider a scenario where the cache stores the answer to "How do I reset my password". When new questions like "How do I reset my login" or "I forgot my password, help" are presented, their embeddings are compared with the stored embedding. If the similarity score exceeds the threshold, the cached answer is reused, bypassing the need for a new model call. Only when a question is deemed unrelated to any cached query does the system generate a new answer and store it for future reference.
Implementing a semantic cache entails creating functions for similarity calculation, cache management, and question answering. The similarity function calculates cosine similarity between embeddings, while the SemanticCache class maintains a list of past answers and determines whether a new question warrants a cached response or a fresh model call.
By incorporating a semantic cache into AI systems, teams can significantly reduce redundant computations and enhance efficiency, ensuring that users receive accurate responses even when queries are expressed in various ways.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
