The Semantic Cache That Made a Free LLM Quota Feel Infinite

Token allowances for free language model tiers are often viewed as a spending budget, but this approach misses a crucial optimization opportunity. Rather than treating the allowance as something to be depleted, it's more effective to think of it as a cache to be managed efficiently. A semantic cache that serves previously generated responses for rephrased queries can reduce token consumption by up to half in typical agent workflows.

This article explains how to implement a zero-dependency semantic cache and fine-tune its parameters for safe production use. MonkeyCode's free offering currently provides a 10-million-token allowance and a free server option, making effective quota management a practical concern. The percentages and latency figures presented below are illustrative measurements from a controlled prototype, not guarantees of specific outcomes.

In most agent workflows, developers assume prompts are mostly unique, leading to unnecessary token consumption and latency. A closer look at request logs reveals that about 35% of prompts are semantically near-duplicates of previous answers. Sending duplicate prompts incurs unnecessary token costs, increased latency, and the risk of inconsistent responses when the model reformulates answers differently.

Instead of expecting the model to become smarter, the solution lies in preventing duplicate requests from reaching it in the first place. A semantic cache placed in front of the endpoint can recognize and return cached responses for similar prompts, reducing the number of calls to the model. The cache does not replace the model; it simply filters out redundant requests.

To implement a semantic cache, the first step is selecting an appropriate similarity metric and threshold. A zero-dependency option is character n-gram Jaccard similarity, which tokenizes text into three-character overlapping shingles and compares their sets. While more advanced methods like embedding models exist, they introduce additional dependencies and API costs.

A threshold of 0.92 was chosen through calibration of one hundred logged prompts, balancing the need to catch duplicates versus the risk of returning incorrect cached responses. The choice of similarity metric depends on prompt length, with character n-grams suitable for short prompts and word-based methods or embeddings potentially better for longer text.

After a response is retrieved from the cache, it should be stored with a time-aware eviction policy to prevent memory leaks. The prototype uses SQLite with a created_at timestamp and a TTL (time-to-live) of one hour, striking a balance between freshness and hit rate. Shorter TTLs protect against stale answers, while longer TTLs improve hit rates but risk serving outdated information.

The schema stores the original prompt, response text, token cost of the original call, creation time, and hit counter. The hit counter enables a popularity-aware eviction strategy, where the cache removes the oldest entries with the lowest hit counts when it exceeds a size limit. This strategy focuses on prompts that actually recur, rather than one-off requests.

The provided Python code demonstrates a complete cache implementation, including initialization, tokenization, similarity scoring, retrieval, and insertion. It is designed to be dependency-free, allowing it to run on any fresh Python environment, including free servers without package installation constraints.