The KV Cache From Scratch
In every part of this series, one thing kept showing up: the notebook . In Part 1 it grew by 72 KB with every word. In Part 3 it was the part of the work that batching couldn't share. In Part 4 it stayed stuck on the memory side of the roof, however many users we added. Engineers call it the KV cache . Today we build one by hand, check that it really gives the same answer, weigh it in bytes, and…
In this report, we delve into the KV cache—a crucial component of language models that we will build manually. The KV cache works by storing question (Q), label (K), and value (V) cards for each word in a text. When generating the next word, the model compares the new word's Q card with the labels of previously generated words. The better the match, the more the content (V) card of the new word is mixed in.
Importantly, a word only looks backward—once a word has created its cards, they cannot be altered by later words. This leads to the creation of a notebook, which stores the K and V cards for all earlier words, enabling the model to read from this storage instead of regenerating the entire text.
To illustrate the efficiency of the KV cache, we compared two versions of a 12-layer GPT-2 small model (12 floors, width 768, 12 heads, random weights) while generating 32 words. The first version ran without the KV cache, with the time per new word ranging from 231.7 to 330.2 ms. The second version, with the KV cache enabled, showed significantly faster performance—20.4 to 21.4 ms per new word.
This represents a notable improvement, with the notebook making the typing process around 15 times faster, especially for longer answers where the difference becomes even more pronounced.
The KV cache's memory footprint is substantial. For each word, the notebook saves two cards (K and V) on every floor and for every head, resulting in a memory cost of approximately 73,728 bytes per token. This amounts to 74 MB for 1,000 tokens, 590 MB for 8,000 tokens, and 2359 MB for 32,000 tokens. Interestingly, once the notebook grows beyond approximately 4,600 words, it surpasses the size of the entire model itself.
Even with real-world models like Llama 3.1 8B, the notebook can consume a significant portion of the GPU memory. For instance, a single 8K-word chat on Llama 3.1 8B would require about 1 GB of notebook memory, which is more than the 16 GB the model's tables occupy. Consequently, this highlights the need for careful management of notebook memory, especially when dealing with long contexts in language models.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
