Why the KV cache, not the weights, decides how many users an LLM server can hold

A 7B Llama-2 model with 32 layers and 32 attention heads requires 13.5 GB of GPU memory for its weights. This leaves around 66 GB of free memory on an 80 GB GPU card. However, the KV cache quickly consumes this free memory, limiting the number of users an LLM server can handle. The KV cache, which stores keys and values from previous tokens, is crucial for attention computation but is not part of the model weights.

A single token's KV cache occupies about 0.5 MiB of memory. A typical 4,096-token context, when batched, requires nearly 68 GB of cache memory. This is more than the memory needed for the model's weights, which are allocated once. With only 66.5 GB of free memory, the server can only manage approximately 30 sequences of 4,096-token conversations before hitting cache memory limits.

Batch processing improves efficiency, allowing a single read of weights to serve all sequences, but each additional sequence consumes additional cache memory, capping the server's throughput. Techniques to reduce the KV cache size include using fewer KV heads, storing the cache in int8 precision, employing a sliding window to retain only recent tokens, and applying paged attention to manage cache memory more efficiently.