{
  "id": 15771616,
  "title": "The KV cache is eating your GPU. MQA and GQA are how models fight back",
  "url": "https://urgent.news/2026/10/20/the-kv-cache-is-eating-your-gpu-mqa-and-gqa-are-how-models-fight-back",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-20T17:16:12.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/dev_jhawar/the-kv-cache-is-eating-your-gpu-mqa-and-gqa-are-how-models-fight-back-3809"
  },
  "original_language": "en",
  "account": "Quick recap from the previous article: the KV cache saves on re-computing past tokens, explaining why generation remains swift. However, it comes at a cost - memory. To quantify this, the formula is used: for every token, you store a Key and a Value for each layer and head. The formula is: bytes per token = 2 x layers x kv_heads x head_dim x bytes_per_number. With fp16 precision being 2 bytes per number, the formula can be implemented in code. Using Llama-2 7B as an example, with 32 layers, 32 heads, and head_dim of 128, the KV cache consumes 512 KB per token. For an 8k context, this amounts to 4 GB. For a 128k context, it reaches 64 GB, all for a single request. The model weights are not included in these calculations. When serving multiple users simultaneously, each user has their own cache, which ultimately uses up the GPU memory. The formula highlights that layers and head_dim dictate the model's quality, while kv_heads is the variable to watch. In traditional multi-head attention (MHA), every query head has its own key and value heads. But is it necessary for all query heads to have separate keys and values? Multi-Query Attention (MQA) shares a single K head and V head across all query heads, reducing memory usage by 32x. The catch is quality - shared keys and values may compromise performance, and training might become more challenging. Grouped-Query Attention (GQA) strikes a balance between MQA and MHA by splitting query heads into groups, each sharing a single KV head. Llama-3 8B, with 32 query heads and 8 KV heads, demonstrates a 4x reduction in cache size without sacrificing quality. This approach allows models to affordably support longer contexts, impacting latency and potentially pricing. When building agents, providers' claims of 128k or 1M context support are often made possible by these GQA-style techniques. For those self-hosting models, KV memory is crucial in determining how many concurrent users can fit on a single GPU. Perform calculations before purchasing hardware. Note that these numbers are for one sequence at fp16 precision. Batch processing multiplies the memory requirements, while quantizing the cache can reduce them. Quality loss is minimal and demonstrated in various papers and model cards. The author has not personally evaluated the quality, and readers should not rely solely on blog posts. The GB figures are derived from the formula, not profiler outputs. Real servers add additional overhead. The subsequent article will explore how the same sentence can incur vastly different costs depending on the tokenizer, leading to more efficient hardware utilization and potentially lower costs.",
  "summary": "Quick recap from the last post: the KV cache saves you from recomputing old tokens, and it's the reason generation is fast. But it has a bill. And the bill is memory. I wanted to put an actual number on it, because \"the cache is big\" doesn't mean much until you see the GB. The formula Per token, you store a Key and a Value for every layer and every head: bytes per token = 2 x layers x kv_heads x…",
  "key_points": [
    "KV cache consumes 512 KB per token for Llama-2 7B",
    "MQA reduces memory usage by 32x compared to traditional MHA",
    "GQA balances MQA and MHA, reducing cache size by 4x"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}