{
  "id": 17596688,
  "title": "Halving the KV Cache Doubles Capacity and Buys 1% of Speed",
  "url": "https://urgent.news/2026/11/01/halving-the-kv-cache-doubles-capacity-and-buys-1-of-speed",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-01T12:23:54.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/jaxsinclair/halving-the-kv-cache-doubles-capacity-and-buys-1-of-speed-4j31"
  },
  "original_language": "en",
  "account": "If the engine uses an FP8 cache, the long-context penalty is exactly halved for every context length, on every card. However, the speed does not double at any long context length. This is derived from the formula that compares the bandwidth divided by bytes read per token. Halving the cache bytes per token increases the gain from 1.00x to 2.00x, depending on the model, card, and context length. The crossover point, where the cache read equals the weight read, determines when the speed gain becomes noticeable. For example, Qwen3-32B on an H100 reaches this point at 128k context length, while Llama-3.3-70B only reaches it at 431k. Quantizing the cache can increase the number of sequences that fit within memory, but it does not change the memory wall or make the model compute-bound. The main effect is to allow more sequences to be processed at once, with a potential speed gain of up to 21% for the Qwen3-32B model at 128k context length.",
  "summary": "A reader ran the script from my last piece against the table, found a bug in my code, and asked a better question than the one I answered. The bug first. For Llama-3.3-70B on an 80 GiB H100 the fit column printed a negative sequence count, about -41 at 4k. The table said \"does not fit\" and the table was right, but the script printed a number, and a number that means nothing invites a diff. It now…",
  "key_points": [
    "Halving KV cache doubles capacity",
    "Speed does not double at any length",
    "Speed gain up to 21% for Qwen3-32B"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}