Halving the KV Cache Doubles Capacity and Buys 1% of Speed
A reader ran the script from my last piece against the table, found a bug in my code, and asked a better question than the one I answered. The bug first. For Llama-3.3-70B on an 80 GiB H100 the fit column printed a negative sequence count, about -41 at 4k. The table said "does not fit" and the table was right, but the script printed a number, and a number that means nothing invites a diff. It now…
If the engine uses an FP8 cache, the long-context penalty is exactly halved for every context length, on every card. However, the speed does not double at any long context length. This is derived from the formula that compares the bandwidth divided by bytes read per token. Halving the cache bytes per token increases the gain from 1.00x to 2.00x, depending on the model, card, and context length.
The crossover point, where the cache read equals the weight read, determines when the speed gain becomes noticeable. For example, Qwen3-32B on an H100 reaches this point at 128k context length, while Llama-3.3-70B only reaches it at 431k. Quantizing the cache can increase the number of sequences that fit within memory, but it does not change the memory wall or make the model compute-bound.
The main effect is to allow more sequences to be processed at once, with a potential speed gain of up to 21% for the Qwen3-32B model at 128k context length.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
