KV cache cut by ~45% with near‑same accuracy

Grouped Value Attention (GVA) slashes the memory usage of transformer key-value caches by about 45% without affecting benchmark performance. This is achieved by storing only grouped values and calculating keys as needed during decoding, eliminating the need for a full key matrix during the decoding process and dramatically reducing persistent cache scalars.

Traditional transformers required large key-value caches that scaled linearly with sequence length, which has been a major bottleneck in autoregressive inference. Even techniques like grouped-query attention still maintained a key for every token, keeping memory pressure as the dominant issue. In a model with 350 million parameters, GVA manages to cut the persistent cache scalars by roughly 46% while maintaining an average task accuracy within 0.01 points of the original GQA.

The 16-dimensional positional variant of GVA reaches an average accuracy of 44.35 across five tasks, compared to GQA's 44.36 and MLA's 43.88. DeepSeek-V4.1-Flash takes KV compression even further, shrinking the global HBM footprint to just 890 bytes per token, about a quarter of its predecessor. Additionally, a deployment optimization called SWA Bounded Replay reduces the persistent KV cache footprint to roughly one-eighth of DeepSeek-V4-Flash's, while still delivering better performance than the baseline.

However, it remains unclear how these memory savings will impact end-to-end latency reductions on commodity GPUs. The custom decoding kernels in GVA are still under evaluation, so real-world throughput gains are yet to be proven. DeepSeek's design incorporates FP4 caching and cross-layer reuse, which could benefit from hardware supporting low-precision arithmetic.

If the reported memory cuts hold true in production, it could be possible to substantially reduce the KV cache allocation for decoder-only models while maintaining quality, potentially enabling longer token contexts on more modest hardware.