{
  "id": 9707190,
  "title": "KV cache cut by ~45% with near‑same accuracy",
  "url": "https://urgent.news/2026/09/25/kv-cache-cut-by-45-with-near-same-accuracy",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-25T05:00:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/olaughter/kv-cache-cut-by-45-with-near-same-accuracy-2b95"
  },
  "original_language": "en",
  "account": "Grouped Value Attention (GVA) slashes the memory usage of transformer key-value caches by about 45% without affecting benchmark performance. This is achieved by storing only grouped values and calculating keys as needed during decoding, eliminating the need for a full key matrix during the decoding process and dramatically reducing persistent cache scalars. Traditional transformers required large key-value caches that scaled linearly with sequence length, which has been a major bottleneck in autoregressive inference. Even techniques like grouped-query attention still maintained a key for every token, keeping memory pressure as the dominant issue. In a model with 350 million parameters, GVA manages to cut the persistent cache scalars by roughly 46% while maintaining an average task accuracy within 0.01 points of the original GQA. The 16-dimensional positional variant of GVA reaches an average accuracy of 44.35 across five tasks, compared to GQA's 44.36 and MLA's 43.88. DeepSeek-V4.1-Flash takes KV compression even further, shrinking the global HBM footprint to just 890 bytes per token, about a quarter of its predecessor. Additionally, a deployment optimization called SWA Bounded Replay reduces the persistent KV cache footprint to roughly one-eighth of DeepSeek-V4-Flash's, while still delivering better performance than the baseline. However, it remains unclear how these memory savings will impact end-to-end latency reductions on commodity GPUs. The custom decoding kernels in GVA are still under evaluation, so real-world throughput gains are yet to be proven. DeepSeek's design incorporates FP4 caching and cross-layer reuse, which could benefit from hardware supporting low-precision arithmetic. If the reported memory cuts hold true in production, it could be possible to substantially reduce the KV cache allocation for decoder-only models while maintaining quality, potentially enabling longer token contexts on more modest hardware.",
  "summary": "Grouped Value Attention slashes transformer KV memory by roughly 45 % without hurting benchmark scores. By storing only grouped values and reconstructing keys on the fly, it eliminates the need to materialize a full key matrix during decode, cutting the persistent cache scalars dramatically. Before these works, long‑context decoding relied on full key–value caches that grew linearly with sequence…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}