DCKV: Learned Transform Coding of the Prompt KV Cache in Hybrid Language Models
Hybrid language models keep a key-value (KV) cache only in their few full-attention layers, but this cache is still the part of a prompt's state that grows with the prompt and that must be stored or moved to reuse it. We present DCKV, a differentiable codec for the KV cache and the first KV-cache codec that is trained end to end. Its transform, its quantizer and its entropy models, which are…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
