CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference
Despite their strong performance, large language models (LLMs) are bottlenecked by KV cache memory traffic during long-context inference. Sparse attention is widely used to accelerate LLM inference by computing exact attention over a selected subset of tokens. To recover the contribution of tokens excluded from exact attention, recent methods apply coarse-grained compensation to the omitted…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
