MoSAR: Mixture of Semantic Attention Regimes for Learning Adaptive and Approximable Attention Geometries
The quadratic complexity of dense self-attention remains a central bottleneck for long-context language modeling. Many efficient alternatives address this cost by deciding in advance where attention should be sparse or local. We argue that attention approximation should instead be approached as a geometric problem, with the relevant interaction geometry learned from data: natural-language…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
