Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers
Efficient attention mechanisms, such as linear and sliding window attention, aim to reduce the computational complexity of Transformers from quadratic to linear. However, their degraded in-context recall often necessitates interleaving full attention layers, leaving the quadratic bottleneck unresolved in long-context processing. To address this, we introduce Orthros attention, which shifts the…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
