How Local Mixing Encodes Relative Position in Global NoPE Attention
The attention operation is naively position invariant. However, positional information is fundamental to natural language, and therefore a variety of explicit position encodings have been developed in transformer-based models, such as rotary position encoding (RoPE). Although explicit position encodings have long been assumed to be required, recent methods that interleave local mixing layers,…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
