MIRA: Mitigating Pole Geometry Bias in Recurrent Language Models
We show that the Transformer's recurrent successors --- such as selective state-space models (SSMs), Receptance Weighted Key Value (RWKV), and Retentive Network (RetNet) --- can be characterized as infinite-impulse-response (IIR) systems. The zero--pole geometries of these IIR systems characterize the temporal modes they can represent. However, to trade parallelism, the algebraic structures of…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
