How to Loop MoE: Flatten the Experts, Untie the Attention
Looped Transformers reuse one block of layers several times: by spending extra computation they push a model of fixed size further, and so use its parameters more fully; while sparse mixture-of-experts (MoE) models activate only a few of many experts for each token. Looped MoE bridges these two design philosophies and gives MoE models new potential for better expert usage, but it raises a…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
