Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts
Sparse mixture-of-experts (MoE) models use an independently parameterized router at each sparse layer to select experts for every token. Prior work has shown that routing decisions across depth can often be predicted from earlier routing signals, suggesting that routing is not fully independent across layers. However, the structure behind this predictability remains unclear. In this work, we…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
