Rethinking Routing Exploration in Modern Mixture-of-Experts
Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token. We identify an…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
