Higher-order pruning of experts in mixture-of-experts language models
Mixture-of-Experts (MoE) language models suffer from large parameter counts, which create a significant memory bottleneck. Expert pruning is the most direct approach for reducing this parameter count, yet existing methods make pruning decisions for each expert independently, and assume experts' contributions are purely additive. In reality, expert usage in MoEs is inherently cooperative. We…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
