Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training
Most parameters of a Mixture-of-Experts (MoE) model belong to its experts, and standard training keeps every expert gradient in GPU memory until the optimizer step. This memory grows with the number and size of the experts, so it rises as MoE models scale. We introduce Sonar-MoE, which updates each expert tensor during the backward pass and frees its gradient right away. It brings…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
