Why MuonGS Beats AdamW and Muon in LLM Pretraining

MuonGS outperforms both AdamW and the standard Muon in Large Language Model (LLM) pretraining, according to recent research. This is due to the innovative Gram-Schmidt projection technique employed by MuonGS, which allows for direct mathematical control over weight exploration, unlike uncontrolled weight-norm growth in AdamW and vanilla Muon.

The technology enables a 15% reduction in token-equivalent compute on both 400 million dense and 1 billion parameter mixture-of-experts (MoE) architectures. Specifically, on these models, MuonGS achieved matched validation perplexity 15% faster than MuonH, which translates to hundreds of GPU cluster hours saved.

A key aspect of MuonGS is the strict norm growth bounds it imposes on weight matrices. By removing the radial update component through Gram-Schmidt projection, the Frobenius norm $|W_t|_F$ grows at most sublinearly. This prevents weight-norm from stalling accumulation logarithmically, ensuring sustained exploration and search ($C_T = \Omega(\sqrt{T})$).

Furthermore, MuonGS requires no additional VRAM allocation compared to other heavy second-order optimizers. It executes purely in-place tensor operations with negligible step latency impact, making it compatible with standard CUDA cgroups.

However, there is a trade-off to consider. MuonGS assumes smooth scale-invariant layers, meaning that non-normalized components like token embeddings and language model output heads still need segregated coordinate-wise schedulers or traditional AdamW branches.

As we look towards 2026, it remains to be seen what foundation pretraining and long-horizon fine-tuning stacks teams will be using. Will you still default to AdamW, or will you be running Muon and SOAP in production? Your team's stack is requested below.