{
  "id": 17043756,
  "title": "Why MuonGS Beats AdamW and Muon in LLM Pretraining",
  "url": "https://urgent.news/2026/10/29/why-muongs-beats-adamw-and-muon-in-llm-pretraining",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-29T10:15:05.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/hordii_bohdan/why-muongs-beats-adamw-and-muon-in-llm-pretraining-4k9l"
  },
  "original_language": "en",
  "account": "MuonGS outperforms both AdamW and the standard Muon in Large Language Model (LLM) pretraining, according to recent research. This is due to the innovative Gram-Schmidt projection technique employed by MuonGS, which allows for direct mathematical control over weight exploration, unlike uncontrolled weight-norm growth in AdamW and vanilla Muon.\n\nThe technology enables a 15% reduction in token-equivalent compute on both 400 million dense and 1 billion parameter mixture-of-experts (MoE) architectures. Specifically, on these models, MuonGS achieved matched validation perplexity 15% faster than MuonH, which translates to hundreds of GPU cluster hours saved.\n\nA key aspect of MuonGS is the strict norm growth bounds it imposes on weight matrices. By removing the radial update component through Gram-Schmidt projection, the Frobenius norm $|W_t|_F$ grows at most sublinearly. This prevents weight-norm from stalling accumulation logarithmically, ensuring sustained exploration and search ($C_T = \\Omega(\\sqrt{T})$).\n\nFurthermore, MuonGS requires no additional VRAM allocation compared to other heavy second-order optimizers. It executes purely in-place tensor operations with negligible step latency impact, making it compatible with standard CUDA cgroups.\n\nHowever, there is a trade-off to consider. MuonGS assumes smooth scale-invariant layers, meaning that non-normalized components like token embeddings and language model output heads still need segregated coordinate-wise schedulers or traditional AdamW branches.\n\nAs we look towards 2026, it remains to be seen what foundation pretraining and long-horizon fine-tuning stacks teams will be using. Will you still default to AdamW, or will you be running Muon and SOAP in production? Your team's stack is requested below.",
  "summary": "Standard AdamW and vanilla Muon waste millions of GPU hours during LLM pretraining because uncontrolled weight-norm growth throttles the effective learning rate. We cannot keep burning H100 clusters on sub-optimal trajectory dynamics when Gram-Schmidt projection gives us direct mathematical control over weight exploration. Here is how MuonGS fundamentally shifts pretraining economics: 15%…",
  "key_points": [
    "MuonGS outperforms AdamW and standard Muon in LLM pretraining",
    "15% reduction in compute for 400M dense and 1B MoE models",
    "Gram-Schmidt projection controls weight exploration sublinearly"
  ],
  "editors_take": "Mathematical control over weight exploration through Gram-Schmidt projection allows researchers to reduce pretraining compute costs without increasing VRAM requirements or sacrificing model performance.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}