{
  "id": 4479816,
  "title": "Adam and AdamW: The Optimizer That Made Modern LLM Training Possible",
  "url": "https://urgent.news/2026/08/30/adam-and-adamw-the-optimizer-that-made-modern-llm-training-possible",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-30T18:26:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/shrsv/adam-and-adamw-the-optimizer-that-made-modern-llm-training-possible-4f3o"
  },
  "original_language": "en",
  "account": "Adam and AdamW represent a powerful optimizer that has become the backbone of modern large language model (LLM) training. Named after the original papers by Diederik Kingma and Jimmy Ba, Adam was first introduced in December 2014, predating the Transformer architecture and the rise of modern LLMs.\n\nAt its core, Adam is an adaptive optimization algorithm that addresses the challenges of training neural networks with parameters of vastly different scales. This is a crucial problem, as different parameters can receive vastly different gradient signals during training. Some parameters may receive dense gradients every step, while others may only receive sparse or intermittent signals. Some directions in parameter space may be noisy, while others are highly consistent.\n\nAdam tackles this issue by maintaining two moving averages for each parameter: the first tracks the direction of recent gradients (momentum), and the second tracks the magnitude of those gradients (gradient scale). By combining these two pieces of information, Adam calculates an adaptive learning rate for each parameter, allowing it to move in the right direction while normalizing the step size based on how volatile the gradients have been.\n\nThe key idea behind Adam is to adapt the learning rate for each parameter based on its gradient history. This is achieved through two key components: a moving average of the gradients (m) and a moving average of the squared gradients (v). These values are updated at each training iteration using exponential moving averages, with hyperparameters beta1 and beta2 controlling the decay rates.\n\nTo ensure unbiased estimates of m and v, especially during the initial training stages, Adam incorporates bias correction terms. These terms adjust the moving averages based on the number of iterations, ensuring that the estimates accurately reflect the true gradient history.\n\nThe Adam optimization algorithm combines these components into a single update rule: theta - theta - alpha * m_hat / (sqrt(v_hat) + epsilon), where theta represents the parameters, alpha is the learning rate, m_hat is the bias-corrected moving average of gradients, v_hat is the bias-corrected moving average of squared gradients, and epsilon is a small constant to prevent division by zero.\n\nThe introduction of Adam and its successor AdamW in 2019 revolutionized the training of Transformers, which are the foundation of modern LLMs. AdamW specifically addresses the challenges of applying regularization in the context of adaptive optimization, making it the preferred choice for training Transformer models. By incorporating these optimizer techniques, researchers and practitioners can train LLMs that are capable of achieving impressive language understanding and generation capabilities, driving the current state of the art in natural language processing.",
  "summary": "Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product. Most people learn neural networks by staring at the model. Weights. Attention. MLPs. LayerNorm. Tokenizers. Context windows. But when you actually…",
  "key_points": [
    "Adam and AdamW are adaptive optimization algorithms for LLM training.",
    "Adam maintains two moving averages per parameter: gradient direction and magnitude.",
    "AdamW addresses regularization in adaptive optimization for Transformer models."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}