Adam and AdamW: The Optimizer That Made Modern LLM Training Possible

Adam and AdamW represent a powerful optimizer that has become the backbone of modern large language model (LLM) training. Named after the original papers by Diederik Kingma and Jimmy Ba, Adam was first introduced in December 2014, predating the Transformer architecture and the rise of modern LLMs.

At its core, Adam is an adaptive optimization algorithm that addresses the challenges of training neural networks with parameters of vastly different scales. This is a crucial problem, as different parameters can receive vastly different gradient signals during training. Some parameters may receive dense gradients every step, while others may only receive sparse or intermittent signals. Some directions in parameter space may be noisy, while others are highly consistent.

Adam tackles this issue by maintaining two moving averages for each parameter: the first tracks the direction of recent gradients (momentum), and the second tracks the magnitude of those gradients (gradient scale). By combining these two pieces of information, Adam calculates an adaptive learning rate for each parameter, allowing it to move in the right direction while normalizing the step size based on how volatile the gradients have been.

The key idea behind Adam is to adapt the learning rate for each parameter based on its gradient history. This is achieved through two key components: a moving average of the gradients (m) and a moving average of the squared gradients (v). These values are updated at each training iteration using exponential moving averages, with hyperparameters beta1 and beta2 controlling the decay rates.

To ensure unbiased estimates of m and v, especially during the initial training stages, Adam incorporates bias correction terms. These terms adjust the moving averages based on the number of iterations, ensuring that the estimates accurately reflect the true gradient history.

The Adam optimization algorithm combines these components into a single update rule: theta - theta - alpha * m_hat / (sqrt(v_hat) + epsilon), where theta represents the parameters, alpha is the learning rate, m_hat is the bias-corrected moving average of gradients, v_hat is the bias-corrected moving average of squared gradients, and epsilon is a small constant to prevent division by zero.

The introduction of Adam and its successor AdamW in 2019 revolutionized the training of Transformers, which are the foundation of modern LLMs. AdamW specifically addresses the challenges of applying regularization in the context of adaptive optimization, making it the preferred choice for training Transformer models. By incorporating these optimizer techniques, researchers and practitioners can train LLMs that are capable of achieving impressive language understanding and generation capabilities, driving the current state of the art in natural language processing.