Stop Thinking of LLMs as Next-Token Predictors

The phrase "LLMs are next-token predictors" is not entirely inaccurate, but it fails to capture the full picture. This simplified view is grounded in the fact that transformer-based language models generate tokens in a sequential manner, predicting the next token based on the previous ones. This process aligns with the concept of a next-token predictor.

During the initial pre-training phase, the model learns to increase the likelihood of the actual subsequent token being sampled. It achieves this through intricate loss functions, gradients, and parameter updates, though the focus here is solely on the outcome: the actual next token becomes more probable.

However, it's essential to note that modern LLMs are not merely base models. They undergo a crucial post-training phase, often involving reinforcement learning with verified rewards (RLVR). During this stage, the model is no longer confined to learning from existing sequences in the training data. Instead, it explores by generating new sequences and gauging their outcomes.

Conceptually, despite the similar underlying mechanism of emitting tokens one at a time, post-trained LLMs have shifted their focus. During pre-training, the model learns to predict actual next tokens due to their presence in the training data. In contrast, during RLVR, it makes explored next tokens more probable because the sequences involving them have earned high rewards.

While the base LLM still functions as a next-token predictor, its post-training phase transforms its learning process. It no longer solely relies on predicting existing text; it also learns from novel sequences produced through its exploration.

This distinction becomes clearer when comparing LLMs to chess engines. A chess engine trained on a large database of grandmaster games learns patterns in their responses to different board positions. Given a new position, it predicts the move a grandmaster would likely play next, embodying a next-move predictor. On the other hand, an idealized chess engine that has explored every possible game knows the probability of winning from every position.

Given a position, it selects the move leading to the highest probability of winning. Unlike the first system, this second chess engine is not merely trained on existing games; it learns from games generated by its own exploration.

The post-training techniques like reinforcement learning from human feedback (RLHF) further illustrate this point. RLHF steers the model away from merely imitating its pre-training data and towards simulating a helpful assistant. RLVR extends this learning even further, enabling an LLM to explore and learn from ideas never encountered in its training data.

Therefore, referring to LLMs as "next-token predictors" is an incomplete mental model. While accurate in describing the mechanism of generating tokens sequentially, it overlooks what this mechanism encodes. This encoding includes simulating a helpful assistant and discovering knowledge through exploration, both of which can be encoded within the same next-token loop.