The Evolution of LLMs, Explained Through What We Train Them to Do

The development of large language models (LLMs) can be understood by examining the purpose for which they were trained at various stages. Initially, models like GPT-1 focused on predicting the next word in text, essentially learning the patterns of language. Subsequent iterations, such as GPT-2 and GPT-3, scaled up in size and began to exhibit abilities like summarization, translation, and answering questions without any additional training.

The breakthrough came with the introduction of post-training, where models were fine-tuned to be more helpful assistants. This entailed teaching them to understand and respond appropriately to user queries, as seen with the introduction of InstructGPT and the subsequent release of ChatGPT.

Following this, there was a shift towards reasoning models, which incorporated a two-step process: first, generating a chain of thought or hidden reasoning, and then producing a concise visible output. This approach allowed for more complex problem-solving and verification of answers, particularly in domains like mathematics and coding.

Models like Codex, o1, and DeepSeek-R1 showcased the power of reinforcement learning with verifiable rewards, enabling them to perform tasks step-by-step, check their work, and learn from their mistakes.

The final act in this evolution involves moving beyond pure text-based reasoning to incorporate actions in the real world. Agents equipped with reasoning capabilities can now search for information, run code, and interact with tools and data. This transition from text-based reasoning to action-based execution marks the next frontier for LLMs, with applications ranging from search engines to coding assistants and general-purpose agents.

The story of LLMs thus unfolds as a series of iterative improvements, each driven by a new objective or capability, culminating in models that can act autonomously and solve complex problems in the real world.