Day 45: Emergent Abilities in LLMs

Emergent abilities refer to skills or behaviors that emerge in a system once it becomes complex enough, even though these abilities weren't directly programmed in. To illustrate this concept, consider water: a single hydrogen or oxygen atom doesn't possess the property of "wetness," but when billions of them combine to form water, the ability to "wet" emerges. This phenomenon is similar in large language models (LLMs), where new skills suddenly appear once a model crosses a certain size or training level.

In the development of LLMs, researchers typically begin with small models having millions of parameters. At this scale, the models can perform basic tasks such as autocomplete or echo simple patterns they've seen before. However, as the models grow—increasing training data, adding more parameters, and requiring longer learning times—they don't just improve incrementally. Sometimes, when the model size crosses a certain threshold, entirely new abilities emerge without any explicit instructions from engineers.

For instance, a model with 100 million parameters might struggle with a math word problem. Yet, a model with 10 billion parameters can suddenly solve many of those problems. This isn't unique to AI; in nature, a colony of ants guided by thousands of individual ants following simple rules can display surprising group behaviors like bridge building or food storage. No single ant knows the full plan, but the collective behavior appears as the system grows.

Researchers have observed specific skills popping up in LLMs only after they become large enough. Arithmetic is one such ability: small models cannot reliably add two numbers stated in a sentence, but larger models can solve two- and three-digit addition problems. Translation abilities also emerge in larger models, allowing them to translate between language pairs with little to no direct training data.

Code generation is another area where larger models excel, producing correct, working code from natural-language instructions.

In-context learning is another example, where large LLMs can extend patterns and generalize given a handful of examples (a method called "few-shot prompting"). Smaller models, however, cannot perform this task. The evidence of arithmetic emerged when a small language model was asked to add 17 and 35, returning an ambiguous response. In contrast, a larger model, once past a certain size, suddenly answers correctly: 17 + 35 = 52.

The emergence of these abilities is not because someone added a math plugin to the model. The skill simply emerges. The reason behind the sudden appearance of these abilities at certain scales is not fully understood, but it highlights the limits of our current understanding of how LLMs function. As LLMs continue to grow in size and data, we can expect not just better performance but the appearance of whole new abilities, sometimes seemingly out of nowhere.