Why Models Make Things Up

Models have a tendency to fabricate information when faced with rare or complex queries. This occurs because language models are fundamentally designed to predict the next plausible word, rather than verifying the accuracy of the facts they generate. During the training process, models learn statistical patterns from vast amounts of data, but they lack an inherent understanding of truth or reality.

Consequently, when a model encounters a fact that appears infrequently or is inconsistent across sources, it may confidently produce an incorrect answer, as it has only seen that fact once in its training data. The phenomenon of hallucination is not a separate failure mode but an inherent aspect of how these models operate. Even highly trained and calibrated models must occasionally generate fabricated information due to the statistical nature of their training objective.

The training data serves as the sole source of information for a language model, and if a fact is scarce or contradictory, the model struggles to learn it accurately. This limitation is not due to data quality issues but is a fundamental aspect of statistical learning. Theoretical research supports this concept, demonstrating that calibrated models must hallucinate on rare facts appearing only once during training, with the hallucination rate directly tied to the frequency of such singleton facts.

Due to the nature of training, language models are prone to a phenomenon called exposure bias. During training, they always have access to the correct previous words, allowing them to generate accurate continuations. However, during inference, the model must rely on its own predictions as context, leading to a mismatch between what it was trained on and what it encounters in real-time.

This discrepancy can cause errors to compound over time, resulting in increasingly inaccurate outputs. For instance, a language model drafting a professional email may start with a slightly casual greeting, then gradually deviate further into an unprofessional tone as its predictions diverge from the original intent.