Hallucination Is a Design Problem, Not a Model Problem

One common question in procurement conversations about hallucination is which model produces the least hallucination. This question is misguided because it treats a structural issue as a vendor feature, leading to systems that fail in ways nobody anticipated. The more pertinent question is what happens when a system confidently produces an incorrect answer.

If the response is “someone notices,” the mechanism should be described. If the response is “it flows into the next step,” then the problem is not with the model but with the design, and no version number can fix it.

Improved models can reduce the gap between accurate and hallucinated answers, but they cannot eliminate it entirely. For instance, OpenAI's research reveals that models hallucinate because standard training and evaluation reward guessing over admitting uncertainty. A confident guess may score, while saying "I do not know" guarantees a zero score.

Consequently, some questions cannot be answered from patterns in text, and accuracy will never reach 100 percent. However, a model can abstain when it is uncertain, which is a fix to how models are trained and scored.

The system surrounding the model plays a crucial role in mitigating the issue. Providing the model with actual documents rather than relying on training data can help. Grounding a model by retrieving the correct document creates a different failure mode that is harder to catch. Measuring retrieval quality separately from answer quality is essential since an answer can be faithful to a wrongly retrieved passage.

Showing the retrieved passage itself, not just a link, allows reviewers to see how the answer was constructed. Additionally, deciding on an acceptable abstention rate and measuring confident errors separately from abstentions is crucial. A system that abstains 15 percent of the time and is dependable on the rest is usually more valuable than one that answers everything and is wrong 5 percent of the time without indicating which 5 percent.

Verification is another important step. For claims that can be checked, verify them in code before anyone sees them. If a model states a balance, read the balance; if it names a contract clause, confirm the clause exists; and if it computes a figure, recompute it. This deterministic approach converts probabilistic claims into deterministic ones at the point where it matters.

However, this method only works when ground truth is reachable. For applications like summarization, drafting, and judgment, there is nothing to check against, and these should be handled with extra caution. Lastly, irreversible actions, such as moving money or assets, or compliance obligations, should be handled with deterministic code, while the model should prepare input for a person or rule engine to act on.