Is GEO only in our heads, or something real?
I was deep in thought about how much of a model hallucination is RLHF-based assumption in the absence of training data or context. Pretraining gives the model the ability to confabulate. Post-training often influences whether it chooses to confabulate rather than say "I don't know." A base language model is trained to predict plausible continuations. If the evidence needed to answer is absent…
GEO, or Generative Explainability, is a discipline that aims to understand and influence various factors that contribute to a model's output, particularly when it comes to hallucinations. Hallucinations occur when a model generates incorrect or implausible information, which can be attributed to various reasons during both pretraining and post-training.
Pretraining provides a base language model with the ability to generate plausible continuations when evidence is missing. This means that even without explicit guidance, a model can produce wrong information. TruthfulQA demonstrated that pre-RLHF models like GPT-3, GPT-2, and GPT-Neo/J can confidently reproduce misconceptions learned from text, and larger models are less truthful than smaller ones.
The broader perspective on hallucination involves five key components: incorrect pretraining data, imperfect retrieval from model weights, inference errors, confabulation under missing information, and post-training induced behavior. These factors can be influenced by the ecosystem in which the model operates, which includes elements such as system prompts, memory, retrieval, tools, planners, execution loops, verifiers, reward shaping, UI constraints, and external state.
Improving the public factual corpus and correcting authoritative third parties can help mitigate the issue of base models inventing facts. Enhancing crawlability, indexing, and freshness signals can address wrong or stale retrieval. Making facts explicit, atomic, unambiguous, and easy to extract can help prevent models from misreading correct evidence.
Designing MCP and API responses with precise tool descriptions, narrow responsibilities, and clear schema can mitigate the risk of wrong tools being selected. Publishing direct answers to likely ambiguous questions and explicit negative facts can prevent planners from inventing assumptions. Keeping the current authoritative state easily retrievable can help address incorrect memory.
Designing API and MCP responses with pagination, completeness, timestamps, status, errors, and coverage boundaries can help when partial tool results are treated as complete. Finally, attaching source URLs, IDs, timestamps, and claim provenance to every fact returned can help with provenance and overcome a bad verifier.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
