How I stop an LLM from hallucinating in production (RAG + entity-match + MCP)
How do you make an LLM answer questions about real-world entities — companies, people, records — without it confidently making things up? I build Verivello , a live AI agent that answers questions about any UK company from official registers (Companies House, HM Land Registry, FCA, The Gazette, sanctions lists). In that domain, a wrong director or ownership figure is worse than no answer at all.…
Ensuring an LLM doesn't confidently invent information when answering queries about real-world entities requires a carefully structured system design. In the case of Verivello, a live AI agent that provides UK company information from official registers, the key is a pipeline approach rather than just Retrieval Augmented Generation (RAG).
The first rule is that the LLM never recollects information from memory. Every fact in its answer must stem from a tool call, with the raw output passed through unchanged. The tool's job is to cite records, not to remember them. Tools return structured JSON records, which are then inserted verbatim into the context. The system prompt instructs the model to only draw from these provided records and to cite them. If the needed information isn't found, the model should not fabricate it.
The second crucial rule is entity-match verification. Even the best name-search APIs can return the wrong record. Simply passing the returned record into the context creates a potential for the model to confidently cite false information. Before allowing any record into the grounded context, it must be re-verified against the specific entity the user asked about.
This involves checking for exact identifier matches first, and if that fails, comparing a normalized version of the name and checking for matching incorporation years.
Any records that fail this verification are dropped and not shown to the model. The grounded context only contains records that have been proven to belong to the queried entity. If no verified records are found, the agent must honestly admit "I don't know" instead of risk providing a plausible but incorrect answer. This 'fail closed' approach is a fundamental product decision - a confident false answer can be far more costly than an honest admission of lack of knowledge.
By separating data sources into specialized Model Context Protocol (MCP) tool servers, the system achieves isolation, testability, and reuse. Each source is an independent server with a typed schema, allowing for thorough unit testing. The model requests only what it needs, while the tool layer handles fetching and verification. This clear boundary simplifies maintenance and enables easy integration across different clients.
In summary, grounding an LLM in production requires a pipeline that ensures verbatim tool output, thorough entity-match verification, and a fail-closed policy. Using MCP tool servers provides a clean, testable, and reusable architecture for integrating real-world data into LLM responses. This approach prevents hallucinations and ensures that a business's trust in AI is justified.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
