Getting the Source Right, Not Just the Fact: Source-Aware Verification for MCP Agents

Source-aware verification for Model Context Protocol (MCP) agents tackles the issue of factuality verification in a more subtle way than traditional methods. Most existing systems pool together all available evidence to assess claim support, but fail to identify which specific source backs each claim or whether the answer cites the correct source.

The ProvenanceGuard system developed in the latest paper aims to address this gap by providing source-aware factuality verification for MCP-based LLM agents. This verification layer operates after an agent generates an answer and runs separately from the agent, preserving the provenance or source identity throughout the pipeline.

The verification process involves several steps. First, the answer is broken down into specific claims. Then, the source most relevant to each claim is identified. Next, a check is performed to determine if that source indeed supports the claim. The source is then compared with the one named or implied in the answer. Finally, a per-claim source verdict is emitted, along with a global answer-level allow or block decision.

ProvenanceGuard applies this verification process to captured MCP traces, including tool outputs and source IDs, without retraining the agent. It can be implemented using local models, making it adaptable to various setups, including cloud services. The system was tested using answers from a medical agent that utilized patient records, research articles, and other tools, resulting in 281 real traces for study.

Human experts evaluated 361 claims from 40 answers not used in system development. ProvenanceGuard correctly identified 138 claims that should not have passed, while correctly allowing 67 claims deemed supported by experts. The system also identified the correct source for identified claims 86% of the time. Comparatively, other support checkers performed less effectively in catching unsupported claims while avoiding unnecessary blocks.