Knowledge Base For AI Agents: What It Must Do

A knowledge base built for AI agents behaves quite differently from those designed for human use. When interacting with an agent, users issue a query, receive passages, and receive an answer directly from the provided content. In contrast, people navigating a knowledge base for themselves engage with the content, skim, and make decisions based on the presented information. This fundamental difference creates a set of requirements for a knowledge base to serve AI agents effectively.

Firstly, the retrieval mechanism should be live, meaning the information should be current and not a stale export. Agents do not have the ability to discern between outdated and up-to-date information like humans do; they take the passages as truth, making a wrong answer glaringly obvious. Secondly, every answer must be properly cited to clearly indicate the source of the information.

This allows for traceability if the answer is incorrect. Thirdly, the system should have a clear failure mode, allowing it to acknowledge when it does not have an answer. An agent that generates an answer in such situations is less reliable than no agent at all, as the lack of response is invisible to the user. Fourthly, permissions should be tailored to the individual asking the question rather than a single service account with access to everything.

This approach helps prevent accidental data leaks. Lastly, a standard connection interface, such as the Model Context Protocol, should be used to provide a consistent read interface for any compatible assistant, eliminating the need for bespoke integrations for each tool.

Testing a knowledge base for AI agents can be done in a single afternoon. Collect 20 recent questions asked by your team and find the relevant documents that should contain the answers. Run all questions through the knowledge base, and score the answers based on correctness, citation to the right document, and honesty when the answer is not present.

It's essential to include questions with no documented answers to assess how the system handles such scenarios. A tool that answers 15 out of 20 questions correctly with proper citations, and declines the remaining 5 questions cleanly, is considered functional. Conversely, a tool that answers all 20 questions, even those without sources, demonstrates a lack of important functionality.

This testing process is quick and provides a clear indication of the knowledge base's performance in supporting AI agents.