The Polaroid Trap: Why Vector Search Fails at Multi-Hop Enterprise Reasoning

The article argues that there are two approaches to multi-hop questions in enterprise RAG, each with its own limitations. First, using plain vector retrieval works until the answer depends on a middle fact that doesn't sound like the question. Second, adding a knowledge graph allows following the middle fact, but this comes with extraction and entity resolution costs, and the graph only returns stored information.

Vector search is effective for single-document lookups but struggles to follow links between fragments, similar to how foreign keys tie together rows. For instance, in a scenario where CHIP-402 is part of Engine-B, which is installed in Titan-EV, and Titan-EV is under contract LON-FLEET-26, a vector search may return the first and third facts but miss the middle one when asked about the chip's impact on contract penalties.

The proposed solution is a hybrid design that keeps vector search while extracting entities and relationships into a property graph. A dependency question identifies the root entity, runs a bounded traversal, and incorporates the relevant path into the model's prompt. However, this approach has limitations: it requires hard hop limits (2 or 3), and extraction errors or stale records can affect the path.

Before implementing a graph, a test set of 20 questions where answers depend on facts from multiple documents should be built and scored alongside single-fact questions. Single-fact questions should perform well, while multi-hop questions should be evaluated against the retrieval structure. Other techniques, such as query decomposition or iterative retrieval, can sometimes fix multi-hop cases without using a graph.

If needed, a bounded graph can be piloted in a high-risk domain with dense dependencies, but an enterprise-wide graph should not be attempted all at once.