Urgent.News

What's breaking now, across thousands of outlets.

Tech

How to Make Enterprise RAG Systems More Reliable

Enterprise RAG needs more than accurate retrieval. Explore four critical gaps in access control, data freshness, continuous evaluation, and cost.

How to Make Enterprise RAG Systems More Reliable

Retrieval-augmented generation (RAG) is based on a straightforward concept: instead of language models generating answers from memory, they first search for relevant documents, contextualize the evidence, and then generate the response. This model, formalized by Lewis and colleagues in 2020, transitioned from research architecture to default enterprise design at an unprecedented pace.

Despite its widespread adoption, many enterprise RAG systems fail to meet the expectations of their users. Upon examining the research literature on RAG architecture as implemented in companies, it becomes clear that the model itself is rarely the bottleneck; rather, the collection of documents is the issue. Enterprise RAG systems require a more reliable, up-to-date, and secure collection of documents, as well as transparent accounts of the costs associated with each answer.

The first major gap in enterprise RAG systems lies in permissions. Most RAG systems retrieve passages without considering who is authorized to view them. As a result, retrieving passages can violate access controls, leading to inconsistent evidence based on different employees' queries. Enforcing per-document permissions during retrieval time is a solution, but it is rare in the research literature.

Post-retrieval filtering, which fetches documents first and then filters them, is inadequate as it leaks ranking signals, wastes computational resources, and introduces the risk of a failure mode that could result in a security breach. Additionally, retrieved documents are not just confidential or public; they are untrusted input, making them vulnerable to indirect prompt injection attacks.

Until permission-aware retrieval becomes a standard and evaluated property of the index, enterprise RAG deployments are essentially assembling a security control from untested parts.

The second gap concerns freshness. Companies constantly update their documents, such as prices, policies, and product details. However, many RAG systems are still designed as if the corpus remains static. Consequently, these systems provide answers based on outdated information, which is far worse than admitting ignorance. Research on FreshLLMs and other related studies demonstrate that models struggle with fast-changing knowledge, and a stale index leads to wrong answers with full confidence.

Although there are techniques to update vector indexes, such as upserts, change-data-capture pipelines, and incremental re-embedding, there is no universally adopted reference architecture that combines these pieces. Furthermore, benchmarks measuring RAG systems against continually changing corpora are scarce, leaving freshness as an afterthought that quietly corrupts the entire system.

To address this issue, service-level objectives need to be established for freshness, similar to latency metrics.

The third significant gap revolves around evaluation. RAG systems have far more evaluation instruments than most other applied AI techniques. However, these tools are rarely used in practice. The standard approach involves evaluating a fixed system against a fixed test set at launch. In most deployments, I observe no continuous evaluation of the system's faithfulness and context precision as the corpus evolves.

Tools like TruLens, LangSmith, and Arize Phoenix can monitor live application traces, online evaluation, and embedding drift, respectively. Nevertheless, there is no standard protocol for continuous evaluation, and no shared benchmark that links a system's scores to the state of its corpus. A study by Barnett and colleagues highlights seven failure points from real-world deployments, including missing content, missed top-ranked documents, answers present in the evidence but never extracted, and wrong specificity.

These failures emphasize the need for continuous evaluation practices. Without a universal standard for evaluating RAG systems in production, enterprises are unable to adapt their models to the evolving knowledge landscape, leading to compromised performance and reliability.

Written by urgent.news from HackerNoon's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at hackernoon.com →

More in Tech

More from Sunday 15 November →