Urgent.News

What's breaking now, across hundreds of outlets.

Tech

Deleted Documents Still Answer

Originally published at Trida AI . You delete the file. The assistant keeps quoting it. That is not a model failure, and it is not a retrieval failure either. Retrieval did what it was asked to do: it searched an index. The index still held the document, because nobody told the index the document was gone. A retrieval-augmented generation (RAG) system has at least two copies of every document:…

In the world of artificial intelligence, retrieval-augmented generation (RAG) systems are innovative tools that search an index to provide answers. However, these systems come with a significant drawback: deleted documents can still be quoted, even if they have been removed from the source. This occurs because RAG systems maintain two copies of every document – the source, where edits and deletions are made, and the index, where chunks and embeddings are read at query time.

Ensuring these two copies remain in agreement is a separate piece of engineering that is often overlooked in initial builds. Consequently, deletions can be challenging to handle, as a sync process that relies on visibility of the data will simply pass over what is missing. This asymmetry is well-documented in platform guidance, such as Microsoft's Azure AI Search indexers, which highlight that deletion detection is not automatic.

To address this issue, a soft-delete strategy must be implemented from the very first indexer run. If deletions are not detected before the removal happens, the index will have no record of the missing document, and a new index with the policy in place from the start can clean up these discrepancies. However, this is a recovery operation rather than a sync strategy, and teams often discover how outdated their index is only after a user raises a question.

Edits, permissions changes, and document movements can also pose problems. When a warranty policy document is edited, with the twenty-four-month warranty period removed, the document splits into ten chunks instead of twelve. If the ingestion process adds the new chunks by position without retiring the old ones, chunks eleven and twelve remain searchable, appearing fluent and on-topic, but superseded.

This issue extends to permission changes as well, as a document moving from a public folder to a restricted one is effectively treated as deleted for some readers. To mitigate these issues, it is crucial to treat a document as a unit, not as loose chunks. Every chunk needs to know which document it came from, its version, and what happened to the document since.

With this identity in place, an edit becomes a replacement process, where new version chunks replace the previous version chunks, and the swap is the smallest unit the sync can complete. If the swap fails halfway, the document should either be unavailable or revert to its old self, never as a combination of both. Managed services like Amazon Bedrock sync handle deletions incrementally, processing only documents added, modified, or deleted since the last sync, thereby re-ingesting changed documents and removing deleted ones from the vector store.

However, this approach sacrifices some control over the replacement process. Before adopting any stack, it is essential to understand how a deletion reaches the index and how long it takes, and to test this unfriendly aspect the hard way by deleting a real document from a staging corpus and observing the responses.

Rewritten by urgent.news; may contain errors — check the original.

Read the original at dev.to →

Report a problem with this story

More in Tech

What Checking Every Scheduled Run, Not Just the Latest, Actually Caught

42 scheduled runs so far, 5 of them failed outright, and not one of those 5 ever surfaced on its own. No email, no notification, no red mark anywhere I'd naturally look.

  • 42 scheduled runs conducted, but 5 failed without alert
  • Failures went undetected until routine manual checks
  • Status check with Get-ScheduledTaskInfo caught issues

I Got Tired of Noisy Terminal Output, So I Built a Tiny CLI in C++

I've been working with different languages and frameworks for years, and there's one thing that has always bothered me a little: how much unnecessary output our development tools produce.

  • Muffle is a lightweight CLI written in C++17 to filter command output
  • Users can run muffle -- npm test to get concise test results
  • Muffle logs complete output to files and preserves original exit codes

More from Saturday 21 November →