When Your AI Confidently Replies to Emails It Shouldn't Touch
A technical investigation into a RAG system that can't tell when it's out of its depth Setup InboxSync is a personal project I built: a multi-account email aggregation API that uses a RAG (Retrieval-Augmented Generation) pipeline to suggest replies. The system indexes emails via IMAP, categorizes them with GPT-4o-mini, and for actionable emails retrieves semantically similar training examples…
A technical investigation reveals that a RAG (Retrieval-Augmented Generation) system designed to suggest email replies is failing to discern when its knowledge is insufficient for the task at hand. The system, built as a personal project to help salespeople respond to leads more efficiently, uses a database of training examples to generate contextually relevant reply suggestions. However, a series of adversarial tests exposing the system's limitations has uncovered a critical flaw.
In each test, the system returned a confidence score of 0.85 for its generated responses, regardless of whether the source material was spam, an auto-reply, a refusal, a legal request, or a question requiring multi-topic understanding. For example, when given a spam query offering a 50% discount, the system replied politely with "Thank you for the exciting offer! I appreciate the heads-up about the Black Friday sale. I'll definitely take a look." Despite the obvious spam content, the system's confidence remained high.
The investigation also revealed that the system simply ignores the relevance score of the retrieved training examples, using a fixed confidence value of 0.85 in all cases. This approach disregards the actual similarity scores computed by the system, which ranged from 0.13 to 0.54 for the various tests. The lack of a relevance threshold means the system treats all retrieved neighbors equally, even when they are completely unrelated to the query.
Moreover, there is no mechanism in place to prevent the system from auto-replying to certain types of queries, such as auto-reply to auto-reply situations. The system's reliance on a hardcoded confidence value, rather than a dynamic relevance assessment, leads to potentially unsafe and unvetted replies being suggested, even for queries it should be unable to handle.
This failure highlights the importance of implementing proper relevance gating and confidence thresholds in RAG systems to prevent inappropriate or harmful responses.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
