Knowledge Poisoning in RAG: Attacking AI Through Its Knowledge Base

Rijul is developing LiveReview, an AI code review tool that is aware of its blast radius. RAG, or Retrieval-Augmented Generation, allows an LLM to answer questions using information from a user's documents. When building RAG systems, it's crucial to consider security aspects. The model's output can be influenced by the knowledge base it retrieves information from.

If the documents contain malicious information, the model may incorporate that incorrect data into its responses, known as knowledge poisoning. There are several types of attacks that can lead to knowledge poisoning: direct injection, prompt injection, embedding hijacking, and gradual drift. Direct injection involves creating a false document that appears legitimate and gets treated as factual by the model.

Prompt injection hides an instruction within the data, which the model may interpret as something to follow, potentially resulting in unintended or harmful outputs. Embedding hijacking adds irrelevant text similar to the target query, which could be retrieved instead of the correct information, leading the model to produce incorrect answers.

Gradual drift is a more subtle attack where the attacker makes small, seemingly harmless changes over time, gradually altering the knowledge base toward incorrect information without any single edit clearly indicating the source of the problem.