AI agent memory vs RAG — what's the difference?

Most teams using Large Language Models (LLMs) implement two patterns within the same codebase: Retrieval-Augmented Generation (RAG) for finding information in a corpus, and a custom memory system for retaining agent interactions or user input. These two systems are often mixed up, leading to wasted engineering time. RAG fetches content the agent doesn't already possess, while memory retrieves context that the agent has previously interacted with.

They both utilize a vector store but serve distinct purposes, store unique data formats, and have varying correctness needs. The two systems share a common foundation: an embedding store, typically pgvector or a specialized vector database, and use approximate nearest neighbor search during retrieval. This is where the similarity ends.

RAG stores document fragments (markdown pages, PDFs, support articles) for retrieval, which return the most semantically similar chunks to a user's query. These chunks are then incorporated into the prompt for context. Memory stores structured records of events, including raw conversation turns, tool calls, and decisions, as well as compiled memories derived from these episodes, complete with provenance back to the source material.

Retrieval from memory returns the most pertinent prior context for a given subject, ranked by factors such as recency, relevance, validity, and similarity. If one only needs to ground answers in static documentation, RAG is sufficient. However, if an agent needs to remember user interactions across sessions, memory is necessary.

RAG was designed for answering questions like "What does our documentation say about X?" and not "What did this user tell us last month?" When teams attempt to use RAG for memory purposes, three issues arise: similarity-based embeddings may not be suitable for decision-making, memory ranking requires additional criteria like kind priority, temporal validity, and explicit provenance, and RAG does not include a compaction mechanism to reduce historical data into manageable facts.

Additionally, RAG lacks a mechanism to mark outdated information as invalid. These limitations can be addressed within application code but result in a memory layer with essentially the same functionality as the retrieval system.

Memory infrastructure goes beyond retrieval by incorporating three key elements on top of the vector store: compilation of raw episodes into typed memories with confidence scores and validity windows, deterministic ranking that combines similarity with kind priority, recency, temporal validity, and token budget considerations, and provenance, which tracks the episodes from which compiled memories are derived, allowing for traceability back to the source events.

These features are not inherent to RAG and are what distinguish memory infrastructure from retrieval over chat logs.

Production agents generally require both RAG and an agent memory system. The grounding corpus, including documentation and knowledge bases, resides within RAG. The user, account, and project context are stored in memory. Attempting to use either pattern for the other's intended purpose is a common architectural mistake, and Statewave aims to prevent this by providing a dedicated memory layer.

Statewave handles the "who you're talking to" aspect, while a traditional RAG framework manages the "what the knowledge base says" component. The architecture documentation provides more details on ranking signals and the difference between compilation and retrieval, while a five-minute Docker Compose guide is available for testing Statewave alongside existing RAG setups.