Agent Memory: Short-Term, Long-Term and What's Just a Database

A model possesses no inherent memory. Instead, it operates using a context window that is repeatedly sent in full on each request and subsequently erased. The term "memory" associated with agent functions is merely a decision-making process about what to include in this window and where the rest of the information is stored. The use of metaphors like "short-term" and "long-term" from cognitive psychology is misleading, as there is no automatic transfer of data between storage locations or a forgetting curve.

When assessing agent memory, it is advisable to consider three technical questions: when the data is written, when it is retrieved, and what determines its removal. Anything that claims memory status without answering these questions is essentially a vector database with aspirations beyond its capabilities.

There are three distinct memory patterns:

1. The Transcript: This is the complete history of every action taken by the model, written automatically and read in its entirety with each request. It expires when it reaches the context limit through a compaction process. This memory type is the most expensive, as its cost increases quadratically with the number of steps.

2. The Fact Store: This type of memory is written either manually by the agent or through a post-run extraction process. It is read by a retrieval system, typically using a top-k approach. The data expires based on various criteria such as time-to-live (TTL), supersession, or may persist indefinitely. It is commonly referred to as long-term memory.

3. External State: This refers to the actual changes made by the agent, such as editing a file, updating a database, or modifying a task list. It is read by the agent when needed and expires when the related information is altered. Unlike the previous two types, this is not considered memory, as it is reliable and requires no embedding model or retrieval tuning.

Effective compaction policies should be established to determine what information should be dropped first. Prioritize removing old tool outputs, superseded results, and middle reasoning steps. Maintain the system prompt, original task, and the final two steps, as dropping these can lead to the agent finishing a task adjacent to the one initially requested.

For a fact store implementation, consider creating a table with columns that facilitate maintainability. Incorporate provenance, supersession, and self-contained sentences. Facts should be limited to five per request to avoid unnecessary costs. Additionally, implement safeguards to prevent the agent from writing facts about unverified content, maintaining source information, and regularly reviewing and updating the memory store.