The Write Policy Is the Hard Part: Promotion Pipelines for Agent Memory

Writing a robust memory promotion pipeline for agent systems proves far more complex than simply retrieving relevant information from a vector database. While retrieval operations are relatively straightforward, the write-path poses numerous challenges that often go overlooked. These include deciding which observations deserve to be stored, sanitizing potentially sensitive data, and ensuring proper access control so agents can only write to authorized storage locations.

Failure modes in this area often stem from distributed system issues, default deny security policies, and uncontrolled growth of vector stores over time. The retrieval side of memory systems has matured considerably, with established techniques for embedding queries, ranking results, and feeding top-k answers back into context. However, the write-side remains fraught with unknowns.

Every observation an agent generates during a session can potentially be a candidate for long-term memory, yet only a fraction of these observations are truly valuable. A single session can generate hundreds of transient facts, ranging from the value of a variable to a command that failed. Storing all of these observations in a permanent store would quickly degrade into a disorganized "landfill" rather than a coherent memory bank.

The real question, therefore, is not how to store these observations, but rather what the write policy should be and how to enforce it securely. One initial approach is to write everything and filter out the noise later during the retrieval phase. This cheap and easy solution relies on similarity ranking to surface high-quality observations while hiding the irrelevant ones.

However, this approach fails due to the inability of embeddings to distinguish between signal and noise. A useful observation on one topic may receive the same ranking as a purely speculative guess made during the same session. Another simplistic method is to maintain a recency window, keeping only the last N observations and discarding the rest.

This approach also falls short, as it treats all recent data equally, regardless of its actual value or relevance. Both of these methods neglect the crucial step of making a decision about whether an observation should be written to the permanent store before it is even inserted. There is no proactive gating mechanism in place to prevent low-quality or duplicate data from contaminating the memory system.

The cleaner solution is to implement a promotion pipeline that acts as a gatekeeper for all incoming observations. Treat the long-term memory store as a production branch that requires validation before any commits can be accepted. Observations should be initially stored in a transient scratchpad that is scoped to each agent session and does not undergo any gating.

The pipeline consists of four stages: Capture the observation into the transient scratchpad. Score each candidate for importance and novelty. Validate and sanitize the observation by removing secrets, deduplicating, and enforcing schema checks. Promote the high-quality observations that pass all checks into the durable long-term storage system, preserving provenance information about how and when the data was inserted.

At the core of this system lies the gatekeeper function, which applies a threshold-based decision process to each incoming observation. The gatekeeper checks for certain criteria such as minimum length or known non-fact patterns. If an observation fails these initial cheap rejections, it is dropped. Next, the gatekeeper assigns an importance score to the observation on a scale from 0.0 to 1.0, with higher scores indicating more valuable information.

Additionally, novelty is measured by comparing the observation's embedding to those already present in the storage system. If the observation is a near-duplicate, it receives a low novelty score. The gatekeeper then checks whether the observation meets a minimum importance threshold or a minimum novelty threshold. Only observations that satisfy both criteria proceed to the next stage.

During the promotion stage, the observation undergoes sanitization to remove any sensitive information before it is written to the durable store. Two key parameters govern the behavior of the gatekeeper: the importance threshold and the novelty check. By adjusting these parameters, the system can be tuned to strike the right balance between strictness and inclusivity.

The importance threshold determines how stringent the promotion criteria must be, while the novelty check guards against the duplication of similar observations. The scoring function used to evaluate observations can employ a variety of lightweight heuristics combined with an optional LLM tiebreaker for borderline cases. Simple pattern matching techniques, such as searching for keywords related to fixes, decisions, root causes, or version numbers, can yield substantial results.

Concrete artifacts like commands, configuration settings, or specific commands are given higher importance scores. User-pinned observations, which are explicitly marked by the user as important, receive additional weight. Observations containing common status responses like "OK," "Done," or "Running" are penalized, as they provide little value to the memory system.

By implementing a structured promotion pipeline with a dedicated gatekeeper function, agent memory systems can effectively separate signal from noise, preserve only the most valuable information, and maintain a high-quality long-term memory store. This approach ensures that agents can rely on their memory systems to provide accurate and relevant information when needed, without the risk of being overwhelmed by irrelevant or duplicated data.