I Gave Claude Code a Brain. Then I Measured What It Costs to Ask for One.

In this report, I will present the findings from a thorough competitor comparison of memory systems, specifically focusing on the performance and cost of using a memory layer alongside an LLM (Language Model). To ensure a fair evaluation, I conducted experiments using a dataset with four arms, five corpus conditions, and graded every outcome by executing a checker against the code the agents actually wrote. No LLM judge was employed in this process.

The corpus consisted of 4,911 documents, with 4,902 adversarial distractors generated from the corpus's own vocabulary and the remaining documents serving as planted material for 36 tasks. In comparison to an earlier small feed, the hit@1 score dropped from 0.485 to 0.182, highlighting the challenges faced by memory systems in retrieving relevant information from a large, disorganized corpus.

The five conditions differed in what was planted within the corpus, with four of the five conditions serving as traps designed to test the memory system's ability to handle outdated, contradictory, inapplicable, or empty information. Only 37 percent of the cells contained clean, retrievable facts, leaving the remaining 63 percent filled with outdated, contradictory, inapplicable, or empty information.

The memory layer's primary function is to not make the situation worse in these cases, which is precisely what I aimed to measure.

To ensure that the comparison was fair, I built a control arm that received the same instruction but had no memory at all. The results showed that telling an agent to consult memory when there was no memory resulted in measurably worse performance, with 17 cells out of 358 representing a borderline significant difference. This finding underscores the importance of having a functional memory system when working with large, complex datasets.

The comparison revealed that MemPalace, a memory system I built, outperformed the instruction-only control in recall (RE-call) by 2 points overall, with a statistically significant gain of +3 on the bare agent baseline. However, when examining the individual conditions, it is clear that MemPalace only performs better when there is something to recall and performs slightly worse in conditions designed to punish retrieval.

This suggests that memory systems can be effective in certain scenarios but may not always provide a clear benefit, particularly in the presence of outdated or contradictory information.

In conclusion, this report highlights the complexities and challenges of using memory systems with LLMs. While memory systems can provide valuable benefits in certain situations, they also come with costs and limitations. The findings suggest that a more nuanced approach to evaluating memory systems, one that accounts for the quality and relevance of the information being retrieved, may be necessary to fully understand their effectiveness.