Agent memory and questions to ask
Tell a coding agent that your project runs tests with npm test . Start a new session and ask it how to run the tests. It answers correctly. Good first result. Now tell it the project has migrated to pnpm. Start another session. Ask the same question. Does it give you the new command? Does it offer both? Can it explain which instruction is current? That second conversation is where I'd start…
To evaluate an agent's memory, start by testing its ability to recall specific instructions based on historical notes. Create two fictional projects, Atlas and Beacon, and set up test environments for each. For Atlas, record the initial fact that tests run with npm test. Then, introduce a correction stating that Atlas now uses pnpm for testing. Repeat the same process for Beacon, noting that its test command remains unchanged.
Begin with a fresh session and ask how to run tests for Atlas on September 12. The correct answer should be pnpm test, reflecting the latest instruction. Then inquire about how Atlas tests were run on September 1, expecting the response to be npm test, identified as the historical command. For Beacon, ask how to run its tests on September 12, expecting the answer to still be npm test.
Lastly, ask why Atlas migrated to pnpm, and verify that it correctly identifies the September 10 maintainer note as the source of the current command.
Break down your evaluation into key aspects: retaining corrections, preserving historical information, distinguishing between different projects, recognizing when information is missing, and tracing answers back to their sources. If the agent fails to provide the correct response at any point, investigate the specific stage where the memory system faltered. For instance, if it returns the September 1 note instead of the corrected one, examine the retrieval path and the logic used to rank and select answers.
Repeat the process by importing an old Atlas README dated September 1, after applying the correction. The agent should still return pnpm test, demonstrating that the retrieval process is not influenced by the order in which the notes were stored. This exercise helps you determine whether your memory system can accurately maintain and retrieve different versions of instructions over time.
Consider how your specific memory implementation—whether it uses a graph, vector, relational database, or a knowledge graph—stores corrections and retrieves the appropriate version of information. Each system has its own method for representing changes, such as attaching a supersession relationship to notes or maintaining separate current-state records. Choose the architecture that best fits your application's needs, ensuring it can answer questions about both the present and the past accurately.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
