How we tested AI agent memory on 30,000 fake keys
13 November 2026 · PADAM NOCTURNE 0.4.17–0.4.18 An agent sees API keys, database passwords and wallet keys while it works. Whatever it sees can end up in its memory, and memory is read back into prompts, logs and exports. A memory layer has to drop the secret before anything is stored. This note explains how we check that, and what the check does not cover. Why fake keys We were about to rotate…
On 13 November 2026, researchers tested how AI agents handle memory of various secret keys. The test involved generating 30,000 fake keys for different services, using ten formats for each type of key. These fake keys were then inserted into various types of sentences that AI agents and humans might generate, such as NAME=value, export NAME=value, Markdown rows, JSON, .env fragments, plain English and Russian text, and more.
After running the test 150 times, 12 phases of testing were conducted, comparing memory performance before and after masking. The results showed that version 0.4.17 of the memory layer had no leaks compared to version 0.4.16, which had 1,927 out of 27,000 phrases leaking secret data. Version 0.4.18 maintained zero leaks, as well as 0 of 12 fake Solana wallet keys, which required a separate test due to the nature of their format.
The researchers also examined the possibility of false alarms, where the masking process might be too aggressive and hinder the AI agent's functionality. In this case, version 0.4.17 produced one false alarm, where a real API key from a public dataset was mistakenly masked. In version 0.4.18, 11 false alarms were identified, including names instead of actual values, which were not masked. However, these false positives did not impact the overall performance of the memory layer.
The hosted version of the memory layer, AIfa Memory, was also tested using the same rules and found to have zero differences from the local implementation, with 0 of 30,000 fake keys leaking any secret data. The tests were conducted using a fixed-seed version of the checks in various CI pipelines.
The main takeaway from this testing is that the memory layer effectively masks secret keys from being stored, while avoiding false alarms that could hinder the AI agent's functionality. However, care should be taken when dealing with secrets that don't follow known formats, as they might still pass the masking test. The full implementation of these tests can be found in the provided GitHub repository.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
