A memory benchmark draft with 17 axes and no headline number: is it worth running?
I work at a company that builds memory infrastructure, so I'm biased: we compete in the thing this benchmark measures. We wrote a long-term memory benchmark called glasshouse, and I'd like to know if you think it is worth running on your own system. Why we made it Memory systems report their own numbers, and the numbers don't agree. The case that pushed us to build this: Zep reported 84% on…
Memory systems report their own numbers, and discrepancies arise - glasshouse is a long-term memory benchmark created to tackle this issue. The benchmark features 2,847 questions over one conversation of 1.97 million tokens, in 10 languages, with 50 photographs. There are 17 axes, each reported separately, instead of a single headline number because a system may excel in one axis but fail in another.
The standard run is full (103,572 turns, 1,547 questions), while a smaller core tier (1,882 turns) serves as the control. Axes in the benchmark include STALE, CONTRADICT, RECONCILE, and FALSE_MEMORY.
STALE assesses a fact's change, calculating full credit for finding the new value, half for confidently repeating the old value, and zero for confidently asserting the old value. CONTRADICT requires correct identification of two incompatible statements from weeks apart, with full credit for correct identification and zero for confidently picking one.
RECONCILE calls for explaining two statements that seem incompatible but hold under different conditions, or picking one and calling the other wrong. FALSE_MEMORY challenges the assertion of something that was never said, awarding full credit for correct identification and zero for confidently stating the incorrect value.
The benchmark also includes cross-language questions, with 800 questions that store in one language and ask in another, across all 10 languages. Partial credit rules apply to incomplete answers and visible gaps, with half credit for a visible gap and zero for a wrong value stated as current if it scores more than 20%.
The benchmark is currently a draft and not yet released. It requires a custom ingest step and an OpenRouter scorer to function, which involves costs. The benchmark has not undergone testing in both answer and memory modes, and the synthetic conversation used for testing does not represent a real user's life.
Despite these shortcomings, the creator of glasshouse invites others to run the benchmark on their own memory systems. Questions arise regarding missing components, which axes appear poorly designed or unfair, and whether one should trust a benchmark run by a company that also sells memory products. For more details, visit the GitHub repository at github.com/wontopos/glasshouse.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
