{
  "id": 14929684,
  "title": "A memory benchmark draft with 17 axes and no headline number: is it worth running?",
  "url": "https://urgent.news/2026/10/16/a-memory-benchmark-draft-with-17-axes-and-no-headline-number-is-it",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-16T10:31:41.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/woochan/a-memory-benchmark-draft-with-17-axes-and-no-headline-number-is-it-worth-running-h6k"
  },
  "original_language": "en",
  "account": "Memory systems report their own numbers, and discrepancies arise - glasshouse is a long-term memory benchmark created to tackle this issue. The benchmark features 2,847 questions over one conversation of 1.97 million tokens, in 10 languages, with 50 photographs. There are 17 axes, each reported separately, instead of a single headline number because a system may excel in one axis but fail in another. The standard run is full (103,572 turns, 1,547 questions), while a smaller core tier (1,882 turns) serves as the control. Axes in the benchmark include STALE, CONTRADICT, RECONCILE, and FALSE_MEMORY.\n\nSTALE assesses a fact's change, calculating full credit for finding the new value, half for confidently repeating the old value, and zero for confidently asserting the old value. CONTRADICT requires correct identification of two incompatible statements from weeks apart, with full credit for correct identification and zero for confidently picking one. RECONCILE calls for explaining two statements that seem incompatible but hold under different conditions, or picking one and calling the other wrong. FALSE_MEMORY challenges the assertion of something that was never said, awarding full credit for correct identification and zero for confidently stating the incorrect value.\n\nThe benchmark also includes cross-language questions, with 800 questions that store in one language and ask in another, across all 10 languages. Partial credit rules apply to incomplete answers and visible gaps, with half credit for a visible gap and zero for a wrong value stated as current if it scores more than 20%.\n\nThe benchmark is currently a draft and not yet released. It requires a custom ingest step and an OpenRouter scorer to function, which involves costs. The benchmark has not undergone testing in both answer and memory modes, and the synthetic conversation used for testing does not represent a real user's life.\n\nDespite these shortcomings, the creator of glasshouse invites others to run the benchmark on their own memory systems. Questions arise regarding missing components, which axes appear poorly designed or unfair, and whether one should trust a benchmark run by a company that also sells memory products. For more details, visit the GitHub repository at github.com/wontopos/glasshouse.",
  "summary": "I work at a company that builds memory infrastructure, so I'm biased: we compete in the thing this benchmark measures. We wrote a long-term memory benchmark called glasshouse, and I'd like to know if you think it is worth running on your own system. Why we made it Memory systems report their own numbers, and the numbers don't agree. The case that pushed us to build this: Zep reported 84% on…",
  "key_points": [
    "Glasshouse is a memory benchmark with 17 separate axes, each reported individually.",
    "Benchmark includes 2,847 questions, 1.97 million tokens, and 50 photographs in 10 languages.",
    "Creator invites others to run the draft benchmark on their own memory systems."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}