Glasshouse v0.1 Is Out: A Memory Benchmark for AI Systems

Glasshouse v0.1 is now available, providing a long-term memory benchmark for AI systems. This benchmark emerged from discussions in developer communities where discrepancies were noted between reported and measured memory benchmark numbers. The benchmark includes 2,847 questions spread across a conversation of 1.97 million tokens, encompassing 10 languages and featuring 50 photographs.

The conversation is offered in four sizes, ranging from 1,882 turns to 103,572 turns. The benchmark highlights the point at which a system's performance starts to degrade as the accumulated history increases. Each metric is reported individually, with no single headline score, as a system may excel in one area while failing in another.

The benchmark extends beyond simple recall; it evaluates how a system handles changed facts, conflicting information, and unanswered questions. For instance, if a fact changes and the system fails to locate the updated value, it receives a score for stating "I don't know" rather than confidently repeating the outdated information.

When two stored facts disagree and no resolution is found, the system's response of "they don't agree" is considered correct, while arbitrarily selecting one fact for the answer would be incorrect. Additionally, if the original fact was never mentioned, the system's response must verify this fact. The Glasshouse v0.1 repository encourages individuals to run the benchmark and submit any identified issues through pull requests or issues, fostering a community-driven approach to benchmark refinement.

While there isn't a specific threshold for individual submissions, any specific error reported should be addressed publicly. The repository also outlines how companies can submit their results or self-identify within the system, providing a structured approach for corporate involvement. All necessary information, including setup instructions, is accessible at the repository link: github.com/wontopos/glasshouse.