The Working Set That Never Saturated
Every experiment in this series had charged each byte of a count table as resident memory. That is a real assumption and nobody had examined it, which I noticed only after reading someone else's project: a third-party runtime that runs 35B and 80B mixture-of-experts models on Apple hardware by keeping a dense core in memory and streaming experts from SSD, reporting 43 to 70% cache hit rates. A…
Multiple experiments in this series have consistently filled every byte of a count table into resident memory, regardless of the size of the model used. This is a significant assumption that was only noticed after comparing it to a third-party runtime that successfully runs large mixture-of-experts models on Apple hardware. The key factor in this phenomenon is the mixture-of-experts architecture, which allows for streaming experts from SSD and maintains a dense core in memory, leading to cache hit rates ranging from 43 to 70%.
The research reveals that instead of focusing on how much accuracy can fit in B bytes, the more pertinent question is how many resident bytes are required for a given level of accuracy. The study's hypothesis proved false, but the more intriguing aspect is the outcome of testing the hypothesis on which it depended. The harness used in the research is not publicly available, but the findings are based on its internal mechanisms.
In the tests, each record stores its own context tokens and undergoes comparison on every probe. A 64-bit hash collision cannot produce an incorrect distribution, ensuring the correctness of the tables. After establishing the correctness gate, 13,500 sampled lookups with hits and misses were conducted, and they perfectly matched the in-memory tables, with no mismatches observed across all 7,200 positions. Consequently, the accuracy remains unchanged, while the cost of memory usage becomes the primary concern.
The working set does not saturate, as evidenced by completing roughly 2,700 consecutive positions within a single file and observing the extent of table pages that are touched. With a size of 142.7 MB, the table represents 3.9 MB of the last quarter added, amounting to only 2.7% of the total. The study further demonstrates that as more data is added, the working set continues to expand, with an average increase of 1 to 3.8 MB in the final quarter of the session.
The locality argument initially suggested that a small working set would suffice, but this notion was debunked upon closer examination. Each position in the table represents a novel context, and the predictor's accuracy relies on the recurrence of the context rather than the token itself. This distinction is crucial as it differs from the 0.88 repeat rate observed in code, which is primarily token-based.
The findings underscore the importance of measuring the actual performance of the working set under real-world conditions. The original verdict, based on a hypothesized number, lacked empirical validation, making it unreliable. By conducting a second pass over the same offsets under a non-cached control, the researchers were able to substantiate their claims and avoid drawing premature conclusions.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
