Three things I got wrong measuring my own cache
A team that produces regulatory documents kept getting the same kind of question from other teams: does the current rule allow X? Answering meant someone reading through memos, manuals and regulations to find where the rule was stated, then writing a summary with the source cited. The work was real, and the answer was almost always already written down somewhere. Retrieval fits this exactly. The…
In the process of measuring my own cache, I made three significant mistakes. First, I overlooked a crucial issue with my negation pairs. I had two of the five pairs that were supposed to be negative interrogatives that require a confirmatory answer. However, they were incorrectly labeled as pairs that should be rejected by the cache.
This mistake was linguistic rather than technical, and it would not have been caught through testing. This oversight led to an incorrect rate of 0.9702 for my negation row, which was included in the report. The corrected version now has an erratum, and these two pairs have been moved to a separate confirmatory category.
Secondly, my control group was too easy. The pairs that my cache was supposed to accept differed by nearly all of their tokens, while the pairs that should be rejected only differed by a single token. This imbalance made it difficult to accurately measure the cache's performance. The experiment inadvertently proved that the embedder could distinguish between near-identical strings and strings with slight differences, rather than semantic differences.
Finally, I initially believed the embedder failed at semantics due to the results I observed. However, I later realized that a function counting shared tokens, unaware of meaning, could produce similar outcomes. This realization meant that the embedder was not failing at semantics as initially thought but rather was succeeding at surface form while I misinterpreted the output as semantic.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
