Tested a Per-Document Cap at Real Scale — and Found I Had No Ground Truth
Entry 07 ended on a loose thread: a document split into 17 chunks crowds out equally relevant single-entry documents, just by having more shots at ranking. The usual fix is a per-document cap, so I flagged it and moved on. Now that the collection holds 741 real chunks instead of 20, it was time to find out whether the problem was real at this size, and whether the fix worked. Measured first,…
A per-document cap was tested at real scale and found to have no ground truth. The collection initially held 20 chunks, which was increased to 741 real chunks for the test. The usual fix for the issue was a per-document cap, but measuring this before and after the cap was applied showed no significant difference. The cap removed extra chunks from relevant articles and replaced them with chunks from unrelated documents, leading to a trade-off between relevance and variety.
The ranking of a specific document, Entry 02, changed from being the top hit to being ranked lower in the capped version. This indicates that Entry 02 was never a correct answer, as it was about system prompts and not pod-status commands, which were the focus of the query. The cap's effectiveness could not be determined from the data, as the query likely has no right answer in the collection.
The next step would be to build a small set of questions with known correct answers to evaluate the cap's performance.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
