I built an oracle to prove my benchmark needed full document coverage. I only tested it from one end.
I had a benchmark of retrieval questions over book-length documents, and a claim that answering them required reading the whole document rather than the part that fits in a context window. I did not want to take that on faith, so I wrote a test with no model in it at all. It scores oracles : agents with perfect extraction and perfect reasoning that can only see the first F of each document. If a…
The writer built an oracle to demonstrate that their benchmark required full document coverage rather than partial document coverage. They created a test where an oracle, which could only see the first quarter of each document, performed well on two tasks: determining who spoke last and whether a certain character spoke in the final quarter.
However, when testing the same tasks with an oracle that could only see the last quarter of each document, the oracle performed almost perfectly. This showed that neither prefix nor suffix readers were necessary for these tasks, as the answers were in either the beginning or the end of the document. The writer's empirical findings indicated that the benchmark tasks did not require full document coverage, and the oracle confirmed this.
The writer then improved their benchmark by dividing the document into ten equal parts and determining who spoke in the greatest number of those parts. This oracle proved that the task could not be recovered proportionally, as a quarter of the document only revealed a quarter of the regions. The writer emphasized that this benchmark was a more honest and accurate representation of the tasks' requirements.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
