{
  "id": 14972687,
  "title": "I built an oracle to prove my benchmark needed full document coverage. I only tested it from one end.",
  "url": "https://urgent.news/2026/10/16/i-built-an-oracle-to-prove-my-benchmark-needed-full-document-coverage",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-16T14:34:50.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/rickeshtn/i-built-an-oracle-to-prove-my-benchmark-needed-full-document-coverage-i-only-tested-it-from-one-5ein"
  },
  "original_language": "en",
  "account": "The writer built an oracle to demonstrate that their benchmark required full document coverage rather than partial document coverage. They created a test where an oracle, which could only see the first quarter of each document, performed well on two tasks: determining who spoke last and whether a certain character spoke in the final quarter. However, when testing the same tasks with an oracle that could only see the last quarter of each document, the oracle performed almost perfectly. This showed that neither prefix nor suffix readers were necessary for these tasks, as the answers were in either the beginning or the end of the document. The writer's empirical findings indicated that the benchmark tasks did not require full document coverage, and the oracle confirmed this. The writer then improved their benchmark by dividing the document into ten equal parts and determining who spoke in the greatest number of those parts. This oracle proved that the task could not be recovered proportionally, as a quarter of the document only revealed a quarter of the regions. The writer emphasized that this benchmark was a more honest and accurate representation of the tasks' requirements.",
  "summary": "I had a benchmark of retrieval questions over book-length documents, and a claim that answering them required reading the whole document rather than the part that fits in a context window. I did not want to take that on faith, so I wrote a test with no model in it at all. It scores oracles : agents with perfect extraction and perfect reasoning that can only see the first F of each document. If a…",
  "key_points": [
    "Writer created an oracle to test benchmark requirements",
    "Oracle with one-quarter view performed well on tasks",
    "Benchmark tasks not requiring full document coverage"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}