{
  "id": 18874332,
  "title": "When do you lock the evaluation set?",
  "url": "https://urgent.news/2026/11/08/when-do-you-lock-the-evaluation-set",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-08T06:11:23.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/wolfoftyresodebug/when-do-you-lock-the-evaluation-set-5gk3"
  },
  "original_language": "en",
  "account": "When scaling collection and annotation for diverse document and file types at Landvex, the primary concern is avoiding contamination of the test set. To mitigate this risk, the preferred approach is to define the ground truth for each document family beforehand, and lock a held-out test set before expanding annotation efforts and GPU training. Only after establishing these locks should the focus be on evaluating whether additional data truly impacts the key performance metrics.\n\nIn high-stakes document and KYC systems, evaluation often involves locking not only the files themselves but also aspects like the label schema, annotator instructions, and the precise questions the model is being scored on. Maintaining strict boundaries around these elements is crucial to prevent any unintended leakage of test items into reference examples that reviewers may examine.",
  "summary": "At Landvex we're about to scale collection and annotation across several document families, mixed image and file inputs. The obvious failure mode is multiplying training material first, then discovering the test set was contaminated — or that \"accuracy\" was measured against a binary genuine/fake label we no longer believe in. I'm leaning toward: define the ground truth per family, lock a held-out…",
  "key_points": [
    "Lock test set before expanding annotation efforts",
    "Lock evaluation aspects like label schema and annotator instructions",
    "Maintain strict boundaries to prevent test leakage"
  ],
  "editors_take": "Establishing rigid boundaries for test sets and label schemas prevents data leakage, ensuring that scaling annotation efforts and model training actually produces measurable improvements in high-stakes document processing systems.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}