When do you lock the evaluation set?
At Landvex we're about to scale collection and annotation across several document families, mixed image and file inputs. The obvious failure mode is multiplying training material first, then discovering the test set was contaminated — or that "accuracy" was measured against a binary genuine/fake label we no longer believe in. I'm leaning toward: define the ground truth per family, lock a held-out…
When scaling collection and annotation for diverse document and file types at Landvex, the primary concern is avoiding contamination of the test set. To mitigate this risk, the preferred approach is to define the ground truth for each document family beforehand, and lock a held-out test set before expanding annotation efforts and GPU training. Only after establishing these locks should the focus be on evaluating whether additional data truly impacts the key performance metrics.
In high-stakes document and KYC systems, evaluation often involves locking not only the files themselves but also aspects like the label schema, annotator instructions, and the precise questions the model is being scored on. Maintaining strict boundaries around these elements is crucial to prevent any unintended leakage of test items into reference examples that reviewers may examine.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
