Cell-level random splits leak group-owned answers in single-cell benchmarks
Machine learning models in single-cell biology increasingly forecast differentiation, reprogramming and therapeutic response from early transcriptomic profiles. Testing whether a model has learned real biology requires held-out cells. Single-cell data, however, are grouped: cells from the same clone, patient or batch share the same label. A random split therefore places relatives of each test…
Single-cell machine learning models are used to predict cell differentiation, reprogramming and therapeutic response from early transcriptomic profiles. However, these models can be prone to leakage if a random split places relatives of a test cell, which share the same label, into the training set. This leads to the model scoring well by memorizing a relative instead of learning a transferable rule.
To address this issue, researchers have developed a method to estimate leakage before training any model. This involves calculating two properties of the data: exposure and retrievability. Exposure is the fraction of test cells with relatives in the training set, while retrievability is how often a nearest-neighbor search returns a relative instead of an unrelated cell.
Exposure determines whether a random split opens a leakage channel, and retrievability determines how much it can inflate the model's score. In cases where cell state has decoupled from ancestry, the inflation is negligible. However, it becomes much larger when clonal sisters remain close in expression space. In a patient cohort large enough, this leakage can even overturn a clinical conclusion.
To help researchers easily compute these properties, the researchers have created a tool called eakcheck. This tool calculates both exposure and retrievability in seconds, before the outcome model is fitted.
Written by urgent.news from bioRxiv's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
