Reliability and disease sensitivity are dissociable properties of EEG foundation-model representations

EEG foundation models (EEG-FMs) are primarily assessed based on their ability to accurately differentiate between diseases. However, a crucial aspect of a clinical biomarker is measurement reliability, which refers to the consistency of repeated measurements on the same individual – a requirement stipulated by regulatory frameworks.

This study examines whether frozen EEG-FM representations maintain stability, whether this stability can be predicted from common model descriptors, and what information supports this stability. The researchers measured test-retest reliability, disease discrimination, and representation distinctiveness for nine frozen representations: six EEG-focused foundation models (five masked, contrastive, and predictive pretraining, handcrafted spectral features, and two general-purpose time-series models with no EEG exposure) across two healthy retest cohorts, at approximately one month and two years, and three neurodegenerative cohorts.

The primary metric for reliability, the intraclass correlation coefficient (ICC), ranged from 0.08 to 0.76, with a coefficient of variation (CV) of 53.0% while disease discrimination, measured by the area under the receiver operating characteristic curve (AUC), exhibited a much narrower range of 5.4% CV across the same nine models, indicating a roughly tenfold difference in relative dispersion.

The variation in reliability was largely unrelated to pretraining paradigm or domain, and even a model without EEG exposure was among the most reliable tested. Alpha-band information was found to disproportionately contribute to reliability, whereas theta-band information was primarily associated with discrimination for Alzheimer's disease and frontotemporal dementia, aligning with established EEG evidence in both conditions.

However, the study found that only the reliability aspect of this contrast is individually significant. Both subspace geometry and band ablation analyses suggest that reliability and discrimination are partially dissociable, not fully so, and that network architecture plays a crucial role in determining whether this dissociation is preserved, traded off, or jointly degraded across model depth.

Importantly, reliability showed no detectable association with discrimination, pretraining paradigm, or domain, and must be measured directly rather than inferred. Based on these findings, the researchers recommend that reliability become a standard evaluation criterion for EEG-FM representations intended for longitudinal or biomarker use, and they have released a reproducible pipeline for this purpose.