Contributions of Site Heterogeneity in Multi-site Brain-Phenotype Predictions

Machine learning studies of brain-phenotype associations commonly report accuracy with a single Pearson correlation coefficient. However, this metric can be influenced by several factors in harmonized multi-site data, including training-set size, feature variability, outcome variability, and true effect heterogeneity. The Reproducible Brain Charts consortium analyzed structural MRI data from 3486 participants across 9 sites.

Three training set compositions were examined: within-site (Local), size-downsampled multi-site (Multi-Down), and full multi-site (Multi-Full) training. A fourth site-agnostic strategy (Overall) pooled all sites into a single dataset, ensuring the target site was included in both training and testing samples. Site-level heterogeneity was substantial, with an approximate I^2 of 95% for age prediction.

The overall correlation of age prediction (r = 0.73) surpassed the within-site meta-analytic estimate (Multi-Full r = 0.63). This improvement in the pooled dataset was primarily due to an increase in outcome variance rather than a true advantage of multi-site training. Surprisingly, increasing the multi-site training sample size did not provide a significant advantage over within-site training.

The perceived benefit of cross-dataset training appears to be more a result of training-set size than inherent multi-site composition. Similar patterns were observed for weaker brain-p-factor signals. These findings suggest that between-site heterogeneity in machine learning brain-phenotype prediction remains even after harmonization.

The authors recommend reporting site-level prediction accuracy, feature dispersion, and the standard deviation of the outcome variable alongside any pooled estimates to enhance inter-site and inter-study comparisons of machine learning prediction accuracy.