Development and cross-cohort evaluation of a multimodal deep-learning model for ER, PR and HER2 status classification in breast cancer

Background information indicates that estrogen receptor (ER), progesterone receptor (PR) and HER2 status are crucial in determining systemic therapy for breast cancer. However, multimodal models are not commonly compared with established biomarker molecular baselines or assessed for cross-cohort calibration. This investigation aimed to evaluate the advantages of a multimodal deep-learning model, its reliance on label-proximal HER2 transcripts, and its ability to apply discrimination and calibration across different cohorts.

The study utilized data from TCGA-BRCA (1051 patients) and CPTAC-BRCA (120 patients) to develop a multimodal deep-learning model that combined whole-slide images, transcriptomes, age, and stage. The model's performance was assessed in both cohorts, with CPTAC cases including 13 HER2-positive patients. Various analyses were conducted, including model comparisons, transcript removal, alternative loss functions, and recalibration.

In terms of internal discrimination, the model achieved AUCs of 0.943 for ER, 0.901 for PR, and 0.860 for HER2. For external discrimination, the AUCs were 0.952 for ER, 0.937 for PR, and 0.918 for HER2. The external macro-AUC was 0.936 for the multimodal model and 0.957 for logistic regression using ESR1, PGR, and ERBB2, resulting in a difference of -0.021.

Removing the 29-transcript 17q12-q21 region impacted the HER2 AUC by -0.035 (-0.058 to -0.013) internally and -0.042 (-0.089 to -0.007) externally, compared to the baseline multimodal model.

When comparing the multimodal model to a 15-model ensemble, the three-transcript model outperformed the ensemble in terms of external expected calibration error (ECE) and Brier score. However, recalibrated HER2 ECE and calibration-in-the-large for both models exceeded the reference ceiling of 0.125, indicating systematic overprediction.

ER and PR models also exhibited overprediction. The findings suggest that although the multimodal deep-learning model demonstrated high discrimination across cohorts, it did not show clear benefits over a parsimonious three-transcript baseline. Cross-cohort calibration remains a significant challenge, emphasizing the need to consider incremental value, calibration, and clinical utility when assessing AI-based prediction models for medical use.