Benchmark validity in graph neural network scoring of metabolic reaction activity on Recon3D: detecting label leakage, memorized noise and input-invariant models

A recent study has uncovered two major flaws in the Graph Neural Network (MetaGNN) scoring system for metabolic reaction activity in cancer cells. The first issue arises when the framework is given threshold-based supervision, achieving an AUROC (Area Under the Receiver Operating Characteristic curve) of 0.9864 +/- 0.0008 on the TCGA breast cancer cohort.

This high score is due to two factors: firstly, 5,925 reactions have labels that are a deterministic threshold of the model's own input, making ranking by the cohort-mean input alone result in a perfect score of 1.000; secondly, 4,675 reactions have stored labels that were reproduced bit for bit from a seeded pseudo-random number generator.

In this case, the model memorizes a patient-invariant label vector, which is fully visible during training, leading to an AUROC of 0.9291 +/- 0.0030.

The second problem was discovered when evaluating the archived models independently of the input data. The models were never exposed to any patient data during training; their released feature tensors were uniformly zero. Even when independently trained models were used, they showed no agreement on which patient deviated where, as measured by the absolute difference between the model's predictions (|r|).

These findings highlight the need for more rigorous validation methods in graph neural network scoring of metabolic reaction activity to avoid label leakage, memorized noise, and input-invariant models.