Probing isn't Causality in Molecular Models

Linear probes are used to determine if a chemical variable can be extracted from a molecular representation, but they do not indicate how that variable influences subsequent predictions. By modifying a scalar component within an intermediate block and measuring the change in a downstream outcome, researchers can assess recoverability, intervention selectivity, and a selectivity advantage associated with pretraining.

In Uni-Mol, alterations made to block-12 result in predicting the sign of observed energy changes in 97.6% of held-out pairs, with a Spearman correlation coefficient of 0.787. Intervals for the primary direction and encoder contrasts, when structured by clusters, do not include zero. However, even a more stringent analysis involving 20 directions does not hold up after adjusting for six cells.

A subsequent review highlights a strong alignment between concepts and outcomes, and intervention effects continue even after removing the same-block outcome direction, but this does not prove that following encoder computation is essential. When examining MolCLR, pretraining enhances dipole recovery by 0.080, with an R-squared value, while the differences between concept-minus-random and sign-accuracy at the two tested blocks are 0.029 and -0.015, respectively.

These paired uncertainties remain unresolved, so they do not establish the nonexistence of selective effects. To address this, the authors propose an operational evidence ladder, whose conclusions depend on the concept, intervention, control family, and fitted readout.