Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models
Large language models (LLMs) are increasingly evaluated on molecular property benchmarks, but accuracy cannot distinguish a model that predicts a property from one that retrieves a published number. We audit 22 frontier models on 12 regression benchmarks for verbatim retrieval and find that it is widespread but relatively benchmark-specific: on five datasets more than $50\%$ of the LLMs show…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
