Protify: A Benchmarking Platform Revealing Task Dependence and Homology Leakage in Protein Language Model Evaluation

Protein Language Models (pLMs) have emerged as indispensable tools for computational protein analysis. However, the rapid growth of architectures, training objectives, and pretraining corpora has led to a fragmented evaluation landscape. Different studies utilize distinct datasets, preprocessing pipelines, and reporting methods, making it difficult to compare results across publications.

To address this issue, researchers have introduced Protify, an open-source platform that standardizes pLM benchmarking across zero-shot, supervised, and transfer-learning scenarios through a unified, low-code interface. Currently, Protify supports 35 pretrained models and 61 curated datasets. It also allows for easy customization of each model and dataset.

By harnessing Protify's high-throughput pipelines, researchers have conducted a comprehensive benchmarking study involving 23 models across ProteinGym. The study revealed that DPLM3B currently stands as the top-performing self-supervised encoder. In a large-scale supervised evaluation encompassing 32 diverse tasks, including enzyme classification, gene ontology annotation, protein-protein interaction, subcellular localization, and other functional properties, no single pLM emerged as the clear winner.

Instead, 14 different models achieved the top score on at least one dataset, with the gaps between the best and second-best models being within the bounds of run-to-run variability.

One particularly notable finding was that a general-purpose probe pipeline, matching or surpassing EpHod, a specialized model chosen from 11,550 training runs, significantly outperformed the latter on enzyme optimal pH prediction. This remarkable result was achieved after just 50 hyperparameter trials (R2 = 0.482 compared to 0.425).

Under a homology-controlled split of the same data, however, every pooled-embedding probe reduced to near-zero R2 values. Only a transformer probe utilizing full residue embeddings managed to recover any meaningful signal (R2 = 0.248). This highlights the fact that model rankings are heavily influenced by the evaluation design, in addition to the models themselves.

By making those design choices transparent, controlled, and reproducible, Protify effectively addresses the challenges posed by the fragmented evaluation practices in the field of protein language model research. The platform is now freely accessible at https://github.com/Gleghorn-Lab/Protify.