Protify: A Benchmarking Platform Revealing Task Dependence and Homology Leakage in Protein Language Model Evaluation

DPLM3B is the top-performing self-supervised encoder on ProteinGym, but model rankings shift strongly with evaluation design and homology control.

• Why it matters: Evaluation across protein language models is fragmented, so results don’t transfer between studies; Protify targets standardized, replayable benchmarking to expose when rankings depend on design choices.
• What they did: Protify benchmarks 35 pretrained models on 61 curated datasets using unified zero-shot, supervised, and transfer-learning pipelines, and it runs high-throughput evaluations across ProteinGym and 32 supervised tasks.
• The result: On enzyme optimal pH prediction, a general probe matched or exceeded EpHod (R2 = 0.482 vs 0.425), but under a homology-controlled split pooled probes collapsed near-zero R2 while a transformer over full residue embeddings reached R2 = 0.248.

AI summary of the abstract · may contain errors · not medical advice · read the paper.