Mechanistic Interpretability of Protein Language Models Reveals Encoded Structural and Functional Properties of Intrinsically Disordered Proteins

ESM-2 shows reduced attention on intrinsically disordered protein regions but still encodes meaningful biological signals, including heightened attention to disease-relevant residues even at high disorder levels.

• Why it matters: Interpretability work on PLMs has largely emphasized folded proteins, leaving unclear how these models represent intrinsically disordered proteins, which are common in the human proteome and linked to disease.
• What they did: The study analyzes ESM-2 attention patterns on disordered versus folded regions and extracts IDP structural features by deriving radius of gyration and individual dynamic contact maps from model logits and embeddings.
• The result: The results indicate ESM-2 can recover key IDP properties from its outputs, suggesting PLMs retain valuable information for IDP biology despite bias toward structured residues.

AI summary of the abstract · may contain errors · not medical advice · read the paper.