Autonomous generalizability auditing of scientific foundation models

AutoSPECTRA audits four biological foundation models and links generalization limits to protein sequence composition, pretraining-data relatedness, perturbation representation, spatial resolution, and residue-level context.

• Why it matters: Reliable use of scientific foundation models requires knowing where they generalize beyond development data, but existing benchmarks hide limits and routine rigorous audits are hard.
• What they did: The authors built AutoSPECTRA, a coding-agent protocol, after reviewing 38 studies and then running audits with measurable generalization axes, prespecified analyses, controlled comparisons, and independent confirmation.
• The result: In audits of ESMFold2, STATE, CONCH, and AlphaMissense, agents tied performance to specific training and representation factors and evaluated alternative explanations, rejecting unsupported hypotheses.

AI summary of the abstract · may contain errors · not medical advice · read the paper.