Autonomous generalizability auditing of scientific foundation models

Authors

Ektefaie, Y., Jain, S., Sabeti, P.

Abstract

Determining where scientific foundation models generalize beyond their development data is essential for reliable use, but aggregate benchmarks obscure performance limits and rigorous audits are difficult to conduct routinely. Here, we introduce AutoSPECTRA, a coding-agent protocol for systematic generalizability auditing. To inform its design, we reviewed 38 studies across five biological domains and evaluated coding agents, which generated hypotheses and empirical analyses but applied inconsistent standards for judging evidence. We then structured AutoSPECTRA around measurable generalization axes, prespecified analyses, controlled comparisons and independent confirmation. Across four biological foundation models, audits linked performance to protein sequence composition and relatedness to pretraining data (ESMFold2), representation of perturbations in training (STATE), effective spatial resolution (CONCH), and residue-level genomic, evolutionary and structural context (AlphaMissense). Agents also evaluated alternative explanations and rejected unsupported hypotheses. AutoSPECTRA provides a systematic approach to discovering and testing the limits of scientific foundation models.

Preprint server: bioRxiv
The authors list and abstract were imported from bioRxiv on 05 Nov 2026.

Stats

1-terrible, 9-excellent. How would you rate this preprint? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.