Overestimating zero-shot fitness prediction: Broad benchmarks mask local failures and practical limitations

Authors

Woolley, P. R., Feller, A., Ellington, A. O., Wilke, C. O.

Abstract

Deep learning models have emerged as promising tools for navigating mutational landscapes in protein engineering. These models can be used to predict mutation fitness without the need for task-specific training, a process known as zero-shot prediction. However, their practical utility remains only partially characterized. Here, we evaluate the zero-shot performance of a panel of protein sequence and structure models across a range of benchmarking conditions, focusing on factors that complicate the interpretation of aggregate metrics. We show that input modality (sequence vs. structure) does not dictate performance on phenotypic tasks. Instead, performance is sensitive to experimental variability and is heavily confounded by correlation between phenotype and protein abundance. While available models may act as coarse filters separating fit mutations from deleterious ones, they cannot meaningfully rank a set of fit mutations or prioritize new-to-nature functions. Ultimately, the practical utility of zero-shot prediction from protein models is narrower than aggregate benchmarks imply.

Preprint server: bioRxiv
The authors list and abstract were imported from bioRxiv on 09 Jun 2026.

Stats

1-terrible, 9-excellent. How would you rate this preprint? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.