Large language models can predict the results of social science experiments.

Authors

Ashwini Ashokkumar, Luke Hewitt, Isaias Ghezae, Robb Willer

Published in

Nature. Jul 08, 2026. Epub Jul 08, 2026.

Abstract

There is growing interest in how large language models (LLMs) can advance social and behavioural science^1-5. Previous work has assessed LLMs' ability to predict survey responses^6-9, but less is known about whether they can predict the outcomes of social science experiments¹⁰, particularly those absent from training data. Here we built an archive of 70 preregistered, nationally representative survey experiments in the USA involving 469 experimental effects and 119,330 participants. We prompted an LLM to simulate how representative samples from American individuals would respond to experimental stimuli, and then we inferred treatment effects by comparing simulated responses across conditions. Predictions derived from GPT-4, whose training-data cutoff predated the publication of many studies in our archive, were strongly correlated with actual treatment effects, achieving accuracy similar to pooled human forecasts. Correlations remained high for studies not published or publicly posted by the model's training-data cutoff date and for predictions from prominent open-weight models. Despite high correlations, predictions systematically overestimated effect sizes. In a secondary archive of 15 megastudies featuring 606 effects, correlations were lower but comparable to those of pooled expert forecasters. To assess implications for scientific practice, we surveyed 460 social scientists about probable uses and perceived risks and used our archives to assess several applications (pilot testing, intervention selection, identifying effects needing replication) and risks (bias, misuse). Together, these results indicate that LLMs can augment experimental methods in science and practice while raising important considerations for responsible use.

PMID:
42420458
Bibliographic data and abstract were imported from PubMed on 09 Jul 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.