Calibration, accuracy, and domain variation across one million collective forecasts from prediction platforms.

Authors

Justin Dickerson, Hilla Shinitzky, Maayan Peer, Mattan S Ben-Shachar, Michael Gilead

Published in

Proceedings of the National Academy of Sciences of the United States of America. Volume 123. Issue 43. Pages e2615627123. Oct 27, 2026. Epub Oct 19, 2026.

Abstract

Using approximately one million resolved contracts and questions across four collective forecasting platforms, twelve domains, and time horizons ranging from hours to years, we evaluate the calibration and accuracy of collective forecasts in real-world settings. We found that forecast probabilities were overall highly calibrated: events assigned a probability of p% occurred approximately p% of the time. This broad pattern was evident across platforms and many domains, although fine-grained analyses revealed systematic tendencies toward over- or underestimation in highly consequential areas. While this does not mean that forecasting platforms provide certainty about individual events, our findings suggest that large-scale aggregation of judgments contains meaningful information about real-world uncertainty; when examined in aggregate, the predictions generated by the platforms provided informative estimates of event likelihood.

PMID:
42896054
Bibliographic data and abstract were imported from PubMed on 20 Oct 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.