Authors
Thijs Snelleman, Holger Hoos, Odd Erik Gundersen
Published in
Patterns (New York, N.Y.). Volume 7. Issue 10. Pages 101640. Oct 09, 2026. Epub Aug 27, 2026.
Abstract
Insufficiently documented publications can make independent reproduction an unnecessarily laborious task. We define the cost of reproducibility as the resources required to reproduce the findings of a study and we propose five dimensions of cost, using objective characteristics such as code and data. We review 918 publications between 2022 and 2024 at top AI venues to evaluate the variation in reproducibility costs, with the hypothesis that publications at venues with stricter standards are less costly. Papers published at machine learning conferences are less costly to reproduce than publication from other venues, 12.91% less costly compared with journals, and 16.52% less costly compared with general AI conferences. Award-winning papers, however, are not less costly to reproduce than regular papers, and 6.72% of implementation repository links were broken. These findings suggest that reproducibility costs can be significantly reduced through compliance-based mechanisms-provided they receive community support. We recommend that venues require the inclusion of reproducibility checklists in their publications, permit appendices for comprehensive documentation, and adopt low reproducibility cost as a key criterion for awards.
PMID:
42891295
Bibliographic data and abstract were imported from PubMed on 18 Oct 2026.
Read full publication at:
Please sign in
to see all details.
Advertisement
Stats
- Recommendations n/a n/a positive of 0 vote(s)
- Views 26
- Comments 0