Model generalizability considerations in development and evaluation of SaMD.

Authors

Ning Leng, Yichen Lu, Jane Fridlyand, Tiffany Tang, Ting Qi, Yingying Wei, Rodrigo Rojo Sanchez, Noah Simon, Gregory Alexander

Published in

Journal of biopharmaceutical statistics. Pages 1-13. Sep 28, 2026. Epub Sep 28, 2026.

Abstract

Software as a Medical Device (SaMD) has been transforming medical practices by improving patient care with more precise and timely information. Key capabilities of SaMD are often powered by Artificial Intelligence (AI) algorithms. However, significant challenges arise in developing robust algorithms for SaMD, as these algorithms can perform well during development but poorly during pivotal validation studies. Due to the rapid advancement in medical research, data from new studies or real-world sources are likely to differ significantly from the legacy data used for development; with this, algorithms need to account for these potential data heterogeneity differences. This paper discusses the shortcomings of conventional cross-validation methods widely used in SaMD algorithm development and demonstrates model performance overestimation in the presence of data heterogeneity. To address this bias, we propose a simple and practical alternative: the leave-one-set-out (LOSO) cross-validation method. Additionally, we outline best practices for designing independent validation pivotal studies.

PMID:
42806662
Bibliographic data and abstract were imported from PubMed on 29 Sep 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.