Normalized risk-based evaluation of machine learning-based classification models: A multiclass approach with applications in medical devices.

Authors

Julius Wiggerthale, Martin Haimerl, Christoph Reich

Published in

The Journal of international medical research. Volume 54. Issue 9. Pages 3000605261486667. Epub Sep 28, 2026.

Abstract

ObjectiveMachine learning is increasingly integrated into safety-critical domains such as medical applications. In this context, regulatory frameworks require the assessment and minimization of risks associated with incorrect model predictions. However, classical evaluation methods often focus on quantifying error frequencies, which do not reflect the heterogeneous impact of different types of errors. To address this deficiency, we elaborate an approach for assessing machine learning-based multiclass classification models that conforms to regulatory requirements in the field of medical devices. Additionally, we aim to demonstrate its applicability in a concrete application scenario as an exemplary reference.MethodsHerein, we provide weighted balanced accuracy $WB{A}_n$ as a risk-based approach for assessing machine learning models for multiclass classification. $WB{A}_n$ is systematically based on regulatory requirements for medical devices. It incorporates core properties such as the representation of risks as a multiplicative combination of likelihood and severity and the relationship between development and real-world scenarios. We demonstrate the application of $WB{A}_n$ using X-ray based detection of lung diseases as a reference scenario.ResultsOur analysis shows that $WB{A}_n$ achieves a risk-based assessment in the lung disease scenario as a reference. $WB{A}_n$ systematically operationalizes the definition of risk from corresponding regulatory requirements and allows the assessment to be adapted according to risk levels that have to be elaborated in the context of the risk-management process.ConclusionAlthough this study uses only a single use case to demonstrate the practical applicability of our approach, it shows the impact of a risk-based approach when assessing the performance of machine learning multiclass classification models in medical applications. Without such an approach, regulatory requirements cannot be implemented in a fully comprehensive way. Consequently, the clinical impact cannot be adequately assessed when applying the model in real-world scenarios.

PMID:
42803731
Bibliographic data and abstract were imported from PubMed on 28 Sep 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.