Large language models generate diagnostic likelihood ratios with low mean bias but wide dispersion.

Authors

Paul Chong, Shuhan He, Kian Samadian, Amal Mohamed, Boyu Peng, Emma Chua, Cory Rohlfsen, Brian W Locke

Published in

Scientific reports. Jul 11, 2026. Epub Jul 11, 2026.

Abstract

Accurate, context-appropriate likelihood ratios (LRs) are needed for Bayesian diagnosis, but empirical LRs are sparse because diagnostic accuracy studies are costly and context-dependent. We evaluated whether large language models (LLMs) can generate LR outputs that align with literature-reported values. We compared LR outputs generated by three OpenAI models (GPT-4o, o3, and GPT-5) with all literature-reported values curated in TheNNT.com dataset. A few-shot prompt was used to elicit numerical LR outputs, and agreement was assessed using Bland-Altman analyses to evaluate mean bias and multiplicative limits of agreement. A total of 700 reported LRs across 30 conditions were compiled, most involving signs or symptoms (59%), historical elements (19%), or test results (16%). All models demonstrated negligible mean bias. GPT-5 had the narrowest 95% limits of agreement (0.26×-3.70×) compared with o3 and GPT-4o. These findings indicate that LLM-generated LRs can be centered near literature values on average, but they are not interchangeable with published estimates for individual high-stakes decisions. Clinical use, including use for unstudied findings, would require prospective, context-specific diagnostic-accuracy validation.

PMID:
42436230
Bibliographic data and abstract were imported from PubMed on 12 Jul 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.