Determinants of Reader-LLM Interaction in Thoracic Radiology: Impact of Model Confidence and Reader Expertise.

Authors

Jiyoung Song, Won Gi Jeong, Dae Hee Han, Hongseok Ko, Soon Ho Yoon, Hyungjin Kim, Phakhanith Naruetook, Wai Ling Leong, Soojin Kim, Hee Eun Moon, Ji Yeong An, So Jung Koo, Meesun Lee, Eui Jin Hwang, Taehee Lee

Published in

Radiology. Volume 320. Issue 2. Pages e253397.

Abstract

Background Large language models (LLMs) provide diagnostic suggestions and rationales, but determinants of successful reader-LLM interaction remain unclear. Purpose To determine how LLM attributes and reader expertise independently and jointly influence the selective integration of diagnostic advice in human-LLM collaboration. Materials and Methods In this retrospective study, 10 readers evaluated 100 chest imaging cases (radiography, CT, MRI, or PET) from the Korean Society of Thoracic Radiology Weekly Case platform (January 2018 to December 2020), unaided (session 1) and randomized to LLM-assisted setups (session 2) at the reader-case level: high accuracy (76% [379 of 500 reader-case pairs]) using OpenAI's GPT-5 or low accuracy (27% [133 of 500]) using OpenAI's GPT-4o (August 2025), providing ranked diagnostic options with rationales against a reference standard established by the case author. The primary outcome was adequate interaction (accepting correct or rejecting incorrect suggestions). Data were analyzed using multivariable generalized estimating equations, adjusted for reader expertise, diagnostic correctness and reader confidence (session 1), model confidence (score assigned to the correct diagnosis), and reference panel-assessed rationale quality. Results A total of 100 patients were included (mean age, 50.0 years ± 16.3 [SD]; 59 male). After multivariable adjustment, model confidence (odds ratio [OR], 3.82 [95% CI: 1.58, 9.25]; P = .003) and reader expertise (OR, 2.06 [95% CI: 1.38, 3.07]; P < .001) were independently associated with adequate interaction, with a weaker effect of confidence among experts (OR, 0.79 [95% CI: 0.67, 0.94]; P = .008). Higher rationale quality reduced rejection of correct suggestions (OR, 0.79 [95% CI: 0.67, 0.93]; P = .005) but increased acceptance of incorrect suggestions (OR, 1.71 [95% CI: 1.47, 1.99]; P < .001). Higher reader expertise (OR, 0.54 [95% CI: 0.41, 0.70]; P < .001) and reader confidence (OR, 0.80 [95% CI: 0.67, 0.94]; P = .007) were protective, reducing acceptance of incorrect suggestions. Conclusion Successful reader-LLM collaboration is associated with model confidence and reader expertise, with rationale quality facilitating correct advice uptake but increasing overreliance on incorrect suggestions. © RSNA, 2026 Supplemental material is available for this article.

PMID:
42578783
Bibliographic data and abstract were imported from PubMed on 11 Aug 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.