Large language model assistance for report-based carotid plaque AHA classification: a multi-reader study.

Authors

Mengfei Wang, Yixiao Tang, Yulu Yang, Yifan Tu, Zhongzheng Cao, Runjianya Ling, Kaiyue Sun, Jian Ma, Zhicheng Zhang, Ruipeng Zhang, Xiaoer Wei, Dan Wang, Yuehua Li

Published in

Insights into imaging. Volume 17. Issue 1. Oct 16, 2026. Epub Oct 16, 2026.

Abstract

To evaluate whether large language model assistance improves radiologists' agreement with a text-based reference standard in inferring the modified American Heart Association carotid plaque classification from free-text MRI reports, and whether a lightweight model can perform well under criteria-guided prompting.
This retrospective study included 433 patients from three institutions. In a report-based reader study, four radiologists classified MRI reports with and without large language model assistance. In standalone evaluation, a rule-based baseline and 10 models were compared under naive and criteria-guided prompting. The reference standard was expert consensus based on the report text.
Among 433 patients (mean age, 65.9 years ± 13.4; 248 males), AI assistance improved six-class agreement with the text-based reference standard. At the vessel level, junior radiologists improved from 61.75% to 83.12% (Qwen3-8B) and 87.75% (DeepSeek-R1); senior radiologists from 77.00% to 87.50% and 92.25%, respectively (all Holm-corrected p < 0.001). Patient-level agreement increased from 41.0% to 78.8% (junior) and from 59.8% to 87.8% (senior) with DeepSeek-R1. Agreement increased (quadratic-weighted κ: 0.231 without AI vs 0.775 with DeepSeek-R1). In standalone evaluation, criteria-guided prompting improved Qwen3-8B agreement from 38.91% to 81.52% (95% CI: 78.87-84.06%), remaining 5.89 percentage points below DeepSeek-R1 (87.41%; 95% CI: 85.10-89.61%).
In this report-based classification setting, large language model assistance improved radiologists' agreement with the text-based reference standard, with larger gains in junior radiologists. When prompts explicitly encoded AHA criteria, an 8B local candidate model performed well, supporting its potential use as a standardization aid for text-based workflows.
Question Report-based modified AHA classification of carotid plaques is experience-dependent and difficult to standardize, as mapping multi-sequence imaging findings to classification conclusions imposes substantial cognitive load. Findings Large language model assistance improved agreement with a text-based reference standard and raised inter-reader consistency (weighted κ 0.231-0.775), narrowing the junior-senior gap. Critical relevance statement Embedding explicit AHA criteria in the prompt let an 8B model approach 671B-model agreement (38.91-81.52%), supporting standardized text-based workflows without fine-tuning.

PMID:
42886459
Bibliographic data and abstract were imported from PubMed on 17 Oct 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.