Diagnostic Accuracy and Fine-Grained Image-Interpretation Performance of Multimodal Large Language Models on Bilingual Radiology Questions: Cross-Sectional Benchmark Study.

Authors

Yijie Liu, Xin Zhang, Jia Xu, Manxin Huang, Yinting Xiong, Wenqian Wu

Published in

Journal of medical Internet research. Volume 28. Pages e98849. Nov 11, 2026. Epub Nov 11, 2026.

Abstract

Multimodal large language models (LLMs) are rapidly expanding in radiology and digital health; however, their diagnostic performance across diverse clinical scenarios and fine-grained visual reasoning tasks requires systematic evaluation.
The study aims to evaluate the diagnostic accuracy and fine-grained visual reasoning performance of 5 multimodal LLMs with bilingual medical imaging questions.
This cross-sectional benchmark study evaluated bilingual medical imaging questions using zero-shot prompting on 5 LLMs: GPT-5.2, Gemini 3 Pro, Doubao, Yuanbao, and Llama 4 Maverick. Subgroup analyses by language (English/Chinese), question type (image/text), imaging modality (computed tomography, radiography, magnetic resonance imaging, ultrasound, and nuclear medicine), and anatomical system (neurologic, respiratory, cardiovascular, digestive, genitourinary, and musculoskeletal) evaluated intermodel accuracy and intramodel consistency. Image-based visual reasoning performance was assessed across modality, anatomy, lesion detection, and localization.
A total of 298 questions (198 image-based, 100 text-based) were evaluated. Gemini 3 Pro achieved the highest overall accuracy (n=268, 89.9%, 95% CI 86.0%-92.9%), significantly outperforming Doubao (n=235, 78.9%; P<.001), GPT-5.2 (n=233, 78.2%; P<.001), Yuanbao (n=216, 72.5%; P<.001), and Llama 4 Maverick (n=216, 72.5%; P<.001). All models performed better on text-based than image-based questions (Yuanbao, P<.001; Doubao, P=.02; GPT-5.2, P<.001; Gemini 3 Pro, P=.002; Llama 4 Maverick, P<.001). GPT-5.2 (P<.001), Llama 4 Maverick (P=.03), and Yuanbao (P=.03) favored English questions, while intramodel performance remained stable across modalities (Yuanbao, P=.46; Doubao, P=.21; GPT-5.2, P=.26; Gemini 3 Pro, P=.96; Llama 4 Maverick, P=.45) and anatomical systems (Yuanbao, P=.82; Doubao, P=.52; GPT-5.2, P=.35; Gemini 3 Pro, P=.13; Llama 4 Maverick, P=.97). Despite near-perfect macroscopic visual recognition (modality and anatomy: 99.0%-100.0%), accuracy dropped in complex tasks. Gemini 3 Pro outperformed Yuanbao (P<.001), GPT-5.2 (P=.002), and Llama 4 Maverick (P<.001) in lesion detection (175/198, 88.4%) and outperformed Yuanbao (P<.001), Doubao (P=.004), and Llama 4 Maverick (P=.004) in lesion localization (189/198, 95.5%).
Gemini 3 Pro demonstrated optimal, comprehensive performance and cross-lingual stability. However, current multimodal LLMs face a technical bottleneck in complex visual interpretation-particularly in fine-grained lesion detection and localization-emphasizing the need to prioritize local feature perception in future model development.

PMID:
43013771
Bibliographic data and abstract were imported from PubMed on 12 Nov 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.