Comparative Evaluation of Open-Weight Large Language Models for Pathological Brain MRI Reports Classification: From Prompts to a Fine-Tuned Model.

Authors

Claudio Demaria, Claudio Crema, Mario Verdicchio, Valentina Brancato, Vincenzo Alfano, Luca Basso, Giuseppe Pontillo, Damiano Archetti, Silvia Fostinelli, Silvia De Francesco, Michela Pievani, Cesare M Baronio, Alberto Boccali, Nicola Toschi, Arturo Brunetti, Alberto Redolfi, Marco Aiello

Published in

Academic radiology. Oct 14, 2026. Epub Oct 14, 2026.

Abstract

Automated semantic classification of radiology reports can enhance clinical workflows and research, but often relies on language- and task-specific models, limiting scalability. This study aims to evaluate and improve the 0-shot baseline performance of open-weight large language models (LLMs) in the semantic classification of free-text magnetic resonance imaging (MRI) reports.
A corpus of 1385 reports from a single neuroradiology center was used to assess four LLMs (GPT-OSS-20B, Magistral-Small-2509, Gemma-4-31B, Qwen3.5-35B-A3B) of comparable scale (∼20-35B parameters). The brain MRI reports were labeled as Pathological or Non-Pathological, with the Pathological ones further classified into five diagnostic categories (Neoformation, Vascular, Inflammation/Infection, Neurodegenerative, Other). The LLMs were evaluated using zero- and few-shot prompting with majority-vote aggregation. Quantized Low-Rank Adaptation (QLoRA) fine-tuning was then applied to the best-performing model across multiple configurations. Performance was assessed using Macro F1, Micro F1, Precision, Recall, and Exact Match Accuracy.
The binary classification task achieved a Macro F1 score of 0.84 on the best-performing Qwen3.5-35B-A3B model, while multilabel classification reached 0.81. Majority voting (MV) outperformed individual-set predictions. QLoRA fine-tuning yielded a Macro F1 score of 0.83 on the multilabel task; the improvement over the 0-shot baseline was statistically supported for Macro F1, Micro F1, Macro Precision, and Exact Match, whose bootstrap 95% confidence intervals excluded zero, but not for Macro Recall. Fine-tuning and few-shot prompting strategies are complementary, with neither consistently outperforming the other. Error analysis showed that the models struggled with borderline annotation cases and often missed secondary labels.
Open-weight LLMs demonstrate strong capability for neuroradiology report classification without extensive task-specific training. Fine-tuning further improves performance, supporting integration into scalable semantic radiology data pipelines. However, the present evidence comes from a single Italian center without external validation, and the fine-tuning gain was estimated on a small held-out test set, for which the Macro F1 improvement, though statistically significant, has a confidence interval with a lower bound close to zero. In this light, the approach is best positioned as an internal indexing and semantic-retrieval tool, supporting faster report retrieval, structured cohort identification, internal audit, and scalable phenotyping for neuroradiology research and data warehousing, as well as efficient active-learning strategies, with broader generalizability requiring additional external multicentre validation before clinical deployment.

PMID:
42875268
Bibliographic data and abstract were imported from PubMed on 15 Oct 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.