Evaluating Locally Deployed Large Language Models for Ga-68 PSMA PET/CT Report Mining in Prostate Cancer.

Authors

Jagrati Chaudhary, Param Dev Sharma, Sanjay Kumar, Nikita Takalkar, Rakesh Kumar, Kunhi Parambath Haresh, Chandan J Das, Ranjit Kumar Sahoo, Seema Kaushal, M Kalaivani, Hemant Khairwa, Anil Kumar Pandey

Published in

Nuclear medicine and molecular imaging. Volume 60. Issue 4. Pages 382-394. Epub Mar 03, 2026.

Abstract

Large Language Models (LLMs) extracts structured data from unstructured Ga-68 Prostate Specific Membrane Antigen Positron Emission Tomography/Computed Tomography (PSMA PET/CT) reports by interpreting context and semantics, enabling reliable conversion of free-text into structured, queryable data. We evaluated whether locally deployed Llama 3.2:3b and Gemma 2:2b models could accurately perform zero-shot extraction of key diagnostic features.
We retrospectively selected 50 de-identified Ga-68 PSMA PET/CT reports of histologically confirmed prostate cancer (01 January 2020-30 June 2024), ensuring rigorous expert annotation and inter-reader agreement. Reports were batch-processed using standardized zero-shot and few-shot prompts applied to locally deployed Llama 3.2:3b and Gemma 2:2b models, with model outputs benchmarked against an adjudicated dual-expert ground truth reference derived from independent annotations and consensus resolution of discrepancies using standard performance metrics.
Both Llama 3.2:3b and Gemma 2:2b achieved rapid, reliable extraction of key diagnostic features from Ga-68 PSMA PET/CT reports, with excellent inter-reader agreement (average κ = 0.882). Llama 3.2:3b offered superior sensitivity (83.7%) and negative predictive value (NPV) (97.3%), while Gemma 2:2b demonstrated greater overall accuracy (86.2%) and specificity (88.5%). Model performance varied with question prevalence and complexity, highlighting distinct strengths for clinical integration.
Open-weight Llama 3.2:3b and Gemma 2:2b LLMs enable rapid extraction of key PET/CT findings (16-20x faster than manual review), with Llama excelling in sensitivity and Gemma in specificity. However, modest performance on rare but clinically important findings and limited positive predictive value indicate these models are best suited for initial screening or human-assisted workflows rather than autonomous extraction.
The online version contains supplementary material available at 10.1007/s13139-026-00991-z.

PMID:
42502653
Bibliographic data and abstract were imported from PubMed on 26 Jul 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.