Evaluation of six generative large language models answering pre-analytical questions in medical microbiology.

Authors

Michelle Bressan, Eline Nadieh Meijer, Vladimira Hinic, Tilia Ellendorff, Jean-Philippe Goldman, Tannon Kew, Oliver Nolte, Gerold Schneider, Adrian Egli, LLM Evaluation Consortium

Published in

Clinical microbiology and infection : the official publication of the European Society of Clinical Microbiology and Infectious Diseases. Oct 13, 2026. Epub Oct 13, 2026.

Abstract

Large language models (LLMs) have the potential to be integrated into diagnostics workflows, yet their reliability in pre-analytical diagnostic stewardship remains uncertain. We aimed to evaluate the performance of six LLMs answering common pre-analytical questions in the domain of medical microbiology. Next, we propose a blueprint to evaluate LLMs in medical microbiology.
We compared general-purpose models from OpenAI (GPT-3.5, GPT-4) and Meta AI (Llama2-7B, Llama2-13B), alongside two domain-enhanced versions (customised GPT-4 agent and Llama2-7B Medical) in a hybrid evaluation framework. First, we quantified the response length (word count), then assessed the similarity between LLM-generated responses and expert-reviewed reference answers (BLEU/ROUGE scores, BERTScore). Finally, thirty-three medical professionals rated the accuracy and completeness of the LLM-generated responses on a 1-7-point scale.
Responses of OpenAI models showed greater similarity to the reference set and significantly outperformed Meta AI-based models in accuracy and completeness. The customized generative pre-trained transformer (GPT) matched performance of OpenAI's general-purpose models, while generating significantly shorter responses (p<0.001). Quality ratings varied markedly between professionals, as medical microbiologists and experienced infectious disease specialists assigned significantly lower scores compared to infectious disease trainees and medical students (p<0.001).
We observed considerable variation in LLM performance and highlight differences in perceived response quality across professional groups, emphasizing the need for critical appraisal when consulting LLMs for pre-analytical guidance.

PMID:
42869292
Bibliographic data and abstract were imported from PubMed on 14 Oct 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.