[Can ChatGPT Responses Be Improved by Incorporating S3 Clinical Practice Guidelines?].

Authors

Jannik Johannsen, Sofia Kourou, Malte Suchan, Christopher Kopp, Kariem Scharaf, Shachi Jenny Sharma, Hans Eckel, Julia van de Loo, Charlotte Klasen, Kevin Hansen, Marcel Mayer, Louis Jansen, Alissa Reisewitz, Anne Nobis, Jens Peter Klußmann, Arthur Charpentier

Published in

Laryngo- rhino- otologie. Sep 08, 2026. Epub Sep 08, 2026.

Abstract

Large Language Models such as ChatGPT are increasingly being discussed as tools to support medical decision-making. The aim of this study was to investigate whether the quality of responses generated by ChatGPT (GPT-4) to guideline-based ENT questions improves when the underlying evidence-based S3 guideline is provided as a PDF together with the respective question.
Thirty guideline-based questions were derived from five current S3 guidelines. ChatGPT answered each question in two runs: once without and once with provision of the respective guideline as a PDF together with the question. Three ENT specialists independently and anonymously evaluated the responses regarding content accuracy and length/conciseness (scale 1-3).
Providing the guideline as a PDF together with the respective question resulted in a significant improvement in accuracy (1.77 vs. 1.26; p = 0.003) and conciseness (1.96 vs. 1.50; p = 0.004). A significant correlation between accuracy and conciseness was observed (p < 0.001). Interrater reliability was fair to moderate.
The targeted provision of evidence-based guidelines as a PDF together with the respective guideline-based ENT question significantly improves response quality in ChatGPT. These findings highlight that the integration of medically sound primary sources is crucial for the accuracy and reliability of LLMs. LLMs therefore appear to be a potentially supportive tool in clinical and scientific assistance but do not replace medical expertise. In particular, potential "hallucinations" require continued critical medical and scientific oversight.

PMID:
42710499
Bibliographic data and abstract were imported from PubMed on 09 Sep 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.