Development of a benchmarking dataset for symptom detection using large language models.

Authors

Joshua Davis, Brigitte N Durieux, Chloe Van Dongen, Kate Sciacca, Juan Manuel Gutierrez, Charlotta Lindvall

Published in

JAMIA open. Volume 9. Issue 4. Pages ooag134. Epub Jul 10, 2026.

Abstract

To develop a pipeline for evaluating large language models (LLMs) on the task of capturing symptoms from clinical encounters.
We created a gold standard dataset of symptom annotations from simulated doctor-patient encounter excerpts (264 encounters; 16 symptoms; double-coded and adjudicated). Nine different LLMs from 4 vendors (OpenAI, Meta, DeepSeek, Moonshot AI) were used as examples to test our evaluation pipeline; outputs were assessed for correct structure and symptom information.
Of 3085 excerpts, 2087 (68%) contained symptoms. Pain, cough, and shortness of breath were most common; LLMs achieved F1 scores ranging 0.66-0.88 for these symptoms with minimal prompt engineering. Of tested models, GPT-4.1 demonstrated the best overall performance.
Our evaluation pipeline and benchmarking dataset are publicly available and applicable to various LLMs, including open-source models.
This work supports the development and optimization of models that seek to improve patient symptom understanding.

PMID:
42436814
Bibliographic data and abstract were imported from PubMed on 12 Jul 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.