Large Language Model-Generated Electronic Consultations: A Blinded Comparative Study of OpenEvidence and Gastroenterologist Electronic Consultations.

Authors

Christopher R Stephenson, Jithinraj Edakkanambeth Varayil, Bradley R Salonen, Elizabeth C Wight, Phil D Sang, Aditya K Ghosh, Dennis L Shung, Ivana T Croghan, Darrell R Schroeder, Ryan T Hurt

Published in

Mayo Clinic proceedings. Digital health. Volume 4. Issue 4. Pages 100404. Epub Sep 18, 2026.

Abstract

To evaluate whether a context-specific large language model, OpenEvidence, can generate gastroenterology/hepatology eConsult responses for metabolic dysfunction-associated steatotic liver disease (MASLD), while characterizing the frequency and potential severity of errors.
In this retrospective cohort study at a tertiary academic medical center, 125 specialist electronic consultations (eConsults) for MASLD (from January 2020 to December 2024) were paired with OpenEvidence-generated eConsults produced using OpenEvidence and standardized prompts. Five blinded physician reviewers evaluated all eConsults using a previously validated instrument. A board-certified gastroenterologist independently reviewed OpenEvidence-generated eConsults to identify errors and assess potential for harm.
OpenEvidence-generated eConsults received higher overall eConsult Specialist Quality of Response scores than specialist eConsults (mean difference, 6.4; 95% CI, 6.0-6.9; P<.001) and higher global quality ratings (odds ratio [OR], 16.1; 95% CI, 12.0-21.5; P<.001). OpenEvidence-generated eConsults were more likely to be rated as providing an appropriate amount of information (70.9% vs 33.9%; OR, 6.4; 95% CI, 4.8-8.5; P<.001). Of 125 OpenEvidence-generated eConsults, 12 contained minor errors and 1 contained a potentially major error.
In this blind comparison of paired eConsults for MASLD, OpenEvidence-generated responses were rated higher in perceived quality and informational adequacy than specialist-authored responses, but occasionally clinically relevant errors occurred. These findings support further evaluation of OpenEvidence-assisted eConsult workflows for protocol-driven conditions.

PMID:
42893001
Bibliographic data and abstract were imported from PubMed on 18 Oct 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.