A medically grounded LLM agent-based tool to detect patient safety events in medical records.

Authors

Diego Trujillo, Dulin Wang, Nathan Bahr, Tina Yi-Jin Hsieh, Byeongyeon Cho, Garth Meckler, Matthew Hansen, Carl Eriksson, Kyu Seo Kim, Steven Bedrick, Xiaoqian Jiang, Jeanne-Marie Guise

Published in

PLOS digital health. Volume 5. Issue 9. Pages e0001174. Epub Sep 17, 2026.

Abstract

Large language models (LLMs) have shown incredible promise in medicine. While LLMs may be particularly useful in areas requiring extensive review of clinical records, their use remains limited due to their tendency to hallucinate and fabricate information. Hallucination issues, as well as their consequences, are exacerbated in low-probability, high-stakes scenarios such as rare adverse safety events or medical errors. We present SAFE-AI (Structured and Automated Framework for Explainable AI), a novel method for clinical decision making that combines the strengths of clinical expert knowledge with LLMs in an ontology-driven model that minimizes hallucinations using strict rules. We test this method to identify medication errors in medical charts. We collected a sample of 18,402 lines of clinical information from 300 EMS clinical charts that were independently dually reviewed by two expert physicians for epinephrine adverse safety events (ASEs), with 96% inter-rater agreement. We tested SAFE-AI against these labels, achieving similar performance to human experts in detecting epinephrine overdoses with 97.9% accuracy, and 91.6% accuracy in identifying delays in epinephrine administration, greatly outperforming baseline LLMs models. Notably, some disagreements between clinicians and the model were found to be justifiable differences in judgment rather than errors. SAFE-AI presents a novel approach for clinical AI applications that addresses two key limitations of current machine learning methods: 1) over-reliance on probabilistic pattern recognition instead of established medical knowledge, and 2) perpetuation of biases present in training data. This framework is easily adaptable to a range of clinical applications, paving the way for provable and trustworthy AI in medicine.

PMID:
42752617
Bibliographic data and abstract were imported from PubMed on 18 Sep 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.