Mechanism-aware benchmarking of large language models as learning aids in pediatric difficult airway training: true hallucinations vs. false contraindications.

Authors

Merve Bulun Yediyıldız, İrem Durmuş

Published in

BMC medical education. Volume 26. Issue 1. Oct 01, 2026. Epub Oct 01, 2026.

Abstract

Large language models (LLMs) are being investigated for their potential applications in perioperative decision-making and as informal "just-in-time" learning tools for trainees. This study evaluated five LLMs in pediatric difficult-airway management using a mechanism-aware scoring approach that distinguishes true hallucinations from false contraindications and examines their suitability for supervised learning.
In a simulation study involving a comparative blinded evaluation, 50 cases of pediatric difficult airway management were presented to five LLMs using a standardized prompt based on the 2022 American Society of Anesthesiologists Difficult Airway Guidelines. Two anesthesiologists evaluated the responses on a modified 0-5 scale.
The inter-rater agreement was deemed to be of the highest caliber, as evidenced by a quadratic weighted κ value of 0.982. A statistically significant variation in overall performance was observed among the models (Friedman χ²(4) = 100.12, p < 0.001). In this single-pass evaluation, GPT-5.2 Thinking had the highest mean score and proportion of acceptable responses, whereas Claude Opus 4.5 had the highest combined critical-error rate (23.0% vs. 6.5% for both GPT models). True hallucinations were most frequent with Claude Opus 4.5 (11.8%; other models 0.5-2.2%), while false contraindications occurred across all models (5.0-13.0%) and clustered around rocuronium. Scenario inadequacy (score ≤ 2) increased with complexity: 32-98% for difficult intubation, 46-94% for difficult ventilation, and 52-92% for CICO.
The contemporary generation of LLMs exhibits model-specific performance characteristics and error patterns in pediatric difficult airway management. Dosing-related failures manifest through different mechanisms of true hallucination vs. false contraindication. The findings reveal a marked inadequacy of the scenarios employed in these models, particularly as complexity and critical error rates increase. Current LLMs may have value as supervised supplementary learning tools, but not as autonomous sources of clinical or educational guidance.

PMID:
42907955
Bibliographic data and abstract were imported from PubMed on 23 Oct 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.