Authors
Hirday Josan, Sina Sayyad, Madeleine Zanin, Daniel Sutton, Tanvi Vakil, Suhrob Mullo Mirzo, Karanbir Sandu, Christopher Meaney, Fok-Han Leung
Published in
Medical science educator. Volume 36. Issue 6. Pages 3169-3177. Epub Sep 05, 2026.
Abstract
Developing high-quality Objective Structured Clinical Examination (OSCE) standardized patient (SP) scripts is time and resource-intensive. Large language models such as GPT-4 may improve script writing, however, their educational adequacy and realism compared with expert-written scripts remain uncertain.
We compared GPT-4 generated OSCE scripts to expert-written scripts across five educational quality domains: detectability of AI authorship, perceived realism, consistency of role depiction, clinical accuracy, and alignment with learning objectives. A secondary exploratory non-inferiority analysis was also conducted.
We developed fifteen OSCE SP scripts using three approaches, expert-written, AI-generated, and AI-generated with human editing, with five scripts per group. Scripts were rated by blinded SPs and physicians. We assessed whether AI-generated scripts were non-inferior to expert-written scripts within 10-percentage-points.
Raters could not reliably distinguish either AI-generated script group from expert-authored scripts. For realism, means were 3.90 (SD 0.79) for expert-written, 3.88 (1.02) for AI-generated, and 3.56 (1.07) for AI+human scripts. For clinical accuracy, means were 4.27 (0.70), 4.07 (0.96), and 4.40 (0.63). For alignment with learning objectives, they were 4.27 (0.80), 3.67 (1.11), and 4.27 (0.80). AI+human scripts were rated highest for clinical accuracy and alignment with learning objectives, and expert-written scripts highest for realism and consistency of role depiction. In the exploratory non-inferiority analysis, non-inferiority to expert-written scripts was not demonstrated for either AI-generated set.
This pilot study provides preliminary evidence on the comparability of GPT-4-generated and expert-written OSCE scripts, with similar ratings across several educational quality domains, though non-inferiority was not demonstrated. Larger, more rigorous studies are needed to conclusively demonstrate AI generated OSCE scripts are non-inferior when compared to human expert generated OSCE scripts.
The online version contains supplementary material available at https://doi.org/10.1007/s40670-026-02877-3.
PMID:
43031789
Bibliographic data and abstract were imported from PubMed on 14 Nov 2026.
Read full publication at:
Please sign in
to see all details.
Advertisement
Stats
- Recommendations n/a n/a positive of 0 vote(s)
- Views 7
- Comments 0