Development and Validation of a ChatGPT-Based Assessment for Conversation-Related Clinical Skills in Occupational Therapy OSCEs.

Authors

Yi-Ching Wang, An-Ju Chen, Mon-Jane How, Chih-Wei Yang, Ya-Chen Lee, Sheau-Ling Huang, Ching-Lin Hsieh

Published in

The American journal of occupational therapy : official publication of the American Occupational Therapy Association. Pages 19437676261500028. Nov 03, 2026. Epub Nov 03, 2026.

Abstract

ImportanceConversation-related clinical skills are central to Objective Structured Clinical Examinations (OSCEs) in occupational therapy. However, real-time examiner scoring is resource-intensive and may overlook students' specific skills. Transcript-based scoring with Chat Generative Pre-Trained Transformer (ChatGPT) may address these limitations.ObjectiveTo develop a ChatGPT-based assessment for chief-complaint interview OSCEs (G-ACE) and examine criterion validity against expert-consensus ratings, test-retest reliability, and inter-rater reliability.DesignSecondary analysis of de-identified transcripts from 8-min chief-complaint interviews with a simulated patient with stroke during an OSCE.SettingsA university in northern Taiwan.ParticipantsFifty-eight de-identified transcripts from undergraduate occupational therapy students.ResultsCriterion validity was acceptable for "Primary Symptoms and Related Feelings" and "Impact on Daily Life" items (weighted kappa ≥ 0.60), but lower for "Rehabilitation Expectations" and "Communication Skills" items (0.18-0.59). For domain and total scores, intraclass correlation coefficients (ICCs) were 0.17 to 0.78 and percentages of mean absolute errors (MAE%s) were 4.6% to 20.7%. Test-retest reliability was acceptable at the item level (weighted kappa = 0.68-1.00), and acceptable to good at the domain and total score levels (ICC = 0.65-0.85; MAE% = 2.3%-9.1%). Inter-rater reliability was acceptable for most domain and total scores (ICCs = 0.52-0.78; MAE%s = 1.3%-12.0%), except for the "Rehabilitation Expectations" domain (ICCs = 0.17-0.30; MAE%s = 1.3%-7.3%).Conclusion and RelevanceThe G-ACE demonstrated generally acceptable test-retest reliability and small absolute differences from expert ratings for domain and total scores. Because some items showed lower consistency with expert ratings, formative feedback should be interpreted cautiously based on transcript evidence.

PMID:
42974814
Bibliographic data and abstract were imported from PubMed on 03 Nov 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.