Leveraging Large Language Models to Summarize Data and Free-Text Comments From Resident Assessments.

Authors

Carisa M Cooney, Rebecca Slattery, Damon S Cooney, Scott D Lifchez, Julie E Park

Published in

Journal of graduate medical education. Volume 18. Issue 4. Pages 477-482. Epub Aug 14, 2026.

Abstract

Summarizing resident assessments for Clinical Competency Committee or mentoring meetings is challenging. This especially applies to raters' free-text comments, which often provide valuable feedback.
To assess feasibility and accuracy of using a large language model (LLM) to label free-text comments as "Strengths" or "Areas for Improvement" from end-of-rotation assessments of residents.
We performed a mixed-methods study of residents' end-of-rotation assessments completed between July 1, 2024 and June 30, 2025. Following data de-identification, we used a private (paid) LLM account (ChatGPT-4o) to compile numerical and label free-text ("Strengths," "Areas for Improvement") feedback. We assessed feasibility using time spent de-identifying data, time for LLMs to return labeled free-text comments, and LLM recall (ability to accurately extract/label free-text feedback). We performed 2 confirmatory trials (free LLMs: MS CoPilot, ChatGPT-4) to ascertain reproducibility. Two subject matter experts labeled free-text comments to determine LLM labeling accuracy. The study was performed in 2025.
Our dataset represented 277 individual assessments for 25 unique residents. Feasibility data demonstrated data de-identification time of ∼60 minutes and LLM processing/labeling times of <7 minutes per LLM. All LLMs consistently compiled all free-text entries, accurately labeling 59.7% of free-text comments. Human analysis of comments (476) identified 388 (81.5%) "Strengths" and 88 (18.5%) "Areas for Improvement." Of these, LLM recall was 67.8% for "Strengths" and 23.9% for "Areas for Improvement."
We designed prompts that feasibly labeled end-of-rotation assessment free-text comments across 3 LLMs with recall for "Strengths" 2.5 times higher than recall for "Areas for Improvement."

PMID:
42602997
Bibliographic data and abstract were imported from PubMed on 15 Aug 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.