New Large Language Models Outperform Older Models and Hallucinate Less in Operative Note Data Extraction.

Authors

Kellen L Mulford, Linjun Yang, Derrick W Williams, Amirali Khosravi, Minh Le, Marisa N Ulrich, Hilal Maradit Kremers, Cody C Wyles

Published in

The Journal of arthroplasty. Nov 12, 2026. Epub Nov 12, 2026.

Abstract

Recent improvements in large language models (LLMs) may provide a scalable solution for registry data curation. However, trust in model outputs remains limited by hallucinations, wherein models invent information. Given the rapid pace of LLM development, the purpose of this study was to compare the accuracy and hallucination rates of a "newer" and an "older" and previously evaluated LLM in extracting total hip arthroplasty (THA) operative variables.
Surgical approach, fixation technique, and bearing surface were manually abstracted from 1,000 primary THA operative notes from 47 different surgeons. Using 25% of the dataset, we iteratively developed prompts to query two locally hosted LLMs (one released in 2024, the older model, and the other released in 2026, the newer model) to extract those three data points. We tested the extraction pipeline for the best prompts on the remaining 75% of the dataset. We calculated performance metrics and manually assessed 100 sample output reasonings from each model for hallucinations.
Overall Performance: The newer model exhibited higher accuracy than the older model across all three data extraction tasks: surgical approach (98 versus 96%), fixation technique (98 versus 96%), and bearing surface (93 versus 89%). For each task, the highest-performing prompt technique for the newer model was comparatively simpler than the highest-performing prompt technique for the older model. Hallucination Rate Analyses: Across all evaluated extraction domains, the newer model demonstrated significantly lower hallucination rates compared with the older model. Hallucination rates for surgical approach, fixation technique, and bearing surface extraction were 5, 3, and 11% for the newer model versus 49, 20, and 49% for the older model, respectively.
The newer LLM extracted THA operative variables more accurately and hallucinated substantially less with simpler prompts than the 2-year-old model. These findings suggest LLMs are improving their ability to follow directions and reason without inventing information. However, lower rates of hallucinations persist, limiting trustworthiness.

PMID:
43020159
Bibliographic data and abstract were imported from PubMed on 13 Nov 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.