A focused PubMed artificial intelligence search program creates better literature reviews than general purpose artificial intelligence and is comparable to trainees in shoulder and elbow surgery.

Authors

James S MacLeod, Michael S Lee, Hayden Baker, Jordan Serotte, Lewis L Shi, Nicholas H Maassen

Published in

JSES international. Volume 10. Issue 6. Pages 101790. Epub Sep 04, 2026.

Abstract

As the use of artificial intelligence (AI) expands into orthopedic surgery, the potential to use AI for review and summation of literature leaves researchers and clinicians questioning the quality of AI-driven literature reviews. This study aims to compare AI-generated literature reviews to those done by orthopedic surgery trainees.
ChatGPT-4o and The-Literature, a directed AI platform, were asked to produce a literature review on 5 shoulder and elbow-related research topics. The primary output produced by ChatGPT-4, The-Literature, a fourth-year medical student, a third-year orthopedic surgery resident, and an orthopedic surgery sports medicine fellow were blinded then graded on a 1-5 Likert Scale for evidence and analysis, conclusion, references, and overall quality by 2 fellowship-trained orthopedic shoulder and elbow surgeons.
The average overall Likert ratings were 2.90 ± 0.88 for the medical student, 3.90 ± 0.74 for the resident, 3.50 ± 0.53 for the fellow, 2.30 ± 1.16 for ChatGPT-4, and 3.40 ± 0.84 for The-Literature. The resident and fellow reviews were significantly better than ChatGPT-4 (P = .0004, P = .0209), while the medical student review was not (P = .5092). None of the trainee reviews were significantly better than The-Literature. The-Literature's reviews were significantly better than ChatGPT-4 (P = .0412). The majority (58.8%) of ChatGPT-4's references were fabricated.
A limited and directed AI search program was comparable to orthopedic surgery trainees and performed better than an open AI large language model. ChatGPT-4 had poor overall utility stemming mainly from its tendency to fabricate references.

PMID:
42893063
Bibliographic data and abstract were imported from PubMed on 18 Oct 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.