Development and Exploratory Evaluation of a Large Language Model-Supported Scoping Review Workflow.

Authors

Frederick H Kuo, Jamie L Fierstein, Madeline de la Concha, Yasmin Saeed, Ryan Shargo, Dina Ashour, Ernest Amankwah, Ellis Crabtree, Mohamed A Rehman, Peter Najjar, Khyzer Aziz, Brant H Tudor, Luis M Ahumada

Published in

Journal of medical systems. Volume 50. Issue 1. Oct 15, 2026. Epub Oct 15, 2026.

Abstract

Large language models (LLMs) have demonstrated human-level performance in literature review screening phases and have been evaluated in systematic-review workflows. However, LLM performance in scoping reviews, which are broader and more heterogeneous, remains understudied. We developed a modular LLM-supported framework for scoping-review workflows and compared the performance of two general-purpose LLMs against human reviewers on a reference scoping review. The reference review's keyword search results (n = 1336) and prompt sets mirroring workflow stages were provided to GPT-5 and gpt-oss-20b (OpenAI, August 2025). Outputs were benchmarked against human decisions for agreement; disagreements were adjudicated. For title-abstract screening and final-inclusion concordance, unanimous and adjudicated decisions were combined into a composite reference standard. In title-abstract screening, GPT-5 and gpt-oss-20b workflows showed substantial interrater agreement with human consensus (AC1: 0.769; 0.722); reference concordance was 90.9% (p = 0.08) and 92.4% (p = 0.004) versus humans' 89.0%. For final inclusion status, interrater agreement was almost-perfect (AC1: 0.948; 0.853); reference concordance was 95.4% (p = 0.23) and 92.4% (p < 0.001) versus humans' 96.1%. Across data categorization/synthesis, interrater agreement was fair to almost-perfect (AC1: 0.493-0.949; 0.342-0.929); total variation distances (representing distribution divergence) were 1.55%-28.0%. All sources found comparably low definition-adherent use rates of the term "human digital twin" (humans 12.1%; GPT-5 13.0%; gpt-oss-20b 18.0%). Both LLM-supported workflows achieved fair to almost-perfect agreement and high screening/final-inclusion reference concordance. The main synthesis finding was directionally consistent across workflows, with all sources identifying infrequent correct use of "human digital twin"; total variation distance may be useful in assessing distribution-level concordance. Limitations include a single domain/vendor, fixed prompts, and predefined taxonomies; broader evaluations are warranted. In this single-domain benchmark, LLM-supported scoping-review workflows showed feasibility for supporting and potentially streamlining selected review stages, with broadly comparable performance on evaluated endpoints and conclusion-level concordance. Generalizability, reproducibility, and operational efficiency beyond this benchmark remain to be established.

PMID:
42878908
Bibliographic data and abstract were imported from PubMed on 16 Oct 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.