Artificial Intelligence in Systematic Reviews and Meta-Analyses: Task-Specific Performance, Residual Error Quantification, and Human Oversight.

Authors

Jules Descamps, Nicolas Bouguennec, Raphael Porcher, Matthieu Resche-Rigon, Pierre-Alban Bouché

Published in

Orthopaedics & traumatology, surgery & research : OTSR. Pages 104793. Jul 16, 2026. Epub Jul 16, 2026.

Abstract

The volume of published research has expanded rapidly, intensifying the need for reliable, regularly updated evidence syntheses. Systematic reviews (SRs) and meta-analyses (MAs) remain the highest level of evidence, but their completion is time-consuming and resource-intensive, often extending beyond one year. Artificial intelligence (AI) is increasingly applied to automate stages of this workflow, yet validation of performance and methodological rigor remains limited. This narrative review addresses three questions: 1) at which stages of the SR/MA workflow can AI be used reliably; 2) how should AI performance be benchmarked for each task; and 3) how can the residual error remaining after human checking be quantified.
We outline the principles and key steps of a traditional meta-analysis and examine the contribution of AI at each stage. Literature was drawn from major biomedical databases; no systematic search was performed. AI performance is summarized by task, with benchmarking and residual-error quantification considered separately.
Across the workflow, AI maturity is heterogeneous. In a 2025 systematic review of 19 comparative studies, generative AI missed 68.0% to 96.0% of relevant studies in searching, made incorrect inclusion decisions in 0.0% to 29.0% and incorrect exclusions in 1.0% to 83.0%, incorrect data extractions in 4.0% to 31.0%, and incorrect risk-of-bias assessments in 10.0% to 56.0%. A 2025 scoping review identified 37 articles: 15/37 (41%) searching, 14/37 (38%) study selection, 11/37 (30%) extraction, 33/37 (89%) GPT-based, 21/37 (57%) validation studies; 20/37 (54%) drew a promising conclusion, none as a validated implementation. Title/abstract screening was most mature (sensitivity 99.2%, specificity 83.6%) versus full-text screening (97.6% and 47.4%). For structured extraction across 30 articles, one tool reached 92.0% precision, recall, and F1, but sensitivity fell to 77.0% to 80.0% for review-specific variables, with confabulations in 4/90 (4%) of data points; across 107 trials, extraction accuracy was 96.2% (97.9% with refinement) while mean time fell from 86.9 to 14.7 minutes per trial. Risk-of-bias assessment was most fragile (kappa 0.51, 95% CI: 0.36-0.66). For residual error, 0 errors among 60 checked items corresponds to an upper one-sided 95% bound of 4.9%.
AI is transforming SR/MA toward hybrid human-machine workflows rather than replacing them. Performance is task-specific, and metrics alone are insufficient: residual uncertainty should be quantified. Current evidence supports AI as an assistive technology under human oversight, not autonomous use.
lV; narrative review.

PMID:
42462811
Bibliographic data and abstract were imported from PubMed on 17 Jul 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.