A biomedical BERT ensemble (TITAN-SR) outperformed active learning and LLM chatbots for systematic-review title and abstract screening: development and external validation.

Authors

Tyler Pitre, Dena Zeraatkar, John Granton, Jason Busse, Bram Rochwerg, Gordon Guyatt

Published in

Journal of clinical epidemiology. Pages 112514. Sep 21, 2026. Epub Sep 21, 2026.

Abstract

Machine learning tools for literature screening have existed for a decade, yet adoption remains limited: active learning needs hundreds of decisions before prioritizing well, and commercial large language model (LLM) chatbots are used without task-specific training. We aimed to develop TITAN-SR (Training Infrastructure for Automated Nomination in Systematic Reviews), a screening tool built on biomedical Bidirectional Encoder Representations from Transformers (BERT) models, and to compare it with the active-learning tool ASReview and commercial LLM chatbots.
We assembled 762,934 citation-label pairs from 19,787 completed reviews and trained TITAN-SR on a prespecified review-level split, testing on held-out reviews. The tool combines biomedical BERT models (PubMedBERT and BioLinkBERT) that read a review's eligibility criteria with each citation's title and abstract; training penalized false negatives 10 times more heavily than false positives. We compared TITAN-SR with four chatbots used zero-shot on a stratified 500-review benchmark (Claude Sonnet 4, GPT-4o, DeepSeek-V3, Gemini 2.0 Flash), with GPT-4o-mini on all 3,434 evaluable test reviews, and with ASReview on 22 temporally independent Cochrane reviews (142,504 records); ASReview received a warm-up of 20% of known includes.
Across 3,434 test reviews, TITAN-SR achieved a median specificity at 99% recall of 0.902 and median area under the receiver operating characteristic curve (AUC) of 0.974. It outperformed all four chatbots (Bonferroni-corrected paired Wilcoxon p ≤ 5 × 10^-31) and GPT-4o-mini. The chatbots showed calibration collapse: 82-99% of confidence scores were ≥ 0.9 regardless of accuracy. On the 22 external reviews performance was virtually identical (median AUC 0.976; specificity at 95% recall 0.897), and at a threshold fixed on the internal validation split TITAN-SR met the pre-registered primary criterion, retaining 99.8% of included studies (95% confidence interval [CI] 99.5 to 100.0%). It outperformed ASReview on 18 of 22 reviews (median paired difference in specificity at 95% recall +0.145, 95% CI +0.101 to +0.201).
A purpose-built screening tool based on biomedical BERT models, developed from nearly 20,000 completed reviews, outperformed ASReview and commercial LLM chatbots evaluated without task-specific training. Temporal validation on 22 held-out Cochrane reviews confirmed generalization to reviews published after the model-development period.

PMID:
42767575
Bibliographic data and abstract were imported from PubMed on 22 Sep 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.