Lightweight open-source reasoning large language model for evidence synthesis screening.

Authors

David M Dávila-García, Amanda Gilbert, Adam B Wilcox

Published in

JAMIA open. Volume 9. Issue 5. Pages ooag210. Epub Oct 23, 2026.

Abstract

To evaluate performance and efficiency of a lightweight open-source reasoning large language model (LLM) against an expert-derived reference standard for screening scientific publication databases.
Two expert reviewers screened subsets of database records across 3 stages: (1) base eligibility (n = 200), (2) obesity focus (n = 200), and (3) women of reproductive age focus (n = 150). The LLM (gpt-oss-20b) screened all records (n = 20 118). Performance (F1-score and sensitivity [recall]) and efficiency (average screening rate) were assessed.
Interreader agreement was near-perfect ( $\kappa$ : 0.99-1.00; all P < .001). LLM performance was high: F1-score, 0.83-0.94; sensitivity, 90%-95%; specificity, 86%-99%; accuracy, 89%-94%; precision, 77%-99%; and negative predictive value (NPV), 89%-96%. LLM screening was faster than reviewers (19.4 vs ∼1.0 records/min).
The LLM achieved high performance and efficiency relative to an expert-derived reference standard for screening.
The use of lightweight open-source reasoning LLMs shows promise to support screening in evidence synthesis workflows.

PMID:
42920447
Bibliographic data and abstract were imported from PubMed on 24 Oct 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.