Streamlining the Abstract Selection Process for the American Society of Breast Surgeons' Annual Meeting by Utilizing Large Language Models.

Authors

Lauren N Cohen, Drake Seibert, Thomas Kosten, Nathan Wolfrath, Zahraa Al-Hilli, Olga Kantor, Puneet Singh, Mediget Teshome, Sarah Blair, Tina J Hieken, Lee G Wilke, Anai Kothari, Chandler S Cortina

Published in

Annals of surgical oncology. Jul 06, 2026. Epub Jul 06, 2026.

Abstract

The American Society of Breast Surgeons (ASBrS) receives an increasing number of abstracts for its annual meeting, resulting in increasing time and effort for peer-review. We aimed to compare artificial intelligence (AI) large language models (LLMs) to human-generated scores to explore if AI-review is a plausible abstract screening process.
Abstracts published from the 2025 ASBrS meeting were assessed with three LLMs: OpenAI's GPT4-o (GPT), Meta's Llama-3.1-405b (Llama) and DeepSeek's DeepSeek-V3 (DeepSeek). Abstracts were analyzed using zero shot (ZS), which included the 2025 ASBrS scoring rubric and few shot (FS) with examples of high- and low-scoring abstracts. Large language model scores were compared with the committee's scores. Model inference was evaluated by using Spearman rank correlation, quartile accuracy, and mean absolute error; 95% CIs were calculated by using bootstrap resampling.
A total of 378 published accepted abstracts were included. Human-generated median abstract score was 21.2 (rubric ranged 0-35). Median scores were higher amongst all LLMs for both the ZS and FS approaches (27.0-29.7, p < 0.001). Spearman values with the ZS approach for GPT, DeepSeek, and Llama were 0.24, 0.30, and 0.31, respectively, and for the FS approach were 0.37, 0.35, and 0.27, respectively (p < 0.001 for all). DeepSeek ranked the first quartile correctly with 53.5% accuracy while Llama ranked the fourth quartile correctly with 42.9% accuracy. The FS approach modestly improved correlation and first quartile accuracy, with greatest improvement seen in GPT.
Large language models showed statistically significant rank correlation with human ASBrS abstract ranking. Large language models may be useful as a preliminary screening tool to prioritize reviewer effort toward higher-quality abstracts.

PMID:
42406210
Bibliographic data and abstract were imported from PubMed on 06 Jul 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.