Comparative performance of experienced general surgeons and large language models in inguinal hernia management: A domain-based benchmarking study.

Authors

Ethem Bilgiç, Erkan Karacan, Eyüp Murat Yılmaz

Published in

Journal of the Formosan Medical Association = Taiwan yi zhi. Nov 09, 2026. Epub Nov 09, 2026.

Abstract

Large language models (LLMs) are increasingly being explored as clinical decision-support tools; however, their performance across different levels of surgical complexity remains unclear. This study compared the performance of experienced general surgeons and contemporary LLMs in the management of inguinal hernia.
Twenty-five inguinal hernia-related clinical questions were developed based on current European Hernia Society and HerniaSurge recommendations and categorized into three domains: Basic Guideline-Based Management, Complex Clinical Decision-Making, and Complications & Emergency Management. Responses were obtained from 43 board-certified general surgeons and four LLMs (ChatGPT-5.3, Gemini-3.1, DeepSeek-v3.2, and Claude Sonnet 4.6). All responses were independently evaluated by three expert surgeons using a 5-point Likert scale. Group comparisons were performed using Friedman tests followed by Conover post hoc analyses with Holm correction.
No significant differences were observed in the Basic Guideline-Based Management domain (χ²(4) = 4.477, p = 0.345). Significant differences were identified in the Complex Clinical Decision-Making domain (χ²(4) = 18.073, p = 0.001) and the Complications & Emergency Management domain (χ²(4) = 11.108, p = 0.025). In both domains, Gemini-3.1 achieved the highest scores and scored significantly higher than human participants and ChatGPT-5.3 in selected pairwise comparisons. Overall performance differed significantly among groups (χ²(4) = 26.983, p < 0.001, Kendall's W = 0.270). Gemini-3.1 achieved the highest overall score (4.45 ± 0.51), followed by DeepSeek-v3.2 (4.05 ± 1.01), Claude Sonnet 4.6 (4.05 ± 0.99), human participants (3.85 ± 0.47), and ChatGPT-5.3 (3.36 ± 0.95). Post hoc analyses demonstrated significantly higher overall performance for Gemini-3.1 compared with both human participants and ChatGPT-5.3.
LLM performance in inguinal hernia management varied according to clinical complexity. While no significant differences were observed for guideline-based questions, performance differences emerged in complex and emergency-related scenarios. Gemini-3.1 achieved the highest and most consistent scores across the evaluated domains; however, these findings reflect performance within a standardized text-based benchmark and should not be interpreted as evidence of superior real-world clinical competence.

PMID:
43003711
Bibliographic data and abstract were imported from PubMed on 10 Nov 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.