Authors
Sare Safi, Amir Golmakani, Saeed Rahmani, Nikoo Bayan, Seyed Mohamadmehdi Moshtaghion, Hadis Najafimehr, Haleh Kangari
Published in
Health informatics journal. Volume 32. Issue 3. Pages 14604582261470624. Epub Jul 20, 2026.
Abstract
PurposeTo assess the accuracy, comprehensiveness, and reliability of Large Language Model chatbots in answering frequently asked questions about refractive errors.MethodsForty-four questions about refractive errors were posed to four chatbots, including Copilot, Perplexity, Gemini, and ChatGPT. Responses to each question were independently evaluated by three experts using a three-point accuracy scale. The readability of the chatbots' responses was evaluated using several indices. Similarity was assessed using Sentence-Bidirectional Encoder Representations from Transformers (SBERT). Inter-rater agreement among the graders was evaluated using the Gwet Agreement Coefficient 1 (AC1) statistic.ResultsThe overall agreement among the graders for all chatbot responses was almost perfect (Gwet AC1: 0.87). All chatbots received scores above 85% in every category, and there was no significant difference in chatbot accuracy scores (P= 0.168). The highest mean comprehensiveness score was observed for Perplexity (8.13 ± 0.95, P = 0.028). The similarity scores of the chatbots were very close to each other. All readability scores showed significant differences (P<0.001) between the chatbots.ConclusionAll four chatbots showed comparable accuracy in answering questions about refractive errors. Readability levels of all chatbots exceeded public health recommended thresholds; however, ChatGPT produced relatively more accessible output.
PMID:
42474254
Bibliographic data and abstract were imported from PubMed on 20 Jul 2026.
Read full publication at:
Please sign in
to see all details.
Advertisement
Stats
- Recommendations n/a n/a positive of 0 vote(s)
- Views 8
- Comments 0