Safety, accuracy, information quality, readability, and supportive communication of LLM-based chatbots in responses to researcher-developed consumer-oriented prompts on recurrent aphthous stomatitis: an exploratory cross-sectional comparative study.

Authors

Yulong Wang, Jingjing Li, Zhaole Gong, Yan Zhuang, Wu Dong

Published in

Frontiers in oral health. Volume 7. Pages 1968034. Epub Sep 30, 2026.

Abstract

Recurrent aphthous stomatitis (RAS), commonly known as oral ulcers, is a frequent and painful oral mucosal condition that often prompts people to seek consumer health information before professional consultation. Although LLM-generated health advice has been studied in several clinical fields, multidimensional evidence focused specifically on RAS remains limited.
To compare the safety, accuracy, supportive communication/empathy, information quality and transparency, and readability of five publicly accessible LLM-based chatbots when responding to researcher-developed consumer-oriented prompts about RAS.
Thirty researcher-developed, consumer-oriented English prompts covering seven RAS domains were submitted once, in separate sessions, to ChatGPT, Copilot, Gemini, Doubao, and DeepSeek, yielding 150 single-turn responses. Safety was the primary clinical outcome; accuracy, supportive communication/empathy, and information-quality measures were secondary, and six formula-based readability indices were exploratory. Five reviewers scored each response independently before consensus adjudication. User-facing product labels, access conditions, prompts, reference checklists, complete responses, and statistical outputs are documented in the manuscript and supplementary materials.
Sixteen of 150 responses (10.7%) were consensus-coded as potentially unsafe or misleading. Safe-response rates ranged from 83.3% to 93.3%, with no detected between-chatbot difference (Cochran's Q = 2.957, P = 0.565). Potential safety concerns were observed in persistent or non-healing ulcers, cancer-related reassurance, home or traditional remedies, supplementation, pregnancy, and pediatric scenarios. No between-chatbot differences were detected for accuracy, supportive communication/empathy, DISCERN, EQIP, JAMA, or GQS. Five of six readability formulas detected differences, but only 31 of 150 responses (20.7%) met the prespecified FKGL target of grade 6 or below.
In this exploratory dataset, no between-chatbot differences were detected for safety, accuracy, supportive communication/empathy, or information-quality scores, whereas formula-dependent readability differences were observed. Non-significant findings do not establish equivalence, and readability formulas do not establish comprehension. The results apply only to the recorded user-facing configurations and single responses to 30 researcher-developed English prompts during the defined access period; they do not demonstrate readiness for autonomous clinical or patient-facing deployment.

PMID:
42876519
Bibliographic data and abstract were imported from PubMed on 15 Oct 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.