When the Chatbot Leaks: Securing Patient-Facing Medical AI in the Age of Dual-Use Large Language Models.

Authors

Alfredo Madrid-García, Beatriz Merino-Barbancho, Miguel Rujas

Published in

NEJM AI. Volume 3. Issue 10. Pages AIp2600583. Epub Aug 17, 2026.

Abstract

AbstractPatient-facing chatbots based on retrieval-augmented generation (RAG) are increasingly promoted to deliver curated, evidence-based health information. However, RAG is not a security boundary; a chatbot providing clinically accurate responses can still expose its instructions, knowledge base, and stored patient conversations through ordinary deployment errors. In an anonymized assessment of a publicly accessible patient-facing RAG chatbot, we used only a standard web browser, its built-in developer tools, and a commercial AI assistant to retrieve the system prompt, the full pipeline configuration, the knowledge base, and 1000 stored patient conversations without authentication or any specialized tool. The same dual-use capabilities that lower the barrier to building patient-facing AI now also lower the barrier to probing it. The health AI deployment process must extend beyond solely evaluating the model. It must evaluate the solution as a secure software system.

PMID:
42932700
Bibliographic data and abstract were imported from PubMed on 27 Oct 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.