Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study.

Authors

Clara Frydman-Gani, Alejandro Arias, Maria Perez-Vallejo, John Daniel Londoño Martínez, Johanna Valencia-Echeverry, Mauricio Castano Ramírez, Alex A T Bui, Nelson B Freimer, Carlos Lopez-Jaramillo, Loes M Olde Loohuis

Published in

Journal of medical Internet research. Volume 28. Pages e93328. Oct 26, 2026. Epub Oct 26, 2026.

Abstract

The accurate detection of clinical symptoms in electronic health records (EHRs) is pivotal for advancing large-scale genetic and longitudinal studies in psychiatry. Free-text clinical notes are an essential source of symptom-level information, particularly in psychiatry; however, the automated extraction of symptoms from clinical text remains challenging.
We aimed to evaluate the ability of open-source generative large language models (LLMs) to detect psychiatric symptom-level concepts from clinical text in Spanish and to develop a new fine-tuned, open-source LLM that can be shared without risk of exposing patient information.
We tested 11 open-source generative LLMs for their ability to detect 109 psychiatric concepts from clinical text, using annotated EHR notes from a psychiatric hospital in Colombia. The LLMs were evaluated both "out-of-the-box" and after fine-tuning and compared against a traditional natural language processing (tNLP) method developed from the same data. To develop a fine-tuned LLM that can be shared without risk of exposing patient information, we created a novel synthetic dataset, leveraging real concept annotations while generating fully synthetic clinical text, and fine-tuned a top-performing LLM on these data. Fine-tuned model generalizability was assessed through cross-site evaluation using data from a second independent hospital in Colombia.
We show that while base LLM performance was poor to moderate (zero-shot: 0.2-0.6 macro-F₁; few-shot: 0.2-0.74 macro-F₁), it improved significantly after fine-tuning (0.75-0.86 macro-F₁), with several fine-tuned LLMs outperforming the tNLP method. In total, 100 psychiatric concepts could be reliably detected (F₁>0.8) using either a fine-tuned LLM or tNLP. Using the synthetic dataset, we developed Mistral-small-psych, a new LLM capable of detecting psychiatric concepts from Spanish clinical text with performance comparable to models trained on real EHR data (macro-F₁=0.79) while remaining fully distributable. In cross-site evaluations, most LLMs generalized well, with small out-of-domain performance losses (macro-F₁ decrease of 0.02-0.16).
This work provides the first large-scale, cross-institutional evaluation of state-of-the-art LLMs for psychiatric concept detection in Spanish EHRs, introduces a privacy-preserving pathway for building shareable clinical natural language processing models, and presents a new open model for extracting symptom-level psychiatric information from Spanish clinical text.

PMID:
42932872
Bibliographic data and abstract were imported from PubMed on 27 Oct 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.