Development and Evaluation of Large Language Model-Assisted Semi-Automated Data Harmonization Pipeline for the Multiple Chronic Disease Disparities Research Consortium: A Proof-of-Concept Study.

Authors

Hyelee Kim, Shuang Liang, Kathy Lanier, Sarit Helman, Nancy Fan Cheng, Niloufar Ameli, Jaysón Davidson, Shivani Mehta, Yulin Hswen, Catherine E Oldenburg, Kim Rhoads, Edwin D Charlebois, Stuart A Gansky, William Brown Iii

Published in

JMIR AI. Aug 31, 2026. Epub Aug 31, 2026.

Abstract

The increasing availability of machine-readable research data has created a growing need for efficient large-scale data harmonization (DH). Although large language models (LLMs) show promise for reducing the time and labor required for DH, their effective integration into harmonization workflows remains an important methodological challenge.
To develop and evaluate a semi-automated, human-in-the-loop (HITL) LLM-assisted DH workflow as a proof of concept across eight projects within a research consortium focused on disparities in multiple chronic conditions.
We developed an LLM-assisted DH workflow that combined preprocessing, semantic mapping, variable response mapping, synthetic data generation, data transformation, and iterative researcher review. Using GPT-4o hosted in a secure academic environment, project-specific survey items were mapped to 114 Consortium Common Data Element (CDE) semantic groups. Semantic mapping accuracy was evaluated based on the Consortium's consensus harmonization results. Complete and incomplete semantic mappings were distinguished according to construct overlap, and incomplete mappings requiring context-dependent human judgement were excluded for mapped item pairs and CDE semantic groups with and without HITL review.
Following preprocessing, 884 survey items were included in the DH workflow, of which 812 were mapped to a mean of 79 CDE semantic groups per project. Mean semantic mapping accuracy for item pairs was 95.9% with HITL review and 86.7% without HITL review. At the CDE semantic-group level, the corresponding accuracies were 98.9% and 94.9%, respectively. Across projects, omission of valid semantic mappings occurred more frequently than incorrect semantic mappings, particularly for heterogeneous response structures and subjective constructs measured using different instruments. Pipeline components-including concept-based preprocessing, iterative prompt refinement, and targeted HITL review-improved mapping completeness and supported variable harmonization across heterogeneous datasets.
This proof-of-concept study demonstrates that the effectiveness of LLM-assisted DH depends more on the design of a structured HITL workflow rather than on the LLM alone. Combining preprocessing, iterative verification, and targeted human oversight improved the efficiency and reliability of DH while preserving researcher judgment for complex mapping decisions.

PMID:
42680206
Bibliographic data and abstract were imported from PubMed on 02 Sep 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.