Authors
Xieji Li, Siyuan Yan, Yingsheng Liu, H Peter Soyer, Monika Janda, Victoria Mar, Zongyuan Ge
Published in
IEEE transactions on medical imaging. Volume PP. Aug 17, 2026. Epub Aug 17, 2026.
Abstract
Vision-language pretraining (VLP) has emerged as a powerful paradigm in medical image analysis, enabling representation learning from large-scale image-text pairs without relying on expensive expertise annotations. However, these challenges are particularly pronounced when VLP is applied to dermatology. The inherent noise and sparsity of web-crawled data hinder effective pretraining, while multifaceted dermatological knowledge is challenging to exploit from long, unstructured descriptions for pretraining optimization. To address these challenges, we propose a dermatological VLP framework integrating a Multi-Agent data GENeration (MAGEN) system and Ontology-based Multi-Aspect Knowledge-Enhanced (O-MAKE) pretraining. First, MAGEN enhances data quality by synthesizing knowledge-enriched descriptions via a foundation model-assisted captioning and retrieval-based verification pipeline. Second, O-MAKE addresses the difficulty of learning from long, unstructured texts by decomposing them into distinct knowledge aspects. This facilitates fine-grained alignment at both global and patch levels, while explicitly modeling medical concept relationships through ontology-guided mechanisms. Comprehensive experiments demonstrate the effectiveness of each component, achieving state-of-the-art zero-shot performance on disease classification and cross-modal retrieval tasks across eight dermatological datasets.Our code and the augmented dataset Derm1M-AgentAug, comprising over 400K skin image-text pairs, are publicly available at https://github.com/XiejiLi/MAGEN-O-MAKE.
PMID:
42606967
Bibliographic data and abstract were imported from PubMed on 18 Aug 2026.
Read full publication at:
Please sign in
to see all details.
Advertisement
Stats
- Recommendations n/a n/a positive of 0 vote(s)
- Views 9
- Comments 0