BioMamba: Domain-Adaptive Biomedical Language Models.

Authors

Ling Yue, Mingzhi Zhu, Sixue Xing, Yunning Cao, Yanbo Wang, Vijil Chenthamarakshan, Shaowu Pan, Payel Das, Tianfan Fu

Published in

Health data science. Volume 6. Pages 0454. Epub Oct 05, 2026.

Abstract

Background: Biomedical language models should improve performance on biomedical text while retaining general-language-modeling fluency. For Mamba-based models, this trade-off has not been systematically studied across biomedical literature and clinical text. Methods: We developed BioMamba, a family of biomedical Mamba2 models at 5 scales (130M, 370M, 780M, 1.3B, and 2.7B) obtained by continued pretraining of released public Mamba2 checkpoints on a balanced 80%/10%/10% mixture of PubMed abstracts, the Colossal Clean Crawled Corpus (C4), and Wikipedia. The architecture is unchanged from Mamba2; the contribution is the adaptation recipe and the accompanying open-weight checkpoints. We evaluated internal language modeling on fixed held-out sets, out-of-domain multiple-choice benchmarks, and 3 downstream tasks across multiple model scales: clinical note completion and discharge summary generation on MIMIC-IV-Note, and biomedical yes/no question answering on Biomedical Semantic Indexing and Question Answering (BioASQ) and PubMed Question Answering (PubMedQA). Results: Across 5 scales, BioMamba consistently lowered PubMed perplexity, improved Wikipedia-style held-out perplexity by 1.46 to 4.72 PPL, and left C4 perplexity essentially unchanged ( $\mid \Delta PPL\mid \le 0.17$ ). On 6 out-of-domain multiple-choice benchmarks, BioMamba stayed within ±3 pp of Mamba2 with no systematic regression. After supervised fine-tuning, BioMamba+SFT matched or exceeded Mamba2+SFT on MIMIC-IV note completion and discharge summary generation at every evaluated scale (paired-bootstrap $P\left(\Delta >0\right)\ge 0.997$ at 2.7B) and improved PubMedQA at every scale. The strongest model (BioMamba-2.7B) reached a PubMed perplexity of 5.28 and accuracies of 90.24% and 73.00% on BioASQ and PubMedQA, respectively. Conclusions: A balanced domain-adaptive continued pretraining recipe strengthens Mamba2 language models on biomedical literature and clinical text while preserving general-language-modeling fluency.

PMID:
42835900
Bibliographic data and abstract were imported from PubMed on 06 Oct 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.