SegJointGene: joint cell segmentation and spatial gene prioritization by information entropy guided convolutional neural networks

Authors

Ma, H., Wang, D.

Abstract

Spatial sequencing technologies enable the study of molecular organization, such as gene and protein expression, at single-cell resolution. Revealing such spatial patterns relies on accurate cell segmentation. Particularly, in complex tissues with dense cell packing, segmentation based solely on nuclear staining is insufficient for accurate cell boundary detection. This limitation arises because accurate segmentation necessitates the delineation of cell morphology, which is driven by molecular activities such as cytoskeletal dynamics, cell-cell adhesion, and intercellular signaling. Thus, integrating molecular information, such as gene or protein expression, is highly likely to enhance segmentation and understanding of molecular spatial organization, which, however, remains challenging. To address this, we developed SegJointGene, a novel deep learning framework that performs joint cell segmentation and spatial gene prioritization. SegJointGene inputs coarse, nuclei-based images and spatial gene (or protein) expression data, integrating them with an information-entropy guided convolutional neural network. Using such a specialized network, we further leveraged a computational information discarding (CID) score to prioritize gene importance for cell-type-specific segmentation. The model iteratively updates the gene scoring and refines cell segmentation, finally outputting both convergent cell segmentation and prioritized spatial genes across cell types. We applied SegJointGene with benchmarking to multiple real datasets, including spatial transcriptomics from the mouse hippocampus and distinct regions from whole mouse brain, and spatial proteomics from human tonsil. SegJointGene outperformed existing methods by 5-20%, in terms of the accuracy of assigning molecular spots to cell boundaries. Furthermore, we demonstrated the robustness of SegJointGene through various ablation tests, including varying gene numbers and imaging resolutions. Moreover, we found that the genes prioritized by SegJointGene uncovered cell-type-specific segmentations, many of which are structural and developmental markers, and were enriched with synaptic signaling pathways and anatomical organization, including glial cell populations.

Preprint server: bioRxiv
The authors list and abstract were imported from bioRxiv on 11 Jan 2026.

Stats

1-terrible, 9-excellent. How would you rate this preprint? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.