Dual-view fusion of log-mel spectrogram encodings for dysarthria detection: use in the context of awake brain surgery.

Authors

Nassib Abdallah, Harrison Misy, Jean-Marie Marion, Chinmayi Kanthila, Celine Panheleux, Vanessa Saliou, Romuald Seizeur, Guillaume Dardenne

Published in

International journal of computer assisted radiology and surgery. Jul 10, 2026. Epub Jul 10, 2026.

Abstract

Intraoperative speech monitoring during awake glioma surgery is critical to detect stimulation-induced motor speech impairments such as dysarthria. However, automatic detection from short speech segments remains challenging due to limited annotations, strong acoustic variability, and transient impairments in operating-room conditions. This study investigates whether combining complementary encodings of the same log-mel representation can improve robustness under such constraints.
We propose a dual-branch EfficientNet-B0 architecture that processes two complementary views of each log-mel spectrogram: (i) a numeric amplitude-preserving representation stored as NPY, and (ii) an RGB spectrogram image stored as PNG using a fixed colormap. The resulting embeddings are fused using four strategies: naïve concatenation, attention-based fusion, and gated fusion (sigmoid and GRU-style). Models are evaluated under two settings: (i) intra-corpus group-wise cross-validation on TORGO (English, controlled) and DATABRASE (French, intraoperative), and (ii) cross-corpus transfer (TORGO $\to$ DATABRASE) to quantify domain shift. In addition, we evaluate DATABRASE with and without Demucs-based denoising to analyze the impact of operating-room noise.
In intra-corpus evaluation, adaptive fusion strategies provide consistent benefits on TORGO, with gated fusion achieving the highest AUC (0.852±0.087). On raw DATABRASE, the PNG unimodal baseline yields the best mean performance (UAR 0.830±0.132; AUC 0.889±0.109), whereas denoising altered the model ranking and improved the performance of attention-based fusion (UAR 0.854±0.106). Cross-corpus testing reveals a marked degradation for all methods, confirming a severe domain shift between TORGO and intraoperative speech.
Dual-encoding fusion can enhance dysarthria detection when training and testing conditions are aligned and noise is controlled, but robust cross-corpus generalization remains unresolved. These results highlight both the potential and current limitations of spectrogram-based deep models for intraoperative speech monitoring, motivating future work on domain-adaptive and self-supervised learning tailored to clinical recordings.

PMID:
42426485
Bibliographic data and abstract were imported from PubMed on 10 Jul 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.