Emotion-specific modality effects in auditory and visual perception of emotion.

Authors

Yongseong Lee, Youngju Lee, Dongha Lee

Published in

Psychological research. Volume 90. Issue 4. Jul 02, 2026. Epub Jul 02, 2026.

Abstract

Emotion is typically perceived in dynamic multisensory contexts. Although previous studies have demonstrated multimodal gains in audiovisual recognition (i.e., general enhancement in audiovisual conditions), they have not clarified how visual and auditory information contribute to affective judgments or whether their contribution differs by emotion categories. The current study investigated how emotion perception varies across modalities and examined whether these variations depend on emotion categories. We conducted an emotion perception experiment using audio-only (AO), video-only (VO), and audio-visual (AV) speech stimuli. Participants evaluated the valence (positive/negative) and arousal (weak/strong) of stimuli depicting seven emotions (angry, calm, disgust, fearful, happy, sad, and surprise). Overall, AO stimuli tended to be perceived as more negative in valence and weaker in arousal compared to VO or AV stimuli. In representational similarity analysis, the response patterns for AV were more similar to VO than to AO. Happy and disgust stimuli were rated as weaker in arousal and less positive or less negative in valence in AO condition than in other conditions. For surprise, VO stimuli were rated more negatively than AO and AV. However, angry, fearful, and sad emotions did not differ between modalities. These findings indicate that affective judgments are not solely driven by general enhancement from combined audiovisual cues; rather, the relative importance of sensory modalities varies by emotion, suggesting that specific unimodal cues can be as informative as multimodal information. These modality-specific contributions to emotion perception have implications for clinical assessment, social communication, and affective neuroscience.

PMID:
42390603
Bibliographic data and abstract were imported from PubMed on 02 Jul 2026.

Read full publication at:
Please sign in to see all details.

Stats

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.