Computational Counterfactuals Reveal Non-Additive Audiovisual Semantics in Natural Movie Responses

Authors

Li, M.

Abstract

Natural audiovisual perception may not be fully captured by decomposing movies into auditory and visual streams. I introduce a computational-counterfactual framework that keeps movie viewing intact while varying only AI-derived descriptions of the same clips. Using 7 Tesla movie fMRI imaging data from 176 participants, I tested whether cortical responses were better predicted by native audiovisual semantics than by a dimension-matched additive reconstruction from audio-only and video-only descriptions. The native model outperformed the matched additive baseline under content-aware purged cross-validation, with strongest gains in auditory, visual, and dorsal attention systems. Representational-similarity, feature-replacement, and content-gating analyses showed that the advantage reflected feature- and network-specific routing linked to coherent audiovisual semantic emergence rather than raw auditory-visual discrepancy. The effect survived stronger temporal purging and repeat-content exclusion, suggesting that intact movie viewing evokes cortical structure aligned with native audiovisual meaning beyond additive unimodal semantics.

Preprint server: bioRxiv
The authors list and abstract were imported from bioRxiv on 18 Jul 2026.

Stats

1-terrible, 9-excellent. How would you rate this preprint? Sign in in to submit your rating.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.