Hierarchical Uncertainty Transformation During Multimodal Comprehension

Uncertainty during multimodal comprehension reorganizes across three levels—within-modality, cross-modal interaction, and multisensory convergence—showing N400-linked neural signatures that vary with semantic information.

• Why it matters: It remains unclear how uncertainty is coordinated when gesture and speech each provide incomplete or ambiguous semantic evidence and representations must converge.
• What they did: Across four gesture-speech experiments, the team combined behavioral measures, information-theoretic modelling, and electroencephalography to track uncertainty transformations at modality-specific, cross-modal, and convergence levels.
• The result: Uncertainty was expressed relatively early within modalities, while cross-modal transformation and convergence were most prominently associated with N400-range activity, with neural expression and statistical coupling changing with available semantic information.

AI summary of the abstract · may contain errors · not medical advice · read the paper.