Bridging behavior and brain: the impact of emotional and linguistic cues on human performance and neural activation
研究AI生成的面部表情、语音语调和语言正式度如何共同影响认知负荷、疲劳、态势感知及大脑激活，为设计自适应多模态AI系统提供神经行为指南。
Human-machine interaction requires seamless integration of multimodal cues to align with users’ cognitive and neural states. This study investigates how AI-generated facial expressions, voice tones, and language formality jointly shape cognitive load, fatigue, situation awareness (SA), and neural activation. In a simulated industrial task (N = 60), behavioural results revealed that negative facial expressions combined with formal language elevated cognitive load regardless of voice tones, while informal language paired with negative voice tones and facial expressions maximised fatigue. SA improved when formal language was paired with positive facial expressions or negative voice tones. Neuroimaging identified three-way interactions in premotor cortex, Broca’s area, and frontal eye fields, alongside two-way interactions in frontopolar and occipital cortices. Critically, reduced SA correlated with diminished activation in visual and executive control regions. These findings provide neurobehavioral guidelines for designing AI systems that adapt multimodal outputs to mitigate cognitive strain and enhance operational safety.