Mitigating modality bias in multi-label emotion recognition
分析了多模态情绪识别中模态偏差的成因，提出一种新的融合框架，包含两种模态加权方法和一个正则项，在三个大型数据集上优于现有方法。
Accurate recognition of emotions has a wide range of applications. Recognizing emotions from multi-modal data usually generates more accurate results than that from uni-modal data. Nonetheless, modality bias widely exists in multi-modal learning, resulting in that one modality dominates the class prediction than the others. Even though several latest methods have been developed to improve the model performance of emotion recognition, theoretical analysis of the learning process of a multi-modal model is still lacking. Moreover, these methods do not have theoretical guarantees for their performance. Regarding these, we pioneer the analysis of the training process of a multi-modal model and find that dominant modalities impede the learning process of other modalities. Based on this analysis, we theoretically propose a condition for multi-modal fusion. Once satisfied, the modality bias will be alleviated. According to these theoretical insights, we develop a new multi-modal fusion framework, including the introduction of two novel modality weighting methods and one regularization term, for the multi-modal multi-label emotion recognition (MMER) tasks. In the experiment, we apply the proposed method to three large MMER datasets. The experimental results demonstrate that our proposed method achieves superior performance compared to existing state-of-the-art methods by mitigating modality bias.