SHAP-Weighted Cross-Modal Expert Fusion for Emotion and Sentiment Recognition

Jul 10, 2026

A new study revisits XAI-guided adaptive fusion (XGAF) for multimodal emotion and sentiment recognition, using TreeSHAP attribution magnitudes to weight unimodal and cross-modal experts. On MELD 7-class emotion recognition, sum-abs XGAF nearly matches early fusion (0.5983 vs 0.6018) and significantly outperforms late fusion (0.4598). On CMU-MOSEI 3-class sentiment, sum-abs XGAF slightly exceeds early fusion (0.6519 vs 0.6485).

Why it matters: This work provides a transparent empirical analysis of how SHAP reduction methods and expert dimensionality affect modular multimodal fusion, offering a principled alternative to monolithic early fusion.

Full story at: arXiv AI/ML ↗