Emotion-Aware Multimodal Fusion for Meme Emotion Detection

Download paper here

Memes are widely used on social media to express opinions, but current methods struggle to capture their emotional dimensions, relying on large datasets and lacking generalization. We introduce MOOD (Meme emOtiOns Dataset) with six emotions and ALFRED (emotion-Aware muLtimodal Fusion foR Emotion Detection), a neural framework that effectively models visual-emotional cues and cross-modal fusion. ALFRED outperforms existing methods by 4.94% F1, excels in the Memotion task, and generalizes well on HarMeme and Dank Memes datasets. It also offers interpretability through attention maps. We address the challenges of analyzing memes due to complex modality-specific cues.

Recommended citation: S. Sharma, R. S, M. S. Akhtar and T. Chakraborty, “Emotion-Aware Multimodal Fusion for Meme Emotion Detection,” in IEEE Transactions on Affective Computing, doi: 10.1109/TAFFC.2024.3378698.