Multimodal Semi-supervised Learning for Sentiment Analysis of Image Macros

Abstract

Memes in the form of image macro are a part of social media content nowadays. The meme usually has an underlying meaning that needs to be sentiment analyzed for censoring harmful content. Meme censoring systems by machine learning raise the need for a semi-supervised learning solution to leverage a massive quantity of unlabeled memes on the internet and reduce the difficulties of the annotation process. Moreover, the machine learning approach should utilize multimodal data because a meme's meaning usually comes from both visual and linguistic. Therefore, in this research, we proposed a multimodal semi-supervised learning approach that outperformed other multimodal semisupervised learning and supervised learning SOTA when comparing the result on the Multimedia Automatic Misogyny Identification (MAMI) dataset of the meme. Besides successfully applying other excellent studies about multimodal data and imbalanced data, such as CLIP and distribution balanced loss, our research presents a new training manner that wisely combines auto-encoder and classification tasks to utilize unlabeled data