Confidence-Aware Multimodal Learning for Trustworthy Fake News Detection

References

• Bai Y, Mei S, Wang H, Xiong C (2021) Don’t just blame over-parametrization for over-confidence: Theoretical analysis of calibration in binary classification. Internat. Conf. Machine Learn. (PMLR, Cambridge, MA), 566–576. Google Scholar
• Balanya SA, Maroñas J, Ramos D (2024) Adaptive temperature scaling for robust calibration of deep neural networks. Neural Comput. Appl. 36:8073–8095.Crossref, Google Scholar
• Boididou C, Papadopoulos S, Zampoglou M, Apostolidis L, Papadopoulou O, Kompatsiaris Y (2018) Detection and visualization of misleading content on Twitter. Internat. J. Multimedia Inform. Retrieval 7(1):71–86.Crossref, Google Scholar
• Brier GW (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Rev. 78(1):1–3.Crossref, Google Scholar
• Castillo C, Mendoza M, Poblete B (2011) Information credibility on Twitter. Proc. 20th Internat. Conf. World Wide Web (Association for Computing Machinery, New York), 675–684.Google Scholar
• Chen X, Zheng X, Sun K, Liu W, Zhang Y (2023) Self-supervised vision transformer-based few-shot learning for facial expression recognition. Inform. Sci. 634:206–226.Crossref, Google Scholar
• Chen Y, Li D, Zhang P, Sui J, Lv Q, Tun L, Shang L (2022) Cross-modal ambiguity learning for multimodal fake news detection. Proc. ACM Web Conf. 2022 (Association for Computing Machinery, New York), 2897–2905.Google Scholar
• Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) ImageNet: A large-scale hierarchical image database. 2009 IEEE Conf. Comput. Vision Pattern Recognition (IEEE, Piscataway, NJ), 248–255.Google Scholar
• Devlin J, Chang MW, Lee K, Toutanova K (2018) BERT: Pre-training of deep bidirectional transformers for language understanding. Preprint, submitted October 11, https://arxiv.org/abs/1810.04805v1.Google Scholar
• Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, et al. (2020) An image is worth 16x16 words: Transformers for image recognition at scale. Preprint, submitted October 22, https://arxiv.org/abs/2010.11929v1.Google Scholar
• Dou Y, Shu K, Xia C, Yu PS, Sun L (2021) User preference-aware fake news detection. Proc. 44th Internat. ACM SIGIR Conf. Res. Development Inform. Retrieval (Association for Computing Machinery, New York), 2051–2055.Google Scholar
• Frenkel L, Goldberger J (2022) Network calibration by temperature scaling based on the predicted confidence. 2022 30th Eur. Signal Processing Conf. EUSIPCO (IEEE, Piscataway, NJ), 1586–1590.Google Scholar
• Ge S, Chen Y (2025) Confidence-aware multimodal learning for trustworthy fake news detection. https://doi.org/10.1287/ijoc.2024.0655.cd, https://github.com/INFORMSJoC/2024.0655.Google Scholar
• Gelfert A (2018) Fake news: A definition. Informal Logic 38(1):84–117.Crossref, Google Scholar
• Ghosh A, Schaaf T, Gormley M (2022) AdaFocal: Calibration-aware adaptive focal loss. Adv. Neural Inform. Processing Systems, vol. 35 (Curran Associates Inc., Red Hook, NY), 1583–1595.Google Scholar
• Guo C, Pleiss G, Sun Y, Weinberger KQ (2017) On calibration of modern neural networks. Internat. Conf. Machine Learn. (PMLR, Cambridge, MA), 1321–1330.Google Scholar
• Han Z, Yang Y, Zhang C, Zhang L, Zhou JT, Hu Q, Yao H (2024) Selective learning: Towards robust calibration with dynamic regularization. Preprint, submitted July 14, https://arxiv.org/abs/2402.08384.Google Scholar
• Hastie T, Tibshirani R, Friedman JH, Friedman JH (2009) The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. (Springer, New York).Crossref, Google Scholar
• He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. Proc. IEEE Conf. Comput. Vision Pattern Recognition (IEEE, Piscataway, NJ), 770–778.Google Scholar
• Hendrycks D, Gimpel K (2016) Gaussian error linear units (GELUs). Preprint, submitted June 27, https://arxiv.org/abs/1606.08415v1.Google Scholar
• Hiramath CK, Deshpande G (2019) Fake news detection using deep learning techniques. 2019 1st Internat. Conf. Adv. Inform. Tech ICAIT (IEEE, Piscataway, NJ), 411–415.Google Scholar
• Hu L, Chen Z, Zhao Z, Yin J, Nie L (2023) Causal inference for leveraging image-text matching bias in multi-modal fake news detection. IEEE Trans. Knowledge Data Engrg. 35(11):11141–11152.Crossref, Google Scholar
• Hu L, Zhao Z, Qi W, Song X, Nie L (2024) Multimodal matching-aware co-attention networks with mutual knowledge distillation for fake news detection. Inform. Sci. 664:120310.Crossref, Google Scholar
• Hunt K, Agarwal P, Al Aziz R, Zhuang J (2020) Fighting fake news during disasters. OR/MS Today 47(1):34–39.Google Scholar
• Imran M, Ofli F, Caragea D, Torralba A (2020) Using AI and social media multimodal content for disaster response and management: Opportunities, challenges, and future directions. Inform. Processing Management 57(5):102261.Google Scholar
• Jang S, Lee I, Weimer J (2021) Improving classifier confidence using lossy label-invariant transformations. Internat. Conf. Artificial Intelligence Statist. (PMLR, Cambridge, MA), 4051–4059.Google Scholar
• Ji B, Jung H, Yoon J, Kim K, Shin Y (2019) Bin-wise temperature scaling (BTS): Improvement in confidence calibration performance through simple scaling techniques. 2019 IEEE/CVF Internat. Conf. Comput. Vision Workshop ICCVW (IEEE, Piscataway, NJ), 4190–4196.Google Scholar
• Jin Z, Cao J, Guo H, Zhang Y, Luo J (2017) Multimodal fusion with recurrent neural networks for rumor detection on microblogs. Proc. 25th ACM Internat. Conf. Multimedia (Association for Computing Machinery, New York), 795–816.Google Scholar
• Joy T, Pinto F, Lim SN, Torr PH, Dokania PK (2023) Sample-dependent adaptive temperature scaling for improved calibration. Proc. 37th AAAI Conf. Artificial Intelligence (AAAI Press, Washington, DC), 14919–14926.Google Scholar
• Khattar D, Goud JS, Gupta M, Varma V (2019) MVAE: Multimodal variational autoencoder for fake news detection. WWW’19 World Wide Web Conf. (Association for Computing Machinery, New York), 2915–2921.Google Scholar
• Kiela D, Bhooshan S, Firooz H, Perez E, Testuggine D (2019) Supervised multimodal bitransformers for classifying images and text. Preprint, submitted September 6, https://arxiv.org/abs/1909.02950v1.Google Scholar
• Kumar A, Liang PS, Ma T (2019) Verified uncertainty calibration. Adv. Neural Inform. Processing Systems, vol. 32 (Curran Associates Inc., Red Hook, NY), 3792–3803.Google Scholar
• Kumar S, Asthana R, Upadhyay S, Upreti N, Akbar M (2020) Fake news detection using deep learning models: A novel approach. Trans. Emerging Telecommunications Tech. 31(2):e3767.Crossref, Google Scholar
• Kumari R, Ekbal A (2021) AMFB: Attention based multimodal factorized bilinear pooling for multimodal fake news detection. Expert Systems Appl. 184:115412.Crossref, Google Scholar
• Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) RoBERTa: A robustly optimized BERT pretraining approach. Preprint, submitted July 26, https://arxiv.org/abs/1907.11692.Google Scholar
• Luo G, Darrell T, Rohrbach A (2021) NewsCLIPpings: Automatic generation of out-of-context multimodal media. Preprint, submitted September 21, https://arxiv.org/abs/2104.05893.Google Scholar
• Ma X, Blaschko MB (2021) Meta-Cal: Well-controlled post-hoc calibration by ranking. Internat. Conf. Machine Learn. (PMLR, Cambridge, MA), 7235–7245.Google Scholar
• Ma J, Gao W, Wong KF (2019) Detect rumors on twitter by promoting information campaigns with generative adversarial learning. WWW’19 World Wide Web Conf. (Association for Computing Machinery, New York), 3049–3055.Google Scholar
• Mohri M, Rostamizadeh A, Talwalkar A (2018) Foundations of Machine Learning (MIT Press, Cambridge, MA).Google Scholar
• Monti F, Frasca F, Eynard D, Mannion D, Bronstein MM (2019) Fake news detection on social media using geometric deep learning. Preprint, submitted February 10, https://arxiv.org/abs/1902.06673.Google Scholar
• Mozafari AS, Gomes HS, Leão W, Janny S, Gagné C (2018) Attended temperature scaling: A practical approach for calibrating deep neural networks. Preprint, submitted October 27, https://arxiv.org/abs/1810.11586v1.Google Scholar
• Mukhoti J, Kulharia V, Sanyal A, Golodetz S, Torr P, Dokania P (2020) Calibrating deep neural networks using focal loss. Adv. Neural Inform. Processing Systems 33:15288–15299.Google Scholar
• Müller R, Kornblith S, Hinton GE (2019) When does label smoothing help? Adv. Neural Inform. Processing Systems, vol. 32 (Curran Associates Inc., Red Hook, NY), 4694–4703.Google Scholar
• Naeini MP, Cooper G, Hauskrecht M (2015) Obtaining well calibrated probabilities using Bayesian binning. Proc. 29th AAAI Conf. Artificial Intelligence (AAAI Press, Washington, DC), 2901–2907.Google Scholar
• Nakamura K, Levy S, Wang WY (2019) r/Fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection. Preprint, submitted November 10, https://arxiv.org/abs/1911.03854v1.Google Scholar
• Peng L, Jian S, Li D, Shen S (2023) MRML: Multimodal rumor detection by deep metric learning. ICASSP 2023-2023 IEEE Internat. Conf. Acoustics Speech Signal Processing ICASSP (IEEE, Piscataway, NJ), 1–5.Google Scholar
• Peng L, Jian S, Kan Z, Qiao L, Li D (2024) Not all fake news is semantically similar: Contextual semantic representation learning for multimodal fake news detection. Inform. Processing Management 61(1):103564.Crossref, Google Scholar
• Pereyra G, Tucker G, Chorowski J, Kaiser Ł, Hinton G (2017) Regularizing neural networks by penalizing confident output distributions. Preprint, submitted January 23, https://arxiv.org/abs/1701.06548.Google Scholar
• Platt J (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv. Large Margin Classifiers 10(3):61–74.Google Scholar
• Qian F, Gong C, Sharma K, Liu Y (2018) Neural user response generator: Fake news detection with collective user intelligence. IJCAI’ 18 Proc. 27th Internat. Joint Conf. Artificial Intelligence (AAAI Press, Washington, DC), 3834–3840.Google Scholar
• Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, et al. (2021) Learning transferable visual models from natural language supervision. Internat. Conf. Machine Learn. (PMLR, Cambridge, MA), 8748–8763.Google Scholar
• Ruchansky N, Seo S, Liu Y (2017) CSI: A hybrid deep model for fake news detection. Proc. 2017 ACM Conf. Inform. Knowledge Management (Association for Computing Machinery, New York), 797–806.Google Scholar
• Sahoo SR, Gupta BB (2021) Multiple features based approach for automatic fake news detection on social networks using deep learning. Appl. Soft Comput. 100:106983.Crossref, Google Scholar
• Shu K, Sliva A, Wang S, Tang J, Liu H (2017) Fake news detection on social media: A data mining perspective. ACM SIGKDD Explorations Newslett. 19(1):22–36.Crossref, Google Scholar
• Singhal S, Pandey T, Mrig S, Shah RR, Kumaraguru P (2022) Leveraging intra and inter modality relationship for multimodal fake news detection. WWW ’ 22 Companion Proc. Web Conf. 2022, (Association for Computing Machinery, New York), 726–734.Google Scholar
• Singhal S, Kabra A, Sharma M, Shah RR, Chakraborty T, Kumaraguru P (2020) SpotFake+: A multimodal framework for fake news detection via transfer learning (student abstract). Proc. 34th AAAI Conf. Artificial Intelligence (AAAI Press, Washington, DC), 13915–13916.Google Scholar
• Singhania S, Fernandez N, Rao S (2017) 3HAN: A deep neural network for fake news detection. Proc. Neural Inform. Processing 24th Internat. Conf. ICONIP Part II (Springer, Cham, Switzerland), 572–581.Google Scholar
• Tao L, Dong M, Xu C (2023) Dual focal loss for calibration. Internat. Conf. Machine Learn.(PMLR, Cambridge, MA), 33833–33849.Google Scholar
• Thota A, Tilak P, Ahluwalia S, Lohia N (2018) Fake news detection: A deep learning approach. SMU Data Sci. Rev. 1(3):10.Google Scholar
• Tomani C, Cremers D, Buettner F (2022) Parameterized temperature scaling for boosting the expressive power in post-hoc uncertainty calibration. Eur. Conf. Comput. Vision ECCV 2022 (Springer, Cham, Switzerland), 555–569.Google Scholar
• Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Adv. Neural Inform. Processing Systems, vol. 30 (Curran Associates Inc., Red Hook, NY), 6000–6010.Google Scholar
• Wang C (2023) Calibration in deep learning: A survey of the state-of-the-art. Preprint, submitted August 2, https://arxiv.org/abs/2308.01222v1.Google Scholar
• Wang DB, Feng L, Zhang ML (2021a) Rethinking calibration of deep neural networks: Do not be afraid of overconfidence. Adv. Neural Inform. Processing Systems, vol. 34 (Curran Associates Inc., Red Hook, NY), 11809–11820.Google Scholar
• Wang S, Pang MS, Pavlou PA (2021b) Cure or poison? Identity verification and the posting of fake news on social media. J. Management Inform. Systems 38(4):1011–1038.Crossref, Google Scholar
• Wang B, Feng Y, Xiong X-c, Wang Y-h, Qiang B-h (2023a) Multi-modal transformer using two-level visual features for fake news detection. Appl. Intelligence 53(9):10429–10443.Crossref, Google Scholar
• Wang L, Zhang C, Xu H, Xu Y, Xu X, Wang S (2023b) Cross-modal contrastive learning for multimodal fake news detection. Proc. 31st ACM Internat. Conf. Multimedia (Association for Computing Machinery, New York), 5696–5704.Google Scholar
• Wang Y, Ma F, Jin Z, Yuan Y, Xun G, Jha K, Su L, Gao J (2018) EANN: Event adversarial neural networks for multi-modal fake news detection. Proc. 24th ACM SIGKDD Internat. Conf. Knowledge Discov. Data Mining (Association for Computing Machinery, New York), 849–857.Google Scholar
• Winterbottom T, Xiao S, McLean A, Moubayed NA (2020) On modality bias in the TVQA dataset. Preprint, submitted December 18, https://arxiv.org/abs/2012.10210.Google Scholar
• Wu L, Long Y, Gao C, Wang Z, Zhang Y (2023) MFIR: Multimodal fusion and inconsistency reasoning for explainable fake news detection. Inform. Fusion 100:101944.Crossref, Google Scholar
• Wu Y, Zhan P, Zhang Y, Wang L, Xu Z (2021) Multimodal fusion with co-attention networks for fake news detection. Findings Assoc. Comput. Linguistics ACL-IJCNLP 2021 (Association for Computational Linguistics, Kerrville, TX), 2560–2569.Google Scholar
• Xiong M, Deng A, Koh PWW, Wu J, Li S, Xu J, Hooi B (2023) Proximity-informed calibration for deep neural networks. Adv. Neural Inform. Processing Systems, vol. 36 (Curran Associates Inc., Red Hook, NY), 68511–68538.Google Scholar
• Xue J, Wang Y, Tian Y, Li Y, Shi L, Wei L (2021) Detecting fake news by exploring the consistency of multimodal data. Inform. Processing Management 58(5):102610.Crossref, Google Scholar
• Yu Y, Bates S, Ma Y, Jordan M (2022b) Robust calibration with multi-domain temperature scaling. Adv. Neural Inform. Processing Systems, vol. 35 (Curran Associates Inc., Red Hook, NY), 27510–27523.Google Scholar
• Yu C, Ma Y, An L, Li G (2022a) BCMF: A bidirectional cross-modal fusion model for fake news detection. Inform. Processing Management 59(5):103063.Crossref, Google Scholar
• Zadrozny B, Elkan C (2001) Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. ICML’01 Proc. 18th Internat. Conf. Machine Learn. (Morgan Kaufmann Publishers, San Francisco), 609–616.Google Scholar
• Zadrozny B, Elkan C (2002) Transforming classifier scores into accurate multiclass probability estimates. Proc. Eighth ACM SIGKDD Internat. Conf. Knowledge Discov. Data Mining (Association for Computing Machinery, New York), 694–699.Google Scholar
• Zanatta ET, Wanderley GPDM, Branco IK, Pereira D, Kato LH, Maluf EMCP (2021) Fake news: The impact of the internet on population health. Rev. Assoc. Med. Brasileira 67:926–930.Crossref, Google Scholar
• Zhang X, Ghorbani AA (2020) An overview of online fake news: Characterization, detection, and discussion. Inform. Processing Management 57(2):102025.Crossref, Google Scholar
• Zhang Z, Sabuncu M (2018) Generalized cross entropy loss for training deep neural networks with noisy labels. Adv. Neural Inform. Processing Systems, vol. 31 (Curran Associates Inc., Red Hook, NY), 8792–8802.Google Scholar
• Zhang J, Kailkhura B, Han TYJ (2020) Mix-n-Match: Ensemble and compositional methods for uncertainty calibration in deep learning. Internat. Conf. Machine Learn. (PMLR, Cambridge, MA), 11117–11128.Google Scholar
• Zhou Y, Yang Y, Ying Q, Qian Z, Zhang X (2023) Multimodal fake news detection via CLIP-guided learning. 2023 IEEE Internat. Conf. Multimedia Expo ICME (IEEE, Piscataway, NJ), 2825–2830.Google Scholar
• Zhu P, Hua J, Tang K, Tian J, Xu J, Cui X (2024) Multimodal fake news detection through intra-modality feature aggregation and inter-modality semantic fusion. Complex Intelligent Systems 10:5851–5863.Crossref, Google Scholar