Knowing (Not) to Know: Explainable Artificial Intelligence and Human Metacognition

References

• Abdel-Karim BM, Pfeuffer N, Carl V, Hinz O (2023) How AI-based systems can induce reflections: The case of AI-augmented diagnostic work. MIS Quart. 47(4):1395–1424.Crossref, Google Scholar
• Ackerman R, Thompson VA (2017) Meta-reasoning: Monitoring and control of thinking and reasoning. Trends Cognitive Sci. 21(8):607–617.Crossref, Google Scholar
• Baird A, Maruping LM (2021) The next generation of research on IS use: A theoretical framework of delegation to and from agentic IS artifacts. MIS Quart. 45(1):315–341.Crossref, Google Scholar
• Bansal G, Nushi B, Kamar E, Lasecki WS, Weld DS, Horvitz E (2019) Beyond accuracy: The role of mental models in human-AI team performance. AAAI Conf. Human Comput. Crowdsourcing, vol. 7 (University of Washington, Seattle).Google Scholar
• Bartling B, Fehr E, Herz H (2014) The intrinsic value of decision rights. Econometrica 82(6):2005–2039.Crossref, Google Scholar
• Bauer K, von Zahn M, Hinz O (2023) Expl(AI)ned: The impact of explainable artificial intelligence on users’ information processing. Inform. Systems Res. 34(4):1582–1602.Link, Google Scholar
• Bauer K, Hinz O, van der Aalst W, Weinhardt C (2021) Expl(AI)n it to me—Explainable AI and information systems research. Bus. Inform. Systems Engrg. 63(2):79–82.Crossref, Google Scholar
• Bouwer A (2022) Under which conditions are humans motivated to delegate tasks to AI? A taxonomy on the human emotional state driving the motivation for AI delegation. Marketing Smart Tech. Proc. ICMarkTech 2021, vol. 1 (Springer, Berlin), 37–53.Google Scholar
• Cecil J, Lermer E, Hudecek MF, Sauer J, Gaube S (2024) Explainability does not mitigate the negative impact of incorrect AI advice in a personnel selection task. Sci. Rep. 14(1):9736.Crossref, Google Scholar
• Chen T, Guestrin C (2016) XGBoost: A scalable tree boosting system. Proc. 22nd ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining (ACM, New York), 785–794.Google Scholar
• Chromik M, Eiband M, Buchner F, Krüger A, Butz A (2021) I think I get your point, AI! The illusion of explanatory depth in explainable AI. 26th Internat. Conf. Intelligent User Interfaces (ACM, New York), 307–317.Google Scholar
• Clayton RB (2022) On the psychophysiological and defensive nature of psychological reactance theory. J. Comm. 72(4):461–475.Crossref, Google Scholar
• Cram WA, Wiener M, Tarafdar M, Benlian A (2022) Examining the impact of algorithmic control on Uber drivers’ technostress. J. Management Inform. Systems 39(2):426–453.Crossref, Google Scholar
• Desender K, Boldt A, Verguts T, Donner TH (2019) Confidence predicts speed-accuracy tradeoff for subsequent decisions. eLife 8:e43499.Crossref, Google Scholar
• Dietvorst BJ, Simmons JP, Massey C (2015) Algorithm aversion: People erroneously avoid algorithms after seeing them err. J. Experiment. Psych. General 144(1):114–126.Crossref, Google Scholar
• Dietvorst BJ, Simmons JP, Massey C (2018) Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them. Management Sci. 64(3):1155–1170.Link, Google Scholar
• Duarte J, Siegel S, Young L (2012) Trust and credit: The role of appearance in peer-to-peer lending. Rev. Financial Stud. 25(8):2455–2484.Crossref, Google Scholar
• Dunning D (2011) The Dunning–Kruger effect: On being ignorant of one’s own ignorance. Olson JM, Zanna MP, eds. Advances in Experimental Social Psychology, vol. 44 (Elsevier, Amsterdam), 247–296.Google Scholar
• European Union (2024) Proposal for a regulation of the European Parliament and of the council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain union legislative acts. Accessed April 9, 2024, https://data.consilium.europa.eu/doc/document/ST-5662-2024-INIT/en/pdf.Google Scholar
• Fernández-Loría C, Provost F, Han X (2022) Explaining data-driven decisions made by AI systems: The counterfactual approach. MIS Quart. 46(3):1635–1660.Crossref, Google Scholar
• Festinger L (1957) Social comparison theory. Selective Exposure Theory 16(401):3.Google Scholar
• Fiedler K, Ackerman R, Scarampi C (2019) Metacognition: Monitoring and controlling one’s own knowledge, reasoning and decisions. Sternberg RJ, Funke J, eds. The Psychology of Human Thought: An Introduction (Heidelberg University Publishing, Heidelberg, Germany), 89–111.Google Scholar
• Flavell JH (1979) Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. Amer. Psychologist 34(10):906–911.Crossref, Google Scholar
• Fleming SM (2024) Metacognition and confidence: A review and synthesis. Annual Rev. Psych. 75(1):241–268.Crossref, Google Scholar
• Fügener A, Grahl J, Gupta A, Ketter W (2022) Cognitive challenges in human–artificial intelligence collaboration: Investigating the path toward productive delegation. Inform. Systems Res. 33(2):678–696.Link, Google Scholar
• Goddard K, Roudsari A, Wyatt JC (2012) Automation bias: A systematic review of frequency, effect mediators, and mitigators. J. Amer. Medical Informatics Assoc. 19(1):121–127.Crossref, Google Scholar
• Goodman B, Flaxman S (2017) European Union regulations on algorithmic decision-making and a “right to explanation”. AI Magazine 38(3):50–57.Crossref, Google Scholar
• GoogleAI (2025) Responsible AI progress report. Accessed June 13, 2025, https://ai.google/static/documents/ai-responsibility-update-published-february-2025.pdf.Google Scholar
• Goyal N, Baumler C, Nguyen T, Daumé H III (2024) The impact of explanations on fairness in human-AI decision-making: Protected vs proxy features. Proc. 29th Internat. Conf. Intelligent User Interfaces (ACM, New York), 155–180.Google Scholar
• Gramegna A, Giudici P (2021) SHAP and LIME: An evaluation of discriminative power in credit risk. Frontiers Artificial Intelligence 4:752558.Crossref, Google Scholar
• Gregor S, Benbasat I (1999) Explanations from intelligent systems: Theoretical foundations and implications for practice. MIS Quart. 23(4):497–530.Crossref, Google Scholar
• Guo T, Bardhan IR, Ding Y, Zhang S (2025) An explainable artificial intelligence approach using graph learning to predict intensive care unit length of stay. Inform. Systems Res. 36(3):1478–1501.Link, Google Scholar
• Gupta A, Kannan K, Sanyal P (2018) Economic experiments in information systems. MIS Quart. 42(2):595–606.Crossref, Google Scholar
• Händel M, De Bruin AB, Dresel M (2020) Individual differences in local and global metacognitive judgments. Metacognition Learn. 15(1):51–75.Crossref, Google Scholar
• Hastie T, Tibshirani R, Friedman JH (2009) The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. (Springer, New York).Crossref, Google Scholar
• Hurlin C, Pérignon C, Saurin S (2024) The fairness of credit scoring models. Management Sci., ePub ahead of print November 14, https://doi.org/10.1287/mnsc.2022.03888.Link, Google Scholar
• Iyer R, Khwaja AI, Luttmer EF, Shue K (2010) Screening in new credit markets: Can individual lenders infer borrower creditworthiness in peer-to-peer lending? Preprint, submitted March 15, http://dx.doi.org/10.2139/ssrn.1570115.Google Scholar
• Ji-Ye Mao IB (2000) The use of explanations in knowledge-based systems: Cognitive perspectives and a process-tracing analysis. J. Management Inform. Systems 17(2):153–179.Crossref, Google Scholar
• Jordan MI, Mitchell TM (2015) Machine learning: Trends, perspectives, and prospects. Science 349(6245):255–260.Crossref, Google Scholar
• Jussupow E, Benbasat I, Heinzl A (2024) An integrative perspective on algorithm aversion and appreciation in decision-making. MIS Quart. 48(4):1575–1590.Crossref, Google Scholar
• Jussupow E, Spohrer K, Heinzl A, Gawlitza J (2021) Augmenting medical diagnosis decisions? An investigation into physicians’ decision-making process with artificial intelligence. Inform. Systems Res. 32(3):713–735.Link, Google Scholar
• Komiak SYX, Benbasat I (2006) The effects of personalization and familiarity on trust and adoption of recommendation agents. MIS Quart. 30(4):941–960.Crossref, Google Scholar
• Koriat A, Levy-Sadot R (2000) Conscious and unconscious metacognition: A rejoinder. Consciousness Cognition 9(2 Pt 1):193–202.Crossref, Google Scholar
• Lebovitz S, Lifshitz-Assaf H, Levina N (2022) To engage or not to engage with AI for critical judgments: How professionals deal with opacity when using AI for medical diagnosis. Organ. Sci. 33(1):126–148.Link, Google Scholar
• Li X, Grahl J, Hinz O (2022) How do recommender systems lead to consumer purchases? A causal mediation analysis of a field experiment. Inform. Systems Res. 33(2):620–637.Link, Google Scholar
• Lichtenstein S, Fischhoff B, Phillips LD (1982) Calibration of probabilities: The state of the art to 1980. Kahneman D, Slovic P, Tversky A, eds. Judgment Under Uncertainty: Heuristics and Biases (Cambridge University Press, Cambridge, UK), 306–334.Crossref, Google Scholar
• Lundberg SM, Lee SI (2017) A unified approach to interpreting model predictions. Conf. Neural Inform. Processing Systems (NIPS) (University of Washington Seattle, Seattle), 1–10.Google Scholar
• Ma S, Wang X, Lei Y, Shi C, Yin M, Ma X (2024) “Are you really sure?” Understanding the effects of human self-confidence calibration in AI-assisted decision making. Proc. 2024 CHI Conf. Human Factors Comput. Systems (ACM, New York).Google Scholar
• Maruping L, Yin D, Chen A, Kankanhalli A, Burton-Jones A, Brown S (2025) Editor’s comments: Quantitative behavioral IS research—A look back and a look forward. MIS Quart. 49(1):iii–xviii.Crossref, Google Scholar
• McKinsey (2024) Building AI trust: The key role of explainability (November 26), https://www.mckinsey.com/capabilities/quantumblack/our-insights/building-ai-trust-the-key-role-of-explainability.Google Scholar
• Meske C, Bunde E, Schneider J, Gersch M (2022) Explainable artificial intelligence: Objectives, stakeholders, and future research opportunities. Inform. Systems Management 39(1):53–63.Crossref, Google Scholar
• Millet K, Buehler F, Du G, Kokkoris MD (2023) Defending humankind: Anthropocentric bias in the appreciation of AI art. Comput. Human Behav. 143:107707.Crossref, Google Scholar
• Molnar C (2020) Interpretable Machine Learning: A Guide for Making Black Box Models Explainable (Christoph Molnar, Munich, Germany).Google Scholar
• Naiseh M, Al-Thani D, Jiang N, Ali R (2023) How the different explanation classes impact trust calibration: The case of clinical decision support systems. Internat. J. Human-Comput. Stud. 169:102941.Crossref, Google Scholar
• Nannini L, Alonso-Moral JM, Catalá A, Lama M, Barro S (2024) Operationalizing explainable artificial intelligence in the European Union regulatory ecosystem. IEEE Intelligent Systems 39(4):37–48.Crossref, Google Scholar
• Nelson TO, Narens L (1990) Metamemory: A theoretical framework and new findings. Psych. Learn. Motivation 26:125–173.Crossref, Google Scholar
• Ochmann J, Michels L, Tiefenbeck V, Maier C, Laumer S (2024) Perceived algorithmic fairness: An empirical study of transparency and anthropomorphism in algorithmic recruiting. Inform. Systems J. 34(2):384–414.Crossref, Google Scholar
• Pinski M, Benlian A (2024) AI literacy for users—A comprehensive review and future research directions of learning methods, components, and effects. Comput. Human Behav. Artificial Humans 2(1):100062.Crossref, Google Scholar
• Poursabzi-Sangdeh F, Goldstein DG, Hofman JM, Wortman Vaughan JW, Wallach H (2021) Manipulating and measuring model interpretability. Proc. 2021 CHI Conf. Human Factors Comput. Systems (ACM, New York).Google Scholar
• Rader E, Cotter K, Cho J (2018) Explanations as mechanisms for supporting algorithmic transparency. Proc. 2018 CHI Conf. Human Factors Comput. Systems (ACM, New York).Google Scholar
• Rozenblit L, Keil F (2002) The misunderstood limits of folk science: An illusion of explanatory depth. Cognitive Sci. 26(5):521–562.Crossref, Google Scholar
• Senoner J, Netland T, Feuerriegel S (2022) Using explainable artificial intelligence to improve process quality: Evidence from semiconductor manufacturing. Management Sci. 68(8):5704–5723.Link, Google Scholar
• Senoner J, Schallmoser S, Kratzwald B, Feuerriegel S, Netland T (2024) Explainable AI improves task performance in human-AI collaboration. Sci. Rep. 14:31150.Google Scholar
• Spitzer P, Kühl N, Goutier M, Kaschura M, Satzger G (2024) Transferring domain knowledge with (X)AI-based learning systems. Avital M, Karahanna E, Themistocleous M, Constantiou ID, Fitzgerald B, Seidel S, eds. 32nd Eur. Conf. Inform. Systems (ECIS).Google Scholar
• Spreitzer GM (1995) Psychological empowerment in the workplace: Dimensions, measurement, and validation. Acad. Management J. 38(5):1442–1465.Crossref, Google Scholar
• Tankelevitch L, Kewenig V, Simkute A, Scott AE, Sarkar A, Sellen A, Rintel S (2024) The metacognitive demands and opportunities of generative AI. Proc. 2024 CHI Conf. Human Factors Comput. Systems (ACM, New York), 1–24.Google Scholar
• Taudien A, Fuegener A, Gupta A, Ketter W (2022) Calibrating users’ mental models for delegation to AI. Proc. 43rd Internat. Conf. Inform. Systems (Association for Information Systems, Atlanta).Google Scholar
• Taudien A, Walzner DD, Fuegener A, Gupta A, Ketter W (2024) Know thyself: The relationship between metacognition and human-AI collaboration. Internat. Conf. Inform. Systems (Association for Information Systems, Atlanta).Google Scholar
• Tchuente D, Nyawa S (2022) Real estate price estimation in French cities using geocoding and machine learning. Ann. Oper. Res. 308:571–608.Crossref, Google Scholar
• Thompson VA, Markovits H (2025) Fast reasoning and metacognition. Psychonomic Bull. Rev. 32(4):1915–1921.Crossref, Google Scholar
• van Rensburg JJ, Santos CM, de Jong SB, Uitdewilligen S (2022) The five-factor perceived shared mental model scale: A consolidation of items across the contemporary literature. Frontiers Psych. 12:784200.Crossref, Google Scholar
• Wang W, Benbasat I (2007) Recommendation agents for electronic commerce: Effects of explanation facilities on trusting beliefs. J. Management Inform. Systems 23(4):217–246.Crossref, Google Scholar
• Yeung N, Summerfield C (2012) Metacognition in human decision-making: Confidence and error monitoring. Philos. Trans. Roy. Soc. London Ser. B Biol. Sci. 367(1594):1310–1321.Google Scholar
• You S, Yang CL, Li X (2022) Algorithmic versus human advice: Does presenting prediction performance matter for algorithm appreciation? J. Management Inform. Systems 39(2):336–365.Crossref, Google Scholar