Warnings and Endorsements: Improving Human–AI Collaboration in the Presence of Outliers

References

• Babic B, Cohen IG, Evgeniou T, Gerke S (2021) When machine learning goes off the rails. Harvard Bus. Rev. https://hbr.org/2021/01/when-machine-learning-goes-off-the-rails.Google Scholar
• Baier L, Jöhren F, Seebacher S (2019) Challenges in the deployment and operation of machine learning in practice. vom Brocke J, Gregor S, Müller O, eds. Proc. 27th Eur. Conf. Inform. Systems (ECIS) (Association for Information Systems, Atlanta), 163.Google Scholar
• Balakrishnan M, Ferreira KJ, Tong J (2025) Human-algorithm collaboration with private information: Naïve advice-weighting behavior and mitigation. Management Sci., ePub ahead of print March 24, https://doi.org/10.1287/mnsc.2022.03850.Link, Google Scholar
• Bayram F, Ahmed BS (2023) A domain-region based evaluation of ML performance robustness to covariate shift. Neural Comput. Appl. 35:17555–17577.Crossref, Google Scholar
• Ben-Gal I (2005) Outlier detection. Maimon O, Rokach L eds. Data Mining and Knowledge Discovery Handbook (Springer, New York), 1117–1130.Crossref, Google Scholar
• Caro F, Saez de Tejada Cuenca A (2023) Believing in analytics: Managers’ adherence to price recommendations from a DSS. Manufacturing Service Oper. Management 25(2):524–542.Link, Google Scholar
• Castelo N, Bos MW, Lehmann DR (2019) Task-dependent algorithm aversion. J. Marketing Res. 56(5):809–825.Crossref, Google Scholar
• Chiang CW, Yin M (2021) You’d better stop! Understanding human reliance on machine learning models under covariate shift. 13th ACM Web Sci. Conf. UK / Virtual Event (Association for Computing Machinery, New York), 120–129.Google Scholar
• De-Arteaga M, Fogliato R, Chouldechova A (2020) A case for humans-in-the-loop: Decisions in the presence of erroneous algorithmic scores. Proc. 2020 CHI Conf. Human Factors Comput. Systems (ACM, New York), 1–12.Google Scholar
• Dietvorst BJ, Simmons JP, Massey C (2015) Algorithm aversion: People erroneously avoid algorithms after seeing them err. J. Experiment. Psych. General 144(1):114–126.Crossref, Google Scholar
• Dietvorst BJ, Simmons JP, Massey C (2018) Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them. Management Sci. 64(3):1155–1170.Link, Google Scholar
• Ferreira KJ, Lee BHA, Simchi-Levi D (2016) Analytics for an online retailer: Demand forecasting and price optimization. Manufacturing Service Oper. Management 18(1):69–88.Link, Google Scholar
• Ghorbani H (2019) Mahalanobis distance and its application for detecting multivariate outliers. Facta Universitatis Ser. Math. Informatics 34(3):583–595.Crossref, Google Scholar
• Ibrahim R, Kim SH, Tong J (2021) Eliciting human judgment for prediction algorithms. Management Sci. 67(4):2314–2325.Link, Google Scholar
• Kawaguchi K (2021) When will workers follow an algorithm? A field experiment with a retail business. Management Sci. 67(3):1670–1695.Link, Google Scholar
• Kesavan S, Kushwaha T (2020) Field experiment on the profit implications of merchants’ discretionary power to override data-driven decision-making tools. Management Sci. 66(11):5182–5190.Link, Google Scholar
• Kouw WM, Loog M (2019) A review of domain adaptation without target labels. IEEE Trans. Pattern Anal. Machine Intelligence 43(3):766–785.Crossref, Google Scholar
• Kwon C, Raman A, Tamayo J (2024) Human-algorithm interactions in labor scheduling decisions. Preprint, submitted July 5, http://dx.doi.org/10.2139/ssrn.4773773.Google Scholar
• Lai V, Tan C (2019) On human predictions with explanations and predictions of machine learning models: A case study on deception detection. Proc. Conf. Fairness Accountability Transparency, 29–38.Google Scholar
• McGrath S, Mehta P, Zytek A, Lage I, Lakkaraju H (2023) When does uncertainty matter? Understanding the impact of predictive uncertainty in ML assisted decision making. Transact. Machine Learn. Res. (OpenReview.net).Google Scholar
• NewVantagePartners (2023) Data and analytics leadership annual executive survey. Technical report, NewVantage Partners, Boston.Google Scholar
• Ovadia Y, Fertig E, Ren J, Nado Z, Sculley D, Nowozin S, Dillon J, Lakshminarayanan B, Snoek J (2019) Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. Adv. Neural Inform. Processing Systems, vol. 32 (ACM, New York).Google Scholar
• Poursabzi-Sangdeh F, Goldstein DG, Hofman JM, Wortman Vaughan JW, Wallach H (2021) Manipulating and measuring model interpretability. Kitamura Y, Quigley A, Isbister K, Igarashi T, Bjørn P, Drucker SM, eds. Proc. 2021 CHI Conf. Human Factors Comput. Systems. Yokohama, Japan / Virtual Event (Association for Computing Machinery, New York).Google Scholar
• Qin X, Jiang Z (2019) The impact of AI on the advertising process: The Chinese experience. J. Advertising 48(4):338–346.Crossref, Google Scholar
• Shimodaira H (2000) Improving predictive inference under covariate shift by weighting the log-likelihood function. J. Statist. Planning Inference 90(2):227–244.Crossref, Google Scholar
• Sniezek JA, Buckley T (1995) Cueing and cognitive conflict in judge-advisor decision making. Organ. Behav. Human Decision Processes 62(2):159–174.Crossref, Google Scholar
• Snyder C, Keppler S, Leider S (2025) Algorithm reliance: Fast and slow. Management Sci., ePub ahead of print May 6, https://doi.org/10.1287/mnsc.2023.01989.Link, Google Scholar
• Sugiyama M, Krauledat M, Müller KR (2007) Covariate shift adaptation by importance weighted cross validation. J. Machine Learn. Res. 8(5):985–1005.Google Scholar
• Sun J, Zhang DJ, Hu H, Van Mieghem JA (2022) Predicting human discretion to adjust algorithmic prescription: A large-scale field experiment in warehouse operations. Management Sci. 68(2):846–865.Link, Google Scholar
• Tversky A, Kahneman D (1974) Judgment under uncertainty: Heuristics and biases. Science 185(4157):1124–1131.Crossref, Google Scholar
• Yeomans M, Shah A, Mullainathan S, Kleinberg J (2019) Making sense of recommendations. J. Behav. Decision Making 32(4):403–414.Crossref, Google Scholar
• Yin M, Wortman Vaughan J, Wallach H (2019) Understanding the effect of accuracy on trust in machine learning models. Proc. 2019 CHI Conf. Human Factors Comput. Systems (ACM, New York), 1–12.Google Scholar
• Zhang Y, Liao QV, Bellamy RK (2020) Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. Proc. 2020 Conf. Fairness Accountability Transparency (ACM, New York), 295–305.Google Scholar