Eliciting Human Judgment for Prediction Algorithms

Rouba Ibrahim
Rouba Ibrahim
[email protected]
https://orcid.org/0000-0001-7643-4604
School of Management, University College London, London E14 5AA, United Kingdom;
Search for more papers by this author
,
Song-Hee Kim
Song-Hee Kim
[email protected]
https://orcid.org/0000-0002-3106-5726
Marshall School of Business, University of Southern California, Los Angeles, California 90089;
Search for more papers by this author
,
Jordan Tong
Jordan Tong
[email protected]
https://orcid.org/0000-0002-1962-058X
Wisconsin School of Business, University of Wisconsin–Madison, Madison, Wisconsin 53706
Search for more papers by this author

School of Management, University College London, London E14 5AA, United Kingdom;

Search for more papers by this author

Song-Hee Kim

[email protected]

https://orcid.org/0000-0002-3106-5726

Marshall School of Business, University of Southern California, Los Angeles, California 90089;

Search for more papers by this author

Jordan Tong

[email protected]

https://orcid.org/0000-0002-1962-058X

Wisconsin School of Business, University of Wisconsin–Madison, Madison, Wisconsin 53706

Search for more papers by this author

Published Online:25 Jan 2021https://doi.org/10.1287/mnsc.2020.3856

Supplemental Material

mnsc.2020.3856.sm1.pdf

The replication files for this article are available HERE.

Cited by
- Is AI an Algorithm by Any Other Name? Behavioral Reactions to AI ‐ and Model‐Based Demand Planning Algorithms
  13 April 2026 | Journal of Operations Management, Vol. 14
- Targeted Automation and Sustaining Human-AI Learning
  7 October 2025 | Production and Operations Management, Vol. 35, No. 4
- Can Providing Algorithmic Performance Information Facilitate Humans’ Inventory Ordering Behaviors?
  Yingda Lu,
  Xueming Luo,
  Liqiang Huang,
  Danni Wang
  11 April 2025 | Information Systems Research, Vol. 37, No. 1
- How Forced Intervention Facilitates AI Adoption
  Xinyu Cao,
  Chenshan Hu,
  Jiankun Sun,
  Dennis J. Zhang
  24 February 2026 | Manufacturing & Service Operations Management, Vol. 0, No. 0
- Algorithmic Decision‐Making Safeguarded by Human Knowledge
  26 January 2026 | Naval Research Logistics (NRL), Vol. 35
- Identity Disclosure and Anthropomorphism in Voice Chatbot Design: A Field Experiment
  Yuqian Xu,
  Hongyan Dai,
  Wanfeng Yan
  26 August 2024 | Management Science, Vol. 72, No. 1
- The Power of Disagreement: A Field Experiment to Investigate Human–Algorithm Collaboration in Loan Evaluations
  Hongchang Wang,
  Yingjie Zhang,
  Tian Lu
  4 July 2025 | Management Science, Vol. 72, No. 1
- Human-Centered Artificial Intelligence: A Field Experiment
  Sebastian Krakowski,
  Darek Haftor,
  Johannes Luger,
  Natallia Pashkevich,
  Sebastian Raisch
  9 June 2025 | Management Science, Vol. 72, No. 1
- Human-Algorithm Collaboration with Private Information: Naïve Advice-Weighting Behavior and Mitigation
  Maya Balakrishnan,
  Kris Johnson Ferreira,
  Jordan Tong
  24 March 2025 | Management Science, Vol. 72, No. 1
- Managerial Insight and “Optimal” Algorithms
  Blair Flicker
  14 August 2025 | Management Science, Vol. 72, No. 1
- Algorithm Reliance: Fast and Slow
  Clare Snyder,
  Samantha Keppler,
  Stephen Leider
  6 May 2025 | Management Science, Vol. 72, No. 1
- Is Your Machine Better Than You? You May Never Know
  Francis de Véricourt,
  Huseyin Gurkan
  25 May 2023 | Management Science, Vol. 72, No. 1
- Behavioral Externalities of Process Automation
  Ruth Beer,
  Anyan Qi,
  Ignacio Rios
  2 July 2025 | Management Science, Vol. 72, No. 1
- Profit Implications of Judgmental Adjustments to Forecast Inputs: Evidence from a Large-Scale Field Experiment
  Saravanan Kesavan,
  Tarun Kushwaha,
  Dayton Steele
  25 July 2025 | Management Science, Vol. 72, No. 1
- Coupling Artificial Intelligence and Supply Chain Operations
  25 September 2025
- AI in operations management: Literature review and future research
- Collaborative Intelligence in Sequential Experiments: A Human-in-the-Loop Framework for Drug Discovery
  Jinghai He,
  Cheng Hua,
  Yingfei Wang,
  Zeyu Zheng
  3 December 2025 | Information Systems Research, Vol. 0, No. 0
- Complementarity in human-AI collaboration: concept, sources, and evidence
  27 August 2025 | European Journal of Information Systems, Vol. 34, No. 6
- Designing AI-Based Work Processes: How the Timing of AI Advice Affects Diagnostic Decision Making
  Jiamin Yin,
  Kee Yuan Ngiam,
  Sharon Swee-Lin Tan,
  Hock Hai Teo
  12 March 2025 | Management Science, Vol. 71, No. 11
- Warnings and Endorsements: Improving Human–AI Collaboration in the Presence of Outliers
  Matthew DosSantos DiSorbo,
  Kris Johnson Ferreira,
  Maya Balakrishnan,
  Jordan Tong
  16 September 2025 | Manufacturing & Service Operations Management, Vol. 27, No. 6
- Democratizing Data Analytics Products to Drive SME Growth: A Natural Experiment
  Zhe Yuan,
  Yitong Wang,
  Tianshu Sun,
  Huilan Xu
  26 May 2025 | Manufacturing & Service Operations Management, Vol. 27, No. 4
- Enhancing AI-Assisted Purchase Decisions: The Role of the Sense of Autonomy
  16 April 2025 | MIS Quarterly, Vol. 29
- Service-level anchoring in demand forecasting: The moderating impact of retail promotions and product perishability
  International Journal of Forecasting, Vol. 41, No. 2
- 1 + 1 > 2? Information, Humans, and Machines
  Tian Lu; ,
  Yingjie Zhang
  2 May 2024 | Information Systems Research, Vol. 36, No. 1
- Supply chain resilience: A review from the inventory management perspective
  Fundamental Research, Vol. 5, No. 2
- Human-AI Ensembles: When Can They Work?
  3 October 2023 | Journal of Management, Vol. 51, No. 2
- OM Forum—Operations Management Research: Relevance and Impact
  Jayashankar M. Swaminathan
  23 October 2024 | Manufacturing & Service Operations Management, Vol. 27, No. 1
- Chicken or the egg? Forecast and order adjustments in supply-chain planning
  1 January 2025 | SSRN Electronic Journal, Vol. 23
- Agentic AI: Service Operations with Augmentation and Automation AI
  1 January 2025 | SSRN Electronic Journal, Vol. 69
- Information Design and Coordination in Cloud Kitchens -How Human-Tech Overrides affect Order Fulfillment time?
  1 January 2025 | SSRN Electronic Journal, Vol. 46
- Managing with Artificial Intelligence: An Integrative Framework
  Academy of Management Annals, Vol. 19, No. 1
- Queue Configurations and Operational Performance: An Interplay Between Customer Ownership and Queue Length Awareness
  Hummy Song,
  Mor Armony,
  Guillaume Roels
  19 August 2024 | Manufacturing & Service Operations Management, Vol. 26, No. 6
- OM Forum—The Best of Both Worlds: Machine Learning and Behavioral Science in Operations Management
  Andrew M. Davis,
  Shawn Mankad,
  Charles J. Corbett,
  Elena Katok
  25 July 2024 | Manufacturing & Service Operations Management, Vol. 26, No. 5
- Judgment in Business and Management Research: Shedding New Light on a Familiar Concept
  Academy of Management Annals, Vol. 18, No. 2
- Human and Machine: The Impact of Machine Input on Decision Making Under Cognitive Limitations
  Tamer Boyacı,
  Caner Canyakmaz,
  Francis de Véricourt
  31 March 2023 | Management Science, Vol. 70, No. 2
- Utilizing people, analytics, and AI for decision making in the digitalized retail supply chain
  6 June 2023 | Journal of Business Logistics, Vol. 45, No. 1
- Generative AI and Copyright: A Dynamic Perspective
  1 January 2024 | SSRN Electronic Journal, Vol. 13
- Value of Autonomous Last-mile Delivery: Evidence from Alibaba
  1 January 2024 | SSRN Electronic Journal, Vol. 22
- The Best of Both Worlds: Machine Learning and Behavioral Science in Operations Management
  1 January 2024 | SSRN Electronic Journal, Vol. 58
- Using Machine Learning to Create an Early Warning System for Welfare Recipients*
  2 April 2023 | Oxford Bulletin of Economics and Statistics, Vol. 85, No. 5
- AI vs. Human Buyers: A Study of Alibaba’s Inventory Replenishment System
  Jiaxi Liu,
  Shuyi Lin,
  Linwei Xin,
  Yidong Zhang
  26 September 2023 | INFORMS Journal on Applied Analytics, Vol. 53, No. 5
- Demand planning for the digital supply chain: How to integrate human judgment and predictive analytics
  15 May 2023 | Journal of Operations Management, Vol. 69, No. 6
- Believing in Analytics: Managers’ Adherence to Price Recommendations from a DSS
  Felipe Caro,
  Anna Sáez de Tejada Cuenca
  24 January 2023 | Manufacturing & Service Operations Management, Vol. 25, No. 2
- Supply Chain Resilience: A Review from the Inventory Management Perspective
  1 January 2023 | SSRN Electronic Journal, Vol. 56
- Ai Enforcement: Examining the Impact of Ai on Judicial Fairness and Public Safety
  1 January 2023 | SSRN Electronic Journal, Vol. 47
- Role of Presentation Explicitness in Human–Artificial Intelligence Collaboration: A Field Study in a Loan Approval Service
  1 January 2023 | SSRN Electronic Journal, Vol. 21
- Optimal Design and Pricing of Sequenced Bundles in the Presence of Satiation
  1 January 2023 | SSRN Electronic Journal, Vol. 65
- Human-AI Co-Creation in Product Ideation: the Dual View of Quality and Diversity
  1 January 2023 | SSRN Electronic Journal, Vol. 10
- When Should Doctors and Patients Use Shared Decision-Making Under Bounded Rationality?
  1 January 2023 | SSRN Electronic Journal, Vol. 42
- Evaluating human behaviour in response to AI recommendations for judgemental forecasting
  European Journal of Operational Research, Vol. 303, No. 3
- Aggregating Human Judgment Probabilistic Predictions of Coronavirus Disease 2019 Transmission, Burden, and Preventive Measures
  25 July 2022 | Open Forum Infectious Diseases, Vol. 9, No. 8
- Applied Machine Learning in Operations Management
  11 May 2021
- Forecasting with Judgment
  8 July 2022
- 1+1>2? Information, Humans, and Machines
  SSRN Electronic Journal, Vol. 33
- Algorithm Reliance Under Pressure: The Effect of Customer Load on Service Workers
  SSRN Electronic Journal, Vol. 9325
- Is your Machine Better than You? You May Never Know
  SSRN Electronic Journal, Vol. 78
- Can Providing Algorithmic Performance Information Facilitate Humans’ Inventory Ordering Behaviors?
  SSRN Electronic Journal, Vol. 31
- Sustained Learning Under Algorithm-Driven Automation
  SSRN Electronic Journal, Vol. 67
- Behavioral Externalities of Process Automation
  SSRN Electronic Journal, Vol. 70
- Chatbots in Logistics: A Field Experiment on Intelligent Freight Dispatching
  SSRN Electronic Journal, Vol. 119
- Improving Human-Algorithm Collaboration: Causes and Mitigation of Over- and Under-Adherence
  SSRN Electronic Journal, Vol. 86
- Algorithm, Human, or the Centaur: How to Enhance Clinical Care?
  SSRN Electronic Journal, Vol. 9
- Crowdsourced Perceptions of Human Behavior to Improve Computational Forecasts of US National Incident Cases of COVID-19: Survey Study
  30 December 2022 | JMIR Public Health and Surveillance, Vol. 8, No. 12
- Human and AI Ensembles: When Can They Work?
  1 January 2021 | SSRN Electronic Journal, Vol. 70
- Queue Configurations and Operational Performance: An Interplay between Customer Ownership and Queue Length Awareness
  1 January 2021 | SSRN Electronic Journal, Vol. 22
- Consumer and AI Co-creation: When and Why Human Participation Improves AI Creation.
  SSRN Electronic Journal, Vol. 25
- From Forced to Voluntary AI Adoption
  SSRN Electronic Journal, Vol. 61
- Human and Machine: The Impact of Machine Input on Decision-Making Under Cognitive Limitations
  SSRN Electronic Journal, Vol. 59

Volume 67, Issue 4

April 2021

Pages v, 1993-2656, iii-iv

Article Information

Supplemental Material

Metrics

Information

Received:May 13, 2020
Accepted:August 10, 2020
Published Online:January 25, 2021

Cite as

Rouba Ibrahim , Song-Hee Kim , Jordan Tong (2021) Eliciting Human Judgment for Prediction Algorithms. Management Science 67(4):2314-2325.

https://doi.org/10.1287/mnsc.2020.3856

Keywords

PDF download

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Eliciting Human Judgment for Prediction Algorithms

Supplemental Material

Volume 67, Issue 4

Article Information

Supplemental Material

Metrics

Information

Cite as

Keywords

Sign Up for INFORMS Publications Updates and News