Help
Cart
Skip main navigation

Available Issues

July 20, 2022 - September 3, 2026

Available Issues

July 20, 2022 - September 3, 2026

Predicting Effects, Missing Distributions: Evaluating Large Language Models as Human-Behavior Simulators in Operations Management

Published Online:20 Oct 2026https://doi.org/10.1287/ijds.2025.0124

References

  • Aher GV, Arriaga RI, Kalai AT (2023) Using large language models to simulate multiple humans and replicate human subject studies . Krause A, Brunskill E, Cho K, Engelhardt B, Sabato S, Scarlett J, eds., Proc. 40th Internat. Conf. Machine Learn.(PMLR, New York), 337–371.Google Scholar
  • Argyle LP, Busby EC, Fulda N, Gubler JR, Rytting C, Wingate D (2023) Out of one, many: Using language models to simulate human samples. Political Anal. 31(3):337–351.Google Scholar
  • Arora N, Chakraborty I, Nishimura Y (2024) Revolutionizing marketing research with a large language model: A hybrid AI-human approach. Preprint, submitted March 27, https://dx.doi.org/10.2139/ssrn.4683054.Google Scholar
  • Balandat M, Karrer B, Jiang DR, Daulton S, Letham B, Wilson AG, Bakshy E (2020) BoTorch: A framework for efficient Monte-Carlo Bayesian optimization. Larochelle H, Ranzato M, Hadsell R, Balcan MF, Lin H, eds. Adv. Neural Inform. Processing Systems, vol. 33 (Curran Associates, Inc., Red Hook, NY), 21524–21538.Google Scholar
  • Cao Y, Liu H, Arora A, Augenstein I, Röttger P, Hershcovich D (2025) Specializing large language models to simulate survey response distributions for global populations. Chiruzzo L, Ritter A, Wang L, eds. Proc. 2025 Conf. Nations Americas Chapter Assoc. Comput. Linguistics Human Language Tech., vol. 1 (ACL, Kerrville, TX), 3141–3154.Google Scholar
  • Charness G, Rabin M (2002) Understanding social preferences with simple tests. Quart. J. Econom. 117(3):817–869.Google Scholar
  • Chen L, Kök AG, Tong JD (2013) The effect of payment schemes on inventory decisions: The role of mental accounting. Management Sci. 59(2):436–451.Link, Google Scholar
  • Chen Y, Kirshner SN, Ovchinnikov A, Andiappan M, Jenkin T (2025) A manager and an AI walk into a bar: Does ChatGPT make biased decisions like we do? Manufacturing Service Oper. Management 27(2):354–368.Link, Google Scholar
  • Chung HW, Hou L, Longpre S, Zoph B, Tay Y, Fedus W, Li Y, et al. (2024) Scaling instruction-finetuned language models. J. Machine Learn. Res. 25(1):70.Google Scholar
  • Croson R, Donohue K (2006) Behavioral causes of the bullwhip effect and the observed value of inventory information. Management Sci. 52(3):323–336.Link, Google Scholar
  • Davis AM, Katok E, Kwasnica AM (2011) Do auctioneers pick optimal reserve prices? Management Sci. 57(1):177–192.Link, Google Scholar
  • Davis AM, Flicker B, Hyndman K, Katok E, Keppler S, Leider S, Long X, Tong JD (2023) A replication study of operations management experiments in Management Science. Management Sci. 69(9):4977–4991.Link, Google Scholar
  • Donohue K, Katok E, Leider S, eds. (2018) The Handbook of Behavioral Operations (John Wiley & Sons, Hoboken, NJ).Google Scholar
  • Engelbrecht-Wiggans R, Katok E (2008) Regret and feedback information in first-price sealed-bid auctions. Management Sci. 54(4):808–819.Link, Google Scholar
  • Gao C, Lan X, Li N, Yuan Y, Ding J, Zhou Z, Xu F, Li Y (2024) Large language models empowered agent-based modeling and simulation: A survey and perspectives. Humanities Soc. Sci. Comm. 11(1):1259.Google Scholar
  • Garnett R (2023) Bayesian Optimization (Cambridge University Press, Cambridge, UK).Google Scholar
  • Goli A, Singh A (2024) Frontiers: Can large language models capture human preferences? Marketing Sci. 43(4):709–722.Link, Google Scholar
  • Ho TH, Zhang J (2008) Designing pricing contracts for boundedly rational customers: Does the framing of the fixed fee matter? Management Sci. 54(4):686–700.Link, Google Scholar
  • Holtzman A, Buys J, Du L, Forbes M, Choi Y (2020) The curious case of neural text degeneration. Eighth Internat. Conf. Learn. Representations (ICLR, Appleton, WI). Google Scholar
  • Horton JJ, Filippas A, Manning BS (2023) Large language models as simulated economic agents: What can we learn from Homo silicus? Preprint, submitted January 18, https://arxiv.org/abs/2301.07543v1.Google Scholar
  • Howard J, Ruder S (2018) Universal language model fine-tuning for text classification. Gurevych I, Miyao Y, eds. Proc. 56th Annual Meeting Assoc. Comput. Linguistics, vol. 1 (ACL, Stroudsburg, PA), 328–339.Google Scholar
  • Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, Wang L, Chen W (2022) LoRA: Low-rank adaptation of large language models. Tenth Internat. Conf. Learn. Representations, (ICLR, Appleton, WI).Google Scholar
  • Ji Z, Lee N, Frieske R, Yu T, Su D, Xu Y, Ishii E, Bang YJ, Madotto A, Fung P (2023) Survey of hallucination in natural language generation. ACM Comput. Surveys 55(12):248.Google Scholar
  • Jin Y, Yang R, Yi Z, Shen X, Peng H, Liu X, Qin J, et al. (2024) Surrealdriver: Designing LLM-powered generative driver agent framework based on human drivers’ driving-thinking data. 2024 IEEE/RSJ Internat. Conf. Intelligent Robots Systems (IROS) (IEEE, Piscataway, NJ), 966–971.Google Scholar
  • Kahneman D, Knetsch JL, Thaler R (1986) Fairness as a constraint on profit seeking: Entitlements in the market. Amer. Econom. Rev. 76(4):728–741.Google Scholar
  • Kremer M, Debo L (2016) Inferring quality from wait time. Management Sci. 62(10):3023–3038.Link, Google Scholar
  • Kremer M, Moritz B, Siemsen E (2011) Demand forecasting behavior: System neglect and change detection. Management Sci. 57(10):1827–1843.Link, Google Scholar
  • Kuhn D, Esfahani PM, Nguyen VA, Shafieezadeh-Abadeh S (2019) Wasserstein distributionally robust optimization: Theory and applications in machine learning. Oper. Res. Management Sci. Age Analytics 2019(October):130–166.Link, Google Scholar
  • Leng Y, Sang Y, Agarwal A (2024) Reduce preference disparity between LLMs and humans: Calibration to compose human-like ensembles. Preprint, submitted April 20, https://doi.org/10.2139/ssrn.4802019.Google Scholar
  • Li P, Castelo N, Katona Z, Sarvary M (2024b) Frontiers: Determining the validity of large language models for automated perceptual analysis. Marketing Sci. 43(2):254–266.Link, Google Scholar
  • Li N, Gao C, Li M, Li Y, Liao Q (2024a) EconAgent: Large language model-empowered agents for simulating macroeconomic activities. Ku LW, Martins A, Srikumar V, eds. Proc. 62nd Annual Meeting Assoc. Comput. Linguistics, vol. 1 (ACL, Kerrville, TX), 15523–15536.Google Scholar
  • Mei Q, Xie Y, Yuan W, Jackson MO (2024) A Turing test of whether AI chatbots are behaviorally similar to humans. Proc. Natl. Acad. Sci. USA 121(9):e2313925121.Google Scholar
  • Nguyen M, Baker A, Neo C, Roush AG, Kirsch A, Shwartz-Ziv R (2025) Turning up the heat: Min-p sampling for creative and coherent LLM outputs. Thirteenth Internat. Conf. Learn. Representations (ICLR, Appleton, WI).Google Scholar
  • Özer O, Zheng Y, Chen KY (2011) Trust in forecast information sharing. Management Sci. 57(6):1111–1137.Link, Google Scholar
  • Samuelson W, Zeckhauser R (1988) Status quo bias in decision making. J. Risk Uncertainty 1(1):7–59. Google Scholar
  • Santurkar S, Durmus E, Ladhak F, Lee C, Liang P, Hashimoto T (2023) Whose opinions do language models reflect? Krause A, Brunskill E, Cho K, Engelhardt B, Sabato S, Scarlett J, eds. Proc. 40th Internat. Conf. Machine Learn. (PMLR, New York), 29971–30004.Google Scholar
  • Schweitzer ME, Cachon GP (2000) Decision bias in the newsvendor problem with a known demand distribution: Experimental evidence. Management Sci. 46(3):404–420.Link, Google Scholar
  • Shahriari B, Swersky K, Wang Z, Adams RP, de Freitas N (2016) Taking the human out of the loop: A review of Bayesian optimization. Proc. IEEE 104(1):148–175.Google Scholar
  • Shunko M, Niederhoff J, Rosokha Y (2018) Humans are not machines: The behavioral impact of queueing design on service time. Management Sci. 64(1):453–473.Link, Google Scholar
  • Suh J, Jahanparast E, Moon S, Kang M, Chang S (2025) Language model fine-tuning on scaled survey data for predicting distributions of public opinions. Che W, Nabende J, Shutova E, Pilehvar MT, eds. Proc. 63rd Annual Meeting Assoc. Comput. Linguistics, vol. 1 (ACL, Kerrville, TX), 21147–21170.Google Scholar
  • Wang A, Morgenstern J, Dickerson JP (2025) Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence 7(3):400–411.Google Scholar
  • Wang M, Zhang DJ, Zhang H (2026) Large language models for market research: A data-augmentation approach. Marketing Sci. 45(4):728–751.Link, Google Scholar
  • Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, Chi E, Le QV, Zhou D (2022a) Chain-of-thought prompting elicits reasoning in large language models. Koyejo S, Mohamed S, Agarwal A, Belgrave D, Cho K, Oh A, eds. Adv. Neural Inform. Processing Systems, vol. 35 (Curran Associates, Inc., Red Hook, NY), 24824–24837.Google Scholar
  • Wei J, Bosma M, Zhao V, Guu K, Yu AW, Lester B, Du N, Dai AM, Le QV (2022b) Finetuned language models are zero-shot learners. Tenth Internat. Conf. Learn. Representations (ICLR, Appleton, WI).Google Scholar
  • Wei J, Tay Y, Bommasani R, Raffel C, Zoph B, Borgeaud S, Yogatama D, et al. (2022c) Emergent abilities of large language models. Trans. Machine Learn. Res., ePub ahead of print August 31, https://openreview.net/forum?id=yzkSU5zdwD.Google Scholar
  • Williams R, Hosseinichimeh N, Majumdar A, Ghaffarzadegan N (2023) Epidemic modeling with generative agents. Preprint, submitted July 11, https://arxiv.org/abs/2307.04986.Google Scholar
  • Xie C, Chen C, Jia F, Ye Z, Lai S, Shu K, Gu J, et al. (2024) Can large language model agents simulate human trust behavior? Globerson A, Mackey L, Belgrave D, Fan A, Paquet U, Tomczak J, Zhang C, eds. Adv. Neural Inform. Processing Systems, vol. 37 (Curran Associates, Inc., Red Hook, NY), 15674–15729.Google Scholar
  • Yang K, Li H, Wen H, Peng TQ, Tang J, Liu H (2024) Are large language models (LLMs) good social predictors? Al-Onaizan Y, Bansal M, Chen YN, eds. Findings Assoc. Comput. Linguistics EMNLP (ACL, Kerrville, TX), 2718–2730.Google Scholar
  • Zhang Z, Wang D, Chen N, Mansur R, Sarhangian V (2026) Comparing exploration-exploitation strategies of LLMs and humans: Insights from standard multi-armed bandit experiments. Preprint, submitted May 1, https://arxiv.org/abs/2505.09901.Google Scholar
  • Zhang T, Wu F, Katiyar A, Weinberger KQ, Artzi Y (2021) Revisiting few-sample BERT fine-tuning. Ninth Internat. Conf. Learn. Representations (ICLR, Appleton, WI).Google Scholar
Previous
Back to Top
Next
INFORMS site uses cookies to store information on your computer. Some are essential to make our site work; Others help us improve the user experience. By using this site, you consent to the placement of these cookies. Please read our Privacy Statement to learn more.
Agree