Hybrid Intelligence Forecasting: A Review

References

• Abbas AE, Hupman AC, Monger R (2026) Assessing the quality of large language models and human inputs to a decision: A proposed framework and two benchmark case studies. Decision Anal. 23(3):205–220.Link, Google Scholar
• Abdel-Karim BM, Pfeuffer N, Rohde G, Hinz O (2020) How and what can humans learn from being in the loop? Invoking contradiction learning as a measure to make humans smarter. KI-Künstliche Intelligenz 34(2):199–207.Crossref, Google Scholar
• Abolghasemi M, Ganbold O, Rotaru K (2025) Humans vs. large language models: Judgmental forecasting in an era of advanced AI. Internat. J. Forecasting 41(2):631–648.Crossref, Google Scholar
• Agrawal A, Gans J, Goldfarb A (2022) Prediction Machines, Updated and Expanded: The Simple Economics of Artificial Intelligence (Harvard Business Review Press, Boston).Google Scholar
• Agarwal N, Moehring A, Rajpurkar P, Salz T (2023) Combining human expertise with artificial intelligence: Experimental evidence from radiology. NBER Working Paper No. 31422, National Bureau of Economic Research, Cambridge, MA.Google Scholar
• Alhaji B, Beecken J, Ehlers R, Gertheiss J, Merz F, Müller JP, Prilla M, et al. (2020) Engineering human–Machine teams for trusted collaboration. Big Data Cognitive Comput. 4(4):35.Crossref, Google Scholar
• Araujo T, Helberger N, Kruikemeier S, De Vreese CH (2020) In AI we trust? Perceptions about automated decision-making by artificial intelligence. AI Soc. 35(3):611–623.Crossref, Google Scholar
• Armstrong JS (2001) Combining forecasts. Armstrong JS, ed. Principles of Forecasting: A Handbook for Researchers and Practitioners, International Series in Operations Research & Management Science, vol. 30 (Springer, Boston), 417–439.Crossref, Google Scholar
• Arvan M, Fahimnia B, Reisi M, Siemsen E (2019) Integrating human judgement into quantitative forecasting methods: A review. Omega 86:237–252.Crossref, Google Scholar
• Bates JM, Granger CW (1969) The combination of forecasts. J. Oper. Res. Soc. 20(4):451–468.Crossref, Google Scholar
• Ben-Assuli O, Heart T, Klempfner R, Padman R (2023) Human-machine collaboration for feature selection and integration to improve congestive heart failure risk prediction. Decision Support Systems 172:113982.Crossref, Google Scholar
• Benjamin DM, Morstatter F, Abbas AE, Abeliuk A, Atanasov P, Bennett S, Beger A, et al. (2023) Hybrid forecasting of geopolitical events. AI Magazine 44(1):112–128.Crossref, Google Scholar
• Blattberg RC, Hoch SJ (1990) Database models and managerial intuition: 50% model + 50% manager. Management Sci. 36(8):887–899.Link, Google Scholar
• Brau R, Aloysius J, Siemsen E (2023) Demand planning for the digital supply chain: How to integrate human judgment and predictive analytics. J. Oper. Management 69(6):965–982.Crossref, Google Scholar
• Broomell SB, Budescu DV (2009) Why are experts correlated? Decomposing correlations between judges. Psychometrika 74(3):531–553.Crossref, Google Scholar
• Carvalho A (2016) An overview of applications of proper scoring rules. Decision Anal. 13(4):223–242.Link, Google Scholar
• Chacon A, Kausel EE, Reyes T, Trautmann S (2025) Preventing algorithm aversion: People are willing to use algorithms with a learning label. J. Bus. Res. 187:115032.Crossref, Google Scholar
• Clemen RT (1989) Combining forecasts: A review and annotated bibliography. Internat. J. Forecasting 5(4):559–583.Crossref, Google Scholar
• Cooke RM (2026) Wisdom/madness of crowds and perils of point forecasts. Decision Anal., ePub ahead of print April 17, https://doi.org/10.1287/deca.2025.0437.Link, Google Scholar
• Davis-Stober CP, Budescu DV, Broomell SB, Dana J (2015) The composition of optimally wise crowds. Decision Anal. 12(3):130–143.Link, Google Scholar
• De Véricourt F, Gurkan H (2023) Is your machine better than you? You may never know. Management Sci. 72(1):558–574.Link, Google Scholar
• Dellermann D, Ebel P, Söllner M, Leimeister JM (2019) Hybrid intelligence. Bus. Inform. Systems Engrg. 61(5):637–643.Crossref, Google Scholar
• Dietterich TG (2000) Ensemble methods in machine learning. Multiple Classifier Systems, MCS 2000, Lecture Notes in Computer Science, vol. 1857 (Springer, Berlin, Heidelberg), 1–15.Google Scholar
• Dong L, Zheng H, Li L, Hao L (2022) Human-machine hybrid prediction market: A promising sales forecasting solution for e-commerce enterprises. Electronic Commerce Res. Appl. 56:101216.Crossref, Google Scholar
• Dorri A, Kanhere SS, Jurdak R (2018) Multi-agent systems: A survey. IEEE Access 6:28573–28593.Crossref, Google Scholar
• Durach CF, Kembro J, Wieland A (2017) A new paradigm for systematic literature reviews in supply chain management. J. Supply Chain Management 53(4):67–85.Crossref, Google Scholar
• Epping GP, Caplin A, Duhaime E, Holmes WR, Martin D, Trueblood JS (2026) Harnessing human uncertainty to train more accurate and aligned AI systems. Decision Anal., ePub ahead of print April 1, https://doi.org/10.1287/deca.2025.0395.Link, Google Scholar
• Fischer T, Krauss C (2018) Deep learning with long short-term memory networks for financial market predictions. Eur. J. Oper. Res. 270(2):654–669.Crossref, Google Scholar
• Fries M, Ludwig T (2024) Why are the sales forecasts so low?’ Socio-technical challenges of using machine learning for forecasting sales in a bakery. Comput. Supported Cooperative Work (CSCW) 33(2):253–293.Crossref, Google Scholar
• Gaba A, Tsetlin I, Winkler RL (2017) Combining interval forecasts. Decision Anal. 14(1):1–20.Link, Google Scholar
• Genest C, Zidek JV (1986) Combining probability distributions: A critique and an annotated bibliography. Statist. Sci. 1(1):114–135.Crossref, Google Scholar
• Goh E, Gallo R, Hom J, Strong E, Weng Y, Kerman H, Cool JA, et al. (2024) Large language model influence on diagnostic reasoning: A randomized clinical trial. JAMA Network Open 7(10):e2440969.Crossref, Google Scholar
• Graefe A, Armstrong JS, Jones RJ Jr, Cuzán AG (2014) Combining forecasts: An application to elections. Internat. J. Forecasting 30(1):43–54.Crossref, Google Scholar
• Granger CW, Machina MJ (2006) Forecasting and decision theory. Elliott G, Granger CWJ, Timmermann AG, eds. Handbook of Economic Forecasting, vol. 1 (North Holland, Amsterdam), 81–98.Crossref, Google Scholar
• Greiner B, Grünwald P, Lindner T, Lintner G, Wiernsperger M (2026) Incentives, framing, and reliance on algorithmic advice: An experimental study. Management Sci. 72(1):302–322.Link, Google Scholar
• Guo T, Chen X, Wang Y, Chang R, Pei S, Chawla N, Wiest O, Zhang X (2024) Large language model based multi-agents: A survey of progress and challenges. Larson K, ed. IJCAI’24: Proc. 33rd Internat. Joint Conf. Artificial Intelligence, 8048–8057.Google Scholar
• Gweon H, Fan J, Kim B (2023) Socially intelligent machines that learn from humans and help humans learn. Philos. Trans. Roy. Soc. A 381(2251):20220048.Crossref, Google Scholar
• Himmelstein M, Budescu DV (2023) Preference for human or algorithmic forecasting advice does not predict if and how it is used. J. Behav. Decision Making 36(1):e2285.Crossref, Google Scholar
• Karstens CD, Correia J Jr, LaDue DS, Wolfe J, Meyer TC, Harrison DR, Cintineo JL, et al. (2018) Development of a human–Machine mix for forecasting severe convective events. Weather Forecasting 33(3):715–737.Crossref, Google Scholar
• Kesavan S, Kushwaha T, Steele D (2026) Profit implications of judgmental adjustments to forecast inputs: Evidence from a large-scale field experiment. Management Sci. 72(1):119–127.Link, Google Scholar
• Korom R, Kiptinness S, Adan N, Said K, Ithuli C, Rotich O, Kimani B, et al. (2025) AI-based clinical decision support for primary care: A real-world study. Preprint, submitted July 22, https://arxiv.org/abs/2507.16947.Google Scholar
• Lamberson P, Page SE (2012) Optimal forecasting groups. Management Sci. 58(4):805–810.Link, Google Scholar
• Lawrence M, Goodwin P, O’Connor M, Önkal D (2006) Judgmental forecasting: A review of progress over the last 25 years. Internat. J. Forecasting 22(3):493–518.Crossref, Google Scholar
• Lehmann CA, Haubitz CB, Fügener A, Thonemann UW (2022) The risk of algorithm transparency: How algorithm complexity drives the effects on the use of advice. Production Oper. Management 31(9):3419–3434.Crossref, Google Scholar
• Li L, Zheng H (2024) How the introduction of machine traders and disclosure of their presence affect prediction accuracy: An online controlled experiment in prediction market. Internat. J. Electronic Commerce 28(4):569–600.Crossref, Google Scholar
• Magnus JR, Vasnev AL (2023) On the uncertainty of a combined forecast: The critical role of correlation. Internat. J. Forecasting 39(4):1895–1908.Crossref, Google Scholar
• McNeese NJ, Schelble BG, Canonico LB, Demir M (2021) Who/what is my teammate? Team composition considerations in human–AI teaming. IEEE Trans. Human-Machine Systems 51(4):288–299.Crossref, Google Scholar
• Mellers BA, McCoy JP, Lu L, Tetlock PE (2024) Human and algorithmic predictions in geopolitical forecasting: Quantifying uncertainty in hard-to-quantify domains. Perspect. Psych. Sci. 19(5):711–721.Crossref, Google Scholar
• Nair D, Huchzermeier A (2024) Predictably unpredictable? How judgmental and machine learning forecasts complement each other. Production Oper. Management 33(5):1214–1234.Crossref, Google Scholar
• Nguyen OK, Washington C, Clark CR, Miller ME, Patel VA, Halm EA, Makam AN (2021) Man vs. Machine: Comparing physician vs. electronic health record–based model predictions for 30-day hospital readmissions. J. General Internal Medicine 36(9):2555–2562.Crossref, Google Scholar
• Park HJ, Kim SH, Choi JY, Cha D (2023) Human–machine cooperation meta-model for clinical diagnosis by adaptation to human expert’s diagnostic characteristics. Sci. Rep. 13(1):16204.Crossref, Google Scholar
• Patel BN, Rosenberg L, Willcox G, Baltaxe D, Lyons M, Irvin J, Rajpurkar P, et al. (2019) Human–machine partnership with artificial intelligence for chest radiograph diagnosis. NPJ Digital Medicine 2(1):111.Crossref, Google Scholar
• Pescetelli N, Reichert P, Rutherford A (2022) A variational-autoencoder approach to solve the hidden profile task in hybrid human-machine teams. PLoS One 17(8):e0272168.Crossref, Google Scholar
• Petropoulos F, Apiletti D, Assimakopoulos V, Babai MZ, Barrow DK, Taieb SB, Bergmeir C, et al. (2022) Forecasting: Theory and practice. Internat. J. Forecasting 38(3):705–871.Crossref, Google Scholar
• Phillips PJ, Yates AN, Hu Y, Hahn CA, Noyes E, Jackson K, Cavazos JG, et al. (2018) Face recognition accuracy of forensic examiners, superrecognizers, and face recognition algorithms. Proc. Natl. Acad. Sci. 115(24):6171–6176.Crossref, Google Scholar
• Prahl A, Van Swol L (2017) Understanding algorithm aversion: When is advice from automation discounted? J. Forecasting 36(6):691–702.Crossref, Google Scholar
• Rajkomar A, Oren E, Chen K, Dai AM, Hajjaj N, Hardt M, Liu PJ, et al. (2018) Scalable and accurate deep learning with electronic health records. NPJ Digital Medicine 1(1):18.Crossref, Google Scholar
• Ranjan R, Gneiting T (2010) Combining probability forecasts. J. Roy. Statist. Soc. Ser. B Statist. Methodology 72(1):71–91.Crossref, Google Scholar
• Ravuri S, Lenc K, Willson M, Kangin D, Lam R, Mirowski P, Fitzsimons M, et al. (2021) Skilful precipitation nowcasting using deep generative models of radar. Nature 597(7878):672–677.Crossref, Google Scholar
• Samuel OW, Asogbon GM, Sangaiah AK, Fang P, Li G (2017) An integrated decision support system based on ANN and Fuzzy_AHP for heart failure risk prediction. Expert Systems Appl. 68:163–172.Crossref, Google Scholar
• Sanders NR, Ritzman LP (2001) Judgmental adjustment of statistical forecasts. Armstrong JS, ed. Principles of Forecasting: A Handbook for Researchers and Practitioners, International Series in Operations Research & Management Science, vol. 30 (Springer, Boston), 405–416.Crossref, Google Scholar
• Satopää V, Palley A, Grushka-Cockayne Y, Persinger C (2024) Machine-moderated judgmental forecasting to improve prediction accuracy. Preprint, submitted August 24, https://doi.org/10.2139/ssrn.4935332.Google Scholar
• Satopää VA, Salikhov M, Tetlock PE, Mellers B (2021) Bias, information, noise: The bin model of forecasting. Management Sci. 67(12):7599–7618.Link, Google Scholar
• Schoenegger P, Park PS (2023) Large language model prediction capabilities: Evidence from a real-world forecasting tournament. Preprint, submitted October 17, https://arxiv.org/abs/2310.13014.Google Scholar
• Schoenegger P, Park PS, Karger E, Trott S, Tetlock PE (2025) AI-augmented predictions: LLM assistants improve human forecasting accuracy. ACM Trans. Interactive Intelligent Systems 15(1):1–25.Crossref, Google Scholar
• Schoenegger P, Tuminauskaite I, Park PS, Bastos RVS, Tetlock PE (2024) Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy. Sci. Adv. 10(45):eadp1528.Crossref, Google Scholar
• Sharma M, Tong M, Korbak T, Duvenaud D, Askell A, Bowman SR, Cheng N, et al. (2024) Towards understanding sycophancy in language models. Proc. 12th Internat. Conf. Learn. Representations.Google Scholar
• Soule D, Grushka-Cockayne Y, Merrick J (2024) A heuristic for combining correlated experts when there are few data. Management Sci. 70(10):6637–6668.Link, Google Scholar
• Stuart NA, Hartfield G, Schultz DM, Wilson K, West G, Hoffman R, Lackmann G, et al. (2022) The evolving role of humans in weather prediction and communication. Bull. Amer. Meteorological Soc. 103(8):E1720–E1746.Crossref, Google Scholar
• Tan CH, Yap KS, Ishibuchi H, Nojima Y, Yap HJ (2014) Application of fuzzy inference rules to early semi-automatic estimation of activity duration in software project management. IEEE Trans. Human-Machine Systems 44(5):678–688.Crossref, Google Scholar
• Tetlock PE (2005) Expert Political Judgment: How Good Is It? How Can We Know? (Princeton University Press, Princeton, NJ).Google Scholar
• Tranfield D, Denyer D, Smart P (2003) Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British J. Management 14(3):207–222.Crossref, Google Scholar
• Tschandl P, Rinner C, Apalla Z, Argenziano G, Codella N, Halpern A, Janda M, et al. (2020) Human–computer collaboration for skin cancer recognition. Nature Medicine 26(8):1229–1234.Crossref, Google Scholar
• Ungar L, Mellers B, Satopää V, Baron J, Tetlock P, Ramos J, Swift S (2012) The good judgment project: A large scale test of different methods of combining expert predictions. Proc. AAAI 2012 Fall Sympos. Ser. (AAAI Press, Washington, DC), 37–42.Google Scholar
• van der Staak B, Basten R, van de Calseyde P, Demerouti E, de Kok A (2025) Light-touch forecasting: A novel method to combine human judgment with statistical algorithms. Internat. J. Forecasting 41(2):440–451.Crossref, Google Scholar
• Vössing M, Kühl N, Lind M, Satzger G (2022) Designing transparency for effective human-AI collaboration. Inform. Systems Frontiers 24(3):877–895.Crossref, Google Scholar
• Wang X, Hyndman RJ, Li F, Kang Y (2023) Forecast combinations: An over 50-year review. Internat. J. Forecasting 39(4):1518–1547.Crossref, Google Scholar
• Winkler RL (1981) Combining probability distributions from dependent information sources. Management Sci. 27(4):479–488.Link, Google Scholar
• Winkler RL, Clemen RT (2004) Multiple experts vs. multiple methods: Combining correlation assessments. Decision Anal. 1(3):167–176.Link, Google Scholar
• Winkler RL, Grushka-Cockayne Y, Lichtendahl KC Jr, Jose VRR (2019) Probability forecasts and their combination: A research perspective. Decision Anal. 16(4):239–260.Link, Google Scholar
• Yimamu A, Li J, Zhang H, Liang L, Feng L, Wang Y, Zhou C, Li S, Gao Y (2024) Computed tomography and guidelines-based human–machine fusion model for predicting resectability of the pancreatic cancer. J. Gastroenterology Hepatology 39(2):399–409.Crossref, Google Scholar
• Yin M, Wortman Vaughan J, Wallach H (2019) Understanding the effect of accuracy on trust in machine learning models. CHI’19: Proc. 2019 CHI Conf. Human Factors Comput. Systems (Association for Computing Machinery, New York), 1–12.Google Scholar
• Zellner M, Abbas AE, Budescu DV, Galstyan A (2021) A survey of human judgement and quantitative forecasting methods. Roy. Soc. Open Sci. 8(2):201187.Crossref, Google Scholar