Probability Forecasts and Their Combination: A Research Perspective

References

• Armstrong SJ (2001) Combining forecasts. Armstrong SJ, ed. Principles of Forecasting: A Handbook for Researchers and Practitioners (Kluwer Academic Publishers, Norwell, MA), 417–439.Crossref, Google Scholar
• Aspinall WP, Cooke RM, Havelaar AH, Hoffmann S, Hald T (2016) Evaluation of a performance-based expert elicitation: WHO global attribution of foodborne diseases. PLoS One 11(3):e0149817.Crossref, Google Scholar
• Atanasov P, Rescober P, Stone E, Swift SA, Servan-Schreiber E, Tetlock P, Ungar L, Mellers B (2017) Distilling the wisdom of crowds: Prediction markets vs. prediction polls. Management Sci. 63(3):691–706.Link, Google Scholar
• Bassamboo A, Cui R, Moreno A (2019) Wisdom of crowds in operations: forecasting using prediction markets. Working paper, Northwestern University, Evanston, IL.Google Scholar
• Bates JM, Granger CWJ (1969) The combination of forecasts. Oper. Res. Quart. 20(4):451–468.Crossref, Google Scholar
• Belleflamme P, Lambert T, Schwienbacher A (2014) Crowdfunding: Tapping the right crowd. J. Bus. Venturing 29(5):585–609.Crossref, Google Scholar
• Bennett J, Lanning S (2007) The Netflix Prize. Proc. KDD Cup Workshop 2007 (ACM, New York), 3–6.Google Scholar
• Bickel JE (2007) Some comparisons among quadratic, spherical, and logarithmic scoring rules. Decision Anal. 4(2):49–65.Link, Google Scholar
• Blume L, Easley D (2006) If you’re so smart, why aren’t you rich? Belief selection in complete and incomplete markets. Econometrica 74(4):929–966.Crossref, Google Scholar
• Breiman L (2001) Random forests. Machine Learn. 45(1):5–32.Crossref, Google Scholar
• Brier GW (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Rev. 78(1):1–3.Crossref, Google Scholar
• Budescu DV, Chen E (2015) Identifying expertise to extract the wisdom of crowds. Management Sci. 61(2):267–280.Link, Google Scholar
• Budescu DV, Du N (2007) Coherence and consistency of investors’ probability judgment. Management Sci. 53(11):1731–1744.Link, Google Scholar
• Carvalho A (2016) An overview of applications of proper scoring rules. Decision Anal. 13(4):223–242.Link, Google Scholar
• Chen Y, Chu CH, Mullen T (2006) Predicting uncertain outcomes using information markets: Trader behavior and information aggregation. New Math. Natural Comput. 2(3):281–297.Crossref, Google Scholar
• Chen Y, Chu CH, Mullen T, Pennock DM (2005) Information markets vs. option pools: An empirical comparison. Proc. 6th ACM Conf. Electronic Commerce (ACM, New York), 58–67.Google Scholar
• Clemen RT (1989) Combining forecasts: A review and annotated bibliography. Internat. J. Forecasting 5(4):559–583.Crossref, Google Scholar
• Clemen RT, Winkler RL (1985) Limits for the precision and value of information from dependent sources. Oper. Res. 33(2):427–442.Link, Google Scholar
• Clemen RT, Winkler RL (1999) Combining probability distributions from experts in risk analysis. Risk Anal. 19(2):187–203.Crossref, Google Scholar
• Cooke RM (1991) Experts in Uncertainty: Opinion and Subjective Probability in Science (Oxford University Press, Oxford, UK).Crossref, Google Scholar
• Cooke RM (2015) The aggregation of expert judgment: Do good things come to those who weight? Risk Anal. 35(1):12–15.Crossref, Google Scholar
• Croushore D (1993) Introducing: The survey of professional forecasters. Bus. Rev. Federal Reserve Bank Philadelphia 3(November/December):3–15.Google Scholar
• Friedman JH (2001) Greedy function approximation: A gradient boosting machine. Ann. Statist. 29(5):1189–1232.Crossref, Google Scholar
• Gaba A, Tsetlin I, Winkler RL (2017) Combining interval forecasts. Decision Anal. 14(1):1–20.Link, Google Scholar
• Galton F (1907) Vox populi. Nature 75(1949):450–451.Crossref, Google Scholar
• Gelman A (2019) Statistical modeling, casual interference, and social science. Accessed May 1, 2019, https://statmodeling.stat.columbia.edu/.Google Scholar
• Genest C, Zidek JV (1986) Combining probability distributions: A critique and an annotated bibliography. Statist. Sci. 1:114–135.Crossref, Google Scholar
• Gneiting T, Raftery AE (2007) Strictly proper scoring rules, prediction, and estimation. J. Amer. Statist. Assoc. 102(477):359–378.Crossref, Google Scholar
• Gneiting T, Balabdaoui F, Raftery AE (2007) Probabilistic forecasts, calibration, and sharpness. J. Roy. Statist. Soc. B 69(2):243–268.Crossref, Google Scholar
• Good IJ (1952) Rational decisions. J. Roy. Statist. Soc. B 14(1):107–114.Crossref, Google Scholar
• Grushka-Cockayne Y, Jose VRR, Lichtendahl KC Jr (2017a) Ensembles of overfit and overconfident forecasts. Management Sci. 63(4):1110–1130.Link, Google Scholar
• Grushka-Cockayne Y, Lichtendahl KC Jr, Jose VRR, Winkler RL (2017b) Quantile evaluation, sensitivity to bracketing, and sharing business payoffs. Oper. Res. 65(3):712–728.Link, Google Scholar
• Hanson R (2007) Logarithmic market scoring rules for modular combinatorial information aggregation. J. Prediction Markets 1(1):3–15.Crossref, Google Scholar
• Hong T, Pinson P, Fan S, Zareipour H, Troccoli A, Hyndman RJ (2016) Probabilistic energy forecasting: Global Energy Forecasting Competition 2014 and beyond. Internat. J. Forecasting 32(3):896–913.Crossref, Google Scholar
• Hora SC (2004) Probability judgments for continuous quantities: Linear combinations and calibration. Management Sci. 50(5):597–604.Link, Google Scholar
• Howe J (2006) The rise of crowdsourcing. Wired 14(6):1–4.Google Scholar
• IARPA (2010) Aggregative Contingent Estimation (ACE) program broad agency announcement. Solicitation IARPA-BAA-10-05, Office of the Director of National Intelligence, Washington, DC.Google Scholar
• IARPA (2016) Hybrid Forecasting Competition (HFC) program broad agency announcement. Solicitation IARPA-BAA-16-02, Office of the Director of National Intelligence, Washington, DC.Google Scholar
• Johnstone DJ (2007a) Economic Darwinism: Who has the best probabilities? Theory Decision 62(1):47–96.Crossref, Google Scholar
• Johnstone DJ (2007b) The pari-mutuel Kelly probability scoring rule. Decision Anal. 4(2):66–75.Link, Google Scholar
• Johnstone DJ, Jose VRR, Winkler RL (2011) Tailored scoring rules for probabilities. Decision Anal. 8(4):256–268.Link, Google Scholar
• Jose VRR, Winkler RL (2008) Simple robust averages of forecasts: Some empirical results. Internat. J. Forecasting 24(1):163–169.Crossref, Google Scholar
• Jose VRR, Winkler RL (2009) Evaluating quantile assessments. Oper. Res. 57(5):1287–1297.Link, Google Scholar
• Jose VRR, Grushka-Cockayne Y, Lichtendahl KC Jr (2014) Trimmed opinion pools and the crowd’s calibration problem. Management Sci. 60(2):463–475.Link, Google Scholar
• Jose VRR, Nau RF, Winkler RL (2008) Scoring rules, generalized entropy, and utility maximization. Oper. Res. 56(5):1146–1157.Link, Google Scholar
• Kelly JL (1956) A new interpretation of information rate. Bell Syst. Tech. J. 35(4):917–926.Crossref, Google Scholar
• Kishore N, Marqués D, Mahmud A, Kiang MV, Rodriguez I, Fuller A, Ebner P, et al.. (2018) Mortality in Puerto Rico after Hurricane Maria. New Engl. J. Med. 379(17):162–170.Crossref, Google Scholar
• Lamberson PJ, Page SE (2012) Optimal forecasting groups. Management Sci. 58(4):805–810.Link, Google Scholar
• Lambert NS, Langford J, Vaughan JW, Chen Y, Reeves DM, Shoham Y, Pennock DM (2015) An axiomatic characterization of wagering mechanisms. J. Econom. Theory 156(March):389–416.Crossref, Google Scholar
• Larrick RP, Soll JB (2006) Intuitions about combining opinions: Misappreciation of the averaging principle. Management Sci. 52(1):111–127.Link, Google Scholar
• Lichtendahl KC Jr, Winkler RL (2007) Probability elicitation, scoring rules, and competition among forecasters. Management Sci. 53(11):1745–1755.Link, Google Scholar
• Lichtendahl KC Jr, Grushka-Cockayne Y, Pfeifer PE (2013a) The wisdom of competitive crowds. Oper. Res. 61(6):1383–1398.Link, Google Scholar
• Lichtendahl KC Jr, Grushka-Cockayne Y, Winkler RL (2013b) Is it better to average probabilities or quantiles? Management Sci. 59(7):1594–1611.Link, Google Scholar
• Lichtenstein S, Fischhoff B, Phillips LD (1982) Calibration of probabilities: The state of the art to 1980. Kahneman D, Slovic P, Tversky A, eds. Judgment Under Uncertainty: Heuristics and Biases (Cambridge University Press, Cambridge, UK), 305–324.Crossref, Google Scholar
• MacLean LC, Ziemba WT, Blazenko G (1992) Growth vs. security in dynamic investment analysis. Management Sci. 38(11):1562–1585.Link, Google Scholar
• Makridakis S, Spiliotis E, Assimakopoulous V (2018) The M4 Competition: Results, findings, conclusion and way forward. Internat. J. Forecasting 34(4):802–808.Google Scholar
• Mandel R, Noyes E (2013) Beyond the NWS: Inside the thriving private weather forecasting industry. Weatherwise 66(1):12–19.Crossref, Google Scholar
• Matheson JE, Winkler RL (1976) Scoring rules for continuous probability distributions. Management Sci. 22(10):1087–1096.Link, Google Scholar
• McCarthy J (1956) Measures of the value of information. Proc. Natl. Acad. Sci. USA 42:654–655.Crossref, Google Scholar
• Meinshausen N (2006) Quantile regression forecasts. J. Machine Learn. Res. 7(6):983–999.Google Scholar
• Merkle EC, Steyvers M (2013) Choosing a strictly proper scoring rule. Decision Anal. 10(4):292–304.Link, Google Scholar
• Miller K (2018) Hurricane season 2018: Slimmer cone reflects forecasting improvements. Palm Beach Post (May 12), https://www.palmbeachpost.com/weather/hurricanes/hurricane-season-2018-slimmer-cone-reflects-forecasting-improvements/0mCl5JTodN8HLSvUuW81AN/.Google Scholar
• Murphy AH, Winkler RL (1984) Probability forecasting in meteorology. J. Amer. Statist. Assoc. 79(387):489–500.Google Scholar
• National Weather Service (2017) National Weather Service Enterprise Analysis Report: Findings on changes in the private weather industry. Report, National Weather Service, Silver Spring, MD, https://www.weather.gov/media/about/Final_NWS%20Enterprise%20Analysis%20Report_June%202017.pdf.Google Scholar
• O’Hagan A, Buck CE, Daneshkhah A, Eiser JR, Garthwaite PH, Jenkinson DJ, Oakley JE, Rakow T (2006) Uncertain Judgements: Eliciting Experts’ Probabilities (John Wiley & Sons, New York).Crossref, Google Scholar
• Ranjan R, Gneiting T (2010) Combining probability forecasts. J. Roy. Statist. Soc. B 72(1):71–91.Crossref, Google Scholar
• Regnier E (2018) Probability forecasts made at multiple lead times. Management Sci. 64(5):2407–2426.Link, Google Scholar
• Sandroni A (2000) Do markets favor agents able to make accurate predictions? Econometrica 68(6):1303–1341.Crossref, Google Scholar
• Savage LJ (1954) The Foundations of Statistics (John Wiley & Sons, New York).Google Scholar
• Savage LJ (1971) Elicitation of personal probabilities and expectations. J. Amer. Statist. Assoc. 66(336):783–801.Crossref, Google Scholar
• Silver N (2012) The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t (Penguin Press, New York).Google Scholar
• Silver N (2018) How FiveThirtyEight’s house model works. FiveThirtyEight (August 16), https://fivethirtyeight.com/features/2018-house-forecast-methodology/.Google Scholar
• Stark T (2013) SPF panelists’ forecasting methods: A note on the aggregate results of a November 2009 special survey. Report, Research Department, Federal Reserve Bank of Philadelphia, Philadelphia.Google Scholar
• Surowiecki J (2005) The Wisdom of Crowds (Anchor Books, New York).Google Scholar
• Tetlock PE, Gardner D (2015) Superforecasting: The Art and Science of Prediction (Crown, New York).Google Scholar
• Turner BM, Steyvers M, Merkle EC, Budescu DV, Wallsten TS (2014) Forecast aggregation via recalibration. Machine Learn. 95(3):261–289.Crossref, Google Scholar
• U.S. National Hurricane Center (2017) National Hurricane Center Product Description Document: A User’s Guide to Hurricane Products. Accessed May 1, 2019, https://www.nhc.noaa.gov/pdf/NHC_Product_Description.pdf.Google Scholar
• Wilson KJ (2017) An investigation of dependence in expert judgement studies with multiple experts. Internat. J. Forecasting 33(1):325–336.Crossref, Google Scholar
• Wilson KJ, Farrow M (2018) Combining judgements from correlated experts. Dias LC, Morton A, Quigley J, eds. Elicitation: The Science and Art of Structuring Judgment, International Series in Operations Research and Management Science, vol. 261 (Springer, New York), 211–240.Crossref, Google Scholar
• Winkler RL (1981) Combining probability distributions from dependent information sources. Management Sci. 27(4):479–488.Link, Google Scholar
• Winkler RL (1994) Evaluating probabilities: Asymmetric scoring rules. Management Sci. 40(11):1395–1405.Link, Google Scholar
• Winkler RL (1996) Scoring rules and the evaluation of probabilities. TEST 5(1):1–60.Crossref, Google Scholar
• Winkler RL, Clemen RT (1992) Sensitivity of weights in combining forecasts. Oper. Res. 40(3):609–614.Link, Google Scholar
• Wolfers J, Zitzewitz E (2004) Prediction markets. J. Econom. Perspect. 18(2):107–126.Crossref, Google Scholar
• Zumbrun J (2018) Growth seen hitting 3% in 2018, but risks to outlook mount after this year. Wall Street Journal (August 9), https://www.wsj.com/articles/growth-seen-hitting-3-in-2018-but-risks-to-outlook-mount-after-this-year-1533823205?redirect=amp#click=https://t.co/4bHOhQERi5.Google Scholar