Regularized Aggregation of One-Off Probability Predictions

References

• Albert JH, Chib S (1993) Bayesian analysis of binary and polychotomous response data. J. Amer. Statist. Assoc. 88(422):669–679.Crossref, Google Scholar
• Arieli I, Babichenko Y, Smorodinsky R (2018) Robust forecast aggregation. Proc. National Acad. Sci. USA 115(52):E12135–E12143.Crossref, Google Scholar
• Atanasov P, Joseph R, Feijoo F, Marshall M, Siddiqui S (2021) Human forest vs. random forest in time-sensitive covid-19 clinical trial prediction. Preprint, submitted December 9, http://dx.doi.org/10.2139/ssrn.3981732.Google Scholar
• Atanasov P, Rescober P, Stone E, Swift SA, Servan-Schreiber E, Tetlock P, Ungar L, et al. (2017) Distilling the wisdom of crowds: Prediction markets vs. prediction polls. Management Sci. 63(3):691–706.Link, Google Scholar
• Balasubramanian V (2006) MDL, Bayesian inference, and the geometry of the space of probability distributions. Grünwald P, Myung IJ, Pitt M, eds. Advances in Minimum Description Length: Theory and Applications (MIT Press, Cambridge, MA), 81–98.Google Scholar
• Bhattacharya RN, Waymire EC (2007) A Basic Course in Probability Theory, vol. 69 (Springer, Berlin).Google Scholar
• Bordley RF (1982) A multiplicative formula for aggregating probability assessments. Management Sci. 28(10):1137–1148.Link, Google Scholar
• Brier GW (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Rev. 78:1–3.Crossref, Google Scholar
• Budescu DV, Por H-H, Broomell SB, Smithson M (2014) The interpretation of ipcc probabilistic statements around the world. Natural Climate Change 4(6):508–512.Crossref, Google Scholar
• Burdzy K, Pal S (2021) Can coherent predictions be contradictory? Adv. Appl. Probabilities 53(1):133–161.Crossref, Google Scholar
• Clarke BS, Barron AR (1994) Jeffreys’ prior is asymptotically least favorable under entropy risk. J. Statist. Planning Inference 41(1):37–60.Crossref, Google Scholar
• Clemen RT (1987) Combining overlapping information. Management Sci. 33(3):373–380.Link, Google Scholar
• Clemen RT, Winkler RL (1999) Combining probability distributions from experts in risk analysis. Risk Anal. 19(2):187–203.Crossref, Google Scholar
• Dawid A, DeGroot M, Mortera J (1995) Coherent combination of experts’ opinions. TEST 4(2):263–313.Crossref, Google Scholar
• Dietrich F (2010) Bayesian group belief. Soc. Choice Welfare 35(4):595–626.Crossref, Google Scholar
• Dobbin K, Simon R (2005) Sample size determination in microarray experiments for class comparison and prognostic classification. Biostatistics 6(1):27–38.Crossref, Google Scholar
• Erev I, Wallsten TS, Budescu DV (1994) Simultaneous over- and underconfidence: The role of error in judgment processes. Psych. Rev. 101(3):519–527.Crossref, Google Scholar
• Ernst P, Pemantle R, Satopää V, Ungar L (2016) Bayesian aggregation of two forecasts in the partial information framework. Statist. Probability. Lett. 119:170–180.Crossref, Google Scholar
• Everett B (2013) An Introduction to Latent Variable Models (Springer Science & Business Media, New York).Google Scholar
• Fischhoff B, Beyth-Marom R (1983) Hypothesis evaluation from a Bayesian perspective. Psych. Rev. 90(3):239.Crossref, Google Scholar
• Friedman JA, Baker JD, Mellers BA, Tetlock PE, Zeckhauser R (2018) The value of precision in probability assessment: Evidence from a large-scale geopolitical forecasting tournament. Internat. Stud. Quart. 62(2):410–422.Google Scholar
• Gelman A, Stern HS, Carlin JB, Dunson DB, Vehtari A, Rubin DB (2013) Bayesian Data Analysis (Chapman and Hall/CRC).Crossref, Google Scholar
• Ghosh M, et al.. (2011) Objective priors: An introduction for frequentists. Statist. Sci. 26(2):187–202.Crossref, Google Scholar
• Gneiting T, Raftery AE (2007) Strictly proper scoring rules, prediction, and estimation. J. Amer. Statist. Assoc. 102(477):359–378.Crossref, Google Scholar
• Hilbert M (2012) Toward a synthesis of cognitive biases: How noisy information processing can bias human decision making. Psych. Bull. 138(2):211.Crossref, Google Scholar
• Hora SC, Fransen BR, Hawkins N, Susel I (2013) Median aggregation of distribution functions. Decision Anal. 10(4):279–291.Link, Google Scholar
• Jeffreys H (1946) An invariant form for the prior probability in estimation problems. Proc. Royal Soc. London A Math. Phys. Sci. 186(1007):453–461.Google Scholar
• Jeffreys H (1961) The Theory of Probability (Oxford University Press, Oxford, UK).Google Scholar
• Jose VRR, Grushka-Cockayne Y, Lichtendahl KC Jr (2013) Trimmed opinion pools and the crowd’s calibration problem. Management Sci. 60(2):463–475.Link, Google Scholar
• Kahana MJ, Aggarwal EV, Phan TD (2018) The variability puzzle in human memory. J. Experiment. Psych. Learn. Memory Cognition 44(12):1857.Crossref, Google Scholar
• Kahneman D, Tversky A (1973) On the psychology of prediction. Psych. Rev. 80(4):237.Crossref, Google Scholar
• Karvetski C, Meinel C, Maxwell D, Lu Y, Mellers B, Tetlock P (2021) Forecasting the accuracy of forecasters from properties of forecasting rationales. Preprint, submitted February 4, https://dx.doi.org/10.2139/ssrn.3779404.Google Scholar
• Lee MD, Danileiko I (2014) Using cognitive models to combine probability estimates. Judgment Decision Making 9(3):259.Crossref, Google Scholar
• Lichtendahl KC Jr, Winkler RL (2007) Probability elicitation, scoring rules, and competition among forecasters. Management Sci. 53(11):1745–1755.Link, Google Scholar
• Lindley D (1985) Reconciliation of discrete probability distributions. Bayesian Statist. 2:375–390.Google Scholar
• Malagò L, Pistone G (2015) Information geometry of the gaussian distribution in view of stochastic optimization. Proc. ACM Conf. on Foundations of Genetic Algorithms XIII (Association for Computing Machinery, New York), 150–162.Google Scholar
• Matheny J, Rieber S (2015) Aggregative contingent estimation (ACE). Accessed July 20, 2020, https://www.iarpa.gov/index.php/research-programs/ace.Google Scholar
• Mellers B, Ungar L, Baron J, Ramos J, Gurcay B, Fincher K, Scott SE, et al.. (2014) Psychological strategies for winning a geopolitical forecasting tournament. Psych. Sci. 25(5):1106–1115.Crossref, Google Scholar
• Morrison DF, Marshall LC, Sahlin HL (1976) Multivariate Statistical Methods, 2nd ed. (McGraw-Hill, New York).Google Scholar
• Murphy AH (1973) A new vector partition of the probability score. J. Appl. Meteorology 12(4):595–600.Crossref, Google Scholar
• Myung IJ, Balasubramanian V, Pitt MA (2000) Counting probability distributions: Differential geometry and model selection. Proc. National Acad. Sci. USA 97(21):11170–11175.Crossref, Google Scholar
• Neal RM (2003) Slice sampling. Ann. Statist. 31:705–741.Crossref, Google Scholar
• O’Hagan A, Buck CE, Daneshkhah A, Eiser JR, Garthwaite PH, Jenkinson DJ, Oakley JE, et al. (2006) Uncertain Judgements: Eliciting Experts’ Probabilities (John Wiley & Sons, Chichester, UK).Crossref, Google Scholar
• Owen DB (1980) A table of normal integrals. Comm. Statist. Simulation Comput. 9(4):389–419.Crossref, Google Scholar
• Palley A, Satopää V (2021) Boosting the wisdom of crowds within a single judgment problem: Weighted averaging based on peer predictions. Preprint, submitted September 1, https://dx.doi.org/10.2139/ssrn.3504286.Google Scholar
• Paulo R (2005) Default priors for Gaussian processes. Ann. Statist. 33(2):556–582.Crossref, Google Scholar
• Ranjan R, Gneiting T (2010) Combining probability forecasts. J. Royal Statist. Soc. Series B Statist. Methodology 72(1):71–91.Crossref, Google Scholar
• Ravishanker N, Dey DK (2001) A First Course in Linear Model Theory (CRC Press, Boca Raton, FL).Google Scholar
• Reichenbach H (1971) The Theory of Probability (University of California Press).Google Scholar
• Satopää VA (2021) Improving the wisdom of crowds with analysis of variance of predictions of related outcomes. Internat. J. Forecasting. 37(4):1728–1747.Crossref, Google Scholar
• Satopää VA, Pemantle R, Ungar LH (2016) Modeling probability forecasts via information diversity. J. Amer. Statist. Assoc. 111(516):1623–1633.Crossref, Google Scholar
• Satopää VA, Salikhov M, Tetlock PE, Mellers B (2021) Bias, information, noise: The BIN model of forecasting. Management Sci. 67(12):7599–7618.Link, Google Scholar
• Satopää VA, Baron J, Foster DP, Mellers BA, Tetlock PE, Ungar LH (2014) Combining multiple probability predictions using a simple logit model. Internat. J. Forecasting 30(2):344–356.Crossref, Google Scholar
• Satopää VA, Jensen ST, Pemantle R, Ungar LH (2017) Partial information framework: Model-based aggregation of estimates from diverse information sources. Electronic J. Statist. 11(2):3781–3814.Crossref, Google Scholar
• Stone M (1961) The opinion pool. Ann. Math. Statist. 32:1339–1342.Crossref, Google Scholar
• Ungar L, Mellers B, Satopää V, Tetlock P, Baron J (2012) The good judgment project: A large scale test of different methods of combining expert predictions. Technical report FS-12-06, Association for the Advancement of Artificial Intelligence, Palo Alto, CA.Google Scholar
• Van Der Bles AM, van der Linden S, Freeman AL, Spiegelhalter DJ (2020) The effects of communicating uncertainty on public trust in facts and numbers. Proc. National Acad. Sci. USA 117(14):7672–7683.Crossref, Google Scholar
• Venn J (1888) The Logic of Chance: An Essay on the Foundations and Province of the Theory of Probability, with Especial Reference to Its Logical Bearings and Its Application to Moral and Social Science, and to Statistics (Macmillan, London).Google Scholar
• Winkler RL (1981) Combining probability distributions from dependent information sources. Management Sci. 27(4):479–488.Link, Google Scholar
• Winkler RL, Poses RM (1993) Evaluating and combining physicians’ probabilities of survival in an intensive care unit. Management Sci. 39(12):1526–1543.Link, Google Scholar
• Wyart V, De Gardelle V, Scholl J, Summerfield C (2012) Rhythmic fluctuations in evidence accumulation during decision making in the human brain. Neuron 76(4):847–858.Crossref, Google Scholar
• Yates JF (1990) Judgment and Decision Making (Prentice-Hall, Upper Saddle River, NJ).Google Scholar