Choosing a Strictly Proper Scoring Rule

References

• Armstrong JS (2001) Principles of Forecasting (Kluwer Academic, Norwell, MA).Crossref, Google Scholar
• Bickel JE (2007) Some comparisons among quadratic, spherical, and logarithmic scoring rules. Decision Anal. 4:49–65.Link, Google Scholar
• Brier GW (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Rev. 78:1–3.Crossref, Google Scholar
• Buja A, Stuetzle W, Shen Y (2005) Loss functions for binary class probability estimation and classification: Structure and applications. Accessed May 2, 2012, http://stat.wharton.upenn.edu/buja/PAPERS/.Google Scholar
• Davis-Stober CP, Budescu DV, Dana J, Broomell SB (2013) When is a crowd wise? Decision. Forthcoming.Google Scholar
• Gneiting T, Raftery AE (2007) Strictly proper scoring rules, prediction, and estimation. J. Amer. Statist. Assoc. 102:359–378.Crossref, Google Scholar
• Hand DJ, Vinciotti V (2003) Local versus global models for classification problems: Fitting models where it matters. Amer. Statistician 57:124–131.Crossref, Google Scholar
• Johnstone DJ (2007) The parimutuel Kelly probability scoring rule. Decision Anal. 4:66–75.Link, Google Scholar
• Johnstone DJ (2011) Economic interpretation of probabilities estimated by maximum likelihood or score. Management Sci. 57:308–314.Link, Google Scholar
• Johnstone DJ, Jose VRR, Winkler RL (2011) Tailored scoring rules for probabilities. Decision Anal. 8:256–268.Link, Google Scholar
• Jose VRR, Nau RF, Winkler RL (2008) Scoring rules, generalized entropy, and utility maximization. Oper. Res. 56:1146–1157.Link, Google Scholar
• Jose VRR, Nau RF, Winkler RL (2009) Sensitivity to distance and baseline distributions in forecast evaluation. Management Sci. 55:582–590.Link, Google Scholar
• Murphy AH (1972) Scalar and vector partitions of the probability score: Part I. Two-state situation. J. Appl. Meteorology 11:273–282.Crossref, Google Scholar
• O'Hagan A, Buck CE, Daneshkhah A, Eiser JR, Garthwaite PH, Jenkinson DJ, Oakley JE, Rakow T (2006) Uncertain Judgements: Eliciting Experts' Probabilities (Wiley, Hoboken, NJ).Crossref, Google Scholar
• R Development Core Team (2013) R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org.Google Scholar
• Schervish MJ (1989) A general method for comparing probability assessors. Ann. Statist. 17:1856–1879.Crossref, Google Scholar
• Selten R (1998) Axiomatic characterization of the quadratic scoring rule. Experiment. Econom. 1:43–62.Crossref, Google Scholar
• Staël von Holstein CAS (1970) Measurement of subjective probability. Acta Psych. 34:146–159.Crossref, Google Scholar
• Steyvers M, Lee MD, Miller BJ (2009) Wisdom of crowds in the recollection of order information. Bengio Y, Schuurmans D, Lafferty J, Williams CKI, Culotta A, eds. Advances in Neural Information Processing Systems, Vol. 22 (MIT Press, Cambridge, MA), 1785–1793.Google Scholar
• Surowiecki J (2005) The Wisdom of Crowds (Anchor, New York).Google Scholar
• Tetlock PE (2005) Expert Political Judgment: How Good Is It? How Can We Know? (Princeton University Press, Princeton, NJ).Google Scholar
• Turner BM, Steyvers M, Merkle EC, Budescu DV, Wallsten TS (2013) Forecast aggregation via recalibration. Machine Learn. Forthcoming.Google Scholar
• Warnaar DB, Merkle EC, Steyvers M, Wallsten TS, Stone ER, Budescu DV, Yates JF, et al. The aggregative contingent estimation system: Selecting, rewarding, and training experts in a wisdom of crowds approach to forecasting. Pantofaru C, Chernova S, Sorokin A, eds. Proc. 2012 Assoc. Advancement of Artificial Intelligence Spring Sympos. Ser. (AAAI Tech. Rep. SS-12-06) (AAAI Press, Palo Alto, CA), 75–76.Google Scholar
• Winkler RL (1971) Probabilistic prediction: Some experimental results. J. Amer. Statist. Assoc. 66:675–685.Crossref, Google Scholar
• Winkler RL (1996) Scoring rules and the evaluation of probabilities. Test 5:1–26.Crossref, Google Scholar
• Winkler RL, Jose VRR (2010) Scoring rules. Cochran JJ, ed. Wiley Encyclopedia of Operations Research and Management Science (John Wiley & Sons, New York).Google Scholar
• Yi SKM, Steyvers M, Lee MD, Dry M (2010) Wisdom of crowds in minimum spanning tree problems. Ohlsson S, Catrambone R, eds. Proc. 32nd Annual Conf. Cognitive Sci. Soc. (Lawrence Erlbaum Associates, Inc., Mahwah, NJ), 1840–1845.Google Scholar