An Overview of Applications of Proper Scoring Rules

References

• Abernethy J, Johnson-Roberson M (2015) Financialized methods for market-based multi-sensor fusion. 2015 IEEE/RSJ Internat. Conf. Intelligent Robots and Systems (Institute of Electrical and Electronics Engineers, Hamburg, Germany), 900–907.Crossref, Google Scholar
• Abernethy J, Chen Y, Vaughan JM (2011) An optimization-based framework for automated market-making. Proc. 12th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 297–306.Crossref, Google Scholar
• Abernethy J, Chen Y, Vaughan JW (2013) Efficient market making via convex optimization, and a connection to online learning. ACM Trans. Econom. Comput. 1(2):1–39.Crossref, Google Scholar
• Abernethy J, Kutty S, Lahaie S, Sami R (2014) Information aggregation in exponential family markets. Proc. 15th ACM Conf. Econom. Comput. (Association for Computing Machinery, New York), 395–412.Crossref, Google Scholar
• Abramowicz M (2006) Deliberative information markets for small groups. Hahn R, Tetlock P, eds. Information Markets: A New Way of Making Decisions (Aei Press, Washington, DC), 101–125.Google Scholar
• Abramovicz M (2007) The hidden beauty of the quadratic market scoring rule: A uniform liquidity market maker, with variations. J. Prediction Markets 1(2):111–125.Crossref, Google Scholar
• Agrawal S, Delage E, Peters M, Wang Z, Ye Y (2009) A unified framework for dynamic pari-mutuel information market design. Proc. 10th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 255–264.Crossref, Google Scholar
• Ahrens B, Walser A (2008) Information-based skill scores for probabilistic forecasts. Monthly Weather Rev. 136(1):352–363.Crossref, Google Scholar
• Akasiadis C, Chalkiadakis G (2013) Agent cooperatives for effective power consumption shifting. 27th AAAI Conf. Artificial Intelligence, 1263–1269.Google Scholar
• Allen F (1987) Discovering personal probabilities when utility functions are unknown. Management Sci. 33(4):542–544.Link, Google Scholar
• Andersen S, Fountain J, Harrison GW, Rutström EE (2014) Estimating subjective probabilities. J. Risk Uncertainty 48(3):207–229.Crossref, Google Scholar
• Armantier O, Treich N (2013) Eliciting beliefs: Proper scoring rules, incentives, stakes and hedging. Eur. Econom. Rev. 62(August):17–40.Crossref, Google Scholar
• Bacon DF, Chen Y, Kash I, Parkes DC, Rao M, Sridharan M (2012) Predicting your own effort. Proc. 11th International Conf. Autonomous Agents and Multiagent Systems, 695–702.Google Scholar
• Bagchi D, Biswas S, Narahari Y, Viswanadham N, Suresh P, Subrahmanya SV (2013) Incentive compatible green procurement using scoring rules. Proc. 2013 IEEE Internat. Conf. Automation Sci. Engrg., 504–509.Crossref, Google Scholar
• Berg H, Proebsting TA (2009) Hanson’s automated market maker. J. Prediction Markets 3(1):45–59.Crossref, Google Scholar
• Bernardo JM, Muñoz J (1993) Bayesian analysis of population evolution. Statistician 42(5):541–550.Crossref, Google Scholar
• Bhola B, Cooke RM, Blaauw HG, Kok M (1992) Expert opinion in project management. Eur. J. Oper. Res. 57(1):24–31.Crossref, Google Scholar
• Bickel JE (2007) Some comparisons among quadratic, spherical, and logarithmic scoring rules. Decision Anal. 4(2):49–65.Link, Google Scholar
• Bickel JE (2010) Scoring rules and decision analysis education. Decision Anal. 7(4):346–357.Link, Google Scholar
• Blanco M, Engelmann D, Koch AK, Normann H-T (2010) Belief elicitation in experiments: Is there a hedging problem? Experiment. Econom. 13(4):412–438.Crossref, Google Scholar
• Blanco M, Engelmann D, Koch AK, Normann HT (2014) Preferences and beliefs in a sequential social dilemma: A within-subjects analysis. Games Econom. Behav. 87:122–135.Crossref, Google Scholar
• Brahma A, Chakraborty M, Das S, Lavoie A, Magdon-Ismail M (2012) A Bayesian market maker. Proc. 13th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 215–232.Crossref, Google Scholar
• Brier GW (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Rev. 78(1):1–3.Crossref, Google Scholar
• Bröcker J, Smith LA (2008) From ensemble forecasts to predictive distribution functions. Tellus A 60(4):663–678.Crossref, Google Scholar
• Brümmer N, du Preez J (2006) Application-independent evaluation of speaker detection. Comput. Speech Language 20(2):230–275.Crossref, Google Scholar
• Brummer N, Van Leeuwen D (2006) On calibration of language recognition scores. Proc. IEEE Speaker and Language Recognition Workshop, 1–8.Crossref, Google Scholar
• Brunet N, Verret R, Yacowar N (1988) An objective comparison of model output statistics and “perfect prog” systems in producing numerical weather element forecasts. Weather and Forecasting 3(4):273–283.Crossref, Google Scholar
• Buhrmester MD, Kwang T, Gosling SD (2011) Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspect. Psych. Sci. 6(1):3–5.Crossref, Google Scholar
• Cai Y, Mahdian M, Mehta A, Waggoner B (2013) Designing markets for daily deals. Chen Y, Immorlica N, eds. Web and Internet Economics, Lecture Notes in Computer Science, Vol. 8289 (Springer-Verlag, New York), 82–95.Crossref, Google Scholar
• Campbell WM, Brady KJ, Campbell JP, Granville R, Reynolds DA (2006) Understanding scores in forensic speaker recognition. Proc. IEEE Speaker and Language Recognition Workshop, 1–8.Crossref, Google Scholar
• Cao CC, Chen L, Jagadish HV (2014) From labor to trader: Opinion elicitation via online crowds as a market. Proc. 20th ACM SIGKDD Internat. Conf. Knowledge Discovery and Data Mining (Association for Computing Machinery, New York), 1067–1076.Crossref, Google Scholar
• Carvalho A (2015) Tailored proper scoring rules elicit decision weights. Judgment Decision Making 10(1):86–96.Google Scholar
• Carvalho A, Larson K (2010) Sharing a reward based on peer evaluations. Proc. 9th Internat. Conf. Autonomous Agents and Multiagent Systems, 1455–1456.Google Scholar
• Carvalho A, Larson K (2011) A truth serum for sharing rewards. Proc. 10th Internat. Conf. Autonomous Agents and Multiagent Systems, 635–642.Google Scholar
• Carvalho A, Larson K (2012) Sharing rewards among strangers based on peer evaluations. Decision Anal. 9(3):253–273.Link, Google Scholar
• Carvalho A, Larson K (2013) A consensual linear opinion pool. Proc. 23rd Internat. Joint Conf. Artificial Intelligence, 2518–2524.Google Scholar
• Carvalho A, Dimitrov S, Larson K (2015) A study on the influence of the number of MTurkers on the quality of the aggregate output. Bulling N, ed. Multi-Agent Systems, Lecture Notes in Computer Science, Vol. 8953 (Springer-Verlag, New York), 285–300.Crossref, Google Scholar
• Carvalho A, Dimitrov S, Larson K (2016a) How many crowdsourced workers should a requester hire? Ann. Math. Artificial Intelligence 78(1):45–72.Crossref, Google Scholar
• Carvalho A, Dimitrov S, Larson K (2016b) Inducing honest reporting of private information in the presence of social projection. Decision, ePub ahead of print March 14, http://dx.doi.org/10.1037/dec0000052.Crossref, Google Scholar
• Casillas-Olvera G, Bessler DA (2006) Probability forecasting and central bank accountability. J. Policy Modeling 28(2):223–234.Crossref, Google Scholar
• Chakraborty S, Ito T (2012) Smart house load management scheme using scoring rule based optimal time dependent pricing. Proc. 2012 Joint Agent Workshop and Sympos. 1–8.Google Scholar
• Chakraborty S, Ito T (2015) Hierarchical scoring rule based smart dynamic electricity pricing scheme. Bai Q, Ren F, Zhang M, Ito T, Tang X, eds. Smart Modeling and Simulation for Complex Systems, Studies in Computational Intelligence, Vol. 564 (Springer, Japan, Tokyo), 113–131.Crossref, Google Scholar
• Chakraborty M, Das S, Peabody J (2015) Price evolution in a continuous double auction prediction market with a scoring-rule based market maker. Proc. 29th AAAI Conf. Artificial Intelligence, 835–841.Google Scholar
• Chakraborty S, Ito T, Hara K (2013a) Incentive based smart pricing scheme using scoring rule. Proc. 4th IEEE/PES Innovative Smart Grid Technologies Europe, 1–5.Crossref, Google Scholar
• Chakraborty S, Ito T, Senjyu T (2014) Smart pricing scheme: A multi-layered scoring rule application. Expert Systems with Applications 41(8):3726–3735.Crossref, Google Scholar
• Chakraborty S, Ito T, Kanamori R, Senjyu T (2013b) Application of incentive based scoring rule deciding pricing for smart houses. Proc. 2013 IEEE Power and Energy Soc. General Meeting, 1–5.Crossref, Google Scholar
• Charba JP, Klein WH (1980) Skill in precipitation forecasting in the national weather service. Bull. Amer. Meteorological Soc. 61(12):1546–1555.Crossref, Google Scholar
• Chen Y (2007) A utility framework for bounded-loss market makers. Proc. 23rd Conf. Uncertainty in Artificial Intelligence, 49–56.Google Scholar
• Chen Y, Pennock DM (2010) Designing markets for prediction. AI Magazine 31(4):42–52.Crossref, Google Scholar
• Chen Y, Vaughan JW (2010a) A new understanding of prediction markets via no-regret learning. Proc. 11th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 189–198.Crossref, Google Scholar
• Chen Y, Vaughan JW (2010b) Connections between markets and learning. ACM SIGecom Exchanges 9(1):6.Crossref, Google Scholar
• Chen Y, Ruberry M, Vaughan JW (2012) Designing informative securities. Proc. 28th Conf. Uncertainty in Artificial Intelligence, 185–195.Google Scholar
• Chen Y, Ruberry M, Vaughan JM (2013) Cost function market makers for measurable spaces. Proc. 14th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 785–802.Crossref, Google Scholar
• Chen Y, Chu CH, Mullen T, Pennock DM (2005) Information markets vs. opinion pools: An empirical comparison. Proc. 6th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 58–67.Crossref, Google Scholar
• Chen Y, Gao XA, Goldstein R, Kash IA (2014) Market manipulation with outside incentives. Autonomous Agents and Multi-Agent Systems 29(2):230–265.Crossref, Google Scholar
• Chen Y, Kash I, Ruberry M, Shnayder V (2011) Decision markets with good incentives. Chen N, Elkind E, Koutsoupias E, eds. Internet and Network Economics, Lecture Notes in Computer Science, Vol. 7090 (Springer-Verlag, New York), 72–83.Crossref, Google Scholar
• Chen Y, Fortnow L, Lambert N, Pennock DM, Wortman J (2008) Complexity of combinatorial market makers. Proc. 9th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 190–199.Crossref, Google Scholar
• Chen Y, Reeves DM, Pennock DM, Hanson RD, Fortnow L, Gonen R (2007) Bluffing and strategic reticence in prediction markets. Deng X, Graham F, eds. Internet and Network Economics, Lecture Notes in Computer Science, Vol. 4858 (Springer, Berlin), 70–81.Crossref, Google Scholar
• Chen Y, Dimitrov S, Sami R, Reeves DM, Pennock DM, Hanson RD, Fortnow L, Gonen R (2010) Gaming prediction markets: Equilibrium strategies with a market maker. Algorithmica 58(4):930–969.Crossref, Google Scholar
• Chmielecki RM, Raftery AE (2011) Probabilistic visibility forecasting using Bayesian model averaging. Monthly Weather Rev. 139(5):1626–1636.Crossref, Google Scholar
• Christensen HM (2015) Decomposition of a new proper score for verification of ensemble forecasts. Monthly Weather Rev. 143(5):1517–1532.Crossref, Google Scholar
• Christensen HM, Moroz IM, Palmer TN (2015) Evaluation of ensemble forecast uncertainty using a new proper score: Application to medium-range and seasonal forecasts. Quart. J. Roy. Meteorological Soc. 141(687):538–549.Crossref, Google Scholar
• Clemen RT (1989) Combining forecasts: A review and annotated bibliography. Internat. J. Forecasting 5(4):559–583.Crossref, Google Scholar
• Conigliani C, Manca A, Tancredi A (2015) Prediction of patient-reported outcome measures via multivariate ordered probit models. J. Roy. Statist. Soc. Ser. A 178(3):567–591.Crossref, Google Scholar
• Conitzer V (2009) Prediction markets, mechanism design, and cooperative game theory. Proc. 25th Conf. Uncertainty in Artificial Intelligence, 101–108.Google Scholar
• Constantinou AC, Fenton NE (2012) Solving the problem of inadequate scoring rules for assessing probabilistic football forecast models. J. Quant. Anal. Sports 8(1):1.Google Scholar
• Cooke RM (1991) Experts in Uncertainty: Opinion and Subjective Probability in Science (Oxford University Press, Oxford, UK).Crossref, Google Scholar
• Costa-Gomes MA, Weizsäcker G (2008) Stated beliefs and play in normal-form games. Rev. Econom. Stud. 75(3):729–762.Crossref, Google Scholar
• Cunningham AA, Martell DL (1976) The use of subjective probability assessments to predict forest fire occurrence. Canadian J. Forest Res. 6(3):348–356.Crossref, Google Scholar
• Danz DN, Fehr D, Kübler D (2012) Information and beliefs in a repeated normal-form game. Experiment. Econom. 15(4):622–640.Crossref, Google Scholar
• Dawid AP (2007) The geometry of proper scoring rules. Ann. Inst. Statist. Math. 59(1):77–93.Crossref, Google Scholar
• Dawid AP, Musio M (2013) Estimation of spatial processes using local scoring rules. AStA Adv. Statist. Anal. 97(2):173–179.Crossref, Google Scholar
• Debnath S, Pennock DM, Giles CL, Lawrence S (2003) Information incorporation in online in-game sports betting markets. Proc. 4th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 258–259.Crossref, Google Scholar
• Deloatch R, Marmarchi A, Kirlik A (2013) Testing the conditions for acquiring intuitive expertise in judgment evidence from a study of NCAA basketball tournament predictions. Proc. Human Factors and Ergonomics Soc. Annual Meeting, Vol. 57, 290–294.Crossref, Google Scholar
• Diebold FX, Mariano RS (2012) Comparing predictive accuracy. J. Bus. Econom. Statist. 20(1):134–144.Crossref, Google Scholar
• Dimitrov S, Sami R (2008) Non-myopic strategies in prediction markets. Proc. 9th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 200–209.Crossref, Google Scholar
• Dimitrov S, Sami R (2010) Composition of markets with conflicting incentives. Proc. 11th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 53–62.Crossref, Google Scholar
• Dolan JG, Bordley DR, Mushlin AI (1986) An evaluation of clinicians’ subjective prior probability estimates. Medical Decision Making 6(4):216–223.Crossref, Google Scholar
• Dudik M, Lahaie S, Pennock DM, Rothschild D (2013) A combinatorial prediction market for the US elections. Proc. 14th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 341–358.Google Scholar
• Egri P, Váncza J (2013) Efficient mechanism for aggregate demand prediction in the smart grid. Multiagent System Technologies, Lecture Notes in Computer Science, Vol. 8076 (Springer, Berlin), 250–263.Crossref, Google Scholar
• Ehm W, Gneiting T (2012) Local proper scoring rules of order two. Ann. Statist. 40(1):609–637.Crossref, Google Scholar
• Epstein ES (1969) A scoring system for probability forecasts of ranked categories. J. Appl. Meteorology 8(6):985–987.Crossref, Google Scholar
• Epstein ES (1988) Long-range weather prediction: limits of predictability and beyond. Weather and Forecasting 3(1):69–75.Crossref, Google Scholar
• Faltings B, Li JJ, Jurca R (2012) Eliciting truthful measurements from a community of sensors. Proc. 3rd Internat. Conf. Internet of Things, 47–54.Crossref, Google Scholar
• Faltings B, Li JJ, Jurca R (2014) Incentive mechanisms for community sensing. IEEE Trans. Comput. 63(1):115–128.Crossref, Google Scholar
• Fischer GW (1982) Scoring-rule feedback and the overconfidence syndrome in subjective probability forecasting. Organ. Behav. Human Performance 29(3):352–369.Crossref, Google Scholar
• Forbes PGM (2012) Compatible weighted proper scoring rules. Biometrika 99(4):989–994.Crossref, Google Scholar
• Fricker TE, Ferro CAT, Stephenson DB (2013) Three recommendations for evaluating climate predictions. Meteorological Appl. 20(2):246–255.Crossref, Google Scholar
• Friederichs P, Hense A (2007) Statistical downscaling of extreme precipitation events using censored quantile regression. Monthly Weather Rev. 135(6):2365–2378.Crossref, Google Scholar
• Friederichs P, Thorarinsdottir TL (2012) Forecast verification for extreme value distributions with an application to probabilistic peak wind prediction. Environmetrics 23(7):579–594.Crossref, Google Scholar
• Friedman D, Massaro DW (1998) Understanding variability in binary and continuous choice. Psychonomic Bull. Rev. 5(3):370–389.Crossref, Google Scholar
• Gao X, Chen Y, Pennock DM (2009) Betting on the real line. Leonardi S, ed. Internet and Network Economics, Lecture Notes in Computer Science, Vol. 5929 (Springer, Berlin), 553–560.Crossref, Google Scholar
• Gao XA, Zhang J, Chen Y (2013) What you jointly know determines how you act: Strategic interactions in prediction markets. Proc. 14th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 489–506.Crossref, Google Scholar
• Garthwaite PH, O’Hagan A (2000) Quantifying expert opinion in the UK water industry: An experimental study. J. Roy. Statist. Soc. Ser. D 49(4):455–477.Crossref, Google Scholar
• Gerding EH, Larson K, Jennings NR (2010) Eliciting expert advice in service-oriented computing. David E, Gerding E, Sarne D, Shehory O, eds. Agent-Mediated Electronic Commerce, Lecture Notes in Business Information Processing, Vol. 59 (Springer, Berlin), 29–43.Crossref, Google Scholar
• Glahn HR, Jorgensen DL (1970) Climatological aspects of the brier P-score. Monthly Weather Rev. 98(2):136–141.Crossref, Google Scholar
• Gneiting T, Raftery AE (2007) Strictly proper scoring rules, prediction, and estimation. J. Amer. Statist. Assoc. 102(477):359–378.Crossref, Google Scholar
• Gneiting T, Ranjan R (2011) Comparing density forecasts using threshold- and quantile-weighted scoring rules. J. Bus. Econom. Statist. 29(3):411–422.Crossref, Google Scholar
• Gneiting T, Balabdaoui F, Raftery AE (2007) Probabilistic forecasts, calibration and sharpness. J. Roy. Statist. Soc. Ser. B 69(2):243–268.Crossref, Google Scholar
• Gneiting T, Stanberry LI, Grimit EP, Held L, Johnson NA (2008) Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds. Test 17(2):211–235.Crossref, Google Scholar
• Grant A, Johnstone D (2010) Finding profitable forecast combinations using probability scoring rules. Internat. J. Forecasting 26(3):498–510.Crossref, Google Scholar
• Grimit EP, Gneiting T, Berrocal VJ, Johnson NA (2006) The continuous ranked probability score for circular variables and its application to mesoscale forecast ensemble verification. Quart. J. Roy. Meteorological Soc. 132:1–17.Crossref, Google Scholar
• Gschlößl S, Czado C (2007) Spatial modelling of claim frequency and claim size in non-life insurance. Scandinavian Actuarial J. 2007(3):202–225.Crossref, Google Scholar
• Guerra G, Zizzo DJ (2004) Trust Responsiveness and Beliefs. J. Econom. Behav. Organ. 55(1):25–30.Crossref, Google Scholar
• Guo M, Pennock DM (2009) Combinatorial prediction markets for event hierarchies. Proc. 8th Internat. Conf. Autonomous Agents and Multiagent Systems, 201–208.Google Scholar
• Hanson R (2003) Combinatorial information market design. Inform. Systems Frontiers 5(1):107–119.Crossref, Google Scholar
• Hanson R (2007) Logarithmic market scoring rules for modular combinatorial information aggregation. J. Prediction Markets 1(1):3–15.Crossref, Google Scholar
• Hara K, Ito T (2015) A scoring rule-based truthful demand response mechanism. 2015 IEEE/ACIS 14th Internat. Conf. Comput. Inform. Sci. (Institute of Electrical and Electronics Engineers), 355–360.Crossref, Google Scholar
• Harrison GW, Martinez-Correa J, Swarthout JT (2014) Eliciting subjective probabilities with binary lotteries. J. Econom. Behav. Organ. 101(May):128–140.Crossref, Google Scholar
• Heath C, Tversky A (1991) Preference and belief: Ambiguity and competence in choice under uncertainty. J. Risk Uncertainty 4(1):5–28.Crossref, Google Scholar
• Hendrickson AD, Buehler RJ (1971) Proper scores for probability forecasters. Ann. Math. Statist. 1916–1921.Crossref, Google Scholar
• Hendry DF, Clements MP (2004) Pooling of forecasts. Econometrics J. 7(1):1–31.Crossref, Google Scholar
• Hirtle B, Lopez JA (1999) Supervisory information and the frequency of bank examinations. Econom. Policy Rev. 5(1).Google Scholar
• Hossain T, Okui R (2013) The binarized scoring rule. Rev. Econom. Stud. 80(3):984–1001.Crossref, Google Scholar
• Howe J (2006) The rise of crowdsourcing. Wired 14(6):1–4.Google Scholar
• Huck S, Weizsäcker G (2002) Do players correctly estimate what others do? Evidence of conservatism in beliefs. J. Econom. Behav. Organ. 47(1):71–85.Crossref, Google Scholar
• Hyndman K, Özbay EY, Schotter A, Ehrblatt W (2012a) Belief formation: An experiment with outside observers. Experiment. Econom. 15(1):176–203.Crossref, Google Scholar
• Hyndman K, Ozbay EY, Schotter A, Ehrblatt WZ (2012b) Convergence: An experimental study of teaching and learning in repeated games. J. Eur. Econom. Assoc. 10(3):573–604.Crossref, Google Scholar
• Iyer K, Johari R, Moallemi CC (2010) Information aggregation in smooth markets. Proc. 11th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 199–206.Crossref, Google Scholar
• Jaun S, Ahrens B (2009) Evaluation of a probabilistic hydrometeorological forecast system. Hydrology and Earth System Sci. 13(7):1031–1043.Crossref, Google Scholar
• Jensen FA, Peterson CR (1973) Psychological effects of proper scoring rules. Organ. Behav. Human Performance 9(2):307–317.Crossref, Google Scholar
• Johnstone DJ (2012) Log-optimal economic evaluation of probability forecasts. J. Roy. Statist. Soc. Ser. A 175(3):661–689.Crossref, Google Scholar
• Johnstone DJ, Jose VRR, Winkler RL (2011) Tailored scoring rules for probabilities. Decision Anal. 8(4):256–268.Link, Google Scholar
• Johnstone DJ, Jones S, Jose VRR, Peat M (2013) Measures of the economic value of probabilities of bankruptcy. J. Roy. Statist. Soc. Ser. A 176(3):635–653.Crossref, Google Scholar
• Jose VRR (2009) A characterization for the spherical scoring rule. Theory Decision 66(3):263–281.Crossref, Google Scholar
• Jose VRR, Winkler RL (2009) Evaluating quantile assessments. Oper. Res. 57(5):1287–1297.Link, Google Scholar
• Jose VRR, Nau RF, Winkler RL (2008) Scoring rules, generalized entropy, and utility maximization. Oper. Res. 56(5):1146–1157.Link, Google Scholar
• Jose VRR, Nau RF, Winkler RL (2009) Sensitivity to distance and baseline distributions in forecast evaluation. Management Sci. 55(4):582–590.Link, Google Scholar
• Jumadinova J, Dasgupta P (2013) Prediction market-based information aggregation for multi-sensor information processing. David E, Kiekintveld C, Robu V, Shehory O, Stein S, eds. Agent-Mediated Electronic Commerce. Designing Trading Strategies and Mechanisms for Electronic Markets, Lecture Notes in Business Information Processing, Vol. 136 (Springer, Berlin), 75–89.Crossref, Google Scholar
• Jurca R, Faltings B (2009) Mechanisms for making crowds truthful. J. Artificial Intelligence Res. 34(1):209–253.Crossref, Google Scholar
• Kamar E, Horvitz E (2012) Incentives for truthful reporting in crowdsourcing. Proc. 11th Internat. Conf. Autonomous Agents and Multiagent Systems, 1329–1330.Google Scholar
• Karni E (2009) A mechanism for eliciting probabilities. Econometrica 77(2):603–606.Crossref, Google Scholar
• Katz RW, Murphy AH (1997) Economic Value of Weather and Climate Forecasts (Cambridge University Press, Cambridge, UK).Crossref, Google Scholar
• Koessler F, Noussair C, Ziegelmeyer A (2012) Information aggregation and belief elicitation in experimental parimutuel betting markets. J. Econom. Behav. Organ. 83(2):195–208.Crossref, Google Scholar
• Kothiyal A, Spinu V, Wakker PP (2011) Comonotonic proper scoring rules to measure ambiguity and subjective beliefs. J. Multi-Criteria Decision Anal. 17(3–4):101–113.Crossref, Google Scholar
• Lad F, Sanfilippo G, Agrò G (2012) Completing the logarithmic scoring rule for assessing probability distributions. AIP Conf. Proc. 1490(March):13–30.Crossref, Google Scholar
• Lawrence S, Glover EJ, Giles CL (2002) Characterizing efficiency and information incorporation in sports betting markets. Proc. 9th Res. Sympos. Emerging Electronic Markets, 45–52.Google Scholar
• Ledyard J, Hanson R, Ishikida T (2009) An experimental test of combinatorial information markets. J. Econom. Behav. Organ. 69(2):182–189.Crossref, Google Scholar
• Lerch S, Thorarinsdottir TL (2013) Comparison of non-homogeneous regression models for probabilistic wind speed forecasting. Tellus A. 65, http://dx.doi.org/10.3402/tellusa.v65i0.21206.Crossref, Google Scholar
• Li X, Vaughan JW (2013) An axiomatic characterization of adaptive-liquidity market makers. Proc. 14th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York),657–674.Crossref, Google Scholar
• Linnet K (1988) A review on the methodology for assessing diagnostic tests. Clinical Chemistry 34(7):1379–1386.Crossref, Google Scholar
• Linnet K (1989) Assessing diagnostic tests by a strictly proper scoring rule. Statist. Medicine 8(5):609–618.Crossref, Google Scholar
• Lopez JA (2001) Evaluating the predictive accuracy of volatility models. J. Forecasting 20(2):87–109.Crossref, Google Scholar
• Machete RL (2013) Contrasting probabilistic scoring rules. J. Statist. Planning Inference 143(10):1781–1790.Crossref, Google Scholar
• Madigan D, Gavrin J, Raftery AE (1995) Eliciting prior information to enhance the predictive performance of Bayesian graphical models. Comm. Statist. - Theory Methods 24(9):2271–2292.Crossref, Google Scholar
• Mahmoud E (1984) Accuracy in forecasting: A survey. J. Forecasting 3(2):139–159.Crossref, Google Scholar
• Manski CF, Neri C (2013) First- and second-order subjective expectations in strategic decision-making: Experimental evidence. Games Econom. Behav. 81(September):232–254.Crossref, Google Scholar
• Mason SJ (2004) On using “climatology” as a reference strategy in the brier and ranked probability skill scores. Monthly Weather Rev. 132(7):1891–1895.Crossref, Google Scholar
• Matheson JE, Winkler RL (1976) Scoring rules for continuous probability distributions. Management Sci. 22(10):1087–1096.Link, Google Scholar
• Mellers B, Ungar L, Baron J, Ramos J, Gurcay B, Fincher K, Scott SE, Moore D, Atanasov P, Swift SA (2014) Psychological strategies for winning a geopolitical forecasting tournament. Psych. Sci. 25(5):1106–1115.Crossref, Google Scholar
• Merkle EC, Steyvers M (2013) Choosing a strictly proper scoring rule. Decision Anal. 10(4):292–304.Link, Google Scholar
• Mhanna S, Verbic G, Chapman AC (2014a) Guidelines for realistic grounding of mechanism design in demand response. Proc. 2014 Australasian Universities Power Engrg. Conf., 1–6.Crossref, Google Scholar
• Mhanna S, Verbic G, Chapman AC (2014b) Towards a realistic implementation of mechanism design in demand response aggregation. IEEE Power Systems Comput. Conf., 1–7.Crossref, Google Scholar
• Mhanna S, Verbic G, Chapman AC (2015) A faithful distributed mechanism for demand response aggregation. IEEE Trans. Smart Grid 7(3):1743–1753.Crossref, Google Scholar
• Miller N, Resnick P, Zeckhauser R (2005) Eliciting informative feedback: The peer-prediction method. Management Sci. 51(9):1359–1373.Link, Google Scholar
• Muradoglu G, Onkal D (1994) An exploratory analysis of portfolio managers’ probabilistic forecasts of stock prices. J. Forecasting 13(7):565–578.Crossref, Google Scholar
• Murphy AH (1974) A sample skill score for probability forecasts. Monthly Weather Rev. 102(1):48–55.Crossref, Google Scholar
• Murphy AH (1985) Decision making and the value of forecasts in a generalized model of the cost-loss ratio situation. Monthly Weather Rev. 113(3):362–369.Crossref, Google Scholar
• Murphy AH (1993) What is a good forecast? An essay on the nature of goodness in weather forecasting. Weather and Forecasting 8(2):281–293.Crossref, Google Scholar
• Murphy AH, Daan H (1984) Impacts of feedback and experience on the quality of subjective probability forecasts: Comparison of results from the first and second years of the Zierikzee experiment. Monthly Weather Rev. 112(3):413–423.Crossref, Google Scholar
• Murphy AH, Winkler RL (1982) Subjective probabilistic tornado forecasts: Some experimental results. Monthly Weather Rev. 110(9):1288–1297.Crossref, Google Scholar
• Murphy AH, Winkler RL (1984) Probability forecasting in meteorology. J. Amer. Statist. Assoc. 79(387):489–500.Google Scholar
• Murphy AH, Winkler RL (1992) Diagnostic verification of probability forecasts. Internat. J. Forecasting 7(4):435–455.Crossref, Google Scholar
• Murphy AH, Brown BG, Chen YS (1989) Diagnostic verification of temperature forecasts. Weather and Forecasting 4(4):485–501.Crossref, Google Scholar
• Murphy JM (1990) Assessment of the practical utility of extended range ensemble forecasts. Quart. J. Royal Meteorological Soc. 116(491):89–125.Crossref, Google Scholar
• Nakazono Y (2013) Strategic behavior of Federal Open Market Committee board members: Evidence from members’ forecasts. J. Econom. Behav. Organ. 93(September):62–70.Crossref, Google Scholar
• Nelson RG, Bessler DA (1989) Subjective probabilities and scoring rules: Experimental evidence. Amer. J. Agricultural Econom. 71(2):363–369.Crossref, Google Scholar
• Nikolova E, Sami R (2007) A strategic model for information markets. Proc. 8th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 316–325.Crossref, Google Scholar
• Nyarko Y, Schotter A (2002) An experimental study of belief learning using elicited beliefs. Econometrica 70(3):971–1005.Crossref, Google Scholar
• O’Carroll FM (1977) Subjective probabilities and short-term economic forecasts: An empirical investigation. Appl. Statist. 26(3):269–278.Crossref, Google Scholar
• Offerman T, Palley AB (2015) Lossed in translation: An off-the-shelf method to recover probabilistic beliefs from loss-averse agents. Experiment. Econom. 19(1):1–30.Crossref, Google Scholar
• Offerman T, Sonnemans J (2004) What’s causing overreaction? An experimental investigation of recency and the hot-hand effect. Scandinavian J. Econom. 106(3):533–553.Crossref, Google Scholar
• Offerman T, Sonnemans J, Schram A (1996) Value orientations, expectations and voluntary contributions in public goods. Econom. J. 106(437):817–845.Google Scholar
• Offerman T, Sonnemans J, Van De Kuilen G, Wakker PP (2009) A truth serum for non-Bayesians: Correcting proper scoring rules for risk attitudes. Rev. Econom. Stud. 76(4):1461–1489.Crossref, Google Scholar
• Oka M, Todo T, Sakurai Y, Yokoo M (2014) Predicting own action: Self-fulfilling prophecy induced by proper scoring rules. Proc. 2nd AAAI Conf. Human Comput. Crowdsourcing.Google Scholar
• Ostrovsky M (2012) Information aggregation in dynamic markets with strategic traders. Econometrica 80(6):2595–2647.Crossref, Google Scholar
• Othman A, Sandholm T (2010) Decision rules and decision markets. Proc. 9th Internat. Conf. Autonomous Agents and Multiagent Systems, 625–632.Google Scholar
• Othman A, Sandholm T (2011) Liquidity-sensitive automated market makers via homogeneous risk measures. Chen N, Elkind E, Koutsoupias E, eds. Internet and Network Economics, Lecture Notes in Computer Science, Vol. 7090 (Springer-Verlag, New York), 314–325.Crossref, Google Scholar
• Othman A, Sandholm T (2013) The Gates Hillman prediction market. Rev. Econom. Design 17(2):95–128.Crossref, Google Scholar
• Othman A, Pennock DM, Reeves DM, Sandholm T (2013) A practical liquidity-sensitive automated market maker. ACM Trans. Econom. Comput. 1(3):14.Google Scholar
• Palfrey TR, Wang SW (2009) On eliciting beliefs in strategic games. J. Econom. Behav. Organ. 71(2):98–109.Crossref, Google Scholar
• Parkes DC, Wellman MP (2015) Economic reasoning and artificial intelligence. Science 349(6245):267–272.Crossref, Google Scholar
• Pennock DM, Debnath S, Glover EJ, Giles CL (2002) Modeling information incorporation in markets, with application to detecting and explaining events. Proc. 18th Conf. Uncertainty in Artificial Intelligence, 405–413.Google Scholar
• Petroliagis TI, Tambke J, Heinemann D, Denhard M, Hagedorn R (2010) How well can we forecast winds at different heights? An assessment of ECMWF IFS & EPS skill of forecasting wind fields at different model levels. Proc. Eur. Wind Energy Assoc. Conf.Google Scholar
• Phillips LD, Edwards W (1966) Conservatism in a simple probability inference task. J. Experiment. Psych. 72(3):346–354.Crossref, Google Scholar
• Pinson P, Nielsen HA, Møller JK, Madsen H, Kariniotakis GN (2007) Non-parametric probabilistic forecasts of wind power: Required properties and evaluation. Wind Energy 10(6):497–516.Crossref, Google Scholar
• Prelec D (2004) A Bayesian truth serum for subjective data. Science 306(5695):462–466.Crossref, Google Scholar
• Radanovic G, Faltings B (2013) A robust Bayesian truth serum for non-binary signals. Proc. Twenty-Seventh AAAI Conf. Artificial Intelligence, 833–839.Google Scholar
• Radanovic G, Faltings B (2015) Incentives for subjective evaluations with private beliefs. Proc. 19th AAAI Conf. Artificial Intelligence, 1014–1020.Google Scholar
• Ray R, Vallam RD, Narahari Y (2013) Eliciting high quality feedback from crowdsourced tree networks using continuous scoring rules. Proc. 12th Internat. Conf. Autonomous Agents and Multiagent Systems, 279–286.Google Scholar
• Robu V, Kota R, Chalkiadakis G, Rogers A, Jennings NR (2012) Cooperative virtual power plant formation using scoring rules. Proc. 11th Internat. Conf. Autonomous Agents and Multiagent Systems, 1165–1166.Google Scholar
• Rose H, Rogers A, Gerding EH (2011) Mechanism design for aggregated demand prediction in the smart grid. Proc. Workshops at the 25th AAAI Conf. Artificial Intelligence.Google Scholar
• Rose H, Rogers A, Gerding EH (2012) A scoring rule-based mechanism for aggregate demand prediction in the smart grid. Proc. 11th Internat. Conf. Autonomous Agents and Multiagent Systems, 661–668.Google Scholar
• Roulston MS, Smith LA (2002) Evaluating probabilistic forecasts using information theory. Monthly Weather Rev. 130(6):1653–1660.Crossref, Google Scholar
• Rutström EE, Wilcox NT (2009) Stated beliefs versus inferred beliefs: A methodological inquiry and experimental test. Games Econom. Behav. 67(2):616–632.Crossref, Google Scholar
• Sakurai Y, Shinoda M, Oyama S, Yokoo M (2015) Flexible reward plans for crowdsourced tasks. Chen Q, Torroni P, Villata S, Hsu J, Omicini A, eds. Proc. 18th Internat. Conf. Principles and Practice of Multi-Agent Systems (Springer International Publishing, Switzerland), 400–415.Crossref, Google Scholar
• Sakurai Y, Okimoto T, Oka M, Shinoda M, Yokoo M (2013) Ability grouping of crowd workers via reward discrimination. Proc. 1st AAAI Conf. Human Comput. Crowdsourcing, 147–155.Google Scholar
• Sanders F (1963) On subjective probability forecasting. J. Appl. Meteorology 2(2):191–201.Crossref, Google Scholar
• Sandroni A, Shmaya E (2013) Eliciting beliefs by paying in chance. Econom. Theory Bull. 1(1):33–37.Crossref, Google Scholar
• Savage LJ (1971) Elicitation of personal probabilities and expectations. J. Amer. Statist. Assoc. 66(336):783–801.Crossref, Google Scholar
• Schervish MJ (1989) A general method for comparing probability assessors. Ann. Statist. 17(4):1856–1879.Crossref, Google Scholar
• Scheuerer M, Hamill TM (2015) Variogram-based proper scoring rules for probabilistic forecasts of multivariate quantities. Monthly Weather Rev. 143(4):1321–1334.Crossref, Google Scholar
• Schlag KH, van der Weele JJ (2013) Eliciting probabilities, means, medians, variances and covariances without assuming risk neutrality. Theoretical Econom. Lett. 3(1):38–42.Crossref, Google Scholar
• Schum DA, Goldstein IL, Howell WC, Southard JF (1967) Subjective probability revisions under several cost-payoff arrangements. Organ. Behav. Human Performance 2(1):84–104.Crossref, Google Scholar
• Selten R (1998) Axiomatic characterization of the quadratic scoring rule. Experiment. Econom. 1(1):43–62.Crossref, Google Scholar
• Selten R, Sadrieh A, Abbink K (1999) Money does not induce risk neutral behavior, but binary lotteries do even worse. Theory Decision 46(3):211–249.Crossref, Google Scholar
• Shi P, Conitzer V, Guo M (2009) Prediction mechanisms that do not incentivize undesirable actions. Leonardi S, ed. Internet and Network Economics, Lecture Notes in Computer Science, Vol. 5929 (Springer, Berlin), 89–100.Crossref, Google Scholar
• Slamka C, Skiera B, Spann M (2013) Prediction market performance and market liquidity: A comparison of automated market makers. IEEE Trans. Engrg. Management 60(1):169–185.Crossref, Google Scholar
• Smith LA, Suckling EB, Thompson EL, Maynard T, Du H (2015) Towards improving the framework for probabilistic forecast evaluation. Climatic Change 132(1):31–45.Crossref, Google Scholar
• Spiegelhalter DJ (1986) Probabilistic prediction in patient management and clinical trials. Statist. Medicine 5(5):421–433.Crossref, Google Scholar
• Spiegelhalter DJ, Franklin RCG, Bull K (1990) Assessment, criticism and improvement of imprecise subjective probabilities for a medical expert system. Proc. 5th Annual Conf. Uncertainty in Artificial Intelligence, 285–294.Crossref, Google Scholar
• Štrumbelj E, Šikonja MR (2010) Online bookmakers’ odds as forecasts: The case of European soccer leagues. Internat. J. Forecasting 26(3):482–488.Crossref, Google Scholar
• Surowiecki J (2005) The Wisdom of Crowds (Anchor, New York).Google Scholar
• Tetlock P (2005) Expert Political Judgment: How Good Is It? How Can We Know? (Princeton University Press, Princeton, NJ).Google Scholar
• Tetlock PE, Kim JI (1987) Accountability and judgment processes in a personality prediction task. J. Personality Soc. Psych. 52(4):700–709.Crossref, Google Scholar
• Thorarinsdottir TL, Gneiting T (2010) Probabilistic forecasts of wind speed: Ensemble model output statistics by using heteroscedastic censored regression. J. Roy. Statist. Soc. Ser. A 173(2):371–388.Crossref, Google Scholar
• Thorarinsdottir TL, Gneiting T, Gissibl N (2013) Using proper divergence functions to evaluate climate models. SIAM/ASA J. Uncertainty Quantification 1(1):522–534.Crossref, Google Scholar
• Toda M (1963) Measurement of subjective probability distributions. Technical Report ESD-TDR-63-407, Decision Sciences Laboratory, Electronic Systems Division, Air Force Systems Command, United States Air Force, Bedford, MA.Google Scholar
• Trautmann ST, van de Kuilen G (2014) Belief elicitation: A horse race among truth serums. Econom. J. 125(589):2116–2135.Google Scholar
• Ugander J, Drapeau R, Guestrin C (2015) The wisdom of multiple guesses. Proc. 16th ACM Conf. Econom. Comput. (Association for Computing Machinery, New York), 643–660.Crossref, Google Scholar
• van Lenthe J (1994) Scoring-rule feedforward and the elicitation of subjective probability distributions. Organ. Behav. Human Decision Processes 59(2):188–209.Crossref, Google Scholar
• Vardi MY (2010) Revisiting the publication culture in computing research. Comm. ACM 53(3):5.Crossref, Google Scholar
• von Holstein CAS (1971a) An experiment in probabilistic weather forecasting. J. Appl. Meteorology 10(4):635–645.Crossref, Google Scholar
• von Holstein CAS (1971b) The effect of learning on the assessment of subjective probability distributions. Organ. Behav. Human Performance 6(3):304–315.Crossref, Google Scholar
• von Holstein CAS (1971c) Two techniques for assessment of subjective probability distributions—An experimental study. Acta Psychologica 35(6):478–494.Crossref, Google Scholar
• von Holstein CAS (1972) Probabilistic forecasting: An experiment related to the stock market. Organ. Behav. Human Performance 8(1):139–158.Crossref, Google Scholar
• Walker KD, Catalano P, Hammitt JK, Evans JS (2003) Use of expert judgment in exposure assessment: Part 2. Calibration of expert judgments about personal exposures to benzene. J. Exposure Sci. Environ. Epidemiology 13(1):1–16.Crossref, Google Scholar
• Wang SW (2011) Incentive effects: The case of belief elicitation from individuals in groups. Econom. Lett. 111(1):30–33.Crossref, Google Scholar
• Weijs SV, Schoups G, van De Giesen N (2010a) Why hydrological predictions should be evaluated using information theory. Hydrology and Earth System Sci. 14:2545–2558.Crossref, Google Scholar
• Weijs SV, van Nooijen R, van de Giesen N (2010b) Kullback-Leibler divergence as a forecast skill score with classic reliability-resolution-uncertainty decomposition. Monthly Weather Rev. 138(9):3387–3399.Crossref, Google Scholar
• Wilks DS (2011) Statistical Methods in the Atmospheric Sciences (Academic Press, Cambridge, MA).Google Scholar
• Wilson LJ, Burrows WR, Lanzinger A (1999) A strategy for verification of weather element forecasts from an ensemble prediction system. Monthly Weather Rev. 127(6):956–970.Crossref, Google Scholar
• Winkler RL (1969) Scoring rules and the evaluation of probability assessors. J. Amer. Statist. Assoc. 64(327):1073–1078.Crossref, Google Scholar
• Winkler RL (1971) Probabilistic prediction: Some experimental results. J. Amer. Statist. Assoc. 66(336):675–685.Crossref, Google Scholar
• Winkler RL (1972) A decision-theoretic approach to interval estimation. J. Amer. Statist. Assoc. 67(337):187–191.Crossref, Google Scholar
• Winkler RL (1994) Evaluating probabilities: Asymmetric scoring rules. Management Sci. 40(11):1395–1405.Link, Google Scholar
• Winkler RL, Murphy AH (1969) “Good” probability assessors. J. Appl. Meteorology 7(5):751–758.Crossref, Google Scholar
• Winkler RL, Murphy AH (1970) Nonlinear utility and the probability score. J. Appl. Meteorology 9(1):143–148.Crossref, Google Scholar
• Winkler RL, Poses RM (1993) Evaluating and combining physicians’ probabilities of survival in an intensive care unit. Management Sci. 39(12):1526–1543.Link, Google Scholar
• Witkowski J, Parkes DC (2012) A robust Bayesian truth serum for small populations. Proc. 26th AAAI Conf. Artificial Intelligence.Google Scholar
• Woo CK, Horowitz I, Martin J (1998) Reliability differentiation of electricity transmission. J. Regulatory Econom. 13(3):277–292.Crossref, Google Scholar
• Yates JF, McDaniel LS, Brown ES (1991) Probabilistic forecasts of stock prices and earnings: The hazards of nascent expertise. Organ. Behav. Human Decision Processes 49(1):60–79.Crossref, Google Scholar
• Zhang H, Horvitz E, Chen Y, Parkes DC (2012) Task routing for prediction tasks. Proc. 11th Internat. Conf. Autonomous Agents and Multiagent Systems, 889–896.Google Scholar