Validating the Contribution-Weighted Model: Robustness and Cost-Benefit Analyses

References

• Atanasov P, Rescober P, Stone E, Swift SA, Servan-Schreiber E, Tetlock P, Ungar L, Mellers B (2016) Distilling the wisdom of crowds: Prediction markets vs. prediction polls. Management Sci., ePub ahead of print April 22, http://dx.doi.org/10.1287/mnsc.2015.2374.Google Scholar
• Arkes HR, Christensen C, Lai C, Blumer C (1987) Two methods of reducing overconfidence. Organ. Behav. Human Decision Processes 39(1):133–144.Crossref, Google Scholar
• Baron J, Ungar L, Mellers B, Tetlock PE (2014) Two reasons to make aggregated probabilities more extreme. Decision Anal. 11(2):133–145.Link, Google Scholar
• Broomell SB, Budescu DV (2009) Why are experts correlated? Decomposing correlations between judges. Psychometrika 74(3):531–553.Crossref, Google Scholar
• Budescu DV, Chen E (2015) Identifying expertise to extract the wisdom of crowds. Management Sci. 61(2):267–280.Link, Google Scholar
• Cohen J (1960) A coefficient of agreement for nominal scales. Educational Psych. Measurement 20(1):37–46.Crossref, Google Scholar
• Fischhoff B (2001) Learning from experience. Armstrong JS, ed. Handbook of Forecasting Principles (Kluwer, Norwell, MA), 484–494.Crossref, Google Scholar
• Gilovich T, Griffin D, Kahneman D (2002) Heuristics and Biases: The Psychology of Intuitive Judgment (Cambridge University Press, Cambridge, UK).Crossref, Google Scholar
• Hackman JR, Morris CG (1975) Group tasks, group interaction process, and group performance effectiveness: A review and proposed integration. Berkowitz L, ed. Advances in Experimental Social Psychology, Vol. 8 (Academic Press, New York), 1–55.Crossref, Google Scholar
• Herzog SM, Hertwig R (2011) The wisdom of ignorant crowds: Predicting sport outcomes by mere recognition. Judgment Decision Making 6(1):58–72.Google Scholar
• Johnson EJ (1988) Expertise and decision under uncertainty: Performance and process. Chi MTH, Glaser R, Farr MJ, eds. The Nature of Expertise (Erlbaum, Hillsdale, NJ), 209–228.Google Scholar
• Jose VRR, Grushka-Cockayne Y, Lichtendahl Jr KC (2014) Trimmed opinion pools and the crowd’s calibration problem. Management Sci. 60(2):63–475.Link, Google Scholar
• Kahneman D, Tversky A (1973) On the psychology of prediction. Psych. Rev. 80(4):237–251.Crossref, Google Scholar
• Karvetski CW, Olson KC, Mandel DR, Twardy CR (2013) Probabilistic coherence weighting for optimizing expert forecasts. Decision Anal. 10(4):305–326.Link, Google Scholar
• Kauko K, Palmroos P (2014) The Delphi method in forecasting financial markets—An experimental study. Internat. J. Forecasting 30(2):313–327.Crossref, Google Scholar
• Koriat A, Lichtenstein S, Fischhoff B (1980) Reasons for confidence. J. Experiment. Psych.: Human Learning Memory 6(2):107–118.Crossref, Google Scholar
• MacGregor D, Armstrong JS (1994) Judgmental decomposition: When does it work? Internat. J. Forecasting 10(4):495–506.Crossref, Google Scholar
• Maciejovsky B, Budescu DV (2013) Markets as a structural solution to knowledge-sharing dilemmas. Organ. Behav. Human Decision Processes 120(2):154–167.Crossref, Google Scholar
• Mandel DR (2015) Instruction in information structuring improves Bayesian judgment in intelligence analysts. Frontiers Psych. 6:387.Google Scholar
• Mannes AE, Soll JB, Larrick RP (2014) The wisdom of selected crowds. J. Personality Soc. Psych. 107(2):276–299.Crossref, Google Scholar
• Mellers BA, Ungar L, Baron J, Ramos J, Gurcay B, Fincher K, Scott SE, Moore D, Atanasov P, Swift SA, Murray T, Stone E, Tetlock PE (2014) Psychological strategies for winning geopolitical forecasting tournaments. Psych. Sci. 25(5):1106–1115.Crossref, Google Scholar
• Mellers BA, Stone E, Murray T, Minster A, Rohrbaugh N, Bishop M, Chen E, Baker J, Hou Y, Horowitz M, Ungar L, Tetlock PE (2015) Identifying and cultivating “superforecasters” as a method of improving probabilistic predictions. Perspectives Psych. Sci. 10(3):267–281.Crossref, Google Scholar
• O’Hagan A, Buck CE, Daneshkhah A, Eiser JR, Garthwaite PH, Jenkinson DJ, Oakley JE, Rakow T (2006) Uncertain Judgments: Eliciting Experts’ Probabilities (Wiley, New York).Crossref, Google Scholar
• Pagliari C, Grimshaw J, Eccles M (2001) The potential influence of small group processes on guideline development. J. Evaluation Clinical Practice 7(2):165–173.Crossref, Google Scholar
• Ravinder HV, Kleinmuntz DN, Dyer JS (1988) The reliability of subjective probability assessments obtained through decomposition. Management Sci. 34(2):186–199.Link, Google Scholar
• Roth R (2009) The impact of user expertise on geographic risk assessment under uncertain conditions. Cartography Geographic Inform. Sci. 36(1):29–43.Crossref, Google Scholar
• Rowe G, Wright G (1999) The Delphi technique as a forecasting tool: Issues and analysis. Internat. J. Forecasting 15(4):353–375.Crossref, Google Scholar
• Sanders NR (1992) Accuracy of judgmental forecasts: A comparison. Omega 20(3):353–364.Crossref, Google Scholar
• Satopaä VA, Baron J, Foster DP, Mellers BA, Tetlock PE, Ungar LH (2014) Combining multiple probability predictions using a simple logit model. Internat. J. Forecasting 30(2):344–356.Crossref, Google Scholar
• Simmons J, Nelson LD, Galak J, Frederick S (2011) Intuitive biases in choice vs. estimation: Implications for the wisdom of crowds. J. Consumer Res. 38(1):1–15.Crossref, Google Scholar
• Stefanovitch N, Alshamsi A, Cebrian M, Rahwan I (2014) Error and attack tolerance of collective problem solving: The DARPA Shredder Challenge. EPJ Data Sci. 3(13):1–27.Google Scholar
• Steyvers M, Wallsten TS, Merkle EC, Turner BM (2014) Evaluating probabilistic forecasts with Bayesian signal detection models. Risk Anal. 34(3):435–452.Crossref, Google Scholar
• Tetlock PE, Mellers BA, Rohrbaugh N, Chen E (2014) Forecasting tournaments: Tools for increasing transparency and improving the quality of debate. Current Directions Psych. Sci. 23(4):290–295.Crossref, Google Scholar
• Turner BM, Steyvers M, Merkle EC, Budescu DV, Wallsten TS (2014) Forecast aggregation via recalibration. Machine Learn. 95(3):261–289.Crossref, Google Scholar
• Tversky A, Kahneman D (1974) Judgment under uncertainty: Heuristics and biases. Science 185(4157):1124–1131.Crossref, Google Scholar
• Wallsten TS, Budescu DV, Erev I, Diederich A (1997) Evaluating and combining subjective probability estimates. J. Behavioral Decision Making 10(3):243–268.Crossref, Google Scholar
• Watson WE, Kumar K, Michaelsen LK (1993) Cultural diversity’s impact on interaction process and performance: Comparing homogeneous and diverse task groups. Acad. Management J. 36(3):590–602.Crossref, Google Scholar