Confidence Calibration in a Multiyear Geopolitical Forecasting Competition

References

• Anderson C, Brion S, Moore DA, Kennedy JA (2012) A social-functional account of overconfidence. J. Personality Soc. Psych. 103(4):718–735.Crossref, Google Scholar
• Armor DA, Sackett AM (2006) Accuracy, error, and bias in predictions for real versus hypothetical events. J. Personality Soc. Psych. 91(4):583–600.Crossref, Google Scholar
• Armstrong JS (2001) Principles of Forecasting: A Handbook for Researchers and Practitioners (Kluwer Academic, Boston).Crossref, Google Scholar
• Atanasov P, Rescober P, Stone E, Swift SA, Servan-Schreiber E, Tetlock P, Unger L, Mellers B (2017) Distilling the wisdom of crowds: Prediction markets vs. prediction polls. Management Sci. 63(3):691–706.Link, Google Scholar
• Baron J, Ungar L, Mellers BA, Tetlock PE (2014) Two reasons to make aggregated probability forecasts more extreme. Decision Anal. 11(2):133–145.Link, Google Scholar
• Bazerman MH, Moore DA (2013) Judgment in Managerial Decision Making, 8th ed. (John Wiley & Sons, New York).Google Scholar
• Ben-David I, Graham JR, Harvey CR (2013) Managerial miscalibration. Quart. J. Econom. 128(4):1547–1584.Crossref, Google Scholar
• Benson PG, Onkal D (1992) The effects of feedback and training on the performance of probability forecasters. Internat. J. Forecasting 8(4):559–573.Crossref, Google Scholar
• Brier GW (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Rev. 78(1):1–3.Crossref, Google Scholar
• Budescu DV, Wallsten TS, Au WT (1997) On the importance of random error in the study of probability judgment: Part II. Applying the stochastic judgment model to detect systematic trends. J. Behavioral Decision Making 10(3):173–188.Crossref, Google Scholar
• Buehler R, Messervey D, Griffin DW (2005) Collaborative planning and prediction: Does group discussion affect optimistic biases in time estimation. Organ. Behav. Human Decision Processes 97(1):47–63.Crossref, Google Scholar
• Burson KA, Larrick RP, Klayman J (2006) Skilled or unskilled, but still unaware of it: How perceptions of difficulty drive miscalibration in relative comparisons. J. Personality Soc. Psych. 90(1):60–77.Crossref, Google Scholar
• Butler AC, Fazio LK, Marsh EJ (2011) The hypercorrection effect persists over a week, but high-confidence errors return. Psychonomic Bull. Rev. 18(6):1238–1244.Crossref, Google Scholar
• Camerer CF, Johnson EJ (1991) The process-performance paradox in expert judgment: How can experts know so much and predict so badly? Ericsson KA, Smith J, eds. Toward a General Theory of Expertise: Prospects and Limits (Cambridge University Press, Cambridge, UK), 195–217.Google Scholar
• Chauvin C, Fischhoff B, eds. (2011) Intelligence Analysis: Behavioral and Social Scientific Foundations (National Academies Press, Washington, DC).Google Scholar
• Clark J, Friesen L (2009) Overconfidence in forecasts of own performance: An experimental study. Econom. J. 119(534):229–251.Google Scholar
• Cooper AC, Woo CY, Dunkelberg WC (1988) Entrepreneurs’ perceived chances for success. J. Bus. Venturing 3(2):97–109.Crossref, Google Scholar
• Dawes RM, Mulford M (1996) The false consensus effect and overconfidence: Flaws in judgment or flaws in how we study judgment? Organ. Behav. Human Decision Processes 65(3):201–211.Crossref, Google Scholar
• Deaves R, Lüders E, Schröder M (2010) The dynamics of overconfidence: Evidence from stock market forecasters. J. Econom. Behav. Organ. 75(3):402–412.Crossref, Google Scholar
• Dobbins GH, Zaccaro SJ (1986) The effects of group cohesion and leader behavior on subordinate satisfaction. Group Organ. Management 11(3):203–219.Crossref, Google Scholar
• Dunlosky J, Rawson KA (2012) Overconfidence produces underachievement: Inaccurate self evaluations undermine students’ learning and retention. Learn. Instruction 22(4):271–280.Crossref, Google Scholar
• Erev I, Wallsten TS, Budescu DV (1994) Simultaneous over- and underconfidence: The role of error in judgment processes. Psych. Rev. 101(3):519–527.Crossref, Google Scholar
• Fischhoff B (1982) Debiasing. Kahneman D, Slovic P, Tversky A, eds. Judgment Under Uncertainty: Heuristics and Biases (Cambridge University Press, Cambridge, MA), 422–444.Crossref, Google Scholar
• Fischhoff B, Slovic P, Lichtenstein S (1977) Knowing with certainty: The appropriateness of extreme confidence. J. Experiment. Psych.: Human Perception Performance 3(4):552–564.Crossref, Google Scholar
• Flyvbjerg B, Holm MKS, Buhl SL (2002) Underestimating costs in public works projects: Error or lie? J. Amer. Planning Assoc. 68(3):279–295.Crossref, Google Scholar
• Gardner D (2010) Future Babble: Why Expert Predictions Fail – and Why We Believe Them Anyway (Random House, New York).Google Scholar
• Gigerenzer G (1991) How to make cognitive illusions disappear: Beyond “heuristics and biases.” Eur. Rev. Soc. Psych. 2(1):83–115.Crossref, Google Scholar
• Gigerenzer G, Hoffrage U, Kleinbölting H (1991) Probabilistic mental models: A Brunswikian theory of confidence. Psych. Rev. 98(4):506–528.Crossref, Google Scholar
• Glaser M, Weber M (2007) Overconfidence and trading volume. Geneva Risk Insurance Rev. 32(1):1–36.Crossref, Google Scholar
• González-Vallejo C, Bonham A (2007) Aligning confidence with accuracy: Revisiting the role of feedback. Acta Psychologica 125(2):221–39.Crossref, Google Scholar
• Griffin DW, Brenner L (2004) Perspectives on probability judgment calibration. Koehler DJ, Harvey N, eds. Blackwell Handbook of Judgment and Decision Making (Blackwell, Malden, MA), 177–199.Crossref, Google Scholar
• Griffin DW, Tversky A (1992) The weighing of evidence and the determinants of confidence. Cognitive Psych. 24(3):411–435.Crossref, Google Scholar
• Hall CC, Ariss L, Todorov A (2007) The illusion of knowledge: When more information reduces accuracy and increases confidence. Organ. Behav, Human Decision Processes 103(2):277–290.Crossref, Google Scholar
• Hansson P, Rönnlund M, Juslin P, Nilsson L-G (2008) Adult age differences in the realism of confidence judgments: Overconfidence, format dependence, and cognitive predictors. Psych. Aging 23(3):531–544.Crossref, Google Scholar
• Haran U, Moore DA, Morewedge CK (2010) A simple remedy for overprecision in judgment. Judgment and Decision Making 5(7):467–476.Google Scholar
• Harvey N (1997) Confidence in judgment. Trends Cognitive Sci. 1(2):78–82.Crossref, Google Scholar
• Heath C, Gonzalez R (1995) Interaction with others increases decision confidence but not decision quality: Evidence against information collection views of interactive decision making. Organ. Behav. Human Decision Processes 61(3):305–326.Crossref, Google Scholar
• Hoelzl E, Rustichini A (2005) Overconfident: Do you put your money on it? Econom. J. 115(503):305–318.Google Scholar
• Jagacinski RJ, Isaac PD, Burke MW (1977) Application of signal detection theory to perceptual-motor skills: Decision processes in basketball shooting. J. Motor Behav. 9(3):225–234.Crossref, Google Scholar
• Janis IL, Mann L (1977) Decision Making: A Psychological Analysis of Conflict, Choice, and Commitment (Free Press, New York).Google Scholar
• Juslin P, Winman A, Olsson H (2000) Naive empiricism and dogmatism in confidence research: A critical examination of the hard-easy effect. Psych. Rev. 107(2):384–396.Crossref, Google Scholar
• Kahneman D (2011) Thinking Fast and Slow (Farrar, Straus and Giroux, New York).Google Scholar
• Kahneman D, Lovallo D (1993) Timid choices and bold forecasts: A cognitive perspective on risk and risk taking. Management Sci. 39(1):17–31.Link, Google Scholar
• Keren G (1987) Facing uncertainty in the game of bridge: A calibration study. Organ. Behav. Human Decision Processes 39(1):98–114.Crossref, Google Scholar
• Keren G (1991) Calibration and probability judgments: Conceptual and methodological issues. Acta Psychologica 77(3):217–273.Crossref, Google Scholar
• Kerr NL, MacCoun RJ, Kramer GP (1996) Bias in judgment: Comparing individuals and groups. Psych. Rev. 103(4):687–719.Crossref, Google Scholar
• Klayman J, Soll JB, Gonzalez-Vallejo C, Barlas S (1999) Overconfidence: It depends on how, what, and whom you ask. Organ. Behav. Human Decision Processes 79(3):216–247.Crossref, Google Scholar
• Koehler DJ, Brenner L, Griffin DW (2002) The calibration of expert judgment: Heuristics and biases beyond the laboratory. Gilovich T, Griffin D, Kahneman D, eds. Heuristics and Biases: The Psychology of Intuitive Judgment (Cambridge University Press, Cambridge, UK), 686–715.Crossref, Google Scholar
• Koriat A, Sheffer L, Ma’ayan H (2002) Comparing objective and subjective learning curves: Judgments of learning exhibit increased underconfidence with practice. J. Experiment. Psych.: General 131(2):147–162.Crossref, Google Scholar
• Kruger J (1999) Lake Wobegon be gone! The “below-average effect” and the egocentric nature of comparative ability judgments. J. Personality Soc. Psych. 77(2):221–232.Crossref, Google Scholar
• Kruger J, Dunning D (1999) Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J. Personality Soc. Psych. 77(6):1121–1134.Crossref, Google Scholar
• Larrick RP (2004) Debiasing. Koehler DJ, Harvey N, eds. Blackwell Handbook of Judgment and Decision Making (Blackwell Publishers, Malden, MA), 316–337.Crossref, Google Scholar
• Larrick RP, Soll JB (2006) Intuitions about combining opinions: Misappreciation of the averaging principle. Management Sci. 52(1):111–127.Link, Google Scholar
• Larrick RP, Burson KA, Soll JB (2007) Social comparison and confidence: When thinking you’re better than average predicts overconfidence (and when it does not). Organ. Behav. Human Decision Processes 102(1):76–94.Crossref, Google Scholar
• Lerner JS, Tetlock PE (1999) Accounting for the effects of accountability. Psych. Bull. 125(2):255–275.Crossref, Google Scholar
• Lichtenstein S, Fischhoff B (1980) Training for calibration. Organ. Behav. Human Decision Processes 26(2):149–171.Crossref, Google Scholar
• Lichtenstein S, Fischhoff B, Phillips LD (1977) Calibration of probabilities: The state of the art. Jungermann H, DeZeeuw G, eds. Decision Making and Change in Human Affairs (D. Reidel, Amsterdam), 275–324.Crossref, Google Scholar
• Mandel DR, Barnes A (2014) Accuracy of forecasts in strategic intelligence. Proc. Natl. Acad. Sci. USA 111(30):10984–10989.Crossref, Google Scholar
• Massey C, Simmons JP, Armor DA (2011) Hope over experience: Desirability and the persistence of optimism. Psych. Sci. 22(2):274–281.Crossref, Google Scholar
• McKenzie CRM, Liersch MJ, Yaniv I (2008) Overconfidence in interval estimates: What does expertise buy you? Organ. Behav. Human Decision Processes 107(2):179–191.Crossref, Google Scholar
• Mellers BA, Stone E, Atanasov P, Rohrbaugh N, Metz SE, Ungar L, Tetlock PE (2015a) The psychology of intelligence analysis: Drivers of prediction accuracy in world politics. J. Experiment. Psych.: Appl. 21(1):1–14.Crossref, Google Scholar
• Mellers BA, Stone ER, Murray T, Minster A, Rohrbaugh N, Bishop MM, Chen MMet al. (2015b) Identifying and cultivating superforecasters as a method of improving probabilistic predictions. Perspect. Psych. Sci. 10(3):267–281.Crossref, Google Scholar
• Mellers BA, Ungar L, Baron J, Ramos J, Gurcay B, Fincher K, Scott SEet al. (2014) Psychological strategies for winning a geopolitical forecasting tournament. Psych. Sci. 25(5):1106–1115.Crossref, Google Scholar
• Moore DA (2007) Not so above average after all: When people believe they are worse than average and its implications for theories of bias in social comparison. Organ. Behav. Human Decision Processes 102(1):42–58.Crossref, Google Scholar
• Moore DA, Healy PJ (2008) The trouble with overconfidence. Psych. Rev. 115(2):502–517.Crossref, Google Scholar
• Moscovici S, Zavalloni M (1969) The group as a polarizer of attitudes. J. Personality Soc. Psych. 12(2):125–135.Crossref, Google Scholar
• Murphy AH, Winkler RL (1977) Can weather forecasters formulate reliable probability forecasts of precipitation and temperature? Natl. Weather Digest 2:2–9.Google Scholar
• Oskamp S (1965) Overconfidence in case-study judgments. J. Consulting Psych. 29(3):261–265.Crossref, Google Scholar
• Park YJ, Santos-Pinto L (2010) Overconfidence in tournaments: Evidence from the field. Theory Decision 69(1):143–166.Crossref, Google Scholar
• Prims J, Moore DA (2015) Overconfidence over the lifespan. Working paper, University of Illinois at Chicago, Chicago.Google Scholar
• Satopää V, Baron J, Foster B, Mellers BA, Tetlock PE, Ungar L (2014a) Combining multiple probability predictions using a simple logit model. Internat. J. Forecasting 30(2):344–356.Crossref, Google Scholar
• Satopää V, Jensen B, Mellers BA, Tetlock PE, Ungar L (2014b) Probability aggregation in time series: Dynamic hierarchical modeling of sparse expert beliefs. Ann. Appl. Statist. 8:1256–1280.Crossref, Google Scholar
• Sieck WR, Arkes HR (2005) The recalcitrance of overconfidence and its contribution to decision aid neglect. J. Behav. Decision Making 18(1):29–53.Crossref, Google Scholar
• Silver N (2012) The Signal and the Noise: Why so Many Predictions Fail—But Some Don’t (Penguin Press, New York).Google Scholar
• Simon M, Houghton SM (2003) The relationship between overconfidence and the introduction of risky products: Evidence from a field study. Acad. Management J. 46(2):139–150.Crossref, Google Scholar
• Sniezek JA, Henry RA (1989) Accuracy and confidences in group judgment. Organ. Behav. Human Decision Processes 4(3):1–28.Crossref, Google Scholar
• Soll JB, Milkman KL, Payne JW (2016) A user’s guide to debiasing. Wu G, Keren G, eds. Handbook of Judgment and Decision Making (John Wiley & Sons, New York), 924–951.Google Scholar
• Stasser G, Davis JH (1981) Group decision making and social influence: A social interaction sequence model. Psych. Rev. 88(6):523–551.Crossref, Google Scholar
• Stasser G, Titus W (1987) Effects of information load and percentage of shared information on the dissemination of unshared information during group discussion. J. Personality Soc. Psych. 53:81–93.Crossref, Google Scholar
• Statman M, Thorley S, Vorkink K (2006) Investor overconfidence and trading volume. Rev. Financial Stud. 19(4):1531–1565.Crossref, Google Scholar
• Stone ER, Opel RB (2000) Training to improve calibration and discrimination: The effects of performance and environmental feedback. Organ. Behav. Human Decision Processes 83(2):282–309.Crossref, Google Scholar
• Tetlock PE (2005) Expert Political Judgment: How Good Is It? How Can We Know? (Princeton University Press, Princeton, NJ).Google Scholar
• Tetlock PE, Gardner D (2015) Superforecasting: The Art and Science of Prediction (Signal, New York).Google Scholar
• Tetlock PE, Mellers BA (2011) Intelligent management of intelligence agencies: Beyond accountability ping-pong. Amer. Psychologist 66(6):542–554.Crossref, Google Scholar
• Tetlock PE, Mellers BA, Rohrbaugh N, Chen E (2014) Forecasting tournaments: Tools for increasing transparency and improving the quality of debate. Current Directions Psychological Sci. 23:290–295.Crossref, Google Scholar
• Tversky A, Koehler DJ (1994) Support theory: A nonextensional representation of subjective probability. Psych. Rev. 101(4):547–567.Crossref, Google Scholar
• Wells GL, Olson EA (2003) Eyewitness testimony. Annual Rev. Psych. 54:277–295.Crossref, Google Scholar
• Wright G, Rowe G, Bolger F, Gammack J (1994) Coherence, calibration, and expertise in judgmental probability forecasting. Organ. Behav. Human Decision Processes 57(1):1–25.Crossref, Google Scholar
• Yates JF (1982) External correspondence: Decompositions of the mean probability score. Organ. Behav. Human Decision Processes 30(1):132–156.Crossref, Google Scholar
• Yates JF, Lee JW, Shinotsuka H, Patalano AL, Sieck WR (1998) Cross-cultural variations in probability judgment accuracy: Beyond general knowledge overconfidence? Organ. Behav. Human Decision Processes 74(2):89–117.Crossref, Google Scholar