Group Discussions Improve Competence Calibration: Making Self-Perceived Competence Valuable When Harnessing the Wisdom of Crowds

References

• Alba JW, Hutchinson JW (2000) Knowledge calibration: What consumers know and what they think they know. J. Consumer Res. 27(2):123–156.Crossref, Google Scholar
• Ariely D, Tung Au W, Bender RH, Budescu DV, Dietz CB, Gu H, Wallsten TS, Zauberman G (2000) The effects of averaging subjective probability estimates between and within judges. J. Experiment. Psych. Appl. 6(2):130–147.Crossref, Google Scholar
• Arkes HR (1991) Costs and benefits of judgment errors: Implications for debiasing. Psych. Bull. 110(3):486–498.Crossref, Google Scholar
• Aspinall W (2010) A route to more tractable expert advice. Nature 463(7279):294–295.Crossref, Google Scholar
• Atanasov P, Rescober P, Stone E, Swift SA, Servan-Schreiber E, Tetlock P, Ungar L, Mellers B (2017) Distilling the wisdom of crowds: Prediction markets vs. prediction polls. Management Sci. 63(3):691–706.Link, Google Scholar
• Baron J, Mellers BA, Tetlock PE, Stone E, Ungar LH (2014) Two reasons to make aggregated probability forecasts more extreme. Decision Anal. 11(2):133–145.Link, Google Scholar
• Benson PG, Önkal D (1992) The effects of feedback and training on the performance of probability forecasters. Internat. J. Forecasting 8(4):559–573.Crossref, Google Scholar
• Bolger F, Rowe G (2015a) The aggregation of expert judgment: Do good things come to those who weight? Risk Anal. 35(1):5–11.Crossref, Google Scholar
• Bolger F, Rowe G (2015b) There is data, and then there is data: Only experimental evidence will determine the utility of differential weighting of expert judgment. Risk Anal. 35(1):21–26.Crossref, Google Scholar
• Budescu DV, Chen E (2015) Identifying expertise to extract the wisdom of crowds. Management Sci. 61(2):267–280.Link, Google Scholar
• Burson KA, Larrick RP, Klayman J (2006) Skilled or unskilled, but still unaware of it: How perceptions of difficulty drive miscalibration in relative comparisons. J. Personality Soc. Psych. 90(1):60–77.Crossref, Google Scholar
• Clemen RT (1989) Combining forecasts: A review and annotated bibliography. Internat. J. Forecasting 5(4):559–583.Crossref, Google Scholar
• Clemen RT, Winkler RL (1986) Combining economic forecasts. J. Bus. Econom. Statist. 4(1):39–46.Crossref, Google Scholar
• Colson AR, Cooke RM (2017) Cross validation for the classical model of structured expert judgment. Reliability Engrg. System Safety 163(July):109–120.Crossref, Google Scholar
• Cooke RM (2015) Commentary: The aggregation of expert judgment: Do good things come to those who weight? Risk Anal. 35(1):12–15.Crossref, Google Scholar
• Cooke RM, Goossens LLHJ (2008) TU Delft expert judgment data base. Reliability Engrg. System Safety 93(5):657–674.Crossref, Google Scholar
• Cowgill B, Zitzewitz E (2015) Corporate prediction markets: Evidence from Google, Ford, and Firm X. Rev. Econom. Stud. 82(4):1309–1341.Crossref, Google Scholar
• Dana J, Atanasov P, Tetlock P, Mellers B (2019) Are markets more accurate than polls? The surprising informational value of “just asking.” Judgment Decision Making 14(2):135–147.Crossref, Google Scholar
• Davis-Stober CP, Budescu DV, Dana J, Broomell SB (2014) When is a crowd wise? Decision 1(2):79–101.Crossref, Google Scholar
• Graefe A, Armstrong JS (2011) Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task. Internat. J. Forecasting 27(1):183–195.Crossref, Google Scholar
• Griffin D, Brenner L (2004) Perspectives on probability judgment calibration. Koehler DJ, Harvey N, eds. Blackwell Handbook of Judgment and Decision Making (Blackwell Publishing Ltd, Malden, MA), 177–199.Crossref, Google Scholar
• Griffin D, Tversky A (1992) The weighing of evidence and the determinants of confidence. Cognitive Psych. 24(3):411–435.Crossref, Google Scholar
• Gürçay B, Mellers BA, Baron J (2015) The power of social influence on estimation accuracy. J. Behav. Decision Making 28(3):250–261.Crossref, Google Scholar
• Hanea AM, McBride MF, Burgman MA, Wintle BC (2018) The value of performance weights and discussion in aggregated expert judgments. Risk Anal. 38(9):1781–1794.Crossref, Google Scholar
• Hanea AM, McBride MF, Burgman MA, Wintle BC, Fidler F, Flander L, Twardy CR, Manning B, Mascaro S (2017) I nvestigate D iscuss E stimate A ggregate for structured expert judgement. Internat. J. Forecasting 33(1):267–279.Crossref, Google Scholar
• Hastie R, Kameda T (2005) The robust beauty of majority rules in group decisions. Psych. Rev. 112(2):494–508.Crossref, Google Scholar
• Healy PJ, Linardi S, Lowery JR, Ledyard JO (2010) Prediction markets: Alternative mechanisms for complex environments with few traders. Management Sci. 56(11):1977–1996.Link, Google Scholar
• Huang S, Broomell SB, Golman R (2024) A hypothesis test algorithm for determining when weighting individual judgments reliably improves collective accuracy or just adds noise. Decision 11(1):7–34.Crossref, Google Scholar
• Jose VRR, Grushka-Cockayne Y, Lichtendahl KC (2014) Trimmed opinion pools and the crowd’s calibration problem. Management Sci. 60(2):463–475.Link, Google Scholar
• Karelitz TM, Budescu DV (2013) The effect of the raters’ marginal distributions on their matched agreement: A rescaling framework for interpreting Kappa. Multivariate Behav. Res. 48(6):923–952.Crossref, Google Scholar
• Keck S, Tang W (2021) Elaborating or aggregating? The joint effects of group decision-making structure and systematic errors on the value of group interactions. Management Sci. 67(7):4287–4309.Link, Google Scholar
• Koo TK, Li MY (2016) Guideline of selecting and reporting intraclass correlation coefficients for reliability research. J. Chiropractic Medicine 15(2):155–163.Crossref, Google Scholar
• Koriat A, Lichtenstein S, Fischhoff B (1980) Reasons for confidence. J. Experiment. Psych. Human Learn. Memory 6(2):107–118.Crossref, Google Scholar
• Kruger J (1999) Lake Wobegon be gone! The “below-average effect” and the egocentric nature of comparative ability judgments. J. Personality Soc. Psych. 77(2):221–232.Crossref, Google Scholar
• Larrick RP, Soll JB (2006) Intuitions about combining opinions: Misappreciation of the averaging principle. Management Sci. 52(1):111–127.Link, Google Scholar
• Larrick RP, Burson KA, Soll JB (2007) Social comparison and confidence: When thinking you’re better than average predicts overconfidence (and when it does not). Organ. Behav. Human Decision Processes 102(1):76–94.Crossref, Google Scholar
• Larson JR Jr, Christensen C (1993) Groups as problem-solving units: Toward a new meaning of social cognition. British J. Soc. Psych. 32(1):5–30.Crossref, Google Scholar
• Lichtenstein S, Fischhoff B, Phillips LD (1982) Calibration of probabilities: The state of the art to 1980. Kahneman D, Slovic P, Tversky A, eds. Judgment Under Uncertainty (Cambridge University Press, Cambridge, UK), 306–334.Crossref, Google Scholar
• Mannes AE, Soll JB, Larrick RP (2014) The wisdom of select crowds. J. Personality Soc. Psych. 107(2):276–299.Crossref, Google Scholar
• Mauksch S, von der Gracht HA, Gordon TJ (2020) Who is an expert for foresight? A review of identification methods. Tech. Forecasting Soc. Change 154(May):119982.Crossref, Google Scholar
• Mellers B, Stone E, Murray T, Minster A, Rohrbaugh N, Bishop MM, Chen E, et al. (2015) Identifying and cultivating superforecasters as a method of improving probabilistic predictions. Perspect. Psych. Sci. 10(3):267–281.Crossref, Google Scholar
• Mellers B, Ungar L, Baron J, Ramos J, Gurcay B, Fincher K, Scott SE, et al. (2014) Psychological strategies for winning a geopolitical forecasting tournament. Psych. Sci. 25(5):1106–1115.Crossref, Google Scholar
• Merkle EC, Saw G, Davis-Stober C (2020) Beating the average forecast: Regularization based on forecaster attributes. J. Math. Psych. 98(September):102419.Crossref, Google Scholar
• Minson JA, Mueller JS, Larrick RP (2018) The contingent wisdom of dyads: When discussion enhances vs. undermines the accuracy of collaborative judgments. Management Sci. 64(9):4177–4192.Link, Google Scholar
• Moore DA, Swift SA, Minster A, Mellers B, Ungar L, Tetlock P, Yang HHJ, Tenney ER (2017) Confidence calibration in a multiyear geopolitical forecasting competition. Management Sci. 63(11):3552–3565.Link, Google Scholar
• Navajas J, Niella T, Garbulsky G, Bahrami B, Sigman M (2018) Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds. Nature Human Behav. 2(2):126–132.Crossref, Google Scholar
• O’Hagan A (2019) Expert knowledge elicitation: Subjective but scientific. Amer. Statist. 73(sup1):69–81.Crossref, Google Scholar
• Palan S, Huber J, Senninger L (2020) Aggregation mechanisms for crowd predictions. Experiment. Econom. 23(3):788–814.Crossref, Google Scholar
• Palley AB, Satopää VA (2023) Boosting the wisdom of crowds within a single judgment problem: Weighted averaging based on peer predictions. Management Sci. 69(9):5128–5146.Link, Google Scholar
• Satopää VA, Salikhov M, Tetlock PE, Mellers B (2021) Bias, information, noise: The BIN model of forecasting. Management Sci. 67(12):7599–7618.Link, Google Scholar
• Schultze T, Mojzisch A, Schulz-Hardt S (2012) Why groups perform better than individuals at quantitative judgment tasks: Group-to-individual transfer as an alternative to differential weighting. Organ. Behav. Human Decision Processes 118(1):24–36.Crossref, Google Scholar
• Sharp GL, Cutler BL, Penrod SD (1988) Performance feedback improves the resolution of confidence judgments. Organ. Behav. Human Decision Processes 42(3):271–283.Crossref, Google Scholar
• Sniezek JA, Henry RA (1989) Accuracy and confidence in group judgment. Organ. Behav. Human Decision Processes 43(1):1–28.Crossref, Google Scholar
• Stern A, Schultze T, Schulz-Hardt S (2017) How much group is necessary? Group-to-individual transfer in estimation tasks. Collabra Psych. 3(1):16.Crossref, Google Scholar
• Trouche E, Sander E, Mercier H (2014) Arguments, more than confidence, explain the good performance of reasoning groups. J. Experiment. Psych. General 143(5):1958–1971.Crossref, Google Scholar
• Urban K, Urban M (2018) Influence of fluid intelligence on accuracy of metacognitive monitoring in preschool children fades with the calibration feedback. Stud. Psych. 60(2):123–136.Google Scholar
• Wilkening T, Martinie M, Howe PDL (2022) Hidden experts in the crowd: Using meta-predictions to leverage expertise in single-question prediction problems. Management Sci. 68(1):487–508.Link, Google Scholar
• Winkler RL (1981) Combining probability distributions from dependent information sources. Management Sci. 27(4):479–488.Link, Google Scholar
• Wolfers J, Zitzewitz E (2004) Prediction markets. J. Econom. Perspect. 18(2):107–126.Crossref, Google Scholar