Hidden Experts in the Crowd: Using Meta-Predictions to Leverage Expertise in Single-Question Prediction Problems
Published Online:10 Mar 2021https://doi.org/10.1287/mnsc.2020.3919
References
- (2020) Follow the money, not the majority: Incentivizing and aggregating expert opinions with Bayesian markets. Working paper, Erasmus School of Economics, Rotterdam, Netherlands. Google Scholar
- (1951) Comparison of experiments. Neyman J, ed. Proc. Second Berkeley Sympos. Math. Statist. Probab. (The Regents of the University of California, Berkeley) 93–102.Google Scholar
- (1953) Equivalent comparisons of experiments. Ann. Math. Statist. 24(2):265–272.Crossref, Google Scholar
- (1979) Theory of Games and Statistical Decisions (Courier Corporation, Toronto).Google Scholar
- (2015) Identifying expertise to extract the wisdom of crowds. Management Sci. 61(2):267–280.Link, Google Scholar
- (1989) Combining forecasts: A review and annotated bibliography. Internat. J. Forecasting 5(4):559–583.Crossref, Google Scholar
- (1991) Experts in Uncertainty: Opinion and Subjective Probability in Science (Oxford University Press, New York).Crossref, Google Scholar
- (1785) Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix (De l’Imprimerie royale, Paris). Google Scholar
- (2015) Using prediction markets to estimate the reproducibility of scientific research. Proc. Natl. Acad. Sci. USA. 112(50):15343–15347.Crossref, Google Scholar
- (1994) Simultaneous over- and underconfidence: The role of error in judgment processes. Psych. Rev. 101(3):519–527.Crossref, Google Scholar
- (1982) Subjective confidence in forecasts. J. Forecasting 1(2):155–172.Crossref, Google Scholar
- (1907) Vox populi (the wisdom of crowds). Nature 75(7):450–451.Crossref, Google Scholar
- (2013) Combining expert forecasts: Can anything beat the simple average? Internat. J. Forecasting 29(1):108–121.Crossref, Google Scholar
- (1984) External validity of laboratory experiments: The frequency-validity relationship. Amer. J. Psych. 97(2):185–195.Crossref, Google Scholar
- (1991) Probabilistic mental models: A Brunswikian theory of confidence. Psych. Rev. 98(4):506–528.Crossref, Google Scholar
- (2018) Two information aggregation mechanisms for predicting the opening weekend box office revenues of films: Box office prophecy and guess of guesses. Econom. Theory 65(1):25–54.Crossref, Google Scholar
- (2004) Perspectives on probability judgment calibration. Koehler DJ, ed. Blackwell Handbook of Judgment and Decision (John Wiley & Sons, Malden, MA), 177–199.Crossref, Google Scholar
- (2012) Tapping into the wisdom of the crowd—With confidence. Sci. 336(6079):303–304.Crossref, Google Scholar
- (1982) Frequency as a nonpropositional attribute of memory. J. Verbal Learn. Verbal Behav. 21(2):128–141.Crossref, Google Scholar
- (2008) Subjective confidence in one’s answers: The consensuality principle. J. Experiment. Psych. Learn. Memory Cognition 34(4):945–959.Crossref, Google Scholar
- (2012) When are two heads better than one and why? Sci. 336(6079):360–362.Crossref, Google Scholar
- (2019) How to detect high-performing individuals and groups: Decision similarity predicts accuracy. Sci. Adv. 5(11):eaaw9011.Crossref, Google Scholar
- (2017) The relationship between crowd majority and accuracy for binary decisions. Judgment Decision Making 12(4):328.Crossref, Google Scholar
- (1993) On the evaluation of probability judgments: Calibration, resolution, and monotonicity. Psych. Bull. 114(1):162–173.Crossref, Google Scholar
- (1968) Economic comparability of information systems. Internat. Econom. Rev. 9(2):137–174.Crossref, Google Scholar
- (1972) Economic Theory of Teams (Cowles Foundation Monograph 22) (Yale University Press, New Haven, CT).Google Scholar
- (2020) Using meta-predictions to identify experts in the crowd when past performance is unknown. PLoS One 15(4):1–11.Crossref, Google Scholar
- (2017) A statistical model for aggregating judgments by incorporating peer predictions. Preprint, submitted March 14, https://arxiv.org/abs/1703.04778.Google Scholar
- (2015) Identifying and cultivating superforecasters as a method of improving probabilistic predictions. Perspect. Psych. Sci. 10(3):267–281.Crossref, Google Scholar
- (2017) The wisdom of crowds in matters of taste. Management Sci. 64(4):1779–1803.Link, Google Scholar
- (2020) Boosting the wisdom of crowds within a single judgment problem: Selective averaging based on peer predictions. Preprint, submitted January 1, https://dx.doi.org/10.2139/ssrn.3504286.Google Scholar
- (2018) Extracting the wisdom of crowds when information is shared. Management Sci. 65(5):2291–2309.Google Scholar
- (2017) A solution to the single-question crowd wisdom problem. Nature 541(7638):532.Crossref, Google Scholar
- (2016) Modeling probability forecasts via information diversity. J. Amer. Statist. Assoc. 111(516):1623–1633.Crossref, Google Scholar
- (2011) Intuitive biases in choice vs. estimation: Implications for the wisdom of crowds. J. Consumer Res. 38(1):1–15.Crossref, Google Scholar
- (2005) The Wisdom of Crowds (Anchor, New York).Google Scholar
- (2019) Improving information aggregation through meta-cognitive judgments. Working paper, Erasmus School of Economics, Rotterdam, Netherlands.Google Scholar
- (2017) Expert Political Judgment: How Good Is It? How Can We Know? (Princeton University Press, Princeton, NJ).Crossref, Google Scholar
