A Bayesian Hierarchical Model of Crowd Wisdom Based on Predicting Opinions of Others

References

• Atanasov P, Rescober P, Stone E, Swift SA, Servan-Schreiber E, Tetlock P, Ungar L, Mellers B (2017) Distilling the wisdom of crowds: Prediction markets vs. prediction polls. Management Sci. 63(3):691–706.Link, Google Scholar
• Baron, J, Mellers BA, Tetlock PE, Stone E, Ungar LH (2014) Two reasons to make aggregated probability forecasts more extreme. Decision Anal. 11(2):133–145.Link, Google Scholar
• Batchelder WH, Romney AK (1988) Test theory without an answer key. Psychometrika 53(1):71–92.Crossref, Google Scholar
• Budescu DV, Chen E (2015) Identifying expertise to extract the wisdom of crowds. Management Sci. 61(2):267–280.Link, Google Scholar
• Budescu DV, Johnson TR (2011) A model-based approach for the analysis of the calibration of probability judgments. Judgment Decision Making 6(8):857–869.Crossref, Google Scholar
• Burgman MA, McBride M, Ashton R, Speirs-Bridge A, Flander L, Wintle B, Fidler F, Rumpff L, Twardy C (2011) Expert status and performance. PLoS One 6(7):e22998.Crossref, Google Scholar
• Carpenter B, Gelman A, Hoffman MD, Lee D, Goodrich B, Betancourt M, Brubaker M, Guo J, Li P, Riddell A (2017) Stan: A probabilistic programming language. J. Statist. Software 76(1):1–32.Crossref, Google Scholar
• Chambers JR (2008) Explaining false uniqueness: Why we are both better and worse than others. Soc. Personality Psych. Compass 2(2):878–894.Crossref, Google Scholar
• Chen KY, Fine LR, Huberman BA (2004) Eliminating public knowledge biases in information-aggregation mechanisms. Management Sci. 50(7):983–994.Link, Google Scholar
• Clemen RT, Winkler RL (2007) Aggregating probability distributions. Edwards W, Miles RF Jr, von Winterfeldt D, eds. Advances in Decision Analysis: From Foundations to Applications (Cambridge University Press, Cambridge, UK), 154–176.Crossref, Google Scholar
• Cooke RM (1991) Experts in Uncertainty: Opinion and Subjective Probability in Science (Oxford University Press, New York).Crossref, Google Scholar
• Cowgill B, Zitzewitz E (2015) Corporate prediction markets: Evidence from google, ford, and firm x. Rev. Econom. Stud. 82(4):1309–1341.Crossref, Google Scholar
• Dawes RM (1989) Statistical criteria for establishing a truly false consensus effect. J. Experiment. Soc. Psych. 25(1):1–17.Crossref, Google Scholar
• Diermann C, Huchzermeier A (2017) Canyon bicycles: Judgmental demand forecasting in direct sales. INFORMS Trans. Ed. 17(2):63–74.Link, Google Scholar
• Evgeniou T, Fang L, Hogarth RM, Karelaia N (2013) Competitive dynamics in forecasting: The interaction of skill and uncertainty. J. Behav. Decision Making 26(4):375–384.Crossref, Google Scholar
• Gillen BJ, Plott CR, Shum M (2017) A pari-mutuel-like mechanism for information aggregation: A field test inside intel. J. Political Econom. 125(4):1075–1099.Crossref, Google Scholar
• Griffiths TL, Tenenbaum JB (2006) Optimal predictions in everyday cognition. Psych. Sci. 17(9):767–773.Crossref, Google Scholar
• Hertwig R (2012) Tapping into the wisdom of the crowd-with confidence. Science 336(6079):303–304.Crossref, Google Scholar
• Jasra A, Holmes CC, Stephens DA (2005) Markov chain Monte Carlo methods and the label switching problem in Bayesian mixture modeling. Statist. Sci. 20(1):50–67.Crossref, Google Scholar
• Karabatsos G, Batchelder WH (2003) Markov chain estimation for test theory without an answer key. Psychometrika 68(3):373–389.Crossref, Google Scholar
• Keuschnigg M, Ganser C (2017) Crowd wisdom relies on agents’ ability in small groups with a voting aggregation rule. Management Sci. 63(3):818–828.Link, Google Scholar
• Kiani R, Corthell L, Shadlen MN (2014) Choice certainty is informed by both evidence and decision time. Neuron 84(6):1329–1342.Crossref, Google Scholar
• Kruger J, Dunning D (1999) Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J. Personality Soc. Psych. 77(6):1121–1134.Crossref, Google Scholar
• Kurvers RHJM, Herzog SM, Hertwig R, Krause J, Moussaid M, Argenziano G, Zalaudek I, Carney PA, Wolf M (2019) How to detect high-performing individuals and groups: Decision similarity predicts accuracy. Sci. Adv. 5(11):eaaw9011.Crossref, Google Scholar
• Lee MD, Danileiko I (2014) Using cognitive models to combine probability estimates. Judgment Decision Making 9(3):259–273.Crossref, Google Scholar
• Makridakis S, Winkler RL (1983) Averages of forecasts: Some empirical results. Management Sci. 29(9):987–996.Link, Google Scholar
• Marr D (1982) Vision: A Computational Investigation into the Human Representation and Processing of Visual Information (W. H. Freeman and Company, New York).Google Scholar
• Merkle EC, Steyvers M, Mellers B, Tetlock PE (2016) Item response models of probability judgments: Application to a geopolitical forecasting tournament. Decision 3(1):1–19.Crossref, Google Scholar
• Moran R, Teodorescu AR, Usher M (2015) Post choice information integration as a causal determinant of confidence: Novel data and a computational account. Cognitive Psych. 78:99–147.Crossref, Google Scholar
• Oravecz Z, Anders R, Batchelder WH (2013) Hierarchical Bayesian modeling for test theory without an answer key. Psychometrika 80(2):341–364.Crossref, Google Scholar
• Oravecz Z, Vandekerckhove J, Batchelder WH (2014) Bayesian cultural consensus theory. Field Methods 26(3):207–222.Crossref, Google Scholar
• Palley AB, Satopää VA (2023) Boosting the wisdom of crowds within a single judgment problem: Weighted averaging based on peer predictions. Management Sci. 69(9):5128–5146.Google Scholar
• Palley AB, Soll JB (2019) Extracting the wisdom of crowds when information is shared. Management Sci. 65(5):2291–2309.Abstract, Google Scholar
• Pleskac TJ, Busemeyer JR (2010) Two-stage dynamic signal detection: A theory of choice, decision time, and confidence. Psych. Rev. 117(3):864–901.Crossref, Google Scholar
• Prelec D, Seung HS, McCoy J (2017) A solution to the single-question crowd wisdom problem. Nature 541(7638):532–535.Crossref, Google Scholar
• Raykar VC, Yu S, Zhao LH, Valadez GH, Florin C, Bogoni L, Moy L (2010) Learning from crowds. J. Machine Learn. Res. 11(April):1297–1322.Google Scholar
• Romney AK, Weller SC, Batchelder WH (1986) Culture as consensus: A theory of culture and informant accuracy. Amer. Anthropologist 88(2):313–338.Crossref, Google Scholar
• Ross L, Greene D, House P (1977) The “false consensus effect”: An egocentric bias in social perception and attribution processes. J. Experiment. Soc. Psych. 13(3):279–301.Crossref, Google Scholar
• Satopää VA, Baron J, Foster DP, Mellers BA, Tetlock PE, Ungar LH (2014) Combining multiple probability predictions using a simple logit model. Internat. J. Forecasting 30(2):344–356.Crossref, Google Scholar
• Simmons JP, Nelson LD, Galak J, Frederick S (2011) Intuitive biases in choice vs. estimation: Implications for the wisdom of crowds. J. Consumer Res. 38(1):1–15.Crossref, Google Scholar
• Sunstein CR (2006) Infotopia: How Many Minds Produce Knowledge (Oxford University Press, New York).Crossref, Google Scholar
• Surowiecki J (2005) The Wisdom of Crowds (Anchor, Palatine, IL).Google Scholar
• Tetlock PE (2005) Expert Political Judgment: How Good Is It? How Can We Know? (Princeton University Press, Princeton, NJ).Google Scholar
• Turner, BM, Steyvers M, Merkle EC, Budescu DV, Wallsten TS (2014) Forecast aggregation via recalibration. Machine Learn. 95(3):261–289.Crossref, Google Scholar
• Wilkening T, Martinie M, Howe PDL (2022) Hidden experts in the crowd: Using meta-predictions to leverage expertise in single-question prediction problems. Management Sci. 68(1):487–508.Link, Google Scholar
• Yu S, Pleskac TJ, Zeigenfuse MD (2015) Dynamics of postdecisional processing of confidence. J. Experiment. Psych. General 144(2):489–510.Crossref, Google Scholar
• Zhang H, Maloney LT (2012) Ubiquitous log odds: A common representation of probability and frequency distortion in perception, action, and cognition. Frontiers Neuroscience 6:1.Crossref, Google Scholar