The Wisdom of Model Crowds

References

• Akaike H (1973) Information theory and an extension of the maximum likelihood principle. Petrov BN, Csaki F, eds. 2nd Internat. Sympos. Inform. Theory (Akadémia Kiadó, Budapest), 267–281.Google Scholar
• Allais M (1953) Le comportement de l’homme rationnel devant le risque: Critique des postulats et axiomes de l’école américaine. Econometrica 21(4):503–546.Crossref, Google Scholar
• Analytis PP, Barkoczi D, Herzog SM (2018) Social learning strategies for matters of taste. Nature Human Behav. 2(6):415–424.Crossref, Google Scholar
• Armstrong JS (2001) Combining forecasts. Armstrong JS, ed. Principles of Forecasting (Springer, Boston), 417–439.Crossref, Google Scholar
• Ashton AH, Ashton RH (1985) Aggregating subjective forecasts: Some empirical results. Management Sci. 31(12):1499–1508.Link, Google Scholar
• Bahrami B, Olsen K, Latham PE, Roepstorff A, Rees G, Frith CD (2010) Optimally interacting minds. Sci. 329(5995):1081–1085.Crossref, Google Scholar
• Becker GM, McClintock CG (1967) Value: Behavioral decision theory. Annual Rev. Psych. 18(1):239–286.Crossref, Google Scholar
• Bell DE (1982) Regret in decision making under uncertainty. Oper. Res. 30(5):961–981.Link, Google Scholar
• Bell DE (1985) Disappointment in decision making under uncertainty. Oper. Res. 33(1):1–27.Link, Google Scholar
• Bell RM, Koren Y (2007) Lessons from the Netflix Prize Challenge. SIGKDD Explorations Newslett. 9(2):75–79.Crossref, Google Scholar
• Bernoulli D (1738) Specimen theoriae novae de mensura sortis. Commentarii Academiae Scientiarum Imperialis Petropolitanae 5:175–192. [Translated by Sommer L (1954) Exposition of a new theory on the measurement of risk. Econometrica 22(1):23–36.]Google Scholar
• Bhatia S, Loomes G (2017) Noisy preferences in risky choice: A cautionary note. Psych. Rev. 124(5):678–687.Crossref, Google Scholar
• Birnbaum MH (1997) Violations of monotonicity in judgment and decision making. Marley AAJ, ed. Choice, Decision, and Measurement: Essays in Honor of R. Duncan Luce (Lawrence Erlbaum Associates Publishers, Mahwah, NJ), 73–100.Google Scholar
• Birnbaum MH (2008) New paradoxes of risky decision making. Psych. Rev. 115(2):463–501.Crossref, Google Scholar
• Bordalo P, Gennaioli N, Shleifer A (2012) Salience theory of choice under risk. Quart. J. Econom. 127(3):1243–1285.Crossref, Google Scholar
• Box GE (1979) Robustness in the strategy of scientific model building. Launer RL, Wilkinson GN, eds. Robustness in Statistics (Academic Press, New York), 201–236.Crossref, Google Scholar
• Breiman L (1996) Bagging predictors. Machine Learn. 24(2):123–140.Crossref, Google Scholar
• Breiman L (1998) Arcing classifier (with discussion and a rejoinder by the author). Ann. Statist. 26(3):801–849.Crossref, Google Scholar
• Budescu DV, Chen E (2014) Identifying expertise to extract the wisdom of crowds. Management Sci. 61(2):267–280.Link, Google Scholar
• Busemeyer JR, Townsend JT (1993) Decision field theory: A dynamic-cognitive approach to decision making in an uncertain environment. Psych. Rev. 100(3):432–459.Crossref, Google Scholar
• Camerer C, Weber M (1992) Recent developments in modeling preferences: Uncertainty and ambiguity. J. Risk Uncertainty 5(4):325–370.Crossref, Google Scholar
• Cavagnaro DR, Gonzalez R, Myung JI, Pitt MA (2013) Optimal decision stimuli for risky choice experiments: An adaptive approach. Management Sci. 59(2):358–375.Link, Google Scholar
• Cavagnaro DR, Aranovich GJ, McClure SM, Pitt MA, Myung JI (2016) On the functional form of temporal discounting: An optimized adaptive test. J. Risk Uncertainty 52(3):233–254.Crossref, Google Scholar
• Clemen RT (1989) Combining forecasts: A review and annotated bibliography. Internat. J. Forecasting 5(4):559–583.Crossref, Google Scholar
• Davis-Stober CP, Budescu DV, Dana J, Broomell SB (2014) When is a crowd wise? Decision 1(2):79–101.Crossref, Google Scholar
• Dawes RM, Faust D, Meehl PE (1989) Clinical vs. actuarial judgment. Sci. 243(4899):1668–1674.Crossref, Google Scholar
• Diecidue E, van de Ven J (2008) Aspiration level, probability of success and failure, and expected utility. Internat. Econom. Rev. 49(2):683–700.Crossref, Google Scholar
• Dyer JS, Jia J (1997) Relative risk–value models. Eur. J. Oper. Res. 103(1):170–185.Crossref, Google Scholar
• Edwards W (1954) The theory of decision making. Psych. Bull. 51(4):380–417.Crossref, Google Scholar
• Edwards W (1955) The prediction of decisions among bets. J. Experiment. Psych. 50(3):201–204.Crossref, Google Scholar
• Edwards W (1961) Behavioral decision theory. Annual Rev. Psych. 12(1):473–498.Crossref, Google Scholar
• Einhorn HJ, Hogarth RM (1981) Behavioral decision theory: Processes of judgement and choice. Annual Rev. Psych. 32(1):53–88.Crossref, Google Scholar
• Einhorn HJ, Hogarth RM, Klempner E (1977) Quality of group judgment. Psych. Bull. 84(1):158–172.Crossref, Google Scholar
• Ellsberg D (1961) Risk, ambiguity, and the Savage axioms. Quart. J. Econom. 75(4):643–669.Crossref, Google Scholar
• Erev I, Ert E, Plonsky O, Cohen D, Cohen O (2017) From anomalies to forecasts: Toward a descriptive model of decisions under risk, under ambiguity, and from experience. Psych. Rev. 124(4):369–409.Crossref, Google Scholar
• Erev I, Ert E, Roth AE, Haruvy E, Herzog SM, Hau R, Hertwig R, Stewart T, West R, Lebiere C (2010) A choice prediction competition: Choices from experience and from description. J. Behav. Decision Making 23(1):15–47.Crossref, Google Scholar
• Erickson MA, Kruschke JK (1998) Rules and exemplars in category learning. J. Experiment. Psych. General 127(2):107–140.Crossref, Google Scholar
• Fiedler S, Glöckner A (2012) The dynamics of decision making in risky choice: An eye-tracking analysis. Frontiers Psych. 3:335.Google Scholar
• Fishburn PC (1977) Mean-risk analysis with risk associated with below-target returns. Amer. Econom. Rev. 67(2):116–126.Google Scholar
• Frederick S, Loewenstein G, O’donoghue T (2002) Time discounting and time preference: A critical review. J. Econom. Literature 40(2):351–401.Crossref, Google Scholar
• Galesic M, Barkoczi D, Katsikopoulos K (2018) Smaller crowds outperform larger crowds and individuals in realistic task conditions. Decision 5(1):1–15.Crossref, Google Scholar
• Galton F (1907) Vox populi (the wisdom of crowds). Nature 75(1949):450–451.Crossref, Google Scholar
• Geman S, Bienenstock E, Doursat R (1992) Neural networks and the bias/variance dilemma. Neural Comput. 4(1):1–58.Crossref, Google Scholar
• Gigerenzer G, Brighton H (2009) Homo heuristicus: Why biased minds make better inferences. Topics Cognitive Sci. 1(1):107–143.Crossref, Google Scholar
• Gilboa I, Schmeidler D (1995) Case-based decision theory. Quart. J. Econom. 110(3):605–639.Crossref, Google Scholar
• Goldstein DG, McAfee RP, Suri S (2014) The wisdom of smaller, smarter crowds. Proc. 15th ACM Conf. Econom. Comput. (ACM, New York), 471–488).Google Scholar
• Golman R, Bhatia S, Kane P (2020) The dual accumulator model of strategic deliberation and decision making. Psych. Rev. 127(4):477–504.Crossref, Google Scholar
• Hastie R (2001) Problems for judgment and decision making. Annual Rev. Psych. 52(1):653–683.Crossref, Google Scholar
• He L, Zhao WJ, Bhatia S (2020) An ontology of decision models. Psych. Rev., ePub ahead of print July 20, https://doi.org/10.1037/rev0000231.Crossref, Google Scholar
• Hertwig R, Erev I (2009) The description–experience gap in risky choice. Trends Cognitive Sci. 13(12):517–523.Crossref, Google Scholar
• Hertwig R, Barron G, Weber EU, Erev I (2004) Decisions from experience and the effect of rare events in risky choice. Psych. Sci. 15(8):534–539.Crossref, Google Scholar
• Herzog SM, von Helversen B (2018) Strategy selection vs. strategy blending: A predictive perspective on single‐and multi‐strategy accounts in multiple‐cue estimation. J. Behav. Decision Making 31(2):233–249.Crossref, Google Scholar
• Hogarth RM (1978) A note on aggregating opinions. Organ. Behav. Human Performance 21(1):40–46.Crossref, Google Scholar
• Hoeting JA, Madigan D, Raftery AE, Volinsky CT (1999) Bayesian model averaging: A tutorial. Statist. Sci. 14(4):382–401.Crossref, Google Scholar
• Hong L, Page SE (2004) Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proc. Natl. Acad. Sci. USA 101(46):16385–16389.Crossref, Google Scholar
• Kahneman D, Tversky A (1979) Prospect theory: An analysis of decision under risk. Econometrica 47(2):363–391.Crossref, Google Scholar
• Karmarkar US (1978) Subjectively weighted utility: A descriptive extension of the expected utility model. Organ. Behav. Human Performance 21(1):61–72.Crossref, Google Scholar
• Keeney RL, Raiffa H (1993) Decisions with Multiple Objectives: Preferences and Value Trade-offs (Cambridge University Press, Cambridge, UK).Crossref, Google Scholar
• Kőszegi B, Rabin M (2006) A model of reference-dependent preferences. Quart. J. Econom. 121(4):1133–1165.Crossref, Google Scholar
• Lamberson PJ, Page SE (2012) Optimal forecasting groups. Management Sci. 58(4):805–810.Link, Google Scholar
• Lattimore PK, Baker JR, Witte AD (1992) The influence of probability on risky choice: A parametric examination. J. Econom. Behav. Organ. 17(3):377–400.Crossref, Google Scholar
• Leland JW (1994) Generalized similarity judgments: An alternative explanation for choice anomalies. J. Risk Uncertainty 9(2):151–172.Crossref, Google Scholar
• Lieder F, Griffiths TL (2017) Strategy selection as rational metareasoning. Psych. Rev. 124(6):762–794.Crossref, Google Scholar
• Lieder F, Griffiths TL, Hsu M (2018) Overrepresentation of extreme events in decision making reflects rational use of cognitive resources. Psych. Rev. 125(1):1–32.Crossref, Google Scholar
• Loewenstein G, O’Donoghue T, Bhatia S (2015) Modeling the interplay between affect and deliberation. Decision 2(2):55–81.Crossref, Google Scholar
• Loewenstein GF, Weber EU, Hsee CK, Welch N (2001) Risk as feelings. Psych. Bull. 127(2):267–286.Crossref, Google Scholar
• Loomes G, Sugden R (1982) Regret theory: An alternative theory of rational choice under uncertainty. Econom. J. 92(368):805–824.Google Scholar
• Loomes G, Sugden R (1986) Disappointment and dynamic consistency in choice under uncertainty. Rev. Econom. Stud. 53(2):271–282.Crossref, Google Scholar
• Luan S, Katsikopoulos KV, Reimer T (2012) When does diversity trump ability (and vice versa) in group decision making? A simulation study. PLoS One 7(2):e31043.Crossref, Google Scholar
• Machina MJ (1982) “Expected utility” analysis without the independence axiom. Econometrica 50(2):277–323.Crossref, Google Scholar
• Makridakis S, Winkler RL (1983) Averages of forecasts: Some empirical results. Management Sci. 29(9):987–996.Link, Google Scholar
• Mannes AE, Soll JB, Larrick RP (2014) The wisdom of select crowds. J. Personality Soc. Psych. 107(2):276–299.Crossref, Google Scholar
• Markowitz H (1952) Portfolio selection. J. Finance 7(1):77–91.Google Scholar
• Mellers B, Schwartz A, Ritov I (1999) Emotion-based choice. J. Experiment. Psych. General 128(3):332–345.Crossref, Google Scholar
• Müller-Trede J, Choshen-Hillel S, Barneron M, Yaniv I (2017) The wisdom of crowds in matters of taste. Management Sci. 64(4):1779–1803.Link, Google Scholar
• Myung JI, Pitt MA (2009) Optimal experimental design for model discrimination. Psych. Rev. 116(3):499–518.Crossref, Google Scholar
• Navarro DJ, Pitt MA, Myung IJ (2004) Assessing the distinguishability of models and the informativeness of data. Cognitive Psych. 49(1):47–84.Crossref, Google Scholar
• Niculescu-Mizil A, Perlich C, Swirszcz G, Sindhwani V, Liu Y, Melville P, Wang D, et al. (2009) Winning the KDD Cup orange challenge with ensemble selection. Dror G, Boullé M, Guyon I, Lemaire V, Vogel D, eds. Proc. KDD-Cup 2009 Competition, Proceedings of Machine Learning Research, Vol. 7 (PMLR, New York), 23–34.Google Scholar
• Oppenheimer DM, Kelso E (2015) Information processing as a paradigm for decision making. Annual Rev. Psych. 66:277–294.Crossref, Google Scholar
• Pachur T, Mata R, Hertwig R (2017) Who dares, who errs? Disentangling cognitive and motivational roots of age differences in decisions under risk. Psych. Sci. 28(4):504–518.Crossref, Google Scholar
• Pachur T, Schulte-Mecklenbeck M, Murphy RO, Hertwig R (2018) Prospect theory reflects selective allocation of attention. J. Experiment. Psych. General 147(2):147–169.Crossref, Google Scholar
• Payne JW, Bettman JR, Johnson EJ (1988) Adaptive strategy selection in decision making. J. Experiment. Psych. Learn. Memory Cognition 14(3):534–552.Crossref, Google Scholar
• Payne JW, Bettman JR, Johnson EJ (1992) Behavioral decision research: A constructive processing perspective. Annual Rev. Psych. 43(1):87–131.Crossref, Google Scholar
• Pitt MA, Kim W, Myung IJ (2003) Flexibility vs. generalizability in model selection. Psychonomic Bull. Rev. 10(1):29–44.Crossref, Google Scholar
• Pitz GF, Sachs NJ (1984) Judgment and decision: Theory and application. Annual Rev. Psych. 35(1):139–164.Crossref, Google Scholar
• Pleskac TJ, Hertwig R (2014) Ecologically rational choice and the structure of the environment. J. Experiment. Psych. General 143(5):2000–2019.Crossref, Google Scholar
• Plonsky O, Erev I, Hazan T, Tennenholtz M (2017) Psychological forest: Predicting human behavior. Proc. 31st AAAI Conf. Artificial Intelligence 31(1), https://ojs.aaai.org/index.php/AAAI/article/view/10613.Google Scholar
• Polikar R (2006) Ensemble based systems in decision making. IEEE Circuits Systems Magazine 6(3):21–45.Crossref, Google Scholar
• Pollatsek A, Tversky A (1970) A theory of risk. J. Math. Psych. 7(3):540–553.Crossref, Google Scholar
• Prelec D (1998) The probability weighting function. Econometrica 66(3):497–528.Crossref, Google Scholar
• Rapoport A, Wallsten TS (1972) Individual decision behavior. Annual Rev. Psych. 23(1):131–176.Crossref, Google Scholar
• Rieskamp J (2008) The probabilistic nature of preferential choice. J. Experiment. Psych. Learn. Memory Cognition 34(6):1446–1465.Crossref, Google Scholar
• Scheibehenne B, Rieskamp J, Wagenmakers E-J (2013) Testing adaptive toolbox models: A Bayesian hierarchical approach. Psych. Rev. 120(1):39–64.Crossref, Google Scholar
• Simonson I, Carmon Z, Dhar R, Drolet A, Nowlis SM (2001) Consumer research: In search of identity. Annual Rev. Psych. 52(1):249–275.Crossref, Google Scholar
• Singmann H, Kellen D, Mizrak E, Öztekin I (2018) Using ensembles of cognitive models to answer substantive questions. Preprint, submitted January 31, https://psyarxiv.com/rg4sz/.Google Scholar
• Slovic P, Fischhoff B, Lichtenstein S (1977) Behavioral decision theory. Annual Rev. Psych. 28(1):1–39.Crossref, Google Scholar
• Smaldino PE (2017) Models are stupid, and we need more of them. Vallacher RR, Read SJ, Nowak A, eds. Computational Social Psychology (Routledge, New York), 311–331.Crossref, Google Scholar
• Starmer C (2000) Developments in non-expected utility theory: The hunt for a descriptive theory of choice under risk. J. Econom. Literature 38(2):332–382.Crossref, Google Scholar
• Stewart N, Hermens F, Matthews WJ (2016) Eye movements in risky choice. J. Behav. Decision Making 29(2–3):116–136.Crossref, Google Scholar
• Stewart N, Reimers S, Harris AJ (2015) On the origin of utility, weighting, and discounting functions: How they get their shapes and how to change their shapes. Management Sci. 61(3):687–705.Link, Google Scholar
• Stott HP (2006) Cumulative prospect theory’s functional menagerie. J. Risk Uncertainty 32(2):101–130.Crossref, Google Scholar
• Surowiecki J (2004) The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economics, Societies, and Nations (Doubleday, New York).Google Scholar
• Thorngate W (1980) Efficient decision heuristics. Behav. Sci. 25(3):219–225.Crossref, Google Scholar
• Turner BM, Schley DR, Muller C, Tsetsos K (2018) Competing theories of multialternative, multiattribute preferential choice. Psych. Rev. 125(3):329–362.Crossref, Google Scholar
• Tversky A, Kahneman D (1992) Advances in prospect theory: Cumulative representation of uncertainty. J. Risk Uncertainty. 5(4):297–323.Crossref, Google Scholar
• Venkatraman V, Payne JW, Huettel SA (2014) An overall probability of winning heuristic for complex risky decisions: Choice and eye fixation evidence. Organ. Behav. Human Decision Processes 125(2):73–87.Crossref, Google Scholar
• Viscusi WK (1989) Prospective reference theory: Toward an explanation of the paradoxes. J. Risk Uncertainty 2(3):235–263.Crossref, Google Scholar
• von Neumann J, Morgenstern O (1944) Theory of Games and Economic Behavior (Princeton University Press, Princeton, NJ).Google Scholar
• Wagenmakers EJ, Farrell S (2004) AIC model selection using Akaike weights. Psychonomic Bull. Rev. 11(1):192–196.Crossref, Google Scholar
• Wagenmakers EJ, Ratcliff R, Gomez P, Iverson GJ (2004) Assessing model mimicry using the parametric bootstrap. J. Math. Psych. 48(1):28–50.Crossref, Google Scholar
• Weber EU, Johnson EJ (2009) Mindful judgment and decision making. Annual Rev. Psych. 60:53–85.Crossref, Google Scholar
• Weber EU, Shafir S, Blais A-R (2004) Predicting risk sensitivity in humans and lower animals: Risk as variance or coefficient of variation. Psych. Rev. 111(2):430–445.Crossref, Google Scholar