Improving Accuracy by Coherence Weighting of Direct and Ratio Probability Judgments

References

• Abbas AE, Budescu DV, Yu H-T, Haggerty R (2008) A comparison of two probability encoding methods: Fixed probability vs. fixed variable values. Decision Anal. 5(4):190–202.Link, Google Scholar
• Baron J, Scott S, Fincher K, Metz SE (2015) Why does the cognitive reflection test (sometimes) predict utilitarian moral judgment (and other things)? J. Appl. Res. Memory Cognition 4(3):265–284.Crossref, Google Scholar
• Batsell R, Brenner L, Osherson D, Tsavachidis S, Vardi MY (2002) Eliminating incoherence from subjective estimates of chance. Proc. 8th Internat. Conf. Principles Knowledge Representation Reasoning (KR). http://www.princeton.edu/∼osherson/papers/coh.pdf.Google Scholar
• Benjamin D, Mandel DR, Kimmelman J (2017) Can cancer researchers accurately judge whether preclinical reports will reproduce? PLoS Biology 15(6):e2002212.Crossref, Google Scholar
• Bennett ST, Benjamin AS, Mistry PK, Steyvers M (2018) Making a wiser crowd: Benefits of individual metacognitive control on crowd performance. Comput. Brain Behav. 1(1):90–99.Google Scholar
• Bolger F, Wright G (1994) Assessing the quality of expert judgment: Issues and analysis. Decision Support Systems 11(1):1–24.Crossref, Google Scholar
• Budescu DV, Chen E (2015) Identifying expertise to extract the wisdom of crowds. Management Sci. 61(2):267–280.Link, Google Scholar
• Budescu DV, Zwick R, Rapoport A (1986) A comparison of the eigenvalue method and the geometric mean procedure for ratio scaling. Appl. Psych. Measurement 10(1):69–78.Crossref, Google Scholar
• Censor Y, Zenios SA (1997) Parallel Optimization: Theory, Algorithms, and Applications (Oxford University Press, New York).Google Scholar
• Chen E, Budescu DV, Lakshmikanth SK, Mellers BA, Tetlock PE (2016) Validating the contribution-weighted model: Robustness and cost-benefit analyses. Decision Anal. 13(2):128–152.Link, Google Scholar
• Chvátal V (1983) Linear Programming (Freeman, New York).Google Scholar
• Clemen RT (2008) Comment on Cooke’s classical method. Reliability Engrg. System Safety 93(5):760–765.Crossref, Google Scholar
• Clemen RT, Winkler RL (1999) Combining probability distributions from experts in risk analysis. Risk Anal. 19(2):187–203.Crossref, Google Scholar
• Cooke RM (1991) Experts in Uncertainty: Opinion and Subjective Probability in Science (Oxford University Press, New York).Crossref, Google Scholar
• Cooke RM, Goossens LL (2008) TU Delft expert judgment database. Reliability Engrg. System Safety 93(5):657–674.Crossref, Google Scholar
• Crawford GB (1987) The geometric mean procedure for estimating the scale of a judgment matrix. Math. Model. 9(3-5):327–334.Crossref, Google Scholar
• Crawford G, Williams C (1985) A note on the analysis of subjective judgment matrices. J. Math. Psych. 29:387–405.Crossref, Google Scholar
• Davis-Stober C, Dana J, Budescu DV (2010) A constrained linear estimator for multiple regression. Psychometrika 75:521–541.Crossref, Google Scholar
• Dawes RM (1979) The robust beauty of improper linear models in decision making. Amer. Psych. 34(7):571–582.Crossref, Google Scholar
• de Finetti B (1974) Theory of Probability, vol. 1. (John Wiley & Sons, New York).Google Scholar
• Fan Y (2018) Estimating subjective probabilities of bounded continuous distributions using the ratio judgment and scaling (RJS) method. Unpublished dissertation, Fordham University, Bronx, NY.Google Scholar
• Galton F (1907) Vox populi. Nature 75:450–451. http://galton.org/essays/1900-1911/galton-1907-vox-populi.pdf.Google Scholar
• Goldstein DG, Rothschild D (2014) Lay understanding of observed distributions. Judgment Decision Making 9(1):1–14.Google Scholar
• Gulliksen H (1959) Mathematical solutions for psychological problems. Amer. Sci. 47(2):178–201.Google Scholar
• Haran U, Moore DA, Morewedge CK (2010) A simple remedy for overprecision in judgment. Judgment Decision Making 5(7):467–476.Crossref, Google Scholar
• Hora SC, Fransen BR, Hawkins N, Susel I (2013) Median aggregation of distribution functions. Decision Anal. 10(4):279–291.Link, Google Scholar
• Jose VR, Grushka-Cockayne Y, Lichtendahl KC Jr (2013) Trimmed opinion pools and the crowd’s calibration problem. Management Sci. 60(2):463–475.Link, Google Scholar
• Kabera MG, Haines LM (2013) A note on the statistical analysis of point judgment matrices. ORiON 29(1):75–86.Crossref, Google Scholar
• Karvetski CW, Olson KC, Mandel DR, Twardy CR (2013) Probabilistic coherence weighting for optimizing expert forecasts. Decision Anal. 10(4):305–326.Link, Google Scholar
• Lau H, Lau AH, Ho C (1998) Improved moment-estimation formulas using more than three subjective fractiles. Management Sci. 44(3):346–351.Link, Google Scholar
• Luenberger DG (1984) Linear and Nonlinear Programming (Addison-Wesley, Reading, MA).Google Scholar
• Mandel DR (2005) Are risk assessments of a terrorist attack coherent? J. Experiment. Psych. Appl. 11(4):277–288.Crossref, Google Scholar
• Mandel DR (2008) Violations of coherence in subjective probability: A representational and assessment processes account. Cognition 106(1):130–156.Crossref, Google Scholar
• Mandel DR, Karvetski CW, Dhami MK (2018) Boosting intelligence analysts’ judgment accuracy: What works, what fails? Judgment Decision Making 13(6):607–621.Google Scholar
• Mellers BA, Ungar L, Baron J, Ramos J, Gurcay B, Fincher K, Scott S, et al. (2014) Psychological strategies for winning a geopolitical forecasting tournament. Psych. Sci. 25(5):1106–1115.Crossref, Google Scholar
• Mellers BA, Baker JD, Chen E, Mandel DR, Tetlock PE (2017) How generalizable is good judgment? A multi-task, multi-benchmark study. Judgment Decision Making 12(4):369–381.Google Scholar
• Merrick JR (2008) Getting the right mix of experts. Decision Anal. 5(1):43–52.Link, Google Scholar
• Moore DA, Healy PJ (2008) The trouble with overconfidence. Psych. Rev. 115(2):502–517.Crossref, Google Scholar
• O’Hagan A, Buck CE, Daneshkhah A, Eiser JR, Garthwaite PH, Jenkinson DJ, Oakley JE, Rakow T (2006) Uncertain Judgments: Eliciting Experts’ Probabilities (John Wiley & Sons, Hoboken, NJ).Crossref, Google Scholar
• Osherson D, Vardi MY (2006) Aggregating disparate estimates of chance. Games Econom. Behav. 56(1):148–173.Crossref, Google Scholar
• Park S, Budescu DV (2015) Aggregating multiple probability intervals to improve calibration. Judgment Decision Making 10(2):130–143.Google Scholar
• Por H, Budescu DV (2017) Eliciting subjective probabilities through pair-wise comparisons. J. Behav. Decision Making 30(2):181–196.Crossref, Google Scholar
• Predd JB, Osherson DN, Kulkarni SR, Poor HV (2008) Aggregating probabilistic forecasts from incoherent and abstaining experts. Decision Anal. 5(4):177–189.Link, Google Scholar
• Saaty TL (1977) A scaling method for priorities in hierarchical structures. J. Math. Psych. 15(3):234–281.Crossref, Google Scholar
• Sharpe WF, Goldstein DG, Blythe PW (2000) The distribution builder: A tool for inferring investor preferences. Accessed March 17, 2017, https://web.stanford.edu/∼wfsharpe/art/qpaper/qpaper.pdf.Google Scholar
• Stael von Holstein CA (1970) Measurement of subjective probability. Acta Psych 34(2-3):146–159.Crossref, Google Scholar
• Surowiecki J (2004) The Wisdom of Crowds (Doubleday, New York).Google Scholar
• Tsai J, Kirlik A (2012) Coherence and correspondence competence: Implications for elicitation and aggregation of probabilistic forecasts of world events. Proc. Human Factors Ergonomics Soc. 56th Annual Meeting 56(1):313–317.Crossref, Google Scholar
• Tversky A, Koehler DJ (1994) Support theory: A nonextensional representation of subjective probability. Psych. Rev. 101(4):547–567.Crossref, Google Scholar
• Van Lenthe J (1993) ELI: An interactive elicitation technique for subjective probability distributions. Organ. Behav. Human Decision Processes 55(3):379–413.Crossref, Google Scholar
• Wallsten TS, Budescu DV (1983) Encoding subjective probabilities: A psychological and psychometric review. Management Sci. 29(2):151–173.Link, Google Scholar
• Wang G, Kulkarni SR, Poor HV, Osherson DN (2011) Aggregating large sets of probabilistic forecasts by weighted coherent adjustment. Decision Anal. 8(2):128–144.Link, Google Scholar
• Williams JJ, Mandel DR (2007) Do evaluation frames improve the quality of conditional probability judgment? McNamara DS, Trafton JG, eds. Proc. 29th Annual Meeting Cognitive Sci. Soc. (Erlbaum, Mahwah, NJ), 1653–1658.Google Scholar
• Winkler RL (1967) The assessment of prior distributions in Bayesian analysis. J. Amer. Statistical Assoc. 62(319):776–800.Crossref, Google Scholar
• Yechiam E, Budescu DV (2006) The sensitivity of probability assessments to time units and performer characteristics. Decision Anal. 3(3):177–193.Link, Google Scholar