Identifying Expertise to Extract the Wisdom of Crowds

David V. Budescu
David V. Budescu
[email protected]
Department of Psychology, Fordham University, Bronx, New York 10458
Search for more papers by this author
,
Eva Chen
Eva Chen
[email protected]
University of Pennsylvania, Philadelphia, Pennsylvania 19104
Search for more papers by this author

David V. Budescu

[email protected]

Department of Psychology, Fordham University, Bronx, New York 10458

Search for more papers by this author

Eva Chen

[email protected]

University of Pennsylvania, Philadelphia, Pennsylvania 19104

Search for more papers by this author

Published Online:23 May 2014https://doi.org/10.1287/mnsc.2014.1909

Supplemental Material

mnsc.2014.1909-sm-data.xlsx (645 KB)

Cited by
- Personality traits, entrepreneurship education, perceived financial access, entrepreneurial intentions and entrepreneurial behavior: a Vietnamese youth perspective
  26 February 2026 | Journal of Global Entrepreneurship Research, Vol. 16, No. 1
- Majority views and confidence information promote informed decisions
  18 February 2026 | Humanities and Social Sciences Communications, Vol. 13, No. 1
- Modeling consumer ratings: The trade-off between quality and consumer preference
  European Journal of Operational Research, Vol. 334, No. 2
- Introducing a Wisdom of the Crowd’s Process Model for Achieving Effective Crowd Performance
  25 May 2026 | Group Decision and Negotiation, Vol. 35, No. 3
- The FABRICS framework: A Bayesian approach to financial quantification of cyber risk
  Computers & Security, Vol. 167
- A collective-intelligence-driven parametric model for measuring and improving group performance
  Information Sciences, Vol. 748
- Crowdsourced versus large language models forecasting: evidence for the accuracy–correlation effect
  16 April 2026 | Philosophical Transactions B, Vol. 381, No. 1948
- Harnessing Human Uncertainty to Train More Accurate and Aligned AI Systems
  Gunnar P. Epping,
  Andrew Caplin,
  Erik Duhaime,
  William R. Holmes,
  Daniel Martin,
  Jennifer S. Trueblood
  1 April 2026 | Decision Analysis, Vol. 0, No. 0
- Aggregating Judgmental Demand Forecasts in Environments with Structural Breaks
  Matthias Seifert,
  Yun Shin Lee,
  Shijith Kumar PM
  16 March 2026 | Decision Analysis, Vol. 0, No. 0
- Evaluating the effectiveness of crowd wisdom and large language models for fantasy cricket team selection
  8 March 2026 | Journal of the Operational Research Society, Vol. 22
- AI in strategic alliance formation: a framework for human–AI collaboration
  30 December 2025 | Journal of Business Strategy, Vol. 47, No. 2
- Fifty years of decision analysis in operational research: A review
  European Journal of Operational Research, Vol. 329, No. 2
- Selecting Product Species for Aquaculture Companies: A Spherical Fuzzy Delphi–CRADIS Framework
  16 February 2026 | Management Science Advances, Vol. 3, No. 1
- Insights into the impact of visual and textual information on investment decision-making: A multimodal business plan analysis via deep representation learning
  Expert Systems with Applications, Vol. 296
- Kairosis: A method for dynamical probability forecast aggregation informed by Bayesian change-point detection
  International Journal of Forecasting, Vol. 42, No. 1
- Decentralized Information Elicitation Without Verification
  IEEE Transactions on Networking, Vol. 34
- Three and a half stages of crowd wisdom
  30 March 2026 | Collective Intelligence, Vol. 5, No. 1
- Bayesian combination of correlated subjective probability estimates
  5 May 2026 | Judgment and Decision Making, Vol. 21
- Individual incentives that promote collective intelligence
  15 December 2025 | Proceedings of the National Academy of Sciences, Vol. 122, No. 51
- Making sense of confidence: do laypeople perceive eyewitness confidence in similar ways?
  19 December 2025 | Psychology, Crime & Law, Vol. 4
- Whether groups value agreement or dissent depends on the strength of consensus
  4 December 2025 | PLOS One, Vol. 20, No. 12
- Which to Trust: Extracting Collective Wisdom Based on Opinion Quality Rank Learning
  Shuai Jiang,
  Yanhong Guo,
  Chuanren Liu,
  Hui Xiong
  2 December 2025 | INFORMS Journal on Computing, Vol. 0, No. 0
- Systemic risk and macroeconomic activity: International evidence
  Journal of Management Science and Engineering, Vol. 10, No. 4
- Minimizing social bias with sequential collaboration: the role of contributor features in dependent judgments
  24 July 2025 | Scientific Reports, Vol. 15, No. 1
- Messengers: breaking echo chambers in collective opinion dynamics with homophily
  1 October 2025 | npj Complexity, Vol. 2, No. 1
- AI can outperform humans in predicting correlations between personality items
  12 February 2025 | Communications Psychology, Vol. 3, No. 1
- Combining Forecasts from Multiple Experts for Multiple Variables
  Zhi Chen,
  Long Zhao
  13 November 2025 | Management Science, Vol. 0, No. 0
- Evaluating crowdsourcing for ICU EEG annotation: A comparison with expert performance
  6 August 2025 | Epilepsia, Vol. 66, No. 11
- Group Discussions Improve Competence Calibration: Making Self-Perceived Competence Valuable When Harnessing the Wisdom of Crowds
  Michael Goedde-Menke,
  Enrico Diecidue,
  Andreas Jacobs,
  Thomas Langer
  24 October 2025 | Management Science, Vol. 0, No. 0
- Revealed Wisdom of the Crowd: Bids Predict Loan Quality
  Jiayu Yao,
  Mingfeng Lin,
  D. J. Wu
  20 January 2025 | Management Science, Vol. 71, No. 10
- Platform arbitration or expert review? Online dispute resolution mechanisms for e-commerce platforms with live-streaming sales channels
  17 September 2025 | Annals of Operations Research, Vol. 59
- Emergent numeracy: How the crowd wisdom of non-rounding survey respondents generates accurate immigration estimates
  17 January 2025 | Acta Sociologica, Vol. 68, No. 3
- Subjective Confidence as a Monitor of the Replicability of the Response
  6 February 2024 | Perspectives on Psychological Science, Vol. 20, No. 4
- Investor sentiment and stock returns: Wisdom of crowds or power of words? Evidence from Seeking Alpha and Wall Street Journal
  Journal of Financial Markets, Vol. 74
- Modeling dependent group judgments: A computational model of sequential collaboration
  6 January 2025 | Psychonomic Bulletin & Review, Vol. 32, No. 3
- MRI Distance Measures as a Predictor of Subsequent Clinical Status During the Preclinical Phase of Alzheimer's Disease and Related Disorders
  24 April 2025 | Human Brain Mapping, Vol. 46, No. 6
- Opting out in computer-supported sequential collaboration
  Computers in Human Behavior, Vol. 165
- Crowd prediction systems: Markets, polls, and elite forecasters
  International Journal of Forecasting, Vol. 41, No. 2
- Measuring probabilistic coherence to identify superior forecasters
  International Journal of Forecasting, Vol. 41, No. 2
- Five Is the Brightest Star. But by how Much? Testing the Equidistance of Star Ratings in Online Reviews
  8 January 2024 | Organizational Research Methods, Vol. 28, No. 2
- AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy
  10 February 2025 | ACM Transactions on Interactive Intelligent Systems, Vol. 15, No. 1
- Working the Crowd: Citizen Forecasting, Sophistication and Diversity in Canadian Federal and Provincial Elections
  4 February 2025 | Canadian Journal of Political Science, Vol. 58, No. 1
- The collective wisdom of behavioral game theory
  12 April 2024 | Economic Theory, Vol. 79, No. 1
- Multiple financial analyst opinions aggregation based on uncertainty-aware quality evaluation
  European Journal of Operational Research, Vol. 320, No. 3
- Critical barriers to material recovery from e-waste in Brazil
  Journal of Hazardous Materials Advances, Vol. 17
- Goal-line oracles: Exploring accuracy of wisdom of the crowd for football predictions
  31 January 2025 | PLOS ONE, Vol. 20, No. 1
- Bayesian Ensembles of Exponentially Smoothed Life-Cycle Forecasts
  Xiaojia Guo; ,
  Kenneth C. Lichtendahl; ,
  Yael Grushka-Cockayne
  30 October 2024 | Manufacturing & Service Operations Management, Vol. 27, No. 1
- Promises and Limitations of Market-Based Mechanisms in Crowdfunding
  26 October 2025
- The Wisdom of the Digital Crowd
  23 November 2025
- Optional disclosure and observational learning
  Journal of Economic Behavior & Organization, Vol. 229
- Accounting for dependencies among performance shaping factors in SPAR-H using a regularized autoencoder and WINGS-AISM
  Nuclear Engineering and Technology, Vol. 57, No. 1
- Crowd Dynamics in Online Lending: Unveiling Indicators of Collective Insight
  IEEE Access, Vol. 13
- The meta-aggregation approaches: Correctly choosing how to choose correctly in groups
  23 January 2025 | Collective Intelligence, Vol. 4, No. 1
- Harnessing text reviews through a multi-granularity heterogeneous graph convolutional network for facilitating multi-attribute crowd decision-making
  Applied Soft Computing, Vol. 167
- A risk assessment of a gas pressure reduction station system with confidence for dealing with imprecisions and unknowns
  Journal of Loss Prevention in the Process Industries, Vol. 92
- Genetically-diverse crowds are wiser
  Personality and Individual Differences, Vol. 231
- The persuasive effects of social cues and source effects on misinformation susceptibility
  20 February 2024 | Scientific Reports, Vol. 14, No. 1
- RETRACTED ARTICLE: Internal communication from a happiness management perspective: state-of-the-art and theoretical construction of a guide for its development
  9 November 2024 | BMC Psychology, Vol. 12, No. 1
- Boosting wisdom of the crowd for medical image annotation using training performance and task features
  20 May 2024 | Cognitive Research: Principles and Implications, Vol. 9, No. 1
- Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy
  Science Advances, Vol. 10, No. 45
- A Heuristic for Combining Correlated Experts When There Are Few Data
  David Soule,
  Yael Grushka-Cockayne,
  Jason Merrick
  15 November 2023 | Management Science, Vol. 70, No. 10
- A new method for enhancing collective intelligence using expert’s knowledge
  20 February 2024 | Journal of Information and Telecommunication, Vol. 8, No. 4
- Finding the Wise and the Wisdom in a Crowd: Estimating Underlying Qualities of Reviewers and Items
  24 May 2024 | The Economic Journal, Vol. 134, No. 663
- The Crowd as a Source of Knowledge - From User Feedback to Fulfilling Requirements
  9 September 2024
- A Bayesian Hierarchical Model of Crowd Wisdom Based on Predicting Opinions of Others
  John McCoy,
  Drazen Prelec
  19 October 2023 | Management Science, Vol. 70, No. 9
- Can crowdsourcing improve prediction accuracy in fashion retail buying?
  Journal of Retailing, Vol. 100, No. 3
- Human and Algorithmic Predictions in Geopolitical Forecasting: Quantifying Uncertainty in Hard-to-Quantify Domains
  29 August 2023 | Perspectives on Psychological Science, Vol. 19, No. 5
- Teachers’ judgment accuracy: A replication check by psychometric meta-analysis
  25 July 2024 | PLOS ONE, Vol. 19, No. 7
- Enhancing interval-valued time series forecasting through bivariate ensemble empirical mode decomposition and optimal prediction
  Engineering Applications of Artificial Intelligence, Vol. 133
- Improving geopolitical forecasts with 100 brains and one computer
  International Journal of Forecasting, Vol. 40, No. 3
- Where’s Waldo, Ohio? Using Cognitive Models to Improve the Aggregation of Spatial Knowledge
  8 April 2024 | Computational Brain & Behavior, Vol. 7, No. 2
- Crowd-Judging on Two-Sided Platforms: An Analysis of In-Group Bias
  Alan P. Kwan,
  S. Alex Yang,
  Angela Huyue Zhang
  7 June 2023 | Management Science, Vol. 70, No. 4
- Group efficiency and individual fairness tradeoff in making wise decisions
  Omega, Vol. 124
- Variability in verbal eyewitness confidence
  19 March 2024 | Applied Cognitive Psychology, Vol. 38, No. 2
- The role of expertise in herding behaviors: evidence from a crowdfunding market
  31 May 2022 | Electronic Commerce Research, Vol. 24, No. 1
- The effect of iterativity on adversarial opinion forming
  Information Processing Letters, Vol. 185
- The Strengths and Weaknesses of Crowds to Address Global Problems
  10 July 2023 | Perspectives on Psychological Science, Vol. 19, No. 2
- Understanding Collective Intelligence: Investigating the Role of Collective Memory, Attention, and Reasoning Processes
  29 August 2023 | Perspectives on Psychological Science, Vol. 19, No. 2
- Exploiting Meta-cognitive Features for a Machine-Learning-Based One-Shot Group-Decision Aggregation
  21 October 2023 | Group Decision and Negotiation, Vol. 33, No. 1
- Aggregating diverse evaluations in group decision making: An approach based on wisdom of crowds
  Information Sciences, Vol. 657
- Behavioral Biases in the Uncertainty Quantification Process
  31 May 2024
- The Social Psychology of the Wisdom of Crowds (with a New Section on Recent Advances)
  31 May 2024
- Operational Collective Intelligence of Humans and Machines
  1 June 2024
- The Joint Weighted Average (JWA) Operator
  10 August 2024
- Bayesian herd detection for dynamic data
  International Journal of Forecasting, Vol. 40, No. 1
- Searching for the Metaverse: Neuroscience of Physical and Digital Communities
  8 January 2024 | Cyberpsychology, Behavior, and Social Networking, Vol. 27, No. 1
- Evaluating Holistic Privacy Risk Posed by Smart Home Ecosystem: A Capability-Oriented Model Accommodating Epistemic Uncertainty and Wisdom of Crowds
  IEEE Transactions on Engineering Management, Vol. 71
- Leveraging Advice-taking and Kernel Density Estimation to Identify A Cluster of Experts and Improve Wisdom of Crowds
  1 January 2024 | SSRN Electronic Journal, Vol. 7
- Predicting the synthesizability of crystalline inorganic materials from the data of known material compositions
  25 August 2023 | npj Computational Materials, Vol. 9, No. 1
- Social influence makes outlier opinions in online reviews offer more helpful information
  27 June 2023 | Scientific Reports, Vol. 13, No. 1
- On the trajectory of discrimination: A meta-analysis and forecasting survey capturing 44 years of field experiments on gender and hiring decisions
  Organizational Behavior and Human Decision Processes, Vol. 179
- Crowdsourcing in patent examination: overcoming patent examiners' local search bias
  28 April 2023 | R&D Management, Vol. 53, No. 5
- Forecast combinations: An over 50-year review
  International Journal of Forecasting, Vol. 39, No. 4
- Boosting the Wisdom of Crowds Within a Single Judgment Problem: Weighted Averaging Based on Peer Predictions
  Asa B. Palley,
  Ville A. Satopää
  9 January 2023 | Management Science, Vol. 69, No. 9
- Task-specific algorithm advice acceptance: A review and directions for future research
  Data and Information Management, Vol. 7, No. 3
- Forecasting pine sawtimber stumpage prices: A comparison between a time series hybrid model and an artificial neural network
  Forest Policy and Economics, Vol. 154
- A novel multi-phase hierarchical forecasting approach with machine learning in supply chain management
  Supply Chain Analytics, Vol. 3
- Multiobjective Optimization-Based Collective Opinion Generation With Fairness Concern
  IEEE Transactions on Systems, Man, and Cybernetics: Systems, Vol. 53, No. 9
- Information Elicitation from Decentralized Crowd Without Verification
- Automating hybrid collective intelligence in open-ended medical diagnostics
  14 August 2023 | Proceedings of the National Academy of Sciences, Vol. 120, No. 34
- Continent‐Ocean Transition or Boundary? Crowd‐Sourced Seismic Interpretations of the East‐India Passive Margin
  8 August 2023 | Tectonics, Vol. 42, No. 8
- Science Communication as a Collective Intelligence Endeavor: A Manifesto and Examples for Implementation
  4 April 2023 | Science Communication, Vol. 45, No. 4
- A two-stage interval-valued carbon price forecasting model based on bivariate empirical mode decomposition and error correction
  3 June 2023 | Environmental Science and Pollution Research, Vol. 30, No. 32
- Reducing uncertainty in Delphi surveys: A case study on immigration to the EU
  6 December 2023 | Demographic Research, Vol. 49
- Extracting the wisdom of a smaller crowd from dependent quantile judgments
  19 July 2022 | IISE Transactions, Vol. 55, No. 6
- Group Structure and Information Distribution on the Emergence of Collective Intelligence
  Ming Tang,
  Huchang Liao
  10 January 2023 | Decision Analysis, Vol. 20, No. 2
- A two-dimensional approach to flexibility degree of XOR numbers with application to group decision making
  Mathematics and Computers in Simulation, Vol. 207
- Hidden Indicators of Collective Intelligence in Crowdfunding
  30 April 2023
- NewsComp: Facilitating Diverse News Reading through Comparative Annotation
  19 April 2023
- Extracting the collective wisdom in probabilistic judgments
  9 August 2022 | Theory and Decision, Vol. 94, No. 3
- Cultural consensus theory for two-dimensional location judgments
  Journal of Mathematical Psychology, Vol. 113
- Beyond collective intelligence: Collective adaptation
  22 March 2023 | Journal of The Royal Society Interface, Vol. 20, No. 200
- Wisdom of Crowds in Quantum Machine Learning
  3 March 2023 | Physical Review Applied, Vol. 19, No. 3
- Casting votes of antecedents play a key role in successful sequential decision-making
  24 February 2023 | PLOS ONE, Vol. 18, No. 2
- Improving the forecasting accuracy of interval-valued carbon price from a novel multi-scale framework with outliers detection: An improved interval-valued time series analysis mode
  Energy Economics, Vol. 118
- Assessing Spatial Synergy Between Integrated Urban Rail Transit System and Urban Form: A BULI-Based MCLSGA Model With the Wisdom of Crowds
  IEEE Transactions on Fuzzy Systems, Vol. 31, No. 2
- Predicting the future with humans and AI
  7 December 2022 | Consumer Psychology Review, Vol. 6, No. 1
- Preference for human or algorithmic forecasting advice does not predict if and how it is used
  20 May 2022 | Journal of Behavioral Decision Making, Vol. 36, No. 1
- Talent Spotting in Crowd Prediction
  3 June 2023
- Performance-Weighted Aggregation: Ferreting Out Wisdom Within the Crowd
  3 June 2023
- The Wisdom of Timely Crowds
  3 June 2023
- Forecasting Large Collections of Time Series: Feature-Based Methods
  21 September 2023
- Combining probabilistic forecasts of COVID-19 mortality in the United States
  European Journal of Operational Research, Vol. 304, No. 1
- Decomposing the effects of crowd-wisdom aggregators: The bias–information–noise (BIN) model
  International Journal of Forecasting, Vol. 39, No. 1
- Effect of linguistic disfluency on consumer satisfaction: Evidence from an online knowledge payment platform
  Information & Management, Vol. 60, No. 1
- Expertise determines frequency and accuracy of contributions in sequential collaboration
  24 March 2023 | Judgment and Decision Making, Vol. 18
- A normative model for Bayesian combination of subjective probability estimates
  24 November 2023 | Judgment and Decision Making, Vol. 18
- Aggregating or Interacting?: The Role of Individual Overconfidence
  1 January 2023 | SSRN Electronic Journal, Vol. 96
- On the round number bias and wisdom of crowds in different response formats for numerical estimation
  17 May 2022 | Scientific Reports, Vol. 12, No. 1
- The distribution of initial estimates moderates the effect of social influence on the wisdom of the crowd
  3 October 2022 | Scientific Reports, Vol. 12, No. 1
- Reimagining peer review as an expert elicitation process
  5 April 2022 | BMC Research Notes, Vol. 15, No. 1
- Estimating probability of visual detection of exotic pests and diseases in the grains industry—An expert elicitation approach
  18 November 2022 | Frontiers in Ecology and Evolution, Vol. 10
- The motor Wisdom of the Crowd
  5 October 2022 | Journal of The Royal Society Interface, Vol. 19, No. 195
- Interventions Based on Social Norms Could Benefit From Considering Adversarial Information Environments: Comment on Constantino et al. (2022)
  13 October 2022 | Psychological Science in the Public Interest, Vol. 23, No. 2
- A novel carbon price combination forecasting approach based on multi-source information fusion and hybrid multi-scale decomposition
  Engineering Applications of Artificial Intelligence, Vol. 114
- Decline, adopt or compromise? A dual hurdle model for advice utilization
  Journal of Mathematical Psychology, Vol. 110
- A consensus building model in group decision making with non-reciprocal fuzzy preference relations
  17 February 2022 | Complex & Intelligent Systems, Vol. 8, No. 4
- Strategic issue diagnosis by top management teams: A multiple-agent model
  29 April 2021 | Strategic Organization, Vol. 20, No. 3
- A new hedging hypothesis regarding prediction interval formation in stock price forecasting
  21 October 2021 | Journal of Forecasting, Vol. 41, No. 4
- Testing-the-Waters Policy With Hypothetical Investment: Evidence From Equity Crowdfunding
  14 July 2020 | Entrepreneurship Theory and Practice, Vol. 46, No. 4
- Red hot frogs: identifying the Australian frogs most at risk of extinction
  20 August 2021 | Pacific Conservation Biology, Vol. 28, No. 3
- Australia's most imperilled vertebrates
  Biological Conservation, Vol. 270
- Convex combinations in judgment aggregation
  European Journal of Operational Research, Vol. 299, No. 2
- Information aggregation and collective intelligence beyond the wisdom of crowds
  25 April 2022 | Nature Reviews Psychology, Vol. 1, No. 6
- The Wisdom of Model Crowds
  Lisheng He,
  Pantelis P. Analytis,
  Sudeep Bhatia
  16 September 2021 | Management Science, Vol. 68, No. 5
- The Crowd Classification Problem: Social Dynamics of Binary-Choice Accuracy
  Joshua Aaron Becker,
  Douglas Guilbeault,
  Edward Bishop Smith
  2 September 2021 | Management Science, Vol. 68, No. 5
- Designing grant-review panels for better funding decisions: Lessons from an empirically calibrated simulation model
  Research Policy, Vol. 51, No. 4
- Community/Public Approach to Earthquake Forecasting in the Era of Big Data: An On-going Endeavor in China
  11 October 2021 | Surveys in Geophysics, Vol. 43, No. 2
- The calculus of dissent: Bias and diversity in FOMC projections
  6 January 2022 | Public Choice, Vol. 191, No. 1-2
- Soft and hard information in equity crowdfunding: network effects in the digitalization of entrepreneurial finance
  3 August 2021 | Small Business Economics, Vol. 58, No. 4
- Failed but validated? The effect of market validation on persistence and performance after a crowdfunding failure
  Journal of Business Venturing, Vol. 37, No. 2
- Anticipated innovations for the blue economy: Crowdsourced predictions for the North Sea Region
  Marine Policy, Vol. 137
- What is a Good Calibration Question?
  16 April 2021 | Risk Analysis, Vol. 42, No. 2
- Hidden Experts in the Crowd: Using Meta-Predictions to Leverage Expertise in Single-Question Prediction Problems
  Tom Wilkening,
  Marcellin Martinie,
  Piers D. L. Howe
  10 March 2021 | Management Science, Vol. 68, No. 1
- Eliciting Social Knowledge for Creditworthiness Assessment
  20 January 2022
- Towards a solver-aware systems architecting framework: leveraging experts, specialists and the crowd to design innovative complex systems
  11 March 2022 | Design Science, Vol. 8
- Extracting the collective wisdom in probabilistic judgments
  SSRN Electronic Journal, Vol. 61
- Getting more Wisdom out of the Crowd: The Case of Competence-Weighted Aggregates
  SSRN Electronic Journal, Vol. 27
- Beyond Feasibility Filters: How Expertise Heterogeneity Enables Innovation Recognition
  1 January 2022 | SSRN Electronic Journal, Vol. 14
- Bayesian Ensembles of Exponentially Smoothed Life-Cycle Forecasts
  SSRN Electronic Journal, Vol. 21
- Incentives for self-extremized expert judgments to alleviate the shared-information problem
  SSRN Electronic Journal, Vol. 61
- Bias, Information, Noise: The BIN Model of Forecasting
  Ville A. Satopää,
  Marat Salikhov,
  Philip E. Tetlock,
  Barbara Mellers
  10 February 2021 | Management Science, Vol. 67, No. 12
- A Technocratic View of Election Forecasting: Weighting Citizens’ Forecasts according to Competence
  10 May 2021 | International Journal of Public Opinion Research, Vol. 33, No. 3
- Will Humans-in-the-Loop Become Borgs? Merits and Pitfalls of Working with AI
  1 September 2021 | MIS Quarterly, Vol. 45, No. 3
- A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment
  4 August 2020 | Business & Information Systems Engineering, Vol. 63, No. 4
- The Paradox of Interaction: Communication Network Centralization, Shared Task Experience, and the Wisdom of Crowds in Online Crowdsourcing Communities
  4 May 2020 | Communication Research, Vol. 48, No. 6
- Elaborating or Aggregating? The Joint Effects of Group Decision-Making Structure and Systematic Errors on the Value of Group Interactions
  Steffen Keck,
  Wenjie Tang
  7 October 2020 | Management Science, Vol. 67, No. 7
- Social Influence Undermines the Wisdom of the Crowd in Sequential Decision Making
  Vincenz Frey,
  Arnout van de Rijt
  8 October 2020 | Management Science, Vol. 67, No. 7
- Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study
  1 June 2021 | Health Technology Assessment, Vol. 25, No. 37
- Collective Choice May Tell Nothing About Anyone’s Individual Preferences
  Muye Chen,
  Michel Regenwetter,
  Clintin P. Davis-Stober
  9 December 2020 | Decision Analysis, Vol. 18, No. 1
- Forecasting forecaster accuracy: Contributions of past performance and individual differences
  1 January 2023 | Judgment and Decision Making, Vol. 16, No. 2
- Algorithm appreciation or aversion? Comparing in-service and pre-service teachers’ acceptance of computerized expert models
  Computers and Education: Artificial Intelligence, Vol. 2
- Guideline for incorporating the Delphi method in the evaluation of nursing theories
  1 January 2021 | Revista Latino-Americana de Enfermagem, Vol. 29
- Decomposing the Effects of Crowd-Wisdom Aggregators: The Bias-Information-Noise (BIN) Model
  SSRN Electronic Journal, Vol. 30
- Revealed Wisdom of the Crowd: Informational Value of Bids
  SSRN Electronic Journal, Vol. 14
- Investor Sentiment, Media and Stock Returns: The Advancement of Social Media
  SSRN Electronic Journal, Vol. 53
- What Types of Crowds Generate More Valuable Content? Evidence from Cross-Platform Posting
  SSRN Electronic Journal, Vol. 24
- A Regularized Weighting Model of Judgment Aggregation
  SSRN Electronic Journal, Vol. 54
- A Parametric Method for Small Crowd Selection
  SSRN Electronic Journal, Vol. 54
- A consensus model for group decision making under additive reciprocal matrices with flexibility
  Fuzzy Sets and Systems, Vol. 398
- Enhancing the Wisdom of the Crowd With Cognitive-Process Diversity: The Benefits of Aggregating Intuitive and Analytical Judgments
  22 September 2020 | Psychological Science, Vol. 31, No. 10
- Crowdvoting Judgment: An Analysis of Modern Peer Review
  Michael R. Wagner
  2 June 2020 | Stochastic Systems, Vol. 10, No. 3
- Aggregation mechanisms for crowd predictions
  14 March 2025 | Experimental Economics, Vol. 23, No. 3
- Beating the average forecast: Regularization based on forecaster attributes
  Journal of Mathematical Psychology, Vol. 98
- Determining the relative importance of titania nanotubes characteristics on bone implant surface performance: A quality by design study with a fuzzy approach
  Materials Science and Engineering: C, Vol. 114
- A PSO-algorithm-based consensus model with the application to large-scale group decision-making
  16 May 2020 | Complex & Intelligent Systems, Vol. 6, No. 2
- Weighting and aggregating expert ecological judgments
  23 March 2020 | Ecological Applications, Vol. 30, No. 4
- Do Teachers Consider Advice? On the Acceptance of Computerized Expert Models
  8 September 2019 | Journal of Educational Measurement, Vol. 57, No. 2
- Adaptive social networks promote the wisdom of crowds
  11 May 2020 | Proceedings of the National Academy of Sciences, Vol. 117, No. 21
- Using meta-predictions to identify experts in the crowd when past performance is unknown
  24 April 2020 | PLOS ONE, Vol. 15, No. 4
- Effect of Crowd Voting on Participation in Crowdsourcing Contests
  16 June 2020 | Journal of Management Information Systems, Vol. 37, No. 2
- Crowd characteristics and crowd wisdom: Evidence from an online investment community
  15 May 2019 | Journal of the Association for Information Science and Technology, Vol. 71, No. 4
- Nominal Response Model Is Useful for Scoring Multiple-Choice Situational Judgment Tests
  25 November 2018 | Organizational Research Methods, Vol. 23, No. 2
- Estimating Uncertainties Using Judgmental Forecasts with Expert Heterogeneity
  Saurabh Bansal,
  Genaro J. Gutierrez
  24 February 2020 | Operations Research, Vol. 68, No. 2
- Improving expert forecasts in reliability: Application and evidence for structured elicitation protocols
  12 December 2019 | Quality and Reliability Engineering International, Vol. 36, No. 2
- Wisdom of stakeholder crowds in complex social–ecological systems
  13 January 2020 | Nature Sustainability, Vol. 3, No. 3
- Expert judgments for performance shaping Factors’ multiplier design in human reliability analysis
  Reliability Engineering & System Safety, Vol. 194
- Waste Reduction Strategies: Factors Affecting Talent Wastage and the Efficacy of Talent Selection in Sport
  10 January 2020 | Frontiers in Psychology, Vol. 10
- Computer-Supported Collaborative Information Search for Geopolitical Forecasting
  30 May 2020
- Why do some combinations perform better than others?
  International Journal of Forecasting, Vol. 36, No. 1
- Bias, Information, Noise: The BIN Model of Forecasting
  SSRN Electronic Journal, Vol. 110
- Shepherding the Herd
  SSRN Electronic Journal, Vol. 46
- A Heuristic for Combining Correlated Experts
  SSRN Electronic Journal, Vol. 135
- Identify Experts through Revealed Confidence: Application to Wisdom of Crowds
  SSRN Electronic Journal, Vol. 117
- Decentralized Governance on Two-Sided Platforms: Crowdsourcing, Learning, and Debiasing
  SSRN Electronic Journal, Vol. 67
- The Role of Crowdsourcing in the Healthcare Industry
- Probability Forecasts and Their Combination: A Research Perspective
  Robert L. Winkler,
  Yael Grushka-Cockayne,
  Kenneth C. Lichtendahl,
  Victor Richmond R. Jose
  6 November 2019 | Decision Analysis, Vol. 16, No. 4
- Knowledge fusion patterns: A survey
  Information Fusion, Vol. 52
- Rate it again: Using the wisdom of many to improve performance evaluations
  10 April 2019 | Journal of Behavioral Decision Making, Vol. 32, No. 4
- Long-term forecasts for energy commodities price: What the experts think
  Energy Economics, Vol. 84
- Improving Accuracy by Coherence Weighting of Direct and Ratio Probability Judgments
  Yuyu Fan,
  David V. Budescu,
  David Mandel,
  Mark Himmelstein
  15 August 2019 | Decision Analysis, Vol. 16, No. 3
- A method for anticipating the disruptive nature of digitalization in the machine-building industry
  Technological Forecasting and Social Change, Vol. 146
- Digitally forecasting new music product success via active crowdsourcing
  Technological Forecasting and Social Change, Vol. 146
- Scientific Utopia III: Crowdsourcing Science
  1 July 2019 | Perspectives on Psychological Science, Vol. 14, No. 5
- A universal method for evaluating the quality of aggregators
  1 January 2023 | Judgment and Decision Making, Vol. 14, No. 4
- Extracting the Wisdom of Crowds When Information Is Shared
  Asa B. Palley,
  Jack B. Soll
  21 February 2019 | Management Science, Vol. 65, No. 5
- The gateway belief model: A large-scale replication
  Journal of Environmental Psychology, Vol. 62
- The wisdom of amateur crowds: Evidence from an online community of sports tipsters
  European Journal of Operational Research, Vol. 272, No. 3
- New technology assessment in entrepreneurial financing – Does crowdfunding predict venture capital investments?
  Technological Forecasting and Social Change, Vol. 139
- Deep Neural Ranking for Crowdsourced Geopolitical Event Forecasting
  10 May 2019
- Semiparametric Correction for Endogenous Truncation Bias With Vox Populi-Based Participation Decision
  IEEE Access, Vol. 7
- Collective Intelligence: A Taxonomy and Survey
  IEEE Access, Vol. 7
- Elaboration and validation of an ICNP® terminology subset for patients with acute myocardial infarction
  1 January 2019 | Revista da Escola de Enfermagem da USP, Vol. 53
- Enhancing the Wisdom of the Crowd With Cognitive Process Diversity: The Benefits of Aggregating Intuitive and Analytical Judgments
  SSRN Electronic Journal, Vol. 78
- The Wisdom of a Kalman Crowd
  SSRN Electronic Journal, Vol. 5
- Evaluating the Underlying Qualities of Items and Raters from a Series of Reviews
  SSRN Electronic Journal, Vol. 90
- Expert Selection Within a Single Judgment Problem
  SSRN Electronic Journal, Vol. 30
- Incorporating Global Medical Knowledge to Solve Healthcare Problems
- Designing Effective Crowdsourcing Systems for the Healthcare Industry
- The Potential of Crowdsourcing in the Health Care Industry
- Designing Effective Crowdsourcing Systems for the Healthcare Industry
- Incorporating Global Medical Knowledge to Solve Healthcare Problems
- The Potential of Crowdsourcing in the Health Care Industry
- Forecasting Price Movements in Betting Exchanges Using Cartesian Genetic Programming and ANN
  Big Data Research, Vol. 14
- When costly voting is beneficial
  Journal of Public Economics, Vol. 167
- When Crowds Give You Lemons
  1 November 2018 | Proceedings of the ACM on Human-Computer Interaction, Vol. 2, No. CSCW
- Information Precision in Online Communities: Player Valuations on www.Transfermarkt.De
  1 November 2018 | International Journal of Sport Finance, Vol. 13, No. 4
- The Persuasive Power of Algorithmic and Crowdsourced Advice
  17 December 2018 | Journal of Management Information Systems, Vol. 35, No. 4
- A Modified Consensus Model in Group Decision Making With an Allocation of Information Granularity
  IEEE Transactions on Fuzzy Systems, Vol. 26, No. 5
- The Value of Performance Weights and Discussion in Aggregated Expert Judgments
  17 April 2018 | Risk Analysis, Vol. 38, No. 9
- How Endogenous Crowd Formation Undermines the Wisdom of the Crowd in Online Ratings
  25 July 2018 | Psychological Science, Vol. 29, No. 9
- Do Experts Or Crowd-Based Models Produce More Bias? Evidence from Encyclopedia Britannica and Wikipedia1
  1 September 2018 | MIS Quarterly, Vol. 42, No. 3
- Designing Effective Crowdsourcing Systems for the Healthcare Industry
  International Journal of Public Health Management and Ethics, Vol. 3, No. 2
- Eliciting improved quantitative judgements using the IDEA protocol: A case study in natural resource management
  22 June 2018 | PLOS ONE, Vol. 13, No. 6
- Net stock issuance anomaly and cash flow explanation: A research note
  17 August 2017 | Australian Journal of Management, Vol. 43, No. 2
- The Wisdom of Crowds in Matters of Taste
  Johannes Müller-Trede,
  Shoham Choshen-Hillel,
  Meir Barneron,
  Ilan Yaniv
  17 February 2017 | Management Science, Vol. 64, No. 4
- Counteracting estimation bias and social influence to improve the wisdom of crowds
  18 April 2018 | Journal of The Royal Society Interface, Vol. 15, No. 141
- Crowd enabled curation and querying of large and noisy text mined protein interaction data
  4 October 2017 | Distributed and Parallel Databases, Vol. 36, No. 1
- Making a Wiser Crowd: Benefits of Individual Metacognitive Control on Crowd Performance
  11 July 2018 | Computational Brain & Behavior, Vol. 1, No. 1
- Weighted Brier score decompositions for topically heterogenous forecasting tournaments
  1 January 2023 | Judgment and Decision Making, Vol. 13, No. 2
- The Selection of Experts for (Probabilistic) Expert Knowledge Elicitation
  18 November 2017
- Time Prediction Methods and Principles
  1 March 2018
- Designing for Information Quality in the Era of Repurposable Crowdsourced User-Generated Content
  5 June 2018
- Testing the Wisdom of Crowds in the field: Transfermarkt valuations and international soccer results
  International Journal of Forecasting, Vol. 34, No. 1
- Hypothetical Investment Bias
  SSRN Electronic Journal, Vol. 113
- Semi-Parametric Correction for Endogenous Truncation Bias with Vox Populi Based Participation Decision
  SSRN Electronic Journal, Vol. 103
- Averaging Probability Forecasts: Back to the Future
  SSRN Electronic Journal, Vol. 11
- Essays on Angel Investing in the Entrepreneurial Ecosystem
  SSRN Electronic Journal, Vol. 27
- Incorporating Global Medical Knowledge to Solve Healthcare Problems
  International Journal of Healthcare Information Systems and Informatics, Vol. 13, No. 1
- Rescuing Collective Wisdom when the Average Group Opinion Is Wrong
  6 November 2017 | Frontiers in Robotics and AI, Vol. 4
- Managing Reliability and Stability Risks in Forest Harvesting
  Miguel A. Lejeune,
  Janne Kettunen
  10 August 2017 | Manufacturing & Service Operations Management, Vol. 19, No. 4
- Forecasting with temporal hierarchies
  European Journal of Operational Research, Vol. 262, No. 1
- Crowd performance in prediction of the World Cup 2014
  European Journal of Operational Research, Vol. 260, No. 2
- The relationship between crowd majority and accuracy for binary decisions
  1 January 2023 | Judgment and Decision Making, Vol. 12, No. 4
- Identifying experts in the crowd for evaluation of engineering designs
  18 April 2017 | Journal of Engineering Design, Vol. 28, No. 5
- Firms, crowds, and innovation
  9 May 2017 | Strategic Organization, Vol. 15, No. 2
- Combining Interval Forecasts
  Anil Gaba,
  Ilia Tsetlin,
  Robert L. Winkler
  21 February 2017 | Decision Analysis, Vol. 14, No. 1
- Bibliographie
- Semi-Parametric Correction for Endogenous Truncation Bias with Vox Populi Based Participation Decision
  SSRN Electronic Journal, Vol. 103
- The Potential of Crowdsourcing in the Health Care Industry
- Validating the Contribution-Weighted Model: Robustness and Cost-Benefit Analyses
  Eva Chen,
  David V. Budescu,
  Shrinidhi K. Lakshmikanth,
  Barbara A. Mellers,
  Philip E. Tetlock
  18 April 2016 | Decision Analysis, Vol. 13, No. 2
- Wisdom or Madness? Comparing Crowds with Expert Evaluation in Funding the Arts
  Ethan Mollick,
  Ramana Nanda
  11 September 2015 | Management Science, Vol. 62, No. 6
- Geography, Diversity, and Accuracy of Crowdsourced Earnings Forecasts
  SSRN Electronic Journal, Vol. 38
- Judgmental Demand Forecasting for Online Sales of Product Collections: Segmentation by Type or Age?
  SSRN Electronic Journal, Vol. 7
- Estimating Demand Uncertainty Using Dispersion of Team Forecasts or Distributions of Forecast Errors
  SSRN Electronic Journal, Vol. 7
- New Technology Assessment in Entrepreneurial Financing - Can Crowdfunding Predict Venture Capital Investments?
  SSRN Electronic Journal, Vol. 24
- The Composition of Optimally Wise Crowds
  Clintin P. Davis-Stober,
  David V. Budescu,
  Stephen B. Broomell,
  Jason Dana
  14 July 2015 | Decision Analysis, Vol. 12, No. 3
- Investor communication in equity-based crowdfunding: a qualitative-empirical study
  Qualitative Research in Financial Markets, Vol. 7, No. 3
- Aggregating multiple probability intervals to improve calibration
  1 January 2023 | Judgment and Decision Making, Vol. 10, No. 2
- Eliciting and Aggregating Forecasts When Information is Shared
  SSRN Electronic Journal, Vol. 11
- Forecasting Tournaments
  4 August 2014 | Current Directions in Psychological Science, Vol. 23, No. 4
- Wisdom or Madness? Comparing Crowds with Expert Evaluation in Funding the Arts
  SSRN Electronic Journal, Vol. 37
- Investor Communication in Crowdfunding: A Qualitative-Empirical Study
  SSRN Electronic Journal, Vol. 14
- Combining Interval Forecasts
  SSRN Electronic Journal, Vol. 5
- Uncovering the Outcomes of Equity Crowdfunding: Post-Funding Outcomes of Equity Crowdfunded Firms in Europe
  SSRN Electronic Journal, Vol. 3

Volume 61, Issue 2

February 2015

Pages iv-vi, 249-486

Article Information

Supplemental Material

Metrics

Information

Received:January 19, 2013
Accepted:November 25, 2013
Published Online:May 23, 2014

Cite as

David V. Budescu, Eva Chen (2014) Identifying Expertise to Extract the Wisdom of Crowds. Management Science 61(2):267-280.

https://doi.org/10.1287/mnsc.2014.1909

Keywords

PDF download

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Identifying Expertise to Extract the Wisdom of Crowds

Supplemental Material

Volume 61, Issue 2

Article Information

Supplemental Material

Metrics

Information

Cite as

Keywords

Sign Up for INFORMS Publications Updates and News