Which to Trust: Extracting Collective Wisdom Based on Opinion Quality Rank Learning

Aggregating diverse human opinions in the digital era is essential in harnessing collective wisdom to unravel intricate management dilemmas. The aggregation process has significant hurdles to overcome due to the heterogeneity in the quality of individual opinions. This study introduces a CrowdRank machine learning architecture to provide an innovative solution to the central problem of opinion aggregation by learning to rank individual opinions. CrowdRank operates through two phases. (1) It leverages a BNN, pretrained on historical data, to conduct pairwise opinion comparisons. This network, designed to capture meaningful interactions between opinion features, adheres to key axiomatic principles to ensure a principled evaluation of opinion quality. (2) CrowdRank employs expectation propagation to synthesize these microassessments into a coherent global ranking of individual opinions. We validated the efficacy of our approach through a large-scale empirical investigation using real-world financial analyst forecasts. The validation results demonstrated the superiority of our approach over existing methods in accurately predicting both pairwise and aggregate opinion rankings. Importantly, CrowdRank significantly improves the objectivity and precision of collective financial analyst forecasts. This study contributes a theoretically robust and practically validated innovation to opinion aggregation and charts a new path in the application of machine learning to enhance the synthesis of collective wisdom.

History: Accepted by Ram Ramesh, Area Editor for Data Science and Machine Learning.

Funding: This work was partly supported by the National Natural Science Foundation of China [Grants 92370204 and 71974031].

Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information (https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2024.0835) as well as from the IJOC GitHub software repository (https://github.com/INFORMSJoC/2024.0835). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/.