Regularized Aggregation of One-Off Probability Predictions

Forecasters predicting the chances of a future event may disagree because of differing evidence or noise. To harness the collective evidence of the crowd, we propose a Bayesian aggregator that is regularized by analyzing the forecasters’ disagreement and ascribing overdispersion to noise. Our aggregator requires no user intervention and can be computed efficiently even for a large number of predictions. To illustrate, we evaluate our aggregator on subjective probability predictions collected during a four-year forecasting tournament sponsored by the U.S. intelligence community. Our aggregator improves the squared error (a.k.a., the Brier score) of simple averaging by around 20% and other commonly used aggregators by 10%–25%. This advantage stems almost exclusively from improved calibration. An R package called braggR implements our method and is available on CRAN.

Supplemental Material: The online appendix is available at https://doi.org/10.1287/opre.2021.2224.