Should Scoring Rules be “Effective”?

A scoring rule is a reward function for eliciting or evaluating forecasts expressed as discrete or continuous probability distributions. A rule is strictly proper if it encourages the forecaster to state his true subjective probabilities, and effective if it is associated with a metric on the set of probability distributions. Recently, the property of effectiveness (which is stronger than strict properness) has been proposed as a desideratum for scoring rules for continuous forecasts, for reasons of “monotonicity” in keeping the forecaster close to his true probabilities, since in practice the forecast must be chosen from a low-dimensional set of “admissible” distributions. It is shown in this paper that what effectiveness implies, beyond strict properness, is not a monotonicity property but a transitivity property, which is difficult to justify behaviorally. The logarithmic scoring rule is shown to violate the transitivity property, and hence is not effective. The L₁ and L_∞ metrics are shown to allow no effective scoring rules. Some potential difficulties in interpreting admissible forecasts are also discussed.