Precisely inaccurate

In their book “Freakonomics,” Levitt and Dubner note that the average adult in a global sample has one breast and one testicle – an extreme example of being precisely inaccurate. Unfortunately, not all sampling produces insights that are precisely inaccurate and as stark and easy to catch. Hence, they go unnoticed or even worse, they are noticed and serve as the basis for decisions that lead to losses and heartache.

Analytics is a combination of science and art (the left and the right brain) where the science may paint the picture of a one-testicle, single-breasted customer, but the art should ensure that the image that is identified is incorrect and therefore should be modified.

I have seen numerous cases where organizations tend to focus only on the science portion of analytics and then fail to realize the true benefits or, in some cases, any benefits. Before moving ahead, let us differentiate between accuracy and precision, which will form a backdrop for the rest of the article.

Precision: How specific is the value predicted by the models versus the range of value

Accuracy: How well a measurement matches or complies with a known standard or actual value

More often than not, one needs a precise and accurate prediction using analytics. Hence, you want one answer (precisely) and you want that one answer to be accurate (close to the truth).

Organizations thus try and look for an answer akin to Figure 1 (accurate and precise). The real-world dynamics and lack of granular and complete data, however, often lead to a somewhat accurate answer but one that is not very precise as in Figure 2. The real issue arises when the organization pushes for accuracy; in that case you could end up with a precise but inaccurate answer (as shown in Figure 3).

The art of analytics, however, lies in finding the right balance between precision and accuracy that can help one make the right decisions.

In all honesty, when you have comprehensive and granular data in a market, it is very much possible to get to a precise and accurate answer and the need for balance is minimized. However, as one looks at problems that are more complex and data sets that may not be complete or granular (such as those in emerging markets), it becomes important to strike the right balance between precision and accuracy.

For example, if one was to predict the impact of marketing on sales in a country such as Brazil, one would face a situation where the data available through syndicated means may not cover a large portion of the market, and the quality of data may not be good. In such a situation, trying to build analytical models or constructs to provide a precise and accurate answer may prove futuile. In fact, in pursuit of precision, one may reach a conclusion that is inaccurate.

The challenge, therefore, is to know when you should avoid the trap of being precisely inaccurate. Two approaches can help organizations set up their analytics programs/projects. The first approach looks at setting the right expectation, and the second approach looks at choosing the right methodology.

Setting the right expectation. The concept of resolution of data is a great way of setting the right expectation from a results perspective. Resolution is the smallest value or amount that can be measured. If one adds to it the comprehensiveness and completeness of the data set, then it is possible to get an idea of the precision that you should expect from the results. This does not mean that the results would be incorrect and not usable, but that rather than a point estimate that is provided, there could have a variation in the real world.

Therefore, one should focus not necessarily on the “how much” will happen/has happened but the “why” of what will happen/has happened. In addition, one should try and develop a tolerance range against the output. This will help ensure decisions are taken in the right direction.

Changing the methodology/output. Analytical approaches recognize the challenge in being precise and accurate at the same time, and hence there are approaches that belong to the probabilistic school of thought that provide a most probable estimate but also a probability distribution of the estimate (thereby providing a range of results that could be true and the most probable one among them).

The two approaches stated above are not the only way to deal with the preciseness vs. accuracy problem, but they are ones that I find easiest to implement and follow.

The next time you are seeing the model results from a market where data resolution is low and the model output shows a result with an accuracy of two decimal places, please think about whether you are creating a single-breasted, one-testicle being.