Neural networks and the case for efficient modeling, Part 2

In Part 1 of this series, we discussed the emerging popularity of neural networks for analyzing more traditional business questions such as customer churn. We outlined an example where a deep learning model was able to outperform a logistic regression on the same data set, in terms of statistical accuracy. But accuracy is not the only way to evaluate models.

Choosing the best model in a business context requires a balance of three criteria: predictive power, explainability and ease of production. Striving to achieve this balance is similar to the concept of efficiency in economics, which suggests that an appropriate term for this type of model measurement might be “efficient modeling.” In this article, we’ll look more deeply at how we might evaluate two candidate models through an “efficient modeling” lens, and what implications that might have on our business decision.

Different types of models can be more or less challenging to deploy, yet this can often be overlooked by data scientists or analysts who just want “the answer.” But a one-time answer usually doesn’t provide as much business value as a model that can be reapplied over time, which implies that even the most accurate models are handicapped if they are not implemented in a production environment.

It’s true that both a logistic regression and a neural network can be productionized in a cloud-based environment such as Azure and AWS, which provide machine-learning platforms that can host either type of model. However, many companies don’t have the luxury of cloud environments – many still utilize on-premise data warehouses that lack comprehensive support or additional services for hosting advanced analytical models. Since logistic regressions can be implemented using standard SQL, it is straightforward to deploy them within this type of environment.

On the other hand, productionizing a neural network can require a more complex approach, and perhaps a more advanced skillset, to implement successfully in a more bare-bones environment. There are other options available such as third-party software companies offering AI as a service, and, when appropriate, in-house solutions can be created by data and software engineers. But these solutions are likely more complicated, and potentially expensive, than building a model in standard SQL that can fit inherently within the environment.

Another factor to take into consideration is how explainable a model is – that is, how well do you understand what is happening under the hood. This is important not only for the creator of the model, but more importantly for any model sponsors and executives who plan to make decisions based on the outcomes. Building trust in a model requires an understanding of how it works, especially for those who are not technically inclined.

Traditionally, neural networks have been thought of as black box models without a lot of understanding built in to how the model arrived at its conclusions, although recent advances in research are beginning to address this barrier. Logistic regressions, on the other hand, are simpler to explain to business users purely because of their structure. As the coefficients in the model change, so too does the output. This means you can easily explain the quantitative relationship between a variable used in the model and the outcome you are trying to predict (customer churn in the example above). This relationship is often obscured when using neural nets – it can be challenging to explain exactly why the model made the choice that it did.

What’s most important though is to strike a balance of predictive power, ease of production and explainability. Reflecting on that balance, and the concept of efficient modeling, it reminded me of how we approach our consulting work at Elicit. Anytime we want to solve a problem internally or with a client, we use our “Geek Nerd Suit” framework to help us organize our thoughts. Basically, it states that for any organization to run at optimal speed, the technology (Geek), analytics (Nerd) and business (Suit) functions must be collaborating and making decisions in lockstep. Measuring a model is no different. The model must be able to be productionized (Geek), be as powerful as possible (Nerd), while still being understood and trusted by key stakeholders (Suit). As simple as that may seem, integrating these three can be a challenge and is often overlooked.

If we look back at the two example models through an “efficient modeling” lens, while the neural network did have slightly better predictive power, the logistic regression is easier to productionize and understand. In this case, efficient modeling would influence a business stakeholder to choose the logistic regression over the neural network, despite the neural network’s slight advantage in accuracy.

In summary, making good choices about model selection requires consideration of all three categories: Geek (productionization), Nerd (statistical power), and Suit (explainability). Neural networks and deep learning may be the future, but for now, having an efficient modeling mindset is your best bet.