The road already taken teaches something new

I recently read a wonderful essay by the writer Pico Iyer entitled “The Beauty of the Ordinary” [1]. This lovely piece invites us to pay attention to the little things that we too often take for granted. Sharing a bit of his own life’s story, which has taken him from England to the United States and finally to Japan, he deftly uses the metaphor of the fading light of autumn to illuminate his broader points. Since reading this article, I have been careful to take notice of things that ordinarily sail right past me – and have frequently found myself surprised as a result.

Surprise No. 1: I recently learned that my university is honoring a group of MBA students for an award for an analytics consulting project that was part of a course that I taught last spring. The client was Culture Amp, a Software-as-a-Service (SaaS) vendor whose software platform helps organizations to manage their people and culture. Last spring, first-year MBA students Mark Crockett, Lian Liu and Katherine Steinberg examined historical data for thousands of customers, built a model to predict the likelihood of contract renewal, and made recommendations for how to prevent high-risk customers from churning.

At the core of the students’ work was a random forest model that utilized a few dozen potential explanatory variables to identify the key triggers for customer churn, an approach that was conceptually and computationally straightforward for the students to implement and for the client to digest. “Working with your team at USF was rewarding for us. They did a great job of analyzing the available data and helped us by uncovering some new insights that have been impactful for us,” wrote Steve Hopkins, vice president of customer success at Culture Amp, in a recent email.

Since the completion of this project, I have been surprised to discover that this type of predictive modeling is neither standard functionality for customer success management software platforms, nor is it standard practice for customer success teams. While “customer health scores” are common, I have learned that these metrics are typically based on a combination of subjective data inputs and business rules rather than on predictive models.

At first, I was puzzled by this. SaaS companies have incredible access to customer software usage data, and customer success management software platforms typically integrate additional customer data from many different systems. Given all of this, in theory there should be lots of raw material available to construct a rich view of the customer experience and therefore predictive modeling should be straightforward. And since most CS teams are chronically understaffed, there would seem to be great value to empirically knowing which customers are at higher risk of churning.

Asking around, I have come to learn that the reality is quite different. Within the world of customer success, data challenges include a lack of access to detailed software usage data, an absence of longitudinal data, resistance from overworked CSMs to spend time inputting key data, and overall questionable data quality. Moreover, as Todd Eby (CEO of SuccessHacker, a leading customer success training, recruiting and consulting firm) pointed out in a recent conversation, “most companies don’t take the time to really build out a detailed model of their customer experience, which makes it almost impossible for them to know which data is important to track.”

These days, there is a lot of talk about AI/ML being applied to all kinds of new problems. However, the challenges of making AI/ML work in the data-rich world of customer success and SaaS provide some insight into some things that need to be sorted out in order for these methods to be effectively unleashed.

Surprise No. 2: As I write this, the Major League Baseball playoffs are in full swing. Last week, three of the four Divisional Playoff series were decided by a winner-take-all Game 5. Since sports fans everywhere believe in the concept of “home field advantage,” it was surprising that the visiting teams actually won two of these three showdowns. But in a recent article [2] prior to this year’s Game 5s, Joe Posnanski points out that the road teams have actually won more than 60% of Game 5s in the Divisional Playoff series prior to this year. Because Posnanski is a sportswriter rather than a statistician, he simply attributes these results to “small sample size.”

Intrigued, I decided to dig a little deeper. While the data on Divisional Playoff series only dates back to 1995, our estimate (including this year’s data) for the probability of the home team winning Game 5 is now 37.5% (N=32) and that using the normal approximation to the binomial we have the upper (lower) 90% confidence estimate at 48.5% (26.5%). Looking at this another way, if our null hypothesis is that the probability of either team winning a decisive Game 5 is 50%, the probability of seeing the road team win as often as they have is only about 10% based on the binomial distribution itself.

So, for any other baseball geeks like me who might be reading this, none of us should be too surprised the next time a road team wins a deciding Game 5. Perhaps there is something else going on here?

Surprise No. 3: The other day, I receive an email with the subject line “You were right; I should have gone to an analytics graduate program” from a young man who I had written about in this column [3] years ago (he had instead gone on to study operations research at Cornell). Since finishing his master’s degree, he has been working as a healthcare industry consultant. While he is quick to point out that his O.R. degree has served him well in many ways, he now finds himself wistfully wondering what might have happened if he had listened to me in the first place.

The surprise here is that I do not write back saying, “I told you so!” Instead, having entered the autumn of my own life, I know all too well that the road not taken is easily imagined, while all too tangible are the painful memories of the bumps and pitfalls that we have actually encountered. I encourage him not to spend any more energy looking backward but rather to take a thoughtful and careful scan of the many opportunities already available to him today and to reflect on what he has learned about the types of experiences that bring him satisfaction and joy. (I also suggest that perhaps he should explore Georgia Tech’s online MS in Analytics [4] as well). I close my email by quoting Pico Iyer quoting the explorer and naturalist John Burroughs: “to learn something new, take the path that you took yesterday.”

Words to live by.

References

1. https://www.nytimes.com/2019/09/20/opinion/aging-marriage-autumn.html
2. https://theathletic.com/1281060/2019/10/09/posnanski-who-is-going-to-win-game-5-heres-what-the-numbers-can-and-cant-tell-us/
3. http://analytics-magazine.org/analyze-this-grad-school-desires-vs-real-world-demands/
4. https://www.gatech.edu/academics/degrees/masters/analytics-online-degree-oms-analytics