Data for the 99 Percent

I recently finished teaching an MBA elective course called “Predictive Analytics.” I have been teaching this class for nearly a decade, but the course title (originally “Applied Business Statistics”), content and structure have significantly evolved over the years.

Nearly three decades since finishing school, I still have a lot of reverence for my own statistics teachers, all of whom had taught the material in a very traditional way. As such, I am a little sheepish about some of the things that have been dropped from this course, including a formal definition of a random sample, a detailed discussion of confidence intervals and a thorough treatise on p-values. There are several reasons for these omissions. The most obvious one: The course is now focused on prediction and covers a variety of nonparametric techniques in addition to classical statistical methods like regression. As such, many of the concepts from inferential statistics are somewhat less critical than they would be for a course that was more oriented explanation rather than prediction. I understand all of this, but nevertheless I often have this nagging feeling that, like Michael Keaton in “Mr. Mom,” I’m “doing it wrong” [1].

As a brief aside, note that statistical significance and p-values have lately been under siege from many sides. A recent article [2] in Nature by Amrhein, Greenland and McShane asserts that, “we should never conclude there is ‘no difference’ or ‘no association’ just because a p-value is larger than a threshold such as 0.05.” In addition to the authors, 854 researchers from 52 countries and dozens of different fields are listed as co-signatories to this article, which is simply entitled “Retire Statistical Significance.” Also, a 2013 paper [3] by Lin, Lucas and Shmueli in Information Systems Research points out that with today’s very large data sets, “p-values go quickly to zero, and solely relying on p-values can lead the researcher to claim support for results of no practical significance.” 

Predictive Analytics Course

In my predictive analytics course we dig into the importance of avoiding overfitting, the mechanics of partitioning data into training and validation sets, and the value of combining results from multiple models, with an emphasis on data preparation, exploratory data analysis and visualization, and an interactive approach to model building and testing. Although most of my business students have little or no coding background, the JMP software from SAS Institute (and the power of today’s laptops) give them the ability to apply a variety of predictive techniques to moderately large datasets. This seven-week introductory class culminates in a simplified Kaggle-like competition in which student teams compete to see who can most accurately predict the value of a portfolio before presenting their models and findings. My role in the class is equal parts instructor, mentor and tormentor.

I recently had a chat with Kord Davis, a longtime tech industry consultant, the author of a book entitled “Ethics of Big Data” [4] and co-founder of Signal. Over the past few years, he has developed and delivered a very effective series of data literacy workshops. In many cases, the people who attend these workshops are employed by governmental organizations that have invested in systems that capture and integrate data for high-minded purposes such as “making better decisions,” “capturing efficiencies” and “improving processes” – though few people in the organization have a clear vision on how these outcomes will actually be realized.

As I lamented about all the stuff that is now missing from my MBA course, Davis just laughed. “More than once, we’ve had participants tell us, ‘I was very nervous about this workshop because I didn’t think I was qualified to do data analysis,” Davis said. “But the collaborative approach that allowed me to work with folks in other departments, on a problem statement I’m personally passionate about, was very useful. The problem-solving framework helped give me some structure to work through a data project in a more rigorous way. Starting with the end in mind and prioritizing the data narrative for end users and different audience segments was great practice. It works.” 

Key to Success

One key to success, Davis and I agreed, was to accept that neither of us is tasked with creating data scientists (if we were, there would be a very different group of students going through very different content). Davis calls this “data for the 99 percent,” and the goal of his workshops is to help people turn their existing domain expertise and passion into better and more effective results. For my MBA students, my real objective is to develop professionals who: (a) are able to effectively work with data, and (b) understand the overall process of turning data into business value. In the weeks since my most recent class ended, I have received some encouraging feedback.

First, one of my current students sent an email to thank me for the experience that he had gained in this year’s predictive analytics course. “Coming out of college, I had taken full semester classes in both probability and statistics and still couldn’t tell you how any of it was actually used. Getting hit with massive amounts of data was a typical day for me in all of the jobs that I had before grad school, but I rarely knew what to do with the data,” he confessed. He went on to elaborate: “I have always wanted to be able to do exactly what we learned in this course. I’ve stressed the importance of predicting outcomes in my past jobs, but I never had the skills to do so. Though it seems like we just scratched the surface, I now feel empowered to utilize these techniques to predict future outcomes, but with an appropriate awareness that there is a lot more to learn.” 

Next, I had an interesting encounter with a friend who has spent the past two decades working for a variety of tech companies. She has an MBA from a prestigious university but no formal technical training, and she has spent most of her career in jobs that sit at the intersection of information systems, business analytics, data science and leadership. When we met up for lunch recently, she was interviewing for a new job where she would be starting a new analytics group, and the company had given her a case study with dataset to analyze. Over pizza, I was able to teach her how to use the CART algorithm, and on her laptop she was quickly able to see that it clearly outperformed her initial (regression) model on her sample data.

A week later, she emailed with good news: her case interview had gone extremely well, in large part because of her ability to explain her model and results to a group of senior executives, and she had accepted their job offer a couple days later. Reflecting on this outcome, I realized that it had been far easier to teach her to use and interpret a new algorithm because of the overall modeling, analysis and communication process that she had learned through years of work experience – and that the students emerging from my class already have a solid understanding of this kind of stuff.

Finally, I received an unexpected email from a former MBA student. He had been listening to a podcast [5] that immediately reminded him of my predictive analytics class. In particular, he cited this quote from Wharton Professor Christopher Ittner:

“What we are finding is that in a lot of companies, there are great data scientists and great business people, but what is missing is business people who know enough data analytics to say, ‘Here is the problem I would like you to help me with.’ And then they can take the outcome from the data scientists and see how they can best leverage it. That is where we must get to in the next couple of years if we want to take advantage of the digital technologies.”

After more than a decade as a business school professor, I’m still trying to figure out how to be a more effective teacher in the context of the rapidly evolving world of data, models and software. But I am slowly starting to feel like I just might be on the right track. In hopes of learning from others who are also on this journey, I’ll also be leading a session at the INFORMS Annual Meeting in October in Seattle entitled “Teaching Data Mining to MBA Students.” Would love to see some of you there.

References

1. https://www.youtube.com/watch?v=1zKWd5_zMcc
2. https://www.nature.com/articles/d41586-019-00857-9
3. https://pdfs.semanticscholar.org/262b/854628d8e2b073816935d82b5095e1703977.pdf
4. https://www.oreilly.com/library/view/ethics-of-big/9781449314873/
5. https://knowledge.wharton.upenn.edu/article/cost-management-in-the-digital-age/