September 1, 2014 in Analyze This!
Students, professionals need ‘data wrangling’ skills
SHARE: PRINT ARTICLE:
https://doi.org/10.1287/LYTX.2014.05.07
As I write this, I am as usual frantically preparing for the new school year, which starts in just a few short days. I am once again teaching exclusively MBA students here at the University of San Francisco. This year, in addition to the core quantitative methods course and my longstanding applied statistics elective, I’m also teaching a new elective entitled “Introduction to Data Mining.”
All of our analytics courses for MBA students are taught with a strong practical bent. Our core MBA course is entitled “Spreadsheets and Business Analytics,” and not surprisingly this course requires students to be very hands-on with Excel in building models and analyzing historical data. In addition, because both of my electives for this fall place a heavy emphasis on data analysis, both of them are built around the JMP software from SAS Institute.
At this point, a short digression for a true confession: I have never liked computer programming. It’s not that I’m not capable of doing this kind of work – just ask me about the integration of GAMS with Mathematica in order to run numerical experiments for my dissertation! – but the reality is that programming is something that I do not for the most part enjoy. In my research, I am always looking for existing software tools (or graduate students) that I might be able to deploy to get my experiments done more quickly, and I will invariably write code from scratch only as a last resort.
Given this aversion to programming, I am able to empathize with my MBA students, the vast majority of whom have little or no background with coding. Moreover, all of the same factors that have driven the proliferation of analytics in the business world have also made it possible – and increasingly easier – for me to teach a lot of relatively sophisticated mathematical and statistical applications to these students. So I am thrilled that thanks to today’s faster hardware and user-friendly software, I can bring some important ideas and techniques to my MBA students in a hands-on way.
A select few of my MBA students ultimately decide to take my “Analytics Consulting Projects” course [1], and that’s where the game changes radically. The reality is that there is virtually no way to do a meaningful real-world analytics project without doing a significant amount of data manipulation work to get the data into shape for whatever analysis needs to be done. As the New York Times’ Steve Lohr recently reported [2], this need for extensive “data wrangling” is as great for professional data science teams as it is for my students.
This is where the fun begins. How do you manipulate data without writing a computer program? The most common first step is to use Excel to manipulate raw data into the required format, often in awkward and unnatural ways. This is not only because of Excel’s increasingly powerful capabilities for sorting, searching and summarizing but also because this is an environment with which they already have extensive experience and comfort. One of last year’s project teams, upon finally determining a particular statistical analysis that would provide the client with some unique insights, spent several hours perusing and posting on http://www.mrexcel.com/ before ultimately figuring out how to do the lookup/summary calculations that were needed to prepare the data.
As the limitations of the Excel platform become apparent, some teams have no choice but to get educated on other tools such as Python, SQL and R. In class, we often hold short workshops to help support this learning process, some led by me and others organized by the students. But all of this consumes valuable time on the project calendars, and this data wrangling is often the source of a great deal of stress, especially since the problem statement and solution methodology is also evolving along the way for most projects.
All of the students in this projects course ultimately learn that these capabilities are truly essential in order for them to be able to answer even moderately challenging business questions whose answers are data-driven. As such, after teaching this class for a second time last spring, I began thinking that a programming prerequisite might be worth creating. The course that I had envisioned would be ruthlessly practical (fundamental programming logic, basics of relational databases and SQL, and examples of how to manipulate data in a few different environments including Python, SAS, SPSS and R). The goal would be to give them a little more skill – and a lot more confidence – in their ability to manage the data on subsequent projects.
It turns out that I’m not the only business faculty member thinking about whether my students should learn to write code. A recent Business Week article entitled “B-Schools Finally Acknowledge: Companies Want MBAs Who Can Code” [3] provides a brief description of what several elite MBA programs are doing to provide their students with opportunities to study programming. Yet almost none of the examples that are cited in the article, ranging from a joint MBA and MS in computer science at Stanford to new elective courses being considered at Harvard, mention data management (with the exception of economist David Backus’ new elective in data visualization and Python being offered at NYU starting this fall). Instead, the focus of the article is on MBAs who want to pursue careers as product managers and tech entrepreneurs and what these elite programs are doing to support them.
I would argue that there is something more profound here for business schools to consider. As Marc Andreessen famously pointed out in 2011, “More and more major businesses and industries are being run on software and delivered as online services – from movies to agriculture to national defense. …Six decades into the computer revolution, four decades since the invention of the microprocessor and two decades into the rise of the modern Internet, all of the technology required to transform industries through software finally works …” In light of this reality, it seems irresponsible of business schools to continue to teach a core curriculum that largely does not reflect this increasingly central role.
While a class or two in software programming will not solve that problem overnight, it is certainly a start. And any class that helps us create graduates that can wrangle a lot more of their own data should also be a big plus for our quantitatively oriented MBA students – and for their future employers.
Notes & References
2. For the complete article, see http://www.nytimes.com/2014/08/18/technology/for-big-data-scientists-hurdle-to-insights-is-janitor-work.html?_r=0
Vijay Mehrotra is a professor in the Department of Business Analytics and Information Systems at the University of San Francisco’s School of Management and a longtime member of INFORMS.
([email protected])