The Evolution of Analytics and Implications for Industry and Academic Programs

Introduction

The past decade has seen tremendous growth and interest in the use of analytics in industry. Reacting to this increased interest from industry, we have also seen a dramatic increase in the number of master’s programs in analytics, business analytics, and data science. In this work, we discuss the evolution of analytics in industry and academia with the goal of drawing attention to potential gaps that may exist or be forthcoming.

Figure 1 shows a Google trends chart that measures web searches for the term “analytics.” It illustrates the dramatic increase in interest in analytics from 2006 until a peak in 2012 (the graph is normalized around peak search). Since 2012, searches have tapered.

Growth in the use of analytics is generally attributed to the growth of data collected by industry through the internet (e.g., web pages, social media), cell phones, and now the internet of things or IOT (i.e., sensors). However, perhaps the best indicator of the increase in the use of analytics in industry is the analytics and data science talent gap. Although several studies have attempted to quantify and predict analytics job growth (e.g., Manyika et al. 2011), a look at the present market gives a realistic picture of the current situation. A search we conducted on March 11, 2018, on the keyword “analytics” on LinkedIn Jobs resulted in 218,866 entries. Of the 218,866 entries, 4,567 were new (one day or less). Clearly, industry demand for analytical skills is strong.

We have seen a response to this talent gap from academia. Although academia has created several undergraduate degrees in analytics and data science, the number of new master’s degrees has exploded (Figure 2).

We note here that three types of programs are tracked in Figure 2: analytics, business analytics, and data science. For purposes of this work, we use the INFORMS definition of analytics as the scientific process of transforming data into insights for making better decisions. As such, it includes the entire spectrum of analytics: descriptive, predictive, and prescriptive methodologies and is decision-centric or problem-centric. We take business analytics to be the same as analytics but note that it specifically targets business as the application. Data science we take to be more of a cross between applied statistics and computer science. As opposed to analytics, data science is more data-centric than problem-centric.

How is academia doing in satisfying industry’s analytics needs? That question is the focus of this study.

In the next section, we discuss industry needs using the results of a text-mining study of job advertisements (ads). We follow that with a discussion of an in-depth study of master’s degree curricula. Next, we compare and contrast industry needs with what we are seeing in academia and then give a comparison of analytics, data science, and operations research (OR) industry skill sets. We finish with a summary and conclusion.

Industry Needs

The term “analytics” can have many different meanings to different constituents. For example, it can mean anything from traditional business intelligence (BI) (e.g., charts, visualization, reports) to predictive modeling (e.g., forecasting, data mining) to operations research (e.g., simulation, optimization) to specific applications of analytics (e.g., web analytics, marketing analytics). Rather than ask practitioners, who often cannot be identified easily and directly within a company, we decided to use job ads to ascertain the analytical needs of industry.

We used the services of Burning Glass Technologies, in particular, its Labor Insight™ Real-Time Labor Market Information Tool to collect 603,424 job postings for the three-year period from January 2015 to December 2017. Jobs were included if their job-title description contained any of the following four terms: analytics, big data, business analyst, or data scientist. We included these terms because we were interested in understanding the difference between analytics and data science, how the corresponding positions differ from a traditional business analyst position, and what might come under the broader banner of the term “big data.” To the best of our knowledge, our study of analytical job ads is the largest sample studied to date.

Based on word-frequency counts of a smaller pilot study and through several rounds of modifications, the author team reached consensus on a set of 16 custom topics. Each topic consisted of from one to 36 terms.

Figure 3 shows the list of 16 custom topics along with the percentage of the 603,424 job ads that contain one or more terms from that custom topic.

Communication and interpersonal skills were by far the skills mentioned most frequently, followed by managerial skills, database, business domain, modeling and analysis, and tools. These results are consistent with the results of the study of Sodhi and Son (2008), who conducted a similar text study of 1,056 job ads for OR. In their study, communication was the most cited skill, and operations management (a business domain) and modeling, statistics, programming, and database skills were also high in the rank order of skills listed in the ads.

In a recent study, Rienzo and Chen (2018) looked at 400 analytical job postings on Indeed.com. Their findings indicated that the top six skills for analytical jobs are SQL, Excel, project management, SAS, statistics, and database. We note that their terms are very specific relative to our custom topics. Excel and SAS were included in our tools topic, and statistics is in our modeling and analysis topic. SQL and database are included in our database topic. Project management is included in our managerial skills topic. Rienzo and Chen (2018) acknowledge that they did not include communication and that it, like project management, is a critical soft skill for analysts.

The emphasis on database and data handling is consistent with reports of how much time data scientists and business analytics professionals spend on data preparation. For example, a survey of 80 data scientists indicated that data preparation accounts for about 80% of a data scientist’s time (Press 2016).

Table 1 shows the breakdown of the results in Figure 3 by the four job-title keywords: analytics, big data, business analyst, and data scientist. This table shows the percentage of job ads that contain at least one word from the custom topic for each job title; we shaded those that are at least 25%. Not surprisingly, the table shows that big data and data scientist are similar; the exception is that big data ads place more emphasis on data handling and data scientist ads place more emphasis on tools. The less technical business analyst role differs substantially from the others in that it emphasizes business domain, managerial skills, database skills, and communication and interpersonal skills (notably, modeling and analysis is less emphasized).

Table 1. Percentage of Documents (Job Ads) that Contain at Least One Term from the Custom Topic by Job-Title Keyword (Analytics, Big Data, Business Analyst, and Data Scientist)

Analytics job ads emphasize database, managerial skills, modeling and analysis, communication and interpersonal skills, and tools. Except for managerial skills, this list is a subset of the skills emphasized in the data scientist ads. However, we note that although business domain did not make the 25% cutoff, it is mentioned much more frequently in analytics than in big data or data scientist ads (23% versus 4.5% and 7.2%, respectively).

Table 1 confirms that the increased emphasis on managerial skills and business domain in Figure 3 is attributed more to jobs in analytics and business analyst positions than to big data or data scientist positions. Furthermore, scripting and programming skills are more frequently listed for big data and data scientist. Database skills and communication and interpersonal skills are important across the board.

The results in Table 1 are consistent with the work of Harris and Mehrotra (2014), who found that the typical educational background of a data scientist is more programming-focused than that of an analyst. Interestingly, using follow-up interviews, Harris and Mehrotra (2014) state that, regarding data scientists, “they don’t see a need to explain or talk about the implications of their insights, which makes it difficult for them to partner effectively with professionals whose business expertise lies outside of the technical realm” (p. 16). Yet, as Table 1 shows, communication and interpersonal skills are often listed in job ads for data scientists.

Because we have such a large sample of job ads, we could do some analysis by industry sector. The seven largest industry sectors in our sample are consulting, manufacturing, finance (including insurance), banking, healthcare, retail, and high technology. Table 2 shows the percentage of job ads having at least one term from the custom topics by industry sector for analytics jobs.

Table 2. Percentage of Documents (Job Ads) that Contain at Least One Term from the Custom Topic by Keyword by Industry Sector for Analytics Jobs

Finance, banking, and healthcare focus on database, managerial skills, modeling and analysis, communication and interpersonal skills, and tools. It is perhaps not so surprising that finance, banking, and healthcare are closely aligned. Each of these sectors is a service sector and deals with highly sensitive information. Manufacturing and retail appear similar and have the same skills as we mention earlier in addition to more emphasis on business domain knowledge and business intelligence. High technology also emphasizes big data and scripting. Consulting has the most topics over 25%. This is not surprising because consulting cuts across all the other sectors.

Table 3 shows the percentage of job ads having at least one term from the custom topics by industry sector for data scientist jobs.

Table 3. Percentage of Documents (Job Ads) that Contain at Least One Term from the Custom Topic by Keyword by Industry Sector for Data Scientist Jobs

Data scientist jobs are very similar across all sectors with just a few differences. Like the results for analytics, data science jobs in finance and banking tend to require the same skills. Only healthcare meets the 25% threshold for data-handling skills (although manufacturing and high technology are close) and similarly for managerial skills.

We found no real differences across sectors for big data or business analyst positions.

We were curious how the emphasis on communication and interpersonal skills for these analytical jobs compares to other professions. Do all professions place so much emphasis on communication, or are analytical jobs different? To try to shed some light on this, we searched several other professions over the same period (2015–2017) for the existence of communication and interpersonal skill terms in ads for these professions. Figure 4 shows the results.

A higher percentage of ads in analytics and data science emphasize communication and interpersonal skills than in any of the other five professions we investigated. Industrial engineering is five percentage points lower, and sales associate, K–12 teacher, and nurse—all professions highly dependent on communication—have percentages lower than analytics. This suggests that, because communication and interpersonal skills are important but too frequently lacking in candidates, analytical job ads need to explicitly mention it more frequently.

Another interesting question is whether mentions of communication and interpersonal skills for analytics jobs are related to required years of experience. Are communication and interpersonal skills listed more often in jobs requiring less experience? We investigated this by segmenting analytics jobs by required years of experience (e.g., zero to two years, three to five years, six to eight years, more than nine years). We found virtually no difference in the percentage of job ads that mention communication and interpersonal skills across these segments. However, managerial skills are mentioned more often in job ads requiring higher years of experience (e.g., six to eight years and nine or more years).

Academic Programs

As Figure 2 shows, we have seen explosive growth in the number of master of science (MS) programs in business analytics, analytics, and data science. Although a number of MS programs in applied statistics, OR, and quantitative analysis have existed for a long time, the first MS analytics program was created in 2007, the first MS business analytics program in 2010, and the first MS data science program in 2013. MS in business analytics programs tend to be housed in schools or colleges of business whereas MS programs in analytics and data science can be stand-alone programs or they can be within a variety of colleges within a university.

To ascertain how well academia is responding to corporate needs, we conducted an audit of the curricula of MS programs listed on the North Carolina State website in April 2015. This consists of 21 MS analytics (MSA) programs, 33 MS business analytics (MSBA) programs, and 10 MS data science (MSDS) programs. We classified required and elective course work into 11 categories (Table 4).

Table 4. Definitions for Categories We Used to Classify Courses in the Curriculum Audit and the Acronyms We Selected

After categorizing courses using the definitions in Table 4, we followed up with two rounds of emails to all program directors requesting feedback on our categorization. Of the 64 directors, 29 provided feedback. In some cases, they confirmed our categorization; in other cases, they suggested changes that, in their opinion, more accurately reflected the content of their courses.

Because the total number of credit hours and total number of elective credit hours varies by program, we report the average percentage of required credit hours by category in Figure 5.

We focus on required credit hours because this is what every graduate is assured of taking by completing the program; as such, they represent the courses that the faculty members feel is fundamental to that type of degree program.

From Figure 5, we see the following:

• Statistics has the largest percentage of required coursework regardless of program type.

• Database management and data mining are prominent categories regardless of program type.

• Business analytics places relatively more emphasis on business fundamentals.

• Data science places relatively more emphasis on computing.

• In terms of experiential learning, programs more often include capstones than practicums.

• Few credit hours are dedicated to building soft skills regardless of program type.

Furthermore, these data perhaps give some support to the notion that data science is data-centric and analytics and business analytics are problem-centric. OR is problem-centric, and at least in a relative sense, analytics and business analytics programs place more emphasis on OR and problem context (i.e., business fundamentals). Data science places more emphasis on computing, data management, and data mining. This is consistent with the results from a recent study of undergraduate programs (Phelps and Szabat 2017). In addition, the data-centric data science programs engage more often in shorter curriculum-based practicum experiences, and analytics and business analytics programs engage most often in client-based capstone experiences that are problem-centric.

Gorman and Klimberg (2014) benchmarked 32 analytics degree programs in fall 2012 and winter 2013, focusing on master’s programs. Based on 21 universities with master’s programs in analytics, business analytics, and related degree programs, they classified the coursework into four categories: statistics, OR/MS, BI and (or) database, and general business. The percentage of coursework in each category was statistics (49%), OR/MS (26%), BI and (or) database (21%), and general business (4%).

As Figure 2 illustrates, the number of programs has increased rapidly since 2012–2013. What has changed in terms of academic content in these types of programs? We combined the categories to more closely match those used by Gorman and Klimberg (2014). From the categories we list in Table 4, we combined BI, COMP, and DB into an MIS-CS (Management Information Systems/Computer Science) category. We combined the STAT (Statistics) and DM into a Stat category and the PRAC and CAP into a category called experiential. Bus fund and OR remained unchanged as did the other (OTH) category. We note that for the Gorman and Klimberg (2014) study, experiential and other were not curricular categories, and program content encompassed the entire program whereas our study focuses only on required courses. Nonetheless, the average percentage of elective courses in our data are only 25.2%, so we feel there is still valuable information in this comparison (Figure 6).

The most startling difference noted in Figure 6 is the difference in the OR content as reported by Gorman and Klimberg (2014) from 2012–2013 data versus existing programs in 2015. The experiential component is a feature of most of the programs. Of the programs in our 2015 program data, 69% have an experiential component. It is possible that these were counted as OR by Gorman and Klimberg (2104); however, the reality is that many of these experiential courses use data mining and predictive analytics rather than OR techniques. In any case, it appears that there are significant differences in the academic content of programs that existed in 2015 when compared with those reported by Gorman and Klimberg (2014) using data from the programs that existed in 2012–2013. Comparing the two, we see the following:

• Heavy emphasis on statistical methods continues.

• Business analytics programs are placing more emphasis on business foundations (domain knowledge) as well as on data handling and programming skills.

• OR methodology is receiving much less emphasis.

This last point is interesting and concerning. A closer look at Figure 2 might offer an explanation as to why there appears to be less OR content now than in the study of Gorman and Klimberg (2014). The largest growth in programs has been in the MS in business analytics. Although some MS business analytics programs are offered by business school departments with a significant OR presence (e.g., the University of Cincinnati and the University of Tennessee), many are not. We believe that this development, coupled with the strong market emphasis on predictive analytics, has led many of the newer programs to focus more on statistics, predictive analytics, and data mining.

How Well Is Academia Satisfying Industry Needs?

Turel and Kapoor (2016) conducted a study of analytics offerings in 104 business schools to ascertain if business schools were providing professionals with the analytical skills needed by industry. That is, they attempted to address the skills gap, specifically for business schools, and assess the schools’ analytics maturity across skills in database, data analysis, business analytics, and data warehousing for both undergraduate and graduate programs. They recommend that business schools increase their offerings in data warehousing and processing, quantitative business analytics, and general and overview courses in business analytics. However, as their title suggests, they discuss analytics curriculum against presumed industry needs. In our study, we likewise address the analytics skills gap; however, we use a large sample of analytical job ads against a survey of curriculum in master’s programs. We do note that the call from Turel and Kapoor (2016) for more emphasis on data warehousing and processing is consistent with our finding that database and data handling appear relatively often in analytical job ads (Figure 3).

A comparison of Figures 3 and 5 provides insights as to the alignment of academic program curriculum with industry needs. Communication and interpersonal skills and managerial skills are the skills most frequently sought by industry (Figure 3), yet, on average, less than 5% of the academic curriculum is dedicated to so-called soft skills regardless of program type (Figure 5). This is an age-old problem that management science and operations research practitioners and academics have discussed for many years. For example, see Woolsey (1979), Levasseur (1991), Fellers (1996), Sodhi and Son (2008), and Liberatore and Luo (2013).

Programs that do include communication and interpersonal skills explicitly with credit hours dedicated to these concepts through a formal course use a variety of course titles. Some of these include communication strategies for analytics, business communication and analytics consulting, and project leadership and communication.

Most programs do not dedicate full courses to these soft skills. Like OR degree programs, analytics and data science programs are typically created by more technically oriented faculty (e.g., OR, applied statistics, information systems, computer science), and although faculty members often understand the importance of communication, they seldom choose it to replace other technical material in these degree programs. In addition, they face market pressure to keep programs as short as possible; therefore, adding more hours is typically not a desired option. Nonetheless, more programmatic emphasis needs to be placed on communication and interpersonal skills, including teamwork and leadership.

As Figure 5 shows, although MS business analytics programs place some emphasis on business domain knowledge, analytics, and data science, master’s programs allocate virtually no credit hours to business domain knowledge. Yet, based on the data in Figure 3, the need for business domain knowledge appears in nearly 30% of the job ads. The related curriculum issue is similar to that of the soft skills dilemma. The question posed when faculty are designing these programs is the following: if we offer more business domain knowledge, which technical courses will we leave out of the program?

Perhaps a creative solution is to include soft skills and business domain knowledge in experiential courses. Both soft skills and business domain knowledge can be incorporated into experiential courses; however, doing so requires focus and the deliberate dedication of an explicit amount of time in these courses to do so effectively. A project course that simply forces the students to be present in class is not likely to provide enough training on communication and interpersonal skills. There are best practices that can and should be taught and reinforced by practice.

To achieve a better understanding of methodological skills gaps, we provide a more detailed analysis of the modeling and analysis category from Figure 3. Figure 7 shows the percentage of job ads that contained the keywords included in the modeling and analysis set. Not surprisingly, machine learning, predictive models, forecasting, and statistical modeling appear high in the list of keywords shown (i.e., those occurring in at least 1% of the ads).

Given the relative lack of coverage of OR in the degree programs, perhaps more surprising is the presence of optimization toward the top of this list. This suggests that academic programs should include more emphasis on OR. With regard to optimization, the term could be interpreted more broadly than math programming. For example, it could also include the set of techniques included in the broader scope of prescriptive analytics, namely, simulation, optimization, and rule-based systems. Also, optimization is at the core of machine learning and in many ways is a prerequisite to understanding the technique. Hence, one could argue that even if not explicitly mentioned, machine learning makes knowledge of optimization more important.

Academic programs have focused on predictive analytics because most of the demand for analytics arises in this practice. This is confirmed by our text analysis of job ads because the combination of machine learning, predictive models, forecasting, and statistical modeling far exceed the other categories in the modeling and analysis set. This is further confirmed by the Google trends graph in Figure 8. The graph shows the extent to which interest in predictive analytics dominates interest in descriptive analytics and prescriptive analytics (as measured by internet searches). In this sense, the university programs have mostly responded promptly to the demand as articulated by industry at the onset of the analytics craze.

Several factors suggest that interest in prescriptive analytics may be increasing. First, there has long been a view that as a company matures in its use of analytics, it naturally progresses from descriptive to predictive to prescriptive analytics (e.g., Puget 2014). A natural consequence of this is that, as more companies mature in their use of analytics, prescriptive analytics, which really is the domain of OR/MS, will become even more important. Second, the interest in applying analytics to data generated from the IOT (Internet of Things) is definitely on the rise. Although much of the demand for predictive analytics has been around the explosion of social media data, point-of-sale data, and mobile-device data, the potential uses of data from the IOT tend to be more prescriptive. For example, GE Aviation uses sensors on aircraft engines to prescribe preventive maintenance and to optimize flight paths, and John Deere offers similar types of optimization services from machine and logistics optimization to help users of its farm equipment make better farming decisions based on sensors in this equipment.

Figure 9 incorporates the data from Figure 8 but excludes predictive analytics.

Is there evidence from job ads that the skill set sought by industry is increasing its emphasis on OR? To investigate this, we expanded the time horizon of our data set back to 2011. This resulted in an expanded time-series data set from 2011 to 2017, which includes 1,144,726 job ads.

We constructed three skill categories called OR (i.e., optimization, stochastic optimization, Markov chains, model building, simulation), traditional predictive methods (i.e., linear regression, generalized linear model, logistic regression, multiple regression, simulation, time series, forecasting), and a third containing data mining and machine learning. Figure 10 confirms that interest in OR is increasing but at about the same rate as the increase in traditional statistics and at a much lower rate than data mining and machine learning.

Although industry interest in prescriptive analytics may never come close to that of predictive analytics, interest in prescriptive is on the rise. Will academia react and include more prescriptive analytics? And will prescriptive analytics be taught only as traditional math programming and (or) optimization or include broader coverage of rule-based systems and perhaps even cognitive computing?

Our expanded data set also allows us to study the shift in software skills required over time. Figure 11 shows the percentage of job ads mentioning the open-source software Python and R as well as commercially available SAS. Mentions of SAS and R are increasing but at a much lower rate than mentions of Python.

Operations Research vs. Analytics and Data Science

We were curious as to how job postings in analytics and data science compare with those for OR. In terms of required skills, are jobs in OR more similar to jobs in analytics or data science, or are they equally dissimilar? Over the same period (January 1, 2015, to December 31, 2017) using the Burning Glass Technologies Labor Insight Real-Time Labor Market Information Tool, we found 4,953 OR job postings. Table 5 is a heat map comparing the percentage of jobs for analytics, data science, and OR for each of our 16 custom topics.

Table 5. Percentage of Documents (Job Ads) that Contain at Least One Term from the Custom Topic by Keyword for Analytics, Data Scientist, and OR

The table indicates that, for business domain, OR falls between analytics and data science, and OR mentions business intelligence, data handling, and database much less frequently than either analytics or data science do. In the areas of managerial skill, modeling and analysis, programming, scripting, and tools, OR is much closer to analytics than data science. Indeed, the median absolute difference over topics for OR is 9.2% for analytics and 14% for data science.

Figure 12 shows more detail for the modeling and analysis special topic for each of the job categories in Table 5. The side-by-side bar chart shows the percentage of job ads that contain a methodology in our modeling and analysis special topic for data scientist, analytics, and OR. The chart shows the heavier emphasis on predictive analytics for data science as opposed to the heavier emphasis on optimization and simulation for OR. Notice that analytics is never the highest percentage of ads on any of these methods; however, it is lowest for optimization, statistical modeling, econometrics, and simulation. For machine learning and data mining, predictive models, and forecasting, the percentage for analytics job ads lies between the other two job categories. Similar to machine learning and data mining for data scientists, the lack of mention of simulation in data science and analytics job ads is a key difference between those two job categories and OR.

Summary and Conclusion

In this work, we studied the industry needs in analytics via text mining of more than 600,000 job ads. The results indicate that communication and interpersonal skills and managerial skills are strongly cited in analytical job ads. This finding is consistent with the study of OR/MS job ads conducted by Sodhi and Son (2008). Furthermore, business domain knowledge and database skills appear in a higher percentage of job ads than modeling and analysis.

In terms of skill sets mentioned, job ads for big data and data science are somewhat similar. Business analyst ads focus more on business domain knowledge, database, communication and interpersonal skills, and managerial skills. In addition to those skills mentioned for business analysts, analytics job ads place an emphasis on modeling and analysis and more emphasis on tools. We found that emphasis on the skills for analytics can differ by industry sector whereas for data scientists, big data, and business analysts, this was not the case.

Although the most frequently mentioned methodologies tend to be more statistical, several OR methodologies (e.g., optimization and simulation) were among the methodologies mentioned more often. Mentions of OR techniques and traditional statistical techniques are increasing but at a much lower rate than data mining and machine learning.

The number of academic programs in analytics, business analytics, and data science have increased significantly since the study of Gorman and Klimberg (2014). Our audit of the curricula shows less emphasis on OR/MS and more emphasis on experiential learning and, in the case of business analytics, more emphasis on business domain knowledge. It appears that academia has responded well to industry needs in emphasizing predictive analytics; however, the long-standing issue of lack of coverage of communication and interpersonal skills remains an issue. The majority of master’s programs now provide some type of experiential learning, which is critical in creating practitioners who can add value from day one. Experiential learning is perhaps a venue that programs can use to place more emphasis on communication, interpersonal skills, teamwork, and leadership without increasing the total number of credit hours required for a degree. However, explicit coverage of these soft skills is required to ensure that proper skill levels are achieved.

Although predictive analytics remain by far the strongest industry need, interest in prescriptive analytics is on the rise. Relative to predictive analytics, there is little coverage of prescriptive analytics in master’s program curricula. Programs may need to adapt to include more OR/MS (e.g., optimization, simulation, decision analysis) as the use of analytics in industry matures.

Finally, by comparing job ads for analytics, data science, and OR, modeling and analysis and communication and interpersonal skills are heavily mentioned across ads for all three job types. Analytics and data science place more emphasis on database skills than does OR. Managerial skill is more emphasized by analytics and OR than data science. In terms of methodology used, the distribution across various methods was more evenly distributed for analytics than for either data science or OR. Data science heavily emphasizes machine learning and data mining; not surprisingly, OR emphasizes optimization, simulation, and statistical modeling. Relative to optimization, simulation receives relatively little mention outside of OR. Perhaps this could be attributed simply to the fact that optimization has entered the vernacular as the goal of much of analytics and data science.