Introducing Prescriptive and Predictive Analytics to MBA Students with Microsoft Excel

1. Introduction

The emergence of business analytics and artificial intelligence (AI) has fundamentally changed the requirements of a management education due to strong industry demand for personnel that have proficiency in business intelligence, quantitative analysis, and data engineering (Zhan et al. 2018, Dean 2020, Mitra et al. 2021). In fact, the field of data science, which did not formally exist in the early 2010s (Press 2013), is projected to grow approximately 28% through 2026 (Schroeder 2021). A nascent area is the manager who has enough low-level technical knowledge to understand the array of sophisticated analytics techniques that can be applied to data stores while also having a high-level perspective to see the managerial implications of the resulting analysis (Fitzgerald 2012). Although this seems to be an incredible opportunity for MBA programs, which have recently seen a decline in the number of applications (Thomas 2022), many business schools are struggling with how to incorporate analytics training into their MBA programming (Stine et al. 2019).

Although many management schools offer graduate degrees in data science, incorporating modernized analytics modules into the MBA curriculum is challenging. From a program design perspective, there are no accreditation standards for developing machine learning and business analytics courses (Xu and Babaian 2021). Furthermore, many business-focused AI textbooks put too much emphasis on the application of analytics and do not give sufficient detail as to how the algorithms actually work (Marr and Ward 2019). Alternatively, most academic AI resources are written for technical and/or mathematically oriented learners (Xu and Babaian 2021). Thus, there is tension between creating MBA courses that balance the need to have students understand analytics algorithms with teaching them how to interpret and communicate their findings (Langley 2019). There is also pressure to ensure that other functional areas of business knowledge are well represented in the curriculum (e.g., marketing, accounting, and finance), to offer unique courses in new and promising industry segments (e.g., digital transformation, sustainability, and entrepreneurship), and to incorporate extracurricular and networking opportunities (Kurczy 2018, Kirkpatrick 2020). As a consequence, there may be scant room for offering multiple courses on the topic of business analytics. The previous difficulties are further exacerbated by three main course-level challenges.

1.1. Students Require Prerequisite Knowledge

Prescriptive and predictive analytics are grounded in several quantitative disciplines such as algebra, mathematical programming, numerical optimization, statistics, and data mining. Furthermore, many introductory analytics courses assume students have some programming experience in Python/R or teach students how to code (Nagpal and Gabrani 2019). However, strong quantitative skills are typically lacking in MBA students (Weathers and Aragón 2019) and because programming is difficult to learn, a considerable time investment must be made (Murray 2014, Holzmen and Gooch 2016). Consequently, in some courses geared toward nontechnical learners, exercises and assignments often rely on partially constructed solutions if they contain difficult mathematical concepts or snippets of code that students do not fully comprehend (Langley 2019). Expecting that students have the required technical knowledge to succeed is clearly an impediment to encouraging them to seek out analytics training. Conversely, proficiency with Excel spreadsheets continues to be a technical area that graduate programs should place more emphasis on (Formby et al. 2017, Seal et al. 2020) even though most MBA programs require applicants to have a working knowledge of Excel on admittance (Questrom School of Business 2022).

1.2. Experiential Education is Valuable Even for Management Training

Businesses must look beyond the data scientist to develop talent such that managers can contribute to producing data-driven solutions (Miller and Hughes 2017). However, to fully understand what data analysts do, what type of insights are possible, and the many pitfalls that may arise during the data engineering and modeling process, students must be trained to perform prescription/prediction tasks. It has also been shown that business students both enjoy, and gain a deeper understanding of, quantitative material when a hands-on approach is used (Xu and Babaian 2021). Nevertheless, when providing MBA students with opportunities to implement analytics models for difficult data-grounded problems, it is essential to have them also acquire skills related to problem conceptualization, model formulation, and the dissemination of insight (Fry 2008). Thus, the ability to craft application-driven learning experiences is crucial to having MBA students understand how to better manage a team of analytics professionals that work together to make data-informed decisions that improve the efficiency and operations of organizations (Longwood University 2021).

1.3. Modern Analytics Workflow

There are many aspects to data-driven decision making that must be touched on even for management-level knowledge. For instance, the education literature suggests that students should become proficient at assessing data quality, identifying relevant managerial questions, determining whether the data can be used to address the proposed questions, managing data sets, performing quantitative analyses, and interpreting model outputs (Weathers and Aragón 2019). Although these concepts aptly summarize the data acquisition, data preparation, and the model engineering (i.e., train, test, evaluate) pipeline, there is also a fundamental difference between building analytics models that are used to extract managerial insight, and deploying a trained model to generate business value (Visengeriyeva et al. 2020). Thus, MBA students should be aware of how analytics models can be deployed in production environments. Although this involves an introduction to machine learning operations (i.e., ML Ops), it also puts more emphasis on the integration of domain specific knowledge, both in the implementation of the AI tool and in determining what high-level insights can be derived from its deployment (Langley 2019).

This paper describes an application-driven course that has been developed to introduce MBA students to optimization and machine learning concepts using the unifying language of mathematical programming. Although this lens has been introduced at a PhD level (Bertsimas and Dunn 2019), it is argued that this perspective is also valuable for teaching at the MBA level. As a consequence, students are not required to have prerequisite programming knowledge nor do they need to demonstrate advanced mathematical proficiency. Furthermore, all analytics models can be implemented with Microsoft Excel, which strengthens their competency with the software. The course provides students with many opportunities to apply the tools they learn and repeatedly emphasizes the business insight that can be obtained from rigorous quantitative analysis. Thus, it differs from the literature in that it presents more detail than many courses designed for nontechnical learners (Way et al. 2017) while also focusing on managerial issues such as quantitative storytelling, the model engineering process, and deployment; these topics are typically omitted from graduate-level curricula (Mike et al. 2020). In sum, the course teaches students how to obtain and communicate high-level analytics-based insight for real-world applications using experiential learning activities.

2. Background and Motivation

In the first year of their MBA at the Schuilch School of Business, students are required to complete two mandatory courses that relate to data analytics and operations management. The first is entitled Quantitative Methods and teaches business numeracy; this includes topics such as descriptive statistics (measures of location and association, distributions, variability), inference (simple and multiple linear regression), and decision trees. The goal is to cultivate a basic understanding of the methods underlying data-driven decision making, have students gain an appreciation of their use in addressing real-world business problems, and provide opportunities to implement these methods using Microsoft Excel. The second course is entitled Operations Management and presents topics in operational strategy, process analysis, quality management, introduction to supply chains and production systems, and project management. The objective is to teach students foundational knowledge in managing operations. Both are quarter-year courses with six lectures, several assignments/case studies, and at least one individual exam. Although the specific topics may vary from year-to-year, variants of these courses can be found in many MBA curricula (Herrington 2010).

The elective half-year course, entitled Models and Applications in Operational Research (MAOR) is the primary focus of this teaching article although several related modules have been created for other courses. MAOR constitutes a survey of topics related to operational research. The emphasis is placed on analyzing practical problems and obtaining business insight rather than on proofs or exploring mathematical properties. The application areas are varied and include, but are not limited to, financial planning and portfolio optimization, inventory and production, marketing analytics, transportation, healthcare, revenue management, supply chains, and economics.

From a technical perspective, MAOR has historically focused on teaching mathematical optimization, decision trees, and Monte Carlo simulation. However, prior to the redesign of the curriculum, enrollment was low (10–15 students) and the examples were somewhat outdated. In fact, there had not been any substantial revision to the course content for many years and several topics overlapped with existing courses. More specifically, AI and its integration with society emerged as a strategic focus of the Schuilch School of Business and the university more broadly (York University 2022a, b). As a consequence, other courses were revised to meet this objective. For instance, the coverage of decision trees was moved to Quantitative Methods and Monte Carlo methods are now taught in Applications of Data Science in Finance. Thus, MAOR needed to be updated and brought in line with the strategic focus of the program, school, and the university.

The philosophical underpinnings behind the revised version of MAOR come from several sources including the author’s own research and educational background, a review of the relevant literature (Warner 2013, Laursen and Thorlund 2016, Carillo 2017), as well as discussions with colleagues within and outside the institution. The following five principles helped guide the redesign:

1. Data, as opposed to models, are the focus. This perspective is championed in the book The Analytics Edge (Bertsimas et al. 2016) and by many industry experts (e.g., Telenti and Jiang 2020).

2. Students should be trained to become “data-savvy managers” (Manyika et al. 2011) that contribute to producing data-driven solutions. That is, students who successfully complete the course should have the requisite knowledge to understand the relationship between business problems and the application of analytical tools used to extract insight (Wilder and Ozgur 2015b). Thus, although students should receive hands-on experience implementing prescriptive and predictive algorithms to fully understand the process associated with performing analytics, the goal is to create managers and not graduates that will eventually become implementation experts (e.g., data scientists).

3. The course should appeal to students without a strong mathematical background. Introducing data analytics to nontechnical learners is difficult due to persistent concerns about the math requirements (Roth and Matherne 2021). By focusing on mathematical programs (prescriptive) and machine learning models formulated as optimization problems (predictive), a unified and modern treatment of optimization and machine learning can be presented (Kopcso and Pachamanova 2018, Bertsimas et al. 2019). Furthermore, only rudimentary mathematical knowledge is required as constrained optimization has been successfully taught in MBA programs for decades (Liberatore and Nydick 1999, Regan 2005). Finally, the approach enables a rich discussion of the difference between prescriptive and predictive models, emphasizes the analytics pipeline and the varying ways in which data are used (Jurney 2017), and allows for the deliberation of the societal concerns associated with AI, such as model interpretability (Bertsimas and Dunn 2017, Bertsimas and Wiberg 2020, Bertsimas et al. 2021), prejudice (Corbett-Davies and Goel 2018, Manyika et al. 2019, Kaushal et al. 2020), and issues with causality (Shah 2019, Cousineau et al. 2023).

4. Contrary to introductory courses that require some prerequisite knowledge of programming (Boutilier and Chan 2023), the course should appeal to students without any formal coding background. Typically, Microsoft Excel is regarded as a software that is exclusively associated with performing descriptive and prescriptive analysis (Paul and MacDonald 2020). However, with the advent of predictive analytics and its widespread usage/appeal, this is slowly changing (Brusco 2022, Naderi et al. 2022). Furthermore, Excel remains one of the primary tools for data analysis (Leong and Cheong 2008, Montoya 2022). Thus, because of its pervasive use in industry and management education (Wilder and Li 2019, Patrick et al. 2019, Huggins et al. 2020), all examples and coursework should be completed within the Microsoft Excel ecosystem.

5. Because it is an elective within the Operations Management and Information Systems department, the course should highlight the deep connections between analytics and topics within the operations and supply chain management field (LeClair 2018). That is, students should become adept at being quantitative storytellers, that is, communicators that have the skill set to liaise with both data scientists and senior management on operations-focused issues (Olsen-Phillips 2022).

The revised course was first taught in Winter 2019 with 26 students (12 completed course evaluations) and Fall 2021 with 28 students (15 completed course evaluations). Although the course is typically offered in-person, the Fall 2021 version was fully online due to the COVID-19 pandemic.

3. Learning Objectives and Course Design

The purpose of MAOR is to provide students with an introduction to modern analytics through the analysis of real-world problems focusing on operations management. The course discusses data (acquisition, validation, and preparation), model engineering (training, testing, and evaluating prescriptive/predictive models), and model deployment. It consists of a mix of mini-lectures (videos) together with discussions of real-world applications (in-class sessions) focusing on decision situations. The central theme of the course is to have all topics motivated by circumstances that require evidenced-based decision making. Thus, students develop a sense of how to approach complex problems of a technical nature. Furthermore, a ubiquitous feature in modern management is the central role that data plays in driving core business processes. Consequently, students learn how to effectively analyze data sets and implement analytical tools using a mathematical programming framework.

The course is organized into two modules. The first module introduces students to optimization technology for prescriptive analytics and includes topics such as linear programming, sensitivity analysis, mixed-integer linear programming, goal programs, and nonlinear programming. Many application areas are covered including inventory modeling, price optimization, production, transportation, process analysis, scheduling, and supply chain management. Several topics in the first half of the course are somewhat standard and are covered in MBA-level operations courses such as decision analysis (Stern School of Business 2021) and spreadsheet modeling (Rotman School of Management 2022). This is purposeful: not only are these topics taught at a level where larger-scale optimization models for more realistic-sized business problems are solved, but they act as a scaffold for creating prediction models using optimization technology.

The second module begins with a discussion of data acquisition, preparation, and wrangling and subsequently takes the view that machine learning is the application of optimization technology on historical data stores. As a result, supervised and unsupervised learning models are formulated as mathematical programming problems (Gambella et al. 2021). Topics include regression (least squares, quantile, least absolute deviation, k-nearest neighbors), regularization (best-subset selection, ridge, lasso), classification (logistic, support vector machines, k-nearest neighbors), unsupervised models (k-means clustering, principal component analysis), machine learning operations, and modern examples of AI systems. Many applications are introduced including human resource management, worker/student performance, revenue management, healthcare process analysis, quality management, and product forecasting. Moreover, the optimization framework is applied to several data sets to obtain managerial insight. The limitations and benefits of using an exact approach are discussed (e.g., certificate of optimality, added expressiveness of the mathematical language), as well as the process of deployment (giving analytics-based recommendations versus deploying a trained model). Finally, the link between prescription and prediction is emphasized (i.e., predict-then-optimize) to connect the perspectives. All computational analysis is performed in Microsoft Excel using the native solver tool and with OpenSolver, an open-source alternative.

The course objectives are as follows:

• To expose students to the main concepts of quantitative reasoning in a business environment.

• To provide students with opportunities to improve problem solving and critical thinking skills and become expert communicators of quantitative information, for example, acquire the ability to effectively liaise with data scientists, machine learning engineers, and senior managers alike.

• To develop a proficiency with mathematical models and computational tools for prescriptive and predictive analytics with a focus on improving their knowledge of Microsoft Excel.

• To understand the relationship between mathematical programming, prescriptive/predictive models, and the role that data plays in problem formulation and model interpretability.

• To facilitate an understanding of the business processes that support data-driven decision making and to make students aware of potential issues (e.g., model bias, data leakage).

3.1. Course Materials

There are no required textbooks for this course. This is because of financial considerations, the lens MAOR uses to introduce concepts in business analytics (i.e., the unifying language of mathematical programming), and because, as discussed in the Introduction, AI textbooks are typically too technical for business students or they do not present enough detail with respect to the underlying algorithms. Instead, students are provided with a detailed slide deck (∼100 slides per meeting session), several handouts, and multiple Excel spreadsheets that have worked examples of all problems discussed in class. In addition, a package of completed problems is provided for self-study (exams). An example slide deck covering mixed-integer linear programming (MILP) models (Week 5) and machine learning regression (Week 9) is included in the supplementary materials, as well as a handout on sensitivity analysis (Week 2) and three Excel files (one example is from Week 5 and two examples are from Week 9).

3.2. Computational Considerations

When solving problems motivated by real-world applications, it is often necessary to formulate large-scale models (e.g., hundreds or thousands of decision variables and/or constraints). For instance, constructing a mathematical program to solve a transportation problem (prescription) with 50 nodes may require thousands of decision variables. Alternatively, creating a quadratic program to solve a soft-margin, support vector machine (prediction) requires there to be at least as many decision variables as the number of data points and features (Gambella et al. 2021).

Unfortunately, Microsoft Excel solver is limited to 200 decision variables and 100 constraints. Thus, to solve moderate- to large-scale problems, the open-source add-in OpenSolver is used (Mason 2012). Not only is it easy to install, but it has no limit on the number of variables or constraints that can be incorporated into a model. The interface is similar to the native solver tool (Figure 1) and the advanced version has access to several powerful nonlinear solvers. This is especially important when formulating machine learning models as their objective functions are frequently nonlinear (e.g., logistic regression, ridge regression). Furthermore, its open-source status means that OpenSolver can be used in industry for noncore projects. Finally, there are Excel connectors for industrial solvers (Gurobi Optimization, LLC 2022) which implies that students may see similar interfaces if they are employed in organizations that have licenses for these software tools.

Nevertheless, it’s important to note that OpenSolver is not a panacea. Although there has been significant speed improvements in constrained optimization solvers, linear and quadratic programs in particular, state-of-the-art implementations of machine learning models as optimization problems still only contain hundreds to a few thousand data points (Bertsimas and King 2016). Although this is sufficient for many applications and for an introductory MBA course on analytics (Pachamanova et al. 2022), the result is that the course does not delve into problems where big data sets are required (Deng et al. 2009). Thus, it is recommended that all models, especially assigned material, be run on several machines (MacOS, Windows) to determine how long it takes OpenSolver, or the native solver tool, to find an optimal solution. Then, an instructor can modify the problem (e.g., reduce the size of the data set, use a different solver) if the runtime is too long.

In Appendix A, more detail is provided regarding the use of OpenSolver, features that have been particularly important for the course, and issues that instructors should be aware of.

3.3. Weekly Sessions

The course is delivered over a 13-week semester. There are 12 two-hour meeting sessions, a midterm, and a final exam. An example course outline is provided in Table 1. Prior to each meeting session, students are required to watch a 20- to 40-minute video that contextualizes the upcoming in-class session. The video focuses on introducing a specific prescriptive/predictive technique, the settings under which it is appropriate, what/how data are used, potential pitfalls, and discusses how the model can be implemented in Excel. During the meeting session, one to three real-world applications that use the technique are introduced. Each example is structured similarly: problem definition, identifying relevant data, data preparation and transformation, model engineering, discussion of managerial insights, and the communication of business knowledge.

Table 1. Example Course Outline

This method of pedagogy is known as the flipped classroom (Bishop and Verleger 2013, Akçayir and Akçayir 2018) and is particular advantageous in this setting as it allows students to learn at their own pace, makes course content more easily accessible, and meeting sessions are almost entirely devoted to (i) introducing a real-world application; (ii) solving a data-driven analytics problem; and (iii) contextualizing the example within the MBA curriculum to build an understanding of the challenges that may be associated with encountering the problem outside the classroom (e.g., issues with data acquisition, combining multiple competing perspectives). Furthermore, for each problem, a spreadsheet associated with the implementation of the solution is provided. Apart from being extensively used during class, it is a great resource for self-study in that students can review the steps associated with data wrangling, identify the actions needed for implementing the analytics model, and experiment with alternative formulations (e.g., adding/removing constraints, different types of regularization). This helps to improve their Excel proficiency and provides a template for producing spreadsheets with a similar structure (e.g., assignments/exams).

3.3.1. Detailed Outline—Prescriptive Analytics Session: MILP Models (Week 5).

From a technical perspective, the purpose of this week’s module is to have students realize that mathematical programs can include many types of decision variables (continuous, integer, discrete). The implication is that very general models can be created whose decisions only depend on what type of managerial insight one would like to learn. That is, models are not limited to one type of decision variable but, instead, depend on the business decisions that need to be understood/made. The prelecture video touches on this idea noting that the computational complexity of the underlying algorithm is no more difficult than solving an integer or binary program (the previous week’s class). What’s difficult about this class of problems, however, is that to faithfully represent the real-world situation, the different types of decision variables need to be linked. Thus, a new class of constraints is required. To understand how these linking constraints manifest in prescriptive problems, several real-world examples are introduced during the in-class session.

During the two-hour class discussion, two to three examples are introduced that pertain to some aspect of operations management. They can include make versus buy decisions (Agrawal 2006), building plan design (Ngowtanasuwan 2012), production and transportation, and volume discounts (Terwiesch and Cachon 2019). Each example is derived from a research project and/or a consulting experience of the instructor or their colleagues, which means that there is a rich discussion of the setting that motivates each problem. All examples follow the same rubric: problem description, definition of decision variables and objective function, creation of constraints, introduction of the linking constraints, formulation in Excel, and analysis of the optimal solution.

The in-class examples give students the opportunity to formulate and solve a mathematical programming problem, and thus, they get hands-on experience at model building in a low-stress environment. Furthermore, the problems incorporate linking constraints in different ways which further demonstrates the generality of mathematical programs. Because the size of the models and level of involvement of the examples can be slightly above the typical introductory operations course offered in MBA programs, students expand their quantitative problem-solving skills while also improving their competency with Excel. Finally, each problem requires data for model parametrization which leads to a nuanced discussion regarding how this information can be collected in practice.

From a managerial point of view, the flipped classroom environment allows for a significant interaction. Furthermore, after solving each problem, there is a rich conversation of next steps (e.g., how to interpret the results, follow-up analyses that could be performed to demonstrate the robustness of the findings, the ways in which the results can be communicated to management). Finally, operational insights motivated by the solution are considered. For example, in the production and transportation example, a dog food manufacturer must decide which production facilities to operate (binary decision variables) and which customers to serve from which operating plant (continuous decision variables). The optimal solution has that a small subset of all production facilities are used. Furthermore, the students observe that some customers receive dog food from production facilities in a distant city even though there is an option to rent a facility in the same city, whereas other customers are not served at all. The example highlights how the high fixed cost of renting a production facility is driving the model to minimize the number of rented plants. In some cases, the total cost of production is such that it makes sense not to serve a customer. However, if variable costs were higher and fixed costs were lower, the insights may differ. Thus, model interpretability is important as it provides visibility into why an analytics tool produced a particular solution.

For a detailed discussion of the Production and Transportation example, see Appendix B.1 and the corresponding Excel spreadsheet provided in the online appendix.

3.3.2. Detailed Outline—Predictive Analytics Session: Regression (Week 9).

From a technical perspective, the purpose of this week’s module is to have students realize that mathematical programs can be formulated to solve prediction tasks. This concept is introduced in the previous two sessions with a discussion of empirical risk minimization and the presentation of ordinary least squares (OLS) regression as an unconstrained quadratic optimization problem. However, it is expanded on in the prelecture video where mathematical programming formulations of regularized regression models (lasso, ridge, and best subset selection) are introduced.

To be more concrete, suppose that a data set has been collected with N instances and J features. The data have already been cleaned and the features transformed to their final values. Let the outcome variable of interest be y_i and define the value of feature j associated with instance i to be x_ij for $i=1,\dots ,N$ and $j=1,\dots ,J$. Then, the objective function for the regularized models are

where α and β_j for $j=1,\dots ,J$ are the decision variables. Suppose t represents a budget parameter that controls the size of the regression coefficients. In contrast to OLS regression, the budget parameter is introduced into the framework along with constraints whose purpose it is to shrink the magnitude of the regression coefficients such that only the features that are most useful for prediction will be nonzero. This addresses problems associated with the multicollinearity of feature variables, overfitting to the training set, feature selection, and helps to obtain a more parsimonious and interpretable model. In particular, the following constraints can be added:where M represents some sufficiently large number and $w_j\in \{0,1\}$ for $j=1,\dots ,J$ are binary decision variables that indicate whether feature j is included in the optimal solution. The constraint for ridge regression is quadratic, whereas the other constraints can be linearized.

In the prelecture video, apart from presenting the mathematical programming formulation for each regression model, insight is given as to the motivation for the constraints, when each model should be applied, and several real-world studies that utilize the different regularization techniques. In the subsequent in-class session, an example is introduced where students apply the models to a historical data set of academic performance associated with high school students in Portugal. The data set must first be cleaned and some of the variables transformed before it is separated into a training and testing set. The final data set contains 33 features and 395 records, which is split using the 80/20 rule. Then, students implement each regularized model in Excel including two benchmarks: (i) predicting the mean response of the training set and (ii) OLS regression. After implementation, students evaluate the results and how changes to the budget parameter affects model performance. The quantitative discussion concludes with students having to choose the best-performing prediction model, motivating that choice (e.g., lowest root mean squared error), and identifying the most important features for predicting grades, that is, home life situation, presence of both parents, parental education levels, rural versus urban location, school reputation, whether the student has university aspirations, number of absences, and the amount of study time.

After the quantitative analysis, students must indicate how they would go about presenting these findings to parents and school administration alike. This discussion focuses on what is controllable (e.g., number of absences, study time) versus intractable factors that cannot be easily changed (e.g., home life situation, presence of both parents, parental education levels, rural versus. urban location). The classroom conversation eventually leads to a broader dialogue on how to incentivize individuals, such as employees, to achieve desirable performance (Sommer and Loch 2009, Lee and Zenios 2012). The discussion also introduces behavioral nudges (Thaler 2018, Fishbane et al. 2020), a more recent mechanism that is used to indirectly influence behavior, such as Uber rewarding badges to incentivize drivers to work longer hours and Deliveroo using push notifications to nudge to their food delivery workers’ into working faster (Möhlmann 2021).

Two additional models are introduced during the in-class session. The first is the formulation of quantile regression which is motivated by the desire to predict the median response (e.g., house prices) and to create prediction intervals. The corresponding model is a linear program:

where u_i and v_i are decision variables that correspond to the amount of overage and underage associated with predicting a quantile τ for instance i, respectively. The model is implemented in Excel and applied to the academic performance data set to create 95% prediction intervals. The subsequent classroom discussion focuses on the difference between confidence intervals and prediction intervals and suggests that providing these values helps with managerial decision-making. That is, prediction intervals help communicate the uncertainty associated with the mean prediction.

Finally, an implicit assumption regarding both quantile and regularized OLS regression is that the approximator is a linear function of the features. Although more complex models can be introduced in advanced classes, such as mathematical programs for decision tree learning (Dunn 2018, Aghaei et al. 2020), their formulation is too complex for the MBA audience. Instead, a nonparametric model (kNN regression) can be formulated where the function class does not need to be specified:

where d_i represents the precomputed distance (e.g., Euclidean) between a point whose response needs to be predicted and each instance in the training data set, and k represents the number of neighbors used to estimate the mean response u. The benefits (e.g., the more data that is collected, the more accurate the model becomes) and drawbacks (e.g., the more data that is collected, the slower the computational performance becomes) of using a nonparametric approach to prediction are presented and the model is, again, implemented in Excel and applied to the academic performance data set. A brief discussion of the effect that k (the number of neighbors used to predict the mean) has on the optimal solution is undertaken and alternative choices to determine the response are suggested (e.g., maximum, minimum, or median value).

The session concludes by presenting an overview of the data wrangling and model engineering process, and contrasts the role of data in creating predictive/prescriptive models. For a detailed discussion of the Regularized Regression and kNN Regression examples, see Appendices B.2 and B.3 and the corresponding Excel spreadsheets provided in the online appendix.

3.3.3. Discussion Sessions (Week 8 and Week 12).

There are two sessions that do not follow the flipped classroom structure. These sessions present high-level content on the data acquisition and preparation process, and model engineering (train, test, evaluate), as well as details associated with deploying analytics models in practice (i.e., user insight vs. the creation of an AI tool). Week 8 acts as a transition session between the presentation of prescriptive (Weeks 1–7) and predictive (Weeks 9–12) models. Students are introduced to noteworthy concepts associated with the process of creating prediction tools such as transforming data, model engineering, assessing out-of-sample performance, data leakage, model bias, and the bias-variance tradeoff. This is especially important for MBA graduates if and when they manage multiple analytics professionals (e.g., data scientists, machine learning engineers) on large AI projects. Furthermore, the final session (Week 12) focuses on the end result of analysis, that is, whether the models are used to obtain managerial insight or are deployed. Concrete suggestions on effective communication strategies are given such as ensuring that recommendations and business insights are clearly communicated, the importance of creating visual aids, the use of outlines to systematically organize complex ideas, and how to balance the amount of detail that is presented (e.g., main text versus appendix). In addition, a few real-world examples of customer-facing AI tools are showcased. The classroom conversation highlights the purpose of each tool and the potential ethical issues associated with its use. A summary of the class content is given in Appendix C.

3.4. Assignments and Exams

There are five deliverables in MAOR: three assignments worth 12% each and two noncumulative exams worth 32% each. Using Table 1 as a guide, the first assignment covers weeks 2–4, the second covers material from weeks 5–7, and the last assignment focuses on weeks 8–11. Because of time constraints, machine learning operations is not assessed. Finally, the first exam focuses entirely on prescriptive modeling, whereas the second exam covers data engineering and predictive modeling.

All work is submitted individually, although students are encouraged to consult with their peers on assignments. Deliverables have the same format: they contain two to three questions that introduce a modern problem, provide real data, and require both technical and managerial answers. Each question has students perform data preprocessing and wrangling, necessitates the implementation of one or more prescriptive/predictive models, requires the evaluation of the proposed models, and asks several high-level questions about the implications of the results. Deliverables are submitted online, and the only difference between assignments and exams is that the latter has a shorter time interval for completion. That is, all deliverables are take-home, open-book assessments that are completed using Microsoft Excel. For more details, see Appendix B.4 and the online appendix, which includes a sample exam and the corresponding Excel spreadsheet.

3.5. Consulting Project

There is currently no culminating project in MAOR. The reason is that it is a second-year elective course and, during this year, MBA students are required to complete an intense capstone project that liaises with a real company and conducts a comprehensive strategic assessment of all functional organizational areas. Because students analyze many aspects of a business and make recommendations to management in a strategic action plan, adding another consulting project to their workload would be placing an undue burden on their already packed schedules. Furthermore, there may also be conflicts of interest that would greatly increase the logistical complexity of MAOR.

Nevertheless, in many courses, consulting projects with real companies are incorporated into the curriculum (Boutilier and Chan 2023). This can be an extremely valuable learning experience as it exposes students to the analytics pipeline (problem definition, data acquisition, model engineering) while emphasizing soft skills such as the writing of technical reports and presenting analytical findings to management. To facilitate these types of experiences, instructors can collaborate with their department’s career center or use online platforms such as Riipen (https://app.riipen.com), which connects educators to industry partners so that a real-world project can be designed as an in-course assignment (both approaches have been successfully used in other courses taught by the author).

4. Student Feedback and Course Modifications

To determine the effectiveness of the course curriculum, feedback from the online student evaluation system is examined. The program solicits both quantitative and qualitative responses on the efficacy of the instructor, the utility of the course content, and how closely the student’s experiences match with their expectations. Changes to the course, driven by both formal/informal student feedback and structural modifications induced by the COVID-19 pandemic, are also discussed.

4.1. Student Evaluations

To determine the success of the curriculum, a subset of the quantitative course evaluation questions that most directly relate to content design and delivery were selected. This includes the structure of the weekly modules, the effectiveness of the deliverables, and the ability of the instructor to achieve the stated course objectives. The mean course score, department score, and faculty score are recorded with the best assessment being seven and the worst being one (i.e., a seven-point scale is used as focus groups at the school indicated that students desired a more fine grained set of distinctions compared with the five-point scale). The same set of questions are given to all courses offered by the business school. Results for the following 10 questions are presented in Table 2.

• Q1: The course materials (e.g., course kits, textbooks, readings, audio visual materials, laboratory manuals, websites, etc.) helped me achieve the course objectives.

• Q2: The course activities (e.g., lectures, discussions, simulations, assignments, exercises and presentations, etc.) helped me achieve the course objectives.

• Q3: The course tests/exams or final paper/essay were directly related to the course objectives.

• Q4: The course helped me grow intellectually.

• Q5: The course encourages critical thinking skills such as problem-solving definition, decision making, judgment, analysis and evaluation.

• Q6: The instructor was organized and well prepared.

• Q7: The instructor presented ideas and concepts clearly.

• Q8: The instructor showed enthusiasm for the subject.

• Q9: The instructor dealt effectively with student’s questions and comments.

• Q10: The instructor created and maintained an environment that was conducive to learning.

Table 2. Student Evaluations Scores (Seven-Point Scale) for 10 Questions Most Directly Related to Content Delivery

For every question, significantly higher evaluations for MAOR are observed compared with other courses offered by the department and within the business school. The final three rows of Table 2 present the average scores for all 10 questions. Student t statistics and p values corresponding to a one-sided hypothesis test are computed. The null hypothesis is that the observations from MAOR and the department/faculty courses come from populations with the same mean but unknown and potentially unequal variances. Although the tests are underpowered (the population size is two), they do provide some statistical evidence to suggest that the average course score is higher than the average department, $t(2)=7.209,p=0.0094$, and faculty score, $t(1)=5.566,p=0.057$.

Informal discussions with students in MAOR indicate that the favorable evaluations are because of the perceived relevance of the topics covered in the course and the accessibility of the curriculum (e.g., Excel versus Python, emphasis on obtaining business insight versus exploring mathematical proofs/structure). The scores from Table 2 suggest that students are very much engaged with, and interested in, the material and that from the student perspective, the course design is effective at developing problem-solving and critical thinking skills. Finally, the analysis underscores the fact that although some aspects of the course are novel (in particular, representing machine learning models as mathematical programming problems) and, in many cases, there are few resources on the topic, students are satisfied with the support they have received (lecture slides, videos, and in-class sessions) and are intellectually stimulated rather than overwhelmed by the content.

Although student evaluations can be a biased indicator of quality (Wachtel 1998) and their validity in evaluating teaching effectiveness has been repeatedly questioned (Hornstein 2017), the results do indicate that the course has been successful at achieving the desired objectives. Furthermore, research suggests that there is a correlation between high course ratings in student evaluations one year, and a subsequent increase in a course’s overall enrollment in the subsequent year (Brown and Kosovich 2015, Yu et al. 2021). Although MAOR has not been consistently offered, the quantitative evaluations indicate that it may be quite popular once teaching schedules stabilize.

In addition to the quantitative analysis of student course evaluation scores, a thematic analysis of student comments was performed (Braun and Clarke 2006). The relevant question, and student responses to that question, are given in Table 3 where each quote best captures multiple students’ sentiment. The qualitative analysis further suggests that the course has been successful at achieving the learning outcomes and following the five philosophical principles that helped guide the redesign. The results indicate that MAOR has made students more excited about the techniques and applications related to the field of data analytics and it is apparent that, after taking the course, student’s still believe that the concepts they learned will be useful in their future careers. A common theme is that student’s appreciated the balance between the technical and managerial content, enjoyed the opportunity to implement analytics models, and valued the materials that were provided to them. In fact, the ability to complete assignments in Excel was especially important.

Table 3. Qualitative Comments from Student Evaluations

4.2. Student-Driven Course Updates

After the Winter 2019 version of the course, a theme emerged that slightly changed the curriculum for Fall 2021. The following comment summarizes the motivation behind these changes:

“I think more time could be spent on predictive analysis (machine learning), as it is more useful and difficult to learn. I feel it needs more time to teach and learn this topic.”

In the Winter 2019 version of MAOR, there was considerably less discussion of empirical risk minimization (Week 7) and machine learning operations (Week 12), whereas dimensionality reduction was not covered at all (Week 11). After adding these topics, the same sentiment was not expressed in the Fall 2021 course evaluations. Nevertheless, additional changes are being considered. In particular, efforts are focused on determining whether goal programming (Week 6) should be replaced. The original motivation for including this topic is because deviational variables are seen in quantile regression (Week 9). However, the discussion of soft constraints can be incorporated into some of the examples associated with linear programming (Weeks 3–5) and be reinforced in Week 9.

Two potential candidates have emerged as replacements: stochastic programming and an additional session on data wrangling. With stochastic programming, more emphasis can be placed on how data are used in prescriptive analytics. There are also connections with the discussion of empirical risk minimization and the structure of supervised learning models. Finally, a similar lecture is given to students in a different graduate program with identical technical backgrounds. Alternatively, by including another session on data acquisition/wrangling, more emphasis can be placed on how to determine what data are required to answer complex business problems. Furthermore, additional detail can be given as to how to clean and visualize data for model engineering.

A few technically proficient students expressed their desire to see Python incorporated into the course curriculum. This feedback has not, and will not, be incorporated for two main reasons. First, a majority of students enjoyed the coverage of Excel (see, for example, Table 3). These students indicated that they were happy to improve their proficiency with spreadsheet modeling and, more importantly, would not take the course if coding knowledge became a prerequisite. This suggests that, for the most part, the course is appealing to the desired audience. Second, there are currently two masters programs offered by the business school that require students to complete more mathematically and computationally challenging analytics courses. Knowledge of Python/R is expected and MBA students are welcome to take those courses as electives if they desire.

4.3. Case-Based Assignments

Although the course deliverables are motivated by real-world applications, none of them are case based. To further bolster the managerial content of MAOR, assignments and/or exams could be derived from case studies. For instance, there are a plethora of cases at Harvard Business Publishing that require mathematical programming and machine learning methods. In fact, the author is exploring whether future iterations of the course will replace the assignment component with case studies. One of the reservations associated with making this change is that, typically, case studies are completed by groups of MBA students. As a consequence, not all students may get hands-on experience preparing and analyzing the data, implementing the prescriptive/predictive models, validating their effectiveness, and reporting the results; this is a serious concern.

4.4. Hybrid Delivery

In the Winter 2019 version of MAOR, there were no prerecorded video lectures. Instead, meeting sessions were three hours long, and all content was presented in a classroom setting. However, halfway through the semester, COVID-19 shut down all in-person lectures. As a result, prelecture videos were introduced as meeting sessions (over Zoom) and were shortened to two hours. This format remained for the Fall 2021 version of the course, where meeting sessions were also delivered over Zoom. The following comment summarizes the consensus observed with this format:

“Liked the class discussions and that [the] materials were provided prior to class.”

Because of the almost unanimous support of the course structure, a hybrid delivery will be adopted going forward. As discussed in Section 3.3, the videos will cover more technical content, whereas the in-person sessions will be devoted to examples that use the methods in real-world applications.

5. Discussion and Conclusion

Mathematical programming constitutes a fundamental quantitative skill that MBA graduates should possess, and because of this, is taught pervasively in operations management programs throughout North America (Martinez de Albeniz Margalef 2011, Mark 2021). At the same time, data analytics is a nascent but in-demand field that has been shown to produce tangible business benefits (Kelly 2022). This paper introduces a new course geared toward MBA students that teaches them the fundamentals of prescriptive and predictive analytics using the unifying perspective of mathematical optimization. The curriculum is primarily based on a flipped classroom design (Akçayir and Akçayir 2018) where technical content is presented as short prelecture videos and real-world examples are solved collectively during meeting sessions. There are many experiential learning components as students get hands-on opportunities to solve quantitative problems with Excel through in-class examples, assignments, and exams. However, particular focus is placed on ensuring that managerial concepts in operations management are emphasized while also describing how to communicate analytics-driven insight to business managers. Finally, an overview of the analytics pipeline is presented, several AI tools (e.g., fraud detection, revenue management) are introduced, and broader ethical issues associated with their deployment are discussed.

Student feedback indicates that this course has achieved its learning objectives and has appropriately balanced the presentation of technical material with insightful commentary on managerial decision-making. They also suggest that the flipped classroom format has been successful and that the use of Excel has been well received. Interestingly, not one student commented on feeling mathematically unprepared, which indicates that the course may have strong appeal to nontechnical learners who want to gain a better understanding of the relationship between business problems and the application of analytical tools used for decision-making. Finally, it was observed that students appreciated the emphasis on real-world problems; this allows for rich discussions that emphasize the connections between analytics and modern topics in operations and supply chain management. This emphasis also means that, as compared with typical introductory operations courses offered in many MBA programs, larger-sized and potentially more intricate models can be formulated.

The course aims to produce “data-savvy” managers (Manyika et al. 2011). To do so, MBA students must learn the fundamentals of analytics and receive hands-on training to frame problems and interpret the results while also connecting these quantitative skills to the broader business discipline (Wilder and Ozgur 2015a). In sum, they must relate the low-level implementation of mathematical models and computational algorithms, to the high-level processes that govern the deployment of AI tools to extract insight, make data-driven decisions, and are used by organizations to interact with their customers. By unifying the presentation of prescriptive and predictive analytics through the use of mathematical optimization, the course de-emphasizes mathematical knowledge and puts the focus on business-level problem solving abilities and critical thinking skills. Furthermore, providing different perspectives—from micro (model engineering) to macro (analytics pipeline) processes—and incorporating multiple real-world examples allows MBA students to gain the confidence to contribute to analytics teams. This is imperative when working on large organizational projects that are far more intricate and technically complex. Although many business schools have different viewpoints on how best to prepare MBA students for such a position and an industry that increasingly values analytics and AI (Kurczy 2018), this course presents, what are hopefully, valuable insights on how such a curriculum can be designed and delivered.