An Operations Research–Based Teaching Unit for Grade 12: The ROAR Experience, Part III

1. Introduction

Ricerca Operativa Applicazioni Reali (ROAR, i.e., in English, Real Applications of Operations Research) is a three-year learning path for higher secondary students whose aim is to increase awareness and interest toward operations research (OR) and mathematics, as well as to develop and consolidate related disciplinary skills. ROAR is designed to be attended by the same students over three years. For this reason, we conducted a long-range experimentation, from the 2020–2021 school year until the 2022–2023 one, with a class from the IIS Antonietti scientific high school in Iseo (Brescia, Italy). From grade 10 to grade 12, students experimented with a project-work that is part of a Percorso per le Competenze Trasversali e Orientamento (PCTO; in English, Path for Transversal Skills and Orientation). This is an innovative Italian teaching method that aims to reinforce the knowledge acquired by students in school through practical experience (Ministero dell’Istruzione, Università e della Ricerca 2018). In Italian higher secondary schools, the PCTO is a mandatory experience that can be implemented in collaboration with local companies or organizations. We described the experimentation of the first unit of ROAR (ROAR I) and some of its results in Colajanni et al. (2023a) and Taranto et al. (2022). Similar details about the experimentation of the second unit (ROAR II) are in Colajanni et al. (2023b). Furthermore, we are working on a paper on ROAR II more dedicated to educational aspects.

This paper focuses on the third and final unit (ROAR III), which took place with grade 12 students from October 2022 to January 2023. In ROAR III, we presented the Python programming language and the open-source library PuLP (Mitchell et al. 2011) to address optimization problems. As in previous units, we applied collaborative learning by dividing students into groups. Differently, in the final project, we used cooperative learning and involved the company Filtrec S.p.A.,¹ a manufacturer known worldwide for producing and distributing hydraulic filters. A manager introduced us to the problem of optimizing milk-run routes to visit contractors several times a week. We adapted and simplified the problem based on students’ skills. We implemented an initial version of the model in class. Then, each group had to formulate and implement a different family of constraints independently. Once these were put together, we obtained and solved the final version using historical data provided by the company. Finally, the students wrote a report and held a presentation at the Filtrec headquarters in Telgate (Bergamo, Italy), which they later visited.

The paper is organized as follows. In Section 2, we present, from a pedagogical point of view, the teaching methodologies adopted in ROAR III, such as cooperative learning and cases. In Section 3, we illustrate the unit design by discussing objectives, student prerequisites, and instructors’ roles. In Section 4, we describe the unit implementation by reporting the positioning of ROAR III in the mathematics and computer science curricula, the teaching methods and digital technologies used, the assessment adopted, and the structure of each lecture. In Section 5, we report on the assessment of a mathematics and an information and communication technologies (ICT) written tests, as well as results and feedback from the students, teachers involved, and the Filtrec company. We draw our conclusions in Section 6, where we also reflect on the whole experimentation and its extensions to provide valuable suggestions and insights. Finally, Online Appendices A and B include problems presented during ROAR III and the format adopted to describe an instance of the problem tackled in the final project, respectively.

2. Background

In this section, we describe the theories and teaching methodologies underpinning ROAR III, new with respect to ROAR I (Colajanni et al. 2023a, section 2) and ROAR II (Colajanni et al. 2023b, section 2).

2.1. Cooperative Learning

Cooperative learning is designed and implemented to develop students’ social strategies and attitudes and improve social relationships within and between groups (Terwel 2011). There is a difference between simply having students work in groups and structuring groups of students to work cooperatively. Cooperation consists of working together in small groups to achieve shared goals. However, it is not enough to put students in groups to make them cooperate. Indeed, cooperation must be structured and managed by the teacher (Roger and Johnson 1994). As pointed out by Johnson et al. (1987), carefully structured learning activities involve students working in groups to achieve a common goal under conditions of positive interdependence (i.e., all members must cooperate to complete the task) and individual and group responsibility (i.e., each member is responsible for the result).

Cooperative learning groups can be distinguished into base, informal, and formal groups (Smith 1996). Base groups are long-term and heterogeneous groups with a stable number of members. They are exploited to provide each student with the support, encouragement, and assistance needed to progress in school. Informal groups are temporary ad hoc groups that last from a few minutes to an hour of class time and focus students’ attention on the material to be learned. They are often organized such that students engage in focused discussions before and after the lecture, interspersing with classmates during it. Formal groups can last from one class period to several weeks to complete specific tasks and assignments, such as learning new conceptual material, making decisions or solving problems, writing a report, conducting a survey or experiment, preparing for an exam, answering questions, or solving problems at home.

In ROAR III, we developed formal cooperative learning groups. In the following, we briefly summarize what an educator (in our case, the class teacher supported by the experimenters) should do to create and lead them appropriately, according to Smith (1996).

1. Specify lecture objectives. Each lecture should have an objective linked to the concepts, strategies, and procedures to be learned and a group work objective specifying the interpersonal or small group skill to be used and mastered.

2. Make a series of teaching decisions. The educator must decide on the size of the groups, the method of assigning the students to the groups, the duration of the groups, the roles to be assigned to students, the materials needed to conduct the lecture, and the classroom arrangement. We specify that the roles to be assigned must be consistent with the task requirements and important for the smooth running of group activities.

3. Explain the task and positive interdependence. The educator must clearly define the task, teach the needed concepts and strategies, and specify positive interdependence and individual responsibility. To make a group project truly cooperative, these must be structured in different and congruent ways. Positive interdependence typically asks each group to prepare a single product (interdependence of objectives) by asking students to ensure that each person in the group can explain the group’s response (interdependence of learning objectives) and by assigning a particular role to each member (interdependence of roles). Individual and group accountability is typically implemented by specifying functions for each role, monitoring groups, and occasionally asking a student to explain the group’s answer or method (individual oral examination), asking each member to sign the group’s report, and giving individual quizzes, exams, and writing assignments.

4. Monitor student learning and intervene within the groups to assist with tasks or to increase students’ group work skills. The educator systematically observes and collects data on each group as it works. If necessary, the educator intervenes to help students complete the task accurately and work together effectively.

5. Assess student learning and help them process how well the group functions. Students’ learning, as well as their performance, is carefully assessed through written examinations, quizzes, and reports.

2.2. Cases

Case studies and teaching cases are two similar teaching methods often confused in the literature. The former presents problems inspired by real life for which a solution is also typically exposed. The latter may not consider real situations and leaves the task of finding a solving approach to the students. In this sense, a case study can foster analysis and discussion, whereas teaching cases enhance the development of problem-solving skills.² As in Raffaele and Gobbi (2021), we adopt the word “cases” to consider both because the two terms are used interchangeably.

In ROAR III, our goal was to present a situation close to the students as the final project, a problem on which they could apply some OR methods to increase their motivation. First, we asked the students to investigate with their parents, relatives, or friends if they had an interesting topic or problem to suggest. In particular, we hoped to interface with a company to identify a real-life optimization problem. A student reported the availability of her father, Dr. E. Giliani, operations manager at Filtrec. Together with him, we simplified the complex and challenging real problem and designed the Filtrec problems to be tackled with cooperative learning. More details are provided in Section 4.4.1.

3. Design of the Third Teaching Unit of ROAR

In this section, we describe the main objectives pursued, the prerequisites needed, and the roles to be played by the instructors in ROAR III.

3.1. Objectives

The first objective is to teach the usage of the Python library PuLP, an advanced modeling toolkit that enables users to implement mathematical programming models through Python. Students are introduced to a tool that allows them to solve large-scale models. This new approach proves to be more efficient in terms of problem-solving compared with tools, such as GeoGebra (2021) or Excel Solver (Microsoft Corporation 2019), which were introduced during ROAR I (Colajanni et al. 2023a, section 4.5).

The second objective is to strengthen programming skills by teaching the fundamentals of the Python language to use PuLP. In particular, introducing a fundamental subset of Python instructions and keywords allows students to learn how to start using an interpreted programming language that is one of the most popular and widely used in professional settings and in many academic paths. Analogies between Python and already familiar programming languages can aid learning and serve as a review of prior computer science coursework. Students can enhance their ability to interpret and modify code, adding new functional requirements.

The third objective is understanding and tackling an OR problem within a business context. Although ROAR I and ROAR II typically presented real world–inspired problems, these were of a smaller scale. In this unit, students analyze and address a problem formulated not by instructors but by industry experts, thus simulating a genuine work environment. This endeavor enlightens students about the complexity and significance of mathematical problems in a corporate setting.

The fourth goal is to reinforce mathematical modeling skills. After grasping the fundamentals and strengthening their competencies over the years, students must interpret an already constructed mathematical model and integrate/modify it to implement new constraints. We aim to teach them how to abstract the types of constraints and variables used in examples presented both in this unit and in previous ones to create new models for implementation and resolution using the PuLP library.

The last two objectives are to enhance specific soft skills. Problem-solving and teamwork skills are enhanced through the group assignments given during the lectures and through the homework assignments to be completed at home (fifth objective). Public speaking is strengthened through interactions with instructors, group members, all class peers, and interactions with the company’s employees who formulated the problem for the final project. In particular, students present their work to the personnel, requiring them to effectively address any clarification or curiosity-based questions (sixth objective).

3.2. Students’ Prerequisites

Differently from ROAR I and ROAR II (see Colajanni et al. (2023a) and Colajanni et al. (2023b), respectively, subsections 3.2), students have to possess some prerequisites to be able to carry out the activities of ROAR III. Indeed, some of the objectives described in the previous subsection require students to be already familiar with mathematical modeling, particularly the paradigms of linear, integer, and mixed integer linear programming introduced in ROAR I. Moreover, because a few problems faced during the lectures of ROAR III were initially proposed in ROAR II, students should have at least some basic notions of graph theory. Finally, to tackle the first and second objectives, it is warmly suggested that students already have some coding and programming language background. It is optional for them to know Python already, but they should already be familiar with the concepts of instructions, loops, functions, and basic data structures (e.g., arrays).

3.3. Instructors’ Roles

As in ROAR I and in ROAR II, we distinguish two roles. An experimenter introduces new topics and guides students through the various activities. Preferably, the person covering this role should be an expert in OR and familiar with Python and modeling languages. At least one experimenter is required to implement the unit. Then, there could be one or more observers, whose main task is to collect data for research purposes. These data include but are not limited to notes on the progress of the lectures and students’ reactions and comments. The presence of an observer is not mandatory.

4. Implementation of the Third Teaching Unit of ROAR

ROAR III, composed of seven lectures, was implemented from October 2022 to January 2023. All lectures were held in the classroom. Because of geographical distance, the two observers were always connected remotely using Microsoft Teams.

4.1. Grade 12 Class

The grade 12 class involved in ROAR III comprised 13 males and 9 females. Before attending ROAR, students had already participated in problem-solving activities. For instance, during ordinary lectures in previous years, they had solved problems related to real-life situations, and some of them had participated in mathematical and informatics competitions, such as the Mathematical and Informatics Olympiads. Regarding digital technologies, students were already familiar with Mentimeter (2021), Kahoot! (2022), and Excel Solver (Microsoft Corporation 2019), especially after ROAR I and II. Also, they had already developed some coding skills, such as implementing simple programs in the C++ language. According to their mathematics teacher, M. Picchi, their mathematical skills were just above average compared with other grade 12 classes in the same school and at the national level. Their ICT skills were also above average compared with other grade 12 classes in the same school. Students’ interest and engagement in ROAR III activities were generally higher than in the previous units. This was because students who liked mathematics or computer science tended to be more involved.

4.2. Instructors

The instructors were the same as in ROAR I and ROAR II. A. Gobbi and A. Raffaele, both researchers in OR, acted as experimenters. G. Colajanni and E. Taranto, researchers in OR and mathematics education, respectively, played the role of observers, as well as the mathematics and physics teacher of the grade 12 class, M. Picchi.

During group work activities, all instructors assumed the role of observers. Thus, the total number of observers was five, as many as the groups of students (see Section 4.4).

During Lecture 4 (see Section 4.8.4 for details), a practitioner in an OR company, Dr. S. Bortolomiol, first held a seminar and then covered the experimenter and observer roles for one activity only. Finally, during the written test of Lecture 6, the ICT teacher, E. Danesi, was also present.

4.3. Positioning in the Mathematics and Informatics Programs

ROAR III took place during the regular school schedule of the grade 12 class in the first part of the school year. At the beginning of its implementation, all students had already acquired basic algebraic skills in solving different equations and inequalities. In particular, they could work with algebraic, exponential, logarithmic, and goniometric functions. In addition, they could solve simple combinatorics and probability problems. As for previous informatics knowledge and skills, they had experience with graph theory, some basic network applications, and programming languages, such as C++ and HTML.

4.4. Teaching Methods

As for the teaching methods adopted during ROAR III, most of them were the same as in ROAR I and ROAR II. Thus, we refer the reader to Colajanni et al. (2023a, section 4.4) for more details about frontal lectures, collaborative learning, homework assignment, and authentic problems, and to Colajanni et al. (2023b, section 4.4) for more details about discovery learning and seminar presentations. Hereafter, we provide the details about how we implemented other teaching methods, new with respect to ROAR I and ROAR II.

4.4.1. Cases and Cooperative Learning.

As a final project, we defined a case in collaboration with Filtrec S.p.A.,³ and we adopted a combination of cooperative and collaborative learning as a teaching method. During Lecture 4, we presented this final project, simply called Filtrec (see Section 4.8.5), as follows.

After introducing the context, we better defined the related optimization problems as follows.

Students were divided into five groups. We defined and shared the following basic objectives:

1. Formulation of the mathematical model of the Filtrec-Model1 problem;

2. Implementation and resolution of the mathematical model of Filtrec-Model1 with Python and PuLP (from now on, “Python+PuLP”); and

3. Formulation, implementation, and resolution of the mathematical model of Filtrec-Model1-bis with Python+PuLP.

   We also defined the following two optional advanced objectives:

4. Formulation of the mathematical model of the Filtrec-Model2 problem; and

5. Implementation and resolution of the Filtrec-Model2 mathematical model with Python+PuLP.

Objectives 1 and 2 would have had to be achieved with the experimenters and observers during Lecture 5. As for Objective 3, each group had to formulate one family of constraints and implement it in Python within two weeks, sending the proposed solution to us via email. Then, we would have to merge all the constraints formulated and implemented by the groups to obtain the final mathematical model of the Filtrec-Model1-bis problem, which we would have shared with the groups. Regarding Objectives 4 and 5, groups were free to send a mathematical formulation and/or implementation in Python+PuLP of the Filtrec-Model2 problem whenever they wanted.

The result of the collaboration within the groups and cooperation among the groups and us had to be presented at Filtrec headquarters during the last lecture of ROAR III. Thus, we asked each group to assign the following roles to their members: the modeler, who would have mainly dealt with the formulation of the mathematical model; the programmer, who would have primarily implemented the mathematical model in Python+PuLP; the programmer’s support worker, who would have offered support for the implementation; the slide creator, who would have taken care of the graphics and contents of the final presentation; and the presenter, who would have spoken during the final presentation.

Moreover, groups had to write a collective class report, which must have contained at least the following:

• The description of the real problem;

• The analysis carried out to arrive at the mathematical models of the Filtrec-Model1 and Filtrec-Model1-bis problems;

• The description of the Python+PuLP implementation;

• The analysis and interpretation of the results of the Python+PuLP resolution; and

• Conclusions and future developments (possible extensions).

We required the groups to deliver a draft of the class report about 10 days before the last lecture and final presentation at Filtrec headquarters in Telgate (Bergamo, Italy). The day after receiving the draft, we sent the students a list of corrections and points to address. Then, the final version of the class report was delivered to Dr. E. Giliani a few days before the final presentation.

4.5. Digital Technologies

As digital technologies, we adopted Solver (Microsoft Corporation 2019), Mentimeter (2021), Kahoot! (2022), Python, Google Colaboratory, Spyder, and PuLP.

Solver is an add-in program of Microsoft Excel employed to solve mathematical optimization models. Mentimeter allows the development of interactive live polls to involve students in the activities. Kahoot! is a learning platform that can be exploited to review students’ knowledge using quizzes. Because we had already used the first three technologies during ROAR I and ROAR II, we refer the reader to Colajanni et al. (2023a, section 3.5) for more details about Solver and Mentimeter and Colajanni et al. (2023b, section 4.5) for more details about Kahoot. Hereafter, we focus on the last four technologies indicated only, which are new with respect to ROAR I and ROAR II.

4.5.1. Python.

Python is a high-level and object-oriented programming language created by the Dutch computer scientist Guido van Rossum in 1991 (Van Rossum and Drake 1995). Python is commonly used for scripting, prototyping, testing, analyses, and data visualization. In recent years, Python has established itself as one of the preferred languages for educational purposes, even in introductory programming courses at the university level. Comparing the difficulty perceived by first-year university students in learning Python and C++, Ateeq et al. (2014) observed that students considered the former more intuitive, flexible, and compact than the latter. In particular, they felt that most language features, including loops and conditionals, were more straightforward than C++. Also, they appreciated that Python can be used as an interactive shell. This was also previously highlighted by Grandell et al. (2006), who investigated which language would be preferable to adopt when teaching programming at the higher secondary school level. They underlined how the Python interactive environment provides immediate feedback to the students. Also, they stated that Python offers a simple and flexible syntax that significantly reduces the notation overhead compared with other languages. Thus, Python lends itself to in-class coding, testing, and demonstration, supporting active and hands-on learning. A critical aspect of Python, noted in Mészárosová (2015), could be its dynamic typing feature, that is, variable types are not declared in advance. However, they are established during execution and can change according to the values assigned. If students are quickly informed about it, it should not be an issue. Last but not least, another advantage of using Python is that it is an open-source language, and plenty of materials are available, among which several libraries extend its applicability.

4.5.2. PuLP.

PuLP (Mitchell et al. 2011) is a free, open-source software written in Python. Its purpose is to represent and solve optimization problems described as mathematical models according to the paradigms of, for example, linear, integer, or mixed integer linear programming. PuLP relies either on its internal solver COIN-OR branch-and-cut solver (CBC; Forrest and Lougee-Heimer 2005) or on other external solvers available on the machine in use. Because of PuLP user-friendly application programming interfaces, students can use a professional and easily understandable instrument that permits them to customize both the model’s input and the data instances to be solved. PuLP documentation is available online.⁴

As far as we know, in the literature, only Sotomayor-Beltran and Delgado (2019) described the exploitation of PuLP in didactic contexts to introduce and solve optimization problems. However, the students involved in that research were from a university course in industrial engineering and not a higher secondary school.

4.5.3. Google Colaboratory.

Produced by Google Research, Google Colaboratory, commonly known as “Colab,” is a hosted Jupyter Notebook service that allows one to write and execute arbitrary Python code through a browser without any setup (Bisong and Bisong 2019). Besides executable code, Colab notebooks can include simple text, as shown in Figure 1, but also images, HTML, LaTeX, and more, in different blocks. To exploit Colab, students must use their Google account because notebooks are stored in Google Drive, and the code is executed in a virtual machine that is private to the account. Also, Colab notebooks can be shared with other users.

When opening a new Colab notebook, the PuLP library, described in Section 4.5.2, can be installed by adding the code block shown in Figure 2.

4.5.4. Spyder.

Spyder is a free and open-source integrated development environment for coding in Python (Raybaut 2009). As shown in Figure 3, the Spyder interface can be customizable with several panels, such as the Editor, the IPython console, the Variable Explorer, the Plots, and more. As far as we know, no works in the literature describe Spyder’s use for didactic purposes at the higher secondary school level.

4.6. Questionnaires

Similarly to what was done at the beginning of ROAR I (Colajanni et al. 2023a, section 3.6) and ROAR II (Colajanni et al. 2023b, section 4.6), one week before meeting the grade 12 class in October 2022, we asked the students to fill in an anonymous questionnaire about their feelings toward mathematics and their expectations on ROAR III. Moreover, we requested the students to use the same nicknames they had adopted in the analogous questionnaires of ROAR I and ROAR II. Another similar questionnaire was given at the end of lectures to receive feedback on the teaching unit and the final project. At the end, we also provided the students with a third questionnaire about the whole ROAR project. All questionnaires were developed with Google Forms, and the related links were shared with the students. The main results from the first two questionnaires are shown in Section 5.2. On the contrary, the comparison among ROAR I, ROAR II, and ROAR III and the comprehensive last questionnaire will be analyzed in a future longitudinal study.

4.7. Assessment

Summative assessment can be defined as an indication of the levels achieved by the learner at a given point in time (Becchi 1995). A synthetic index (grades or levels) often identifies it as one or more aspects of learning: knowledge, skills, and competencies. Formative assessment can be defined as information communicated to the learners aimed at changing their study strategies to improve their learning Shute (2008). Formative assessment aims to provide feedback to the learner and the teacher on the evolution of the learning process (Scriven 1967, Parmigiani 2018). A desire to harmonize assessment’s summative and formative aspects emerges in Black and Wiliam (1998) and Wininger (2005). Indeed, these two assessment modes can become teaching strategies to be offered to students to help them develop a process of self-assessment and self-direction.

In December 2022, toward the end of the teaching unit (Section 4.8.6), we carried out an individual summative assessment as a written test. In particular, the test was divided into two parts. Initially, students were tasked with modeling the “An Important Conference” problem using the integer linear programming approach. Then, they had to code a Python script to implement their formulation and get an optimal solution using PuLP. The first part was corrected by the math teacher, M. Picchi, whereas the second part was checked by the ICT teacher, E. Danesi. To perform the test, students used school laptops without an Internet connection, thus relying on Spyder. Also, they were allowed to consult all the material from the previous lectures of ROAR III. Eventually, every student obtained two grades: one in mathematics and the other in ICT (see Section 5.1 for the detailed assessment results). In the last meeting of ROAR III (Section 4.8.7), during the final presentation at Filtrec S.p.A. headquarters, we evaluated the public-speaking skills of some students (about one or two per group) through a formative assessment, assigning a score using the criteria established by the teachers (for example, clarity of presentation, adequate use of specific terms, respect for time, fluency, etc.).

4.8. Organization of the Lectures

Hereafter follow the details of the seven lectures held in the grade 12 class at IIS Antonietti, from October 24, 2022, to January 25, 2023. For each lecture, we indicate the date, an overall indicative duration, the structure (i.e., all the activities performed), the homework assigned, the teaching methods adopted, and the digital technologies used. We specify that the lessons proposed are never totally transmissive. They always include a moment of explanation by the experimenters (i.e., frontal teaching), accompanied by group work (whether in collaborative, cooperative, or coopetitive mode). Furthermore, as far as duration is concerned, we would like to point out that the hours indicated are never conducted without a break, there are always moments of recess or change of class teacher (approximately every two hours) in which the students have time to refresh their bodies and minds for about ten minutes. Should one wish to replicate a lesson that has been taught over five hours (e.g., lesson 3), it should not be assumed that these must necessarily be consecutive. The lesson may be broken into two or more moments. The “pattern” to be respected must always be frontal lecture and group work, in which the lectures are applied (see paragraph 4.4. in Colajanni et al. 2023a).

4.8.1. Lecture 1.

In the first lecture of ROAR III, as shown in Table 1, we first recapped the main topics and features of ROAR I and ROAR II. Then, we described the outline and objectives of ROAR III. We introduced the programming language Python using a Google Colab notebook. Successively, we presented the library PuLP. In particular, we retrieved the mathematical formulation of a problem from ROAR I students were already familiar with and showed them how to solve it by implementing the model in Python+PuLP. Successively, we assigned the first group work. We ended the lecture by giving some homework for Lecture 2 and making a live interactive poll with Mentimeter.

Table 1. Details of Lecture 1

4.8.2. Lecture 2.

As summarized in Table 2, after checking the implementation in Python+PuLP of the problems assigned as homework, we started the second lecture by teaching more advanced Python and PuLP features with respect to the ones seen in Lecture 1. In particular, using the jobs assignment problem (Colajanni et al. 2023a, appendix A), already known by students, we introduced dictionaries in Python, how to define them, and how to access their stored data. About PuLP, we showed the students how to use the method dicts to define a set of variables together rather than one by one. Also, we described how to use the function lpSum to compute the sum of a list of linear expressions and how to exploit for-loops to add a family of constraints in one line. Then, we assigned two group works to let students experiment with the new features and strengthen their coding skills. As for digital technologies, students could use either Google Colab on any browser or the Spyder integrated development environment on the school laptops. We ended the lecture by assigning homework for the successive meeting and asking a few questions about Python and PuLP in a live interactive poll.

Table 2. Details of Lecture 2

4.8.3. Lecture 3.

We assigned the students two group works during the third lecture of ROAR III, as described in Table 3. The former was about a new problem to be modeled, implemented, and solved using Python+PuLP, whereas the latter (i.e., the Vehicle Routing Problem Challenge; Colajanni et al. 2023b, section 4.4.1) was already known by the grade 12 class. Indeed, we had proposed it as the final project of ROAR II. On that occasion, the groups had to develop their approach (either applying mathematical modeling or developing heuristic algorithms) to tackle the problem and solve three instances of increasing difficulty. This time, on the contrary, we provided the groups with a Python script containing a partial implementation of the mathematical model and a method to read an instance from a text file and an instance itself. In particular, the partial implementation already included the call to the reading method, the definition of the decision variables, and some model constraints. What was missed were other constraints and the objective function, for which we wrote some comments as suggestions. Each group had to complete the script and try running the code on the given instance. After about one hour, we corrected the missing part of the implementation together. Then, we ended the lecture as usual by assigning a new problem as homework and testing students’ Python and PuLP knowledge through Mentimeter.

Table 3. Details of Lecture 3

4.8.4. Lecture 4.

We divided Lecture 4 activities into two parts, as described in Table 4.

Table 4. Details of Lecture 4

In the first part, we presented the final project of ROAR II about the case designed together with Filtrec S.p.A. (Section 4.4.1). As homework for the successive lecture, we asked the students to think about the analogies and differences between the Filtrec problem and the VRP Challenge tackled as the final project of ROAR II (Colajanni et al. 2023b, section 4.4.1) and taken up in the previous lecture with its model formulation.

In the second part, we organized a seminar held by Dr. S. Bortolomiol, a practitioner working at OPTIT s.r.l., one of the leading companies in Italy in the development of decision support systems, methodologies, and solutions for forecasting, data analysis, simulation, and optimization of complex systems.⁵ The seminar, which had divulgation and orientation purposes, was also opened to other grade 11 and grade 12 classes of the IIS Antonietti and lasted about one hour. Dr. S. Bortolomiol presented different applications of OR OPTIT s.r.l. had worked on. After that, based on his presentation, Dr. S. Bortolomiol supervised a group work involving the students of ROAR II only. In particular, students had the opportunity to choose one application and think about the corresponding problem definition and model formulation, specifying variables, constraints, and the objective function. Then, we discussed all the applications together.

4.8.5. Lecture 5.

The fifth lecture, summarized in Table 5, was wholly dedicated to the final project of this unit.

Table 5. Details of Lecture 5

We started applying cooperative learning by discussing, together with the students, analogies and differences between the VRP Challenge and Filtrec problems. By deepening the features of the two problems, students could observe that both problems can be modeled using graphs. In particular, a student noticed that the Filtrec problems could be cast under the family of vehicle routing problems. Indeed, in both problems, there is (at least) a depot and a set of customers to serve by making deliveries with a fleet of available vehicles. Actually, in the real problem faced by Filtrec S.p.A., customers ask for deliveries and pickups. We simplified this aspect by considering only deliveries. In both applications, vehicles have a start and end depot, limited capacity, a maximum travel time allowed, and a fixed cost if used. However, this fixed cost in the Filtrec problems already includes the fuel cost. Thus, the objective function has a different expression than the one in the VRP Challenge.

After this comparison, we assigned a first group work about the model formulation of the Filtrec-Model1 problem. To face this activity, students were given the model formulation of the VRP Challenge, seen in Lecture 4. Then, we corrected the formulation together and asked them, as second group work, to implement it in Python+PuLP.

4.8.6. Lecture 6.

As mentioned in Section 4.7, during the sixth lecture, we evaluated the students’ skills by having them take an individual written test on the An important conference problem (see Table 6).

Table 6. Details of Lecture 6

4.8.7. Lecture 7.

The last lecture, summarized in Table 7, was held at the Filtrec headquarters in Telgate (Bergamo, Italy). First, Dr. G. Modina, Filtrec’s general manager, and Dr. E. Giliani, Filtrec’s operations manager, held a brief seminar introducing the company to the students. Then, some students gave a class presentation on the final project of ROAR III. In particular, they explained what the groups could do to tackle the Filtrec problems. They described the first version of the model implemented in the classroom (see Lecture 5) and the different families of constraints the groups had to implement individually (see Lecture 5). They thus illustrated the final version of the Filtrec-Model1 model and the solution they obtained by using the historical data given by the company. After the presentation, there was an interesting discussion among Filtrec’s employees, the students, and us experimenters. Successively, we had the opportunity to visit the Filtrec’s factory and warehouse. Finally, to conclude, every student filled in two final questionnaires about ROAR III and the whole ROAR project, respectively.

Table 7. Details of Lecture 7

4.9. Linking Implementation to Objectives

In Table 8, we summarize how the objectives of the teaching unit listed in Section 3.1 were achieved and in which lectures.

Table 8. Linking the Objectives of the Teaching Unit to the Implementation Described in Section 4

5. Results and Feedback

This section first reports on the mathematics and ICT written tests used as summative assessments. Then, we analyze the principal results and feedback from the two questionnaires students filled in. Finally, we provide some qualitative feedback on ROAR III and the final project, received by other teachers of the class and some employees of Filtrec.

5.1. Written Tests

The mathematics test consisted of two questions. The former required formulating an integer linear programming model for a variant of the knapsack problem (see Online Appendix A). The second question asked to integrate the previously formulated model with three additional constraints: a constraint on the maximum number of selectable items (c1), a logical constraint linking two binary variables (c2), and a constraint to ensure that the number of selected items was even (c3). We remark that the knapsack problem was the first problem ever presented to the students during ROAR I and that has been revisited in various forms during ROAR II and ROAR III. Thus, students were already familiar with the first two constraints. On the contrary, the experimenters never addressed the third constraint or required in any exercise. Nonetheless, students had all the theoretical foundations and skills to formulate it.

The test was evaluated (with a final score from 0 to 100) by summing the scores obtained for the following aspects: a correct variable definition, an adequate formulation of the base constraint and the objective function (each with a score from 0 to 20), the formulation of the additional constraints c1, c2, and c3 (with a score from 0 to 10 for the first two, and a bonus score for c3), and the use of an appropriate discipline-specific language (with a score from 0 to 20).

The box-plot diagram in Figure 4 illustrates, for each assessed aspect of the test, the score percentage interval obtained by the students (where × indicates the mean, whereas – indicates the median). On average, they recognized the knapsack problem and were able to correctly formulate the objective function and base constraint, defining the needed binary variables. The mistake made more frequently was the wrong computed value of the knapsack capacity, which was not given in the text of the problem but had to be derived from other data. Moreover, students had difficulty in formulating the additional constraints. In particular, the constraint c1 is the one that most students managed to formulate more accurately. Most students only managed to outline c2, understanding its meaning but incorrectly using mathematical operators. However, two outliers, likely drawing from previously covered classroom examples, differed from this trend. No one, however, succeeded in formulating the constraint c3. This may indicate that students were not able to abstract the concepts learned from other examples and apply them to this new task. Some students attempted to formulate the constraint using the integer remainder operator %, forgetting that a statement written in a programming language is, from a modeling perspective, devoid of meaning.

The ICT test also consisted of two parts, mirroring those of the mathematics test. In the first part, we asked the students to write a Python+PuLP script to implement and solve the previously formulated knapsack problem on paper. The second question required writing additional code to implement constraints c1, c2, and c3. For this test, rather than assessing the use of appropriate language, the ability to print solutions was assessed (with a score from 0 to 10). From Figure 5, we can note that students had difficulty writing and implementing the constraints, whereas they managed in correctly implementing the objective function, as well as printing the optimal results.

Finally, Figure 6 shows the number of students who scored in each range, from [0, 10] (i.e., very poor) to [91, 100] (i.e., excellent), in the mathematics (blue) and ICT (orange) tests. The average score obtained in the mathematics test is 69.77 (17 sufficient votes and 5 marks below, given a threshold of 60). In contrast, in the ICT test the average score was 58.59 (with 12 sufficient votes and 11 insufficiencies). We can make the following observations. (i) There are no mathematics marks less than 30, whereas two students got a mark in [11, 20] in the ICT test. (ii) Five students obtained a mark in [41, 50] in the ICT test and only two in the mathematics test. (iii) Only one student obtained a mark in [81, 90] in the ICT test, whereas four students obtained marks in [81, 90] and one student in [91, 100] in the mathematics test.

5.2. Feedback from Students

We gave students two questionnaires: one before the first lecture and one at the end of the unit, filled in by 19 and 20 students, respectively. As mentioned earlier, we encouraged students to use the same anonymous nickname for both questionnaires. This allowed us to identify the student who did not complete the former questionnaire and remove the corresponding answers from the data to compare the answers to the two questionnaires consistently.

We first investigated the students’ knowledge and preferences on topics and methodologies, and then their opinions referred to the company visit and the milk-run problem. Lastly, we show some final considerations.

5.2.1. Knowledge and Preferences on Topics and Methodologies.

From the initial situation of the whole group class (i.e., before ROAR III), students seemed to have a fairly high knowledge of the C and C++ programming languages, and only one student knew Java (Figure 7). As shown in detail by Figure 8, 74% of students declared that they did not know how to use Python (at all), 21% did not know very much, and only one student stated to be able to use it enough.

After completing ROAR III, as shown in Figure 9, more than 90% of the students stated that they understood the basic notions of Python fairly or completely (for example, variables, functions, screen printing, etc.); approximately 57% understood the advanced notions (for example, dictionaries, reading from files, etc.); and 76% understood the PuLP library for implementing and solving optimization problems.

As reported in Figure 10, the most appreciated aspect appears to be the basic notions of Python (around 67% place it in first or second place), followed by the implementation of optimization problems (around 57% of students place it in first or second place). However, the least appreciated aspect is the advanced notions.

Figure 11 shows that the most appreciated teaching methodology is the working group (during lectures); in fact, 85% of the students absolutely or very much appreciated this methodology. With the exception of homework and group work in extracurricular hours, all methodologies were appreciated by the majority of students.

5.2.2. Company Visit.

As previously described, the last lecture took place at the premises of the Filtrec S.p.A. company, where the students first attended the company presentation by the managers, illustrated their final project, and finally visited Filtrec’s factory and warehouse.

Seventy-five percent of the students completely appreciated the visit to the Filtrec company, and 20% appreciated it very much. Only one student did not appreciate it at all (Figure 12).

Most of the students (30% a lot, 45% enough) believe that it was useful to have worked on the Filtrec problems before visiting the company, as shown by Figure 13. Indeed, they commented that they “had the opportunity to learn about the company and came prepared” and that “it was useful to know how to apply the notions learned during ROAR III in a real corporate environment.” On the other hand, some students believe that “it did not help the company very much” or that “having worked on just one constraint did not make me understand the problem in general and I do not think it helped me much.”

To the question “What did this corporate experience leave you with?,” most students responded that it was a useful experience to understand how a company is organized and who are the main figures and for having a first approach to the working world. Below are some student comments: “I understood what it means to work in a company from all points of view,” “Now I have a clearer idea of what it means to work in a company,” “More confidence in speaking and presenting,” and “it was interesting to see, even if quickly, how the factory and laboratory are managed, as well as to see the notions learned during ROAR III applied in reality,” but also “It gave me just a small taste of the working reality of that single company.”

5.2.3. Filtrec Case.

Feedback on the case presented by the company Filtrec and adapted by the researchers are illustrated in Figure 14. The case was considered challenging by almost all students (95%). Anyway, it was pretty or wholly understood by 90% of the students. The contribution from individual students was high, especially for formulating the constraints (assigned to the groups), followed by their implementations. The feedback from the experimenters turned out to be quite or very useful (60% and 15%, respectively).

As shown in Figure 15, the class group work was challenging to complete in terms of writing for most students (90%); furthermore, according to 80% of the students, it was difficult to coordinate with the entire class group. According to most of the students, the final report would not have been acceptable without the experimenters’ feedback (according to 75% of the students), and with more time available, the report level would have been better (for 55% of the students).

Among the students’ comments, the most common is that “drawing up a single final report was, perhaps, the most difficult part, but it helped us to discuss and listen to each other”; in fact, “the difficulty in writing the final report and then preparing the presentation involved having to coordinate with the whole class.”

As previously described in Section 4.4.1, an additional advanced objective concerned the formulation and implementation of the Filtrec-Model2 extension. Among the students, 65% of them declared that they had not worked on this variant due to lack of time and complexity; about 20% of them stated they had partly worked on it; and the remaining 15% of them had worked on it. The difficulties were related to understanding and implementing the extension using a complex and complicated computer language.

We asked the students to provide a percentage, in their opinion, of how well they had managed to formulate and implement the model correctly. Figure 16 shows the results. Just about 28% of students who worked on the Model2 variant believe they were able to formulate and implement the model almost entirely (i.e., from 61% to 80% of the task). More than 57% and around 43% of students think that they have succeeded enough (i.e., from 41% to 60%) in formulating and implementing the model, respectively. Conversely, more than 14% and 28% of students believe they were not sufficiently successful (i.e., from 21% to 40%) in the two tasks, respectively.

Among students, 75% think that what they learned during ROAR III will be useful in the future (Figure 17), because “it will be handy at university and, in the future, also at work”. Even those who don’t know which university path they will undertake believe that “ROAR III has taught me a method for tackling various types of problems and working in a group”. On the contrary, students who do not consider the ROAR III to be useful justify their response by stating that “these topics are not included in my study plans”, or “I would like to study abstract mathematics at university, not applied”.

ROAR III has changed the mathematics and its real-world applications idea of most students (70%; Figure 18). Indeed, students report that “it allowed me to learn new methods for solving complex problems” or it revealed that “mathematics has real-world applications.” Furthermore, we noticed that among the positive answers (Yes), one student changed his mind, but negatively as they answered “I didn’t like it.” This is the same student who said in the previous question that they prefer abstract mathematics. Finally, students who declare that they have not changed their idea about mathematics and its real-world applications say that their concept has remained unchanged or that “I think this path was based more on computer science than mathematics. I prefer the mathematics we do during curricular lectures.” Furthermore, as can be seen from the comparison between Figures 16 and 19 and between Figures 17 and 20, there is no significant change between ROAR II and ROAR III. This means that the usefulness that students perceived of the new content presented during ROAR III remained constant.

From Figure 21, we can see, by combining colors 3 and 4, how all skills (mathematical formulation, information technology, solution evaluation, group and class cooperation, and public speaking) were acquired by more than 50% of the students.

Comparing the initial and final questionnaires, as shown by Figure 22, we can see that, when asked if they prefer to rely on their computing skills rather than digital technologies when solving mathematical problems, three students changed their preference in favor of digital technologies.

5.3. Feedback from Teachers

The mathematics teacher underlined how ROAR has allowed to enhance students’ different aptitudes through group work. She declared that “The resolution of real problems, also with the use of information technology, has allowed the students to understand the importance of mathematical modeling.” In addition, she noted an increasing relationship of trust between experimenters and students: Experimenters also became a guide for future university choices. Finally, she took stock of the project saying that “ROAR was perfectly coherent with the course of study of the class: It made it possible to deepen the characterizing subjects of the scientific high school and enhance different types of skills of the students.”

Other teachers also gave positive feedback about the project. In particular, the art teacher said that “students experienced academic and corporate worlds facing a real problem thanks to accurate research methods.” The ICT teacher, who collaborated in the assessment during Lecture 6, appreciated the laboratory approach of the lectures in which students had to immediately apply to concrete problems the Python features introduced by the experts. Finally, the English teacher found the project well organized and appreciated the attention to the active involvement of all students.

5.4. Feedback from the Company

Our Filtrec contact person, the operations manager Dr. E. Giliani, commented on the outcome of the project as follows: “An excellent example of how academic notions can find practical feedback within the industrial business. Thanks to the path undertaken, students have demonstrated their ability to develop a concrete project, contextualize it, and present it to a potential end user. The project represented an opportunity for collaboration and transversal growth but, above all, a good starting point for future developments.”

Dr. N. Freri, one of Filtrec external consultants involved in the milk-run design, underlined that this is a very complex task because there are many variables to take into account and that customers and contractors may change their demands and when they need to be served. Thus, “the idea of developing a tool that would allow milk-run routes to be quickly redesigned whenever the need arose became very interesting.” About students’ work, he said that “they understood the purpose of the project well, demonstrating considerable organizational skills and very promising specific skills. It matters little if the model the students have worked on is a simplified model: the relevant aspect is that, with their work, they have brilliantly set up a modeling work that can certainly be further developed.”

Finally, according to Dr. G. Modina, Filtrec’s general manager, the project was “a successful experience both for the results achieved and for the working method used. Students tackled the problem with enthusiasm, fully achieving the company’s objectives. In such a dynamic and changeable context as the current one, a simple and parametrizable simulation tool will allow us to promptly reschedule critical transports within our supply chain promptly.”

6. Conclusions

In this last section, we first draw our conclusions on ROAR III. Then, we illustrate the main extensions of the ROAR projects we developed throughout these years and briefly discuss future work. Finally, we share a few suggestions and insights we learned from designing, developing, and carrying out the project under all its forms.

6.1. On ROAR III

In this paper, we finished describing ROAR, a three-year didactic project addressed to higher secondary school students whose goal was to present and exploit OR, a branch of applied mathematics, to favor students’ understanding of mathematics’s relevance, impact, significance, and appreciation.

In particular, we focused on ROAR’s third and last teaching unit, carried out between October 2022 and January 2023, in a grade 12 class at IIS Antonietti in Iseo (Brescia, Italy). The purpose of ROAR III was to introduce the Python programming language and the Python open-source library called PuLP to tackle optimization problems by exploiting mathematical programming and a solver. Similar to ROAR I and ROAR II, we organized the unit into several lectures by balancing different teaching methods, such as frontal teaching and group work. We put particular emphasis on collaboration by making students work all the time in groups. In this unit, we also applied cooperative learning by working with the students to face a problem inspired by a company’s case. Indeed, we involved Filtrec S.p.A. in the design of the final project of ROAR III to offer a case-study experience to the grade-12 class. The final project required optimizing milk-run routes to deliver some goods to a set of contractors. This problem can be modeled as a vehicle routing problem. Students could present their work at the company’s headquarters and visit the company offices, warehouse, and research and development (R&D) department. Still related to possible applications of OR, during another lecture, students had the chance to learn about the company OPTIT s.r.l., thanks to the seminar held by Dr. S. Bortolomiol.

The OR topics studied in ROAR III were familiar with what was already presented during ROAR I and ROAR II because students were already familiar with linear, integer, and mixed-integer linear programming paradigms. We made this so that we would have more time to introduce the students to the primary and more advanced features of Python and PuLP while having a good starting point. Thus, this allowed students to learn a new programming language and digital technologies while strengthening existing modeling and problem-solving skills.

Feedback received from students and teachers, as well as from the company’s representatives, was quite positive. In terms of results, in the written tests, despite some mistakes and difficulties, students demonstrated to have acquired good competence. From the final questionnaires, most students believe they have learned basic notions of Python and PuLP. This knowledge, especially coding in a prevalent programming language, will be helpful not only in tackling optimization problems. Indeed, the main feature of OR is to be interdisciplinary, and being able to abstract and model a problem is critical in any scientific work.

Similarly to what we did for the previous units, all didactic material developed for ROAR III lectures is available in a public repository,⁶ also accessible from the official ROAR project website (only in Italian).⁷ The material is under the CC-BY-SA license. Anyone, especially any teacher or researcher, can use it directly as it is (e.g., following the organization of lectures in Section 4.8) or design activities customized to the students involved.

6.2. ROAR Extensions and Future Work

Since the idea of developing the ROAR initially arose in 2019, we have been carrying out the ROAR project as described in this work and in previous ones. We have also been implementing single or multiple ROAR units several times, involving other classes beyond the one of IIS Antonietti in different Italian regions. More precisely, we presented a course that takes inspiration from ROAR I twice (from November 2021 to March 2022 and from April to June 2023) to other grade 12 classes at the IIS “Majorana Cascino” in Piazza Armerina (Enna, Italy) and some grade 10–12 classes at the linguistic high school “Abramo Lincoln” in Enna. In April 2023, at the University of Milano-Bicocca, we introduced several high school students to ROAR I and ROAR II topics. Moreover, ROAR I will be carried out once more at IIS Antonietti in Spring 2024 and for the first time in Catania from December 2023 to April 2024. Furthermore, we designed several ROAR-based teacher training courses. In particular, we implemented a course on ROAR I in Catania from October to November 2021, from October 2022 to April 2023, and from October 2023 to April 2024, with participants from 14 higher secondary schools from eastern Sicily. In October 2022, we contributed to the OPS4Math project (Boccia et al. 2022) by holding two seminars in Naples. Finally, from February 2023 to May 2023, we held four meetings at the ITSOS Marie Curie higher secondary school in Cernusco sul Naviglio, where we introduced teachers to mathematical programming (ROAR I) and graph theory (ROAR II). We specify that all the schools where we replicated ROAR are schools that contacted us following seminars that we held at a national level and in which we presented the ROAR project and the results achieved. Some Sicilian schools, in particular, were part of existing teacher training projects, whose project managers welcomed the idea of training teachers on ROAR topics naturally.

As for further work, we will keep experimenting with ROAR and its extensions at several institutes in different Italian regions. Moreover, we will evaluate more deeply the impact of the entire three-year implementation at IIS Antonietti by performing a longitudinal study based on the results of all questionnaires filled in by students since ROAR I.

6.3. Final Suggestions and Insights

First, we want to remark that ROAR is for everyone. It is not suited only for excellent students or those who already have strong motivation in mathematics. Thus, we do not suggest implementing ROAR by only addressing a small, reserved group of students. From the results of this work, indeed, we can see that 70% of the students in our grade 12 class think that ROAR impacted them.

No matter the OR topics one chooses, the actual relevance is how these can transmit to students the usefulness of mathematics in practice. However, one must also be careful while doing this because the beauty of OR (or, in general, mathematics) does not depend only on its applicability. This feature should be exploited to increase interest in mathematics such that students will be more motivated to study more abstract and theoretical concepts without stopping at applications.

Then, to make students curious, one could present OR classical problems without telling all the details, sometimes not even the name of the problem class itself. Students may look for this information themselves if encouraged enough. This happened during ROAR II when we never mentioned “vehicle routing problems” when presenting the final project, but a student found out the definition by himself.

Regarding teaching methods, ROAR is entirely based on active learning. This was one of the successful aspects. At least in this three-year experimentation, we observed that the less traditional frontal lectures, the better. Students enjoyed the possibility of working in groups, collaborating, coopeting, or cooperating. Also, they positively evaluated project-based learning and the several digital technologies we made them use, from GeoGebra and Solver to Spyder and Colab. Interactive live quizzes with Mentimeter and Kahoot! always engaged them by creating a healthy and fun competitive environment.

Typically, a ROAR lecture should last at least two hours and include at least the presentation of one OR topic and the related group work (also in reversed order to favor discovery learning). As mentioned in Section 4.8, lectures are versatile: They can be adapted, always maintaining a proper alternation between frontal lectures and group work. In our experience, the biggest issue we experimenters faced was time management. In particular, during ROAR I, we underestimated the time for each activity. Another difficulty we faced was recording student group work on Teams when groups were close together or all in the same room. Because the audio results were unclear, this did not facilitate the analysis of the collected data. However, this is not necessary except for research purposes.

According to what emerged from the questionnaires, students appreciated the opportunity to better know the academia and industry environments during ROAR II and ROAR III, respectively. In this sense, we could have done more if we had been more experienced. For instance, we could have involved companies since ROAR I or organized a workshop at a university during each teaching unit. This kind of activity makes experimentation appealing to the students, especially during their last high school years when they wonder about and evaluate possible future paths.

Throughout the ROAR experimentation, students developed both technical and soft skills, such as problem solving, teamwork, time management, and public speaking, which they will find helpful in the future, regardless of what path they choose. Indeed, after ROAR, we do not expect all of them to pursue careers in STEM disciplines. Anyway, we hope that the myriad applications of OR fascinated them and, mostly, that we have given them a scientific approach to tackle any problem they face. Moreover, we hope that the publications about ROAR have contributed to bringing the community of researchers and teachers closer to these themes and that others will be inspired by this project and decide to present paths similar to or equal to ours.