Teaching Novices to Model

Professor Joel Sokol's "Issues in Education" column, "Teaching O.R. Modeling," in the December 2005 issue of OR/MS Today raised questions that are important for the future health and effectiveness of O.R. He asked how best to teach the core skill of modeling and how to know that our courses are effective.

Sokol's concern (and ours) is not with the teaching of algorithmic or mathematical knowledge. Instead, it is with solving realistic, ill-structured problems in which O.R. techniques can make important contributions. In brief, our concern is not with the type of problems encountered at the end of textbook chapters, but rather with the gnarly type encountered on the job.

Sokol's questions have broad implications because of the democratization of modeling. The need to build and use models is no longer restricted to experts. Novice modelers build spreadsheet models every day in thousands of businesses and nonprofit organizations in support of managerial decision-making. Novices also routinely encounter models built by others. However, novices typically have little or no training in modeling per se. Little is known about how they go about their tasks, whether they succeed, or on how best to train novice O.R. modelers.

Fortunately, little is not the same as none. There has been some exploratory research on how experts make models, how novice modelers differ from experts, and how to evaluate courses that emphasize modeling. This note summarizes key findings from several studies: Willemain 1994, Willemain 1995, Waisel et al. 2005, Willemain and Powell 2005, Powell and Willemain 2005, Wang and Brooks 2005.

1. Experts share certain ways of approaching modeling problems. They think broadly, not getting hung up on details. They conceive of multiple approaches. They use sketches to visualize alternatives. They range broadly "upstream" of the modeling to understand the client and his problem and "downstream" to plot how to reify and implement their models. They reserve a significant portion of their brainpower for monitoring and assessing their progress, with reference to their client's needs, deadlines and other practical constraints, and their own professional standards. They change course when necessary.
   
   
2. Novices have different ways of approaching modeling problems. They think narrowly and are easily distracted by details. They often settle prematurely on a single approach and follow it uncritically, often to a bad conclusion. They take shortcuts to a specific answer instead of developing a general model that could provide better justification and more insight. They are reluctant to create variables and relationships from a problem statement, though such abstraction is the essence of modeling. (Creating variables is not the same as manipulating variables that someone else gives to them.) They do not monitor their own progress, making critical evaluations and keeping an eye on the clock. They often confuse talking about a problem or brainstorming possible solutions with actually modeling.
   
   
3. Courses devoted to teaching modeling can be evaluated with some rigor. Powell executed an evaluation of his "Art of Modeling" course (Powell 1995, 1998) that compared treatment and control groups before and after the course. Students "thought aloud" as they worked special 30-minute modeling exercises. Their comments were tape recorded, transcribed and coded using a schema that has also been used with experts in the United States (Willemain 1995) and with both novices and experts in the United Kingdom (Wang and Brooks 2005).

Comparison of the coded transcripts with corresponding expert transcripts documented the contrasts between novice and expert modelers noted above. Likewise, comparisons were made between experts and novices in their use of modeling keywords. This pilot study had low statistical power, but did show weak statistical evidence of benefit. Qualitative analysis of the transcripts of the four subjects from the course showed that some key modeling concepts (e.g., prototyping) did seep into the students' consciousness. Overall, though, the evaluation failed to show dramatic improvements, so Sokol's question about how best to teach modeling remains to be answered. But the details presented in Willemain and Powell (2005) and Powell and Willemain (2005) permit anyone interested to replicate Powell's study in a different setting, with a different course, and with more subjects.

References

1. Powell, S. G., 1995, "Teaching the art of modeling to MBA students," Interfaces, Vol. 25, pgs. 88-94.
2. Powell, S. G., 1998, "The studio approach to teaching the craft of modeling," Annals of Operations Research, Vol. 82, pgs. 29-47.
3. Powell, S. G. and Willemain, T. R., 2005, "These open-ended problems are bloody difficult: How MBA students formulate models," Technical Report 38-05-520, Department of Decision Sciences and Engineering Systems, Rensselaer Polytechnic Institute, 2005.
4. Waisel, L. B., Wallace, W. A. and Willemain, T. R., 2005, "Visualization and model formulation: An analysis of the sketches of expert modelers," under review.
5. Wang, W. and Brooks, R., 2005, "Improving the understanding of conceptual modeling," working paper, Lancaster University Department of Management Science, November 2005.
6. Willemain, T. R., 1994, "Insights on modeling from a dozen experts," Operations Research, Vol. 42, pgs. 213-222.
7. Willemain, T. R., 1995, "Model formulation: What experts think about and when," Operations Research, Vol. 43, pgs. 916-932.
8. Willemain, T. R. and Powell, S. G., 2005, "How novices formulate models: A quantitative description of behavior," Technical Report 38-05-519, Department of Decision Sciences and Engineering Systems, Rensselaer Polytechnic Institute, 2005.