Decision Analysis @ 20

This leadoff article in the very first issue of Decision Analysis provides an exhaustive survey of decision analysis applications published during the period 1990-2001 in major English-language operations research and closely related journals, as well as perspective on trends and developments in decision analysis applications. Comments on this article are included in the same issue, as well as our response to those comments. A supplemental technical report provides more details about each application.

This type of article is useful to a variety of readers. Decision analysis instructors can use it to demonstrate the range of applications where decision analysis has been applied and to find illustrative applications for their classes. Students can use it to identify fields where decision analysis has been used, as well as specific applications that may be of personal interest. Theoretically-oriented decision analysts can use it to find applications that may give them ideas about what new theory might be particularly useful to develop to support decision analysis applications, as well as possible illustrative applications for new theory. Applied decision analysts who may have focused on a particular type of application can use it to broaden their knowledge about applications areas where decision analysis has proved especially useful, as well as which specific methods have been most useful. Since there are no equations or other very technical material in the article, it can be profitably read by people with widely different levels of knowledge about decision analysis.

Because the article is of interest to all these different classes of readers, it is not surprising that it is a heavily cited article from the first volume of Decision Analysis. Looking back at the article from twenty years later, it still provides useful information about applications for these different types of readers, as well as a historical record of decision analysis applications during the period 1990-2001. Combined with an earlier application review article referenced in this article that covers the period 1970-1989, the two articles provide an overview of many of the published decision analysis application during three decades when decision analysis was first being widely applied.

In 1964, I (James Matheson) encoded my first probability distribution, one on royalties from a new Westinghouse TV tube. To my surprise, it was bimodal! When I asked why, the subject said that certain companies usually licensed from Westinghouse, but it also might become the industry standard. So, I broke the distribution into two conditional probabilities, plus a probability of becoming the standard.

This was a forerunner of influence diagrams, which are simply a directed signal-flow graph of conditional probabilities. The important feature in constructing them is minimizing the number of arrows (conditions) flowing into each node to minimize the complexity of assessment. Reversing an arrow is the application of Bayes’ law, which can be done only if the two nodes being reversed have the same inputs.

In the late 1970s, a team within the Stanford Research Institute Decision Analysis Group was working on ways to characterize the probabilistic relationships among political events in the Middle East. The work led to the formalization and publication of influence diagrams. Initially, some reviewers said they were too simple, and others said they were too complex, so we had to self-publish. Over two decades later, when INFORMS finally published this paper, it quickly became the most referenced paper in the journal.

Another key feature was the addition of decision nodes. This allowed the graphical formulation of sequential decision problems. It also allowed easy consideration of the value of perfect information by simply adding an arrow for an uncertainty to a decision. This has been generalized to the value of control, and finally to both imperfect information and control.

Surface-to-air missile attacks on large commercial airplanes by terrorists were considered a real threat in the mid-2000s. Around 2004, the Department of Homeland Security invested $200 million per year to study the costs and benefits of airplane-mounted countermeasures to defend against such attacks. Motivated by the large price tag of $25 to $35 billion life-cycle cost, we wanted to explore whether this cost was worth the risk reduction. We initiated an independent study, funded by the Center for Risk and Economic Analysis of Terrorism Events at the University of Southern California, and re-examined the life-cycle cost as well as the reduction of risks to passengers, planes, and the national economy. We found that it was difficult to make a case for installing these countermeasures, as shown in detail in our paper. Partly as a result of this analysis, the countermeasures program was cancelled in 2006.

I wrote this paper to help decision analysts choose among the quadratic (Q), spherical (S), and logarithmic (L) scoring rules. Specifically, I wanted others to understand and appreciate some of the differences of among these rules, even though they are all strictly proper and were designed to encourage assessors to reveal their true beliefs.

In particular, I thought that L scoring was underappreciated because analysts felt the possibility of an assessor earning a negative infinity was too severe. I also felt that analysts did not appreciate the rank-order problems that were bound to arise through the use of Q and S scoring.

In the paper, I examine ex ante and ex post properties of Q, S, and L scoring. From an ex-post perspective, I show that under Q or S scoring, an assessor can assign a higher probability to the event that occurs, but receive a lower score than another assessor who assigned a lower probability to that outcome. This introduces a degree of arbitrariness into the forecast evaluation process that I think is difficult to defend. From an ex-ante perspective, I show that L scoring is the least affected by the assessor’s risk aversion. This was surprising, because L holds the possibility of unbounded losses.

I also endeavored to carefully review and synthesize the scoring-rules literature up to that point. I think this feature of the paper is likely what readers have found to be of most value. I am pleased that others have found my work to be useful, and I am grateful to the referees and editors who helped me improve my work and to the Decision Analysis journal for publishing it.

Several factors contributed to the recognition of this paper and its high level of citations:

1. Given its pedagogical nature, the article has been systematically used as a structured example of the application of the Measuring Attractiveness by a Categorical Based Evaluation Technique (MACBETH) approach to thousands of students of various nationalities, who have been attending masters, PhD, and other specialization courses.
2. Also related to its style of writing, the article has served as a guide for the application of its approach to various companies and organizations, allowing the learning of the fundamental principles of multicriteria evaluation processes and the transposition of theoretical concepts to their application in real decision-support cases. In particular, the article provides protocols of questioning – which are not often explained in articles – and explains the pros and cons of the adopted protocols of questioning and which other protocols could be used.
3. Scientifically, the evaluation problem proposed and implemented in the manuscript is developed in a real setting with a client in the energy industry– the Portuguese Electric Transmission Company. Accordingly, it describes how to develop a reusable model for evaluating call for tenders, which is a problem context relevant to many other companies and cases; it portrays a model that was implemented in practice; and both the evaluation approach and the developed models entail innovative features. Such features include modelling features adopted to produce a reusable model and to deal with the requirement of a 50% weight for costs and benefits (without incurring the most common critical mistake).

My prospect-theory model of resource allocation started with an idea that I could derive simple predictions from prospect theory about resource allocation to risky activities. An extremely large number of papers claim that prospect theory predicts risk aversion above, and risk seeking below, the reference point. My simulation with the actual model showed this was not correct, but had a complex set of results.

This paper led me to do a simpler paper, “Looking at Prospect Theory” (Strategic Management Journal, 2010), which just plotted risk propensities for various gambles. P. Bromiley and Rau, “Some Problems in Using Prospect Theory to Explain Strategic Management Issues” (Academy of Management Perspectives, 2022), extended the analysis.

This led me to conclude that the prospect-theory literature has several problems:

1. It talks only about the implications of the value function, ignoring the probability-weighting function. For gambles with probabilities not near 1 or 0, the probability-weighting function is risk averse.
2. Even looking at just the value function, the literature ignores these properties of the model:
1. The model predicts extreme risk aversion for mixed gambles (i.e., those with both positive and negative potential outcomes). The normal predictions hold only for gambles with all positive or all negative outcomes, but almost all real-world risky situations involve mixed gambles. (Indeed, an individual with numerous all-positive outcome choices would eventually raise the reference point to make some choices have negative potential outcomes. That is, if someone averages 25% return on investments, pretty soon the reference point becomes 25%.)
2. The model’s risk-propensity predictions hold only close to the reference point. Further from the reference point, the model predicts risk neutrality. However, exactly what “close to” and “far from” mean is unclear, and probably varies across individuals or organizations. A small gamble for a wealthy person (or large company) is a big gamble for a poor person (or small company).

Anyway, almost all prospect-theory papers had ignored these features of the model. Note that I’m talking only about what the model predicts – whether it matches reality is a separate issue.

The majority of citations to the paper come from the management field. There is also a smattering of citations from accounting, decision analysis, and even veterinary medicine. So, I guess the paper got a lot of citations because it showed that what people thought was not correct, and tied into a large research community in management dealing with risk.

In finance, people make risky choices by trading off risk and return of alternatives. Risk attitude determines this trade-off. Hardly anyone uses risk attitude derived from the curvature of a utility function to rationalize the trade-off.

We analyzed how subjects make (hypothetical) stock market investments, and found that decisions could best be explained by subjective risk and return judgments of investment alternatives. As a measure of risk attitude, a simple measure (asking subjects’ willingness to take risk, on a scale from 1 to 5) worked best, whereas risk attitude derived from lottery choices (i.e., curvature of utility function) had no significant explanatory power.

I wrote this paper to clarify the hard evidence and formal analyses on the relative performance of heuristic and optimization models, where there is a correct, knowable decision. When does it pay to go simple, as laypeople and experts typically do, and when to make more complex calculations, as academics are prone to? This question has caused a commotion in our and related fields, from psychology and economics to statistics and machine learning. I am glad to see that the objectivity and balanced presentation of the paper were appreciated.

When we wrote this paper, we had started to develop the first concepts and ideas about adversarial risk analysis (ARA), but we had developed mainly theory. We were looking for an interesting case study to illustrate the ideas and better understand the required modeling. The problem referred to a boat being hijacked in the Somali coast, a current topic in Spain at the time, with several recent incidents of the kind, and our Navy contributing to the protecting international coalition. The topic made it to popular tech magazines like Wired and even to the famous film Captain Phillips, starring Tom Hanks. It was relatively easy to get data and experts willing to provide the required judgements, and we were able to deliver the paper. The underlying methodology was in line with the ARA templates we had developed before. We also had to provide a model for the question “Why me?” (i.e., why the bad guys are attacking me and not my neighbor), which we addressed informally. I think the interest in the paper was because it was one of the first applications of ARA, and has served to inspire many other later applications. Incidentally, with the growing interest in risk-analysis issues with AI-infused systems, such “Why me” questions will become increasingly important, requiring more formal answers.

Kenneth Arrow’s famous impossibility theorem proved that there was no logical way to create a logical group ranking of alternatives by combining the individual group members rankings for those alternatives. Some individuals thought that this implied that there was no logical way to make group decisions. As a decision analyst, I felt that if each individual in a group used decision analysis to calculate the expected utilities of the alternatives considered for a decision, there would be a logically sound “group decision analysis” to combine the group member’s utilities to obtain a group expected utility for each alternative. The article “Foundations of Group Decision Analysis” presents this result.

This paper arose from our work on a geopolitical forecasting tournament, where we wrestled with ways to decide which forecasters would provide the best out-of-sample forecasts. Because proper scoring rules were important in this tournament, we focused on the extent to which the “proper” attribute of scoring rules constrained forecaster evaluation. The paper is basically a sensitivity analysis of loss functions, as applied to forecasting, and is not the most groundbreaking paper out there (which a reviewer also pointed out!). We think it continues to receive attention because it addresses a problem that is relevant to both basic and applied forecasting research, and because we tried to write the paper in an accessible fashion.

This paper arose from a collaboration that was lots of fun. It began when Lyle Ungar announced in a meeting that probabilistic forecasts got better Brier scores if they were all raised to 1 or lowered to 0 (whichever was closer). I (Jonathan Baron) replied, “That can’t be right.” Barbara Mellers then suggested a function like the pi function in prospect theory. I checked this, and it worked, but it turned out that all this had been discovered before, including the function we used, which had some surprisingly nice properties. The real contribution of the paper was to ask why extremizing (sometimes) helped. One reason is the compression of probability judgments near 0 and 1, but we discovered that this was insufficient, so we came up with a “second reason.”

The paper was almost never cited because of its actual contribution. Rather, it was cited for the (earlier) idea of extremizing. Surely, it came to people’s attention from other documents about the Good Judgment Project, which attracted a great deal of interest. It was the only paper in that series that was published in Decision Analysis. Still, I like this paper, including the appendices, and am happy for the attention.

In “The Composition of Optimally Wise Crowds,” we aimed to formally characterize psychological traits that result in “wiser” crowds. For example, some individuals may be systematically biased in their judgments, but if these judgments are uncorrelated with those of other group members, a wiser crowd may result from their inclusion. Our paper builds upon prior work by P.J. Lamberson and S. E. Page, “Optimal Forecasting Groups” (Management Science, 2012), who examined a similar question, by extending their results to more general (and, as such, more realistic) conditions. Across many domains, such as business and politics, there is a tangible need for understanding how to construct the most effective group possible. Our paper provides a practical approach for thinking about this problem.

Since their inception, network-interdiction problems have evolved, integrating realistic elements that bridge the gap between theory and practical applications. In our research, we took on the task of addressing a significant gap in this field – the lack of information available to the interdictor. A notable aspect of our work, which we believe is of great interest to the community, is our approach to closing this gap from an operations-research perspective rather than a statistical one. Instead of solely focusing on asymptotic cost estimation, our approach centers on the challenges associated with reconstructing the optimal solution, illuminating the inherent difficulties involved.

The study demonstrates the usage of the industry’s emerging social media analytics tools and techniques. It explains how these approaches have been used to address issues unique to each industry, and their benefits over conventional techniques. The paper addresses how social media analytics has been applied to decision analysis for cross-functional tasks and various business functions in different industries. Thereby, the study moves beyond a pure theoretical perspective, which makes the study relevant for advanced students who would want to understand the applications in industrial contexts. Further, the study connects social media analytics to traditional data-mining activities, which would enable a decision maker to make data-driven decisions.

Social media as a platform elicits honest signals from different stakeholders regarding business contexts. It allows sending of these signals through environment scanning. Subsequently, after removing noise, it allows us to build explanatory models for modelling the needs of different stakeholders based on user generated content. These models have high generalizability, as the sample sizes would be large. The input from various information sources can affect a decision maker’s conclusions. Based on shared preferences, values, expertise, and situations, these collaborative platforms for individuals and groups produce a wide range of options that might be more practical, holistic, and complete to build predictive models through a rather inductive research methodology. Social media analytics is a fast-emerging field to aid businesses in boosting their efforts at process improvement through various business processes.

Our aim when writing this paper was to argue that our community needs to realize that in many circumstances, Bayesian decision analysis will only be one input to the political process that leads to the actual societal decision. For complex decisions, there will often be many reports contributed to the political decision makers, both by official bodies and by many different stakeholder groups. These may be based on different assumptions, data, and evidence, and may use different, sometimes contradictory methodologies. Some will be broadcast on social and other media, some kept confidential within government. The decision makers need to be cognizant of all, and subsequently advise stakeholders and the general public of their decision. This led us to argue that our community of decision analysts needs to deconstruct our paradigm, and attend more to communicating the result of the analysis in comparison with other inputs to the societal decision, so that the evidence can be fairly understood and weighed accordingly.

Reflecting on the many societal decisions taken during the Covid-19 pandemic, we are now even more convinced of this. In the United Kingdom, we saw many groups (some official, some outside established political circles) running analyses, and contributing them to the general debate and discussion of actions to take. These groups were comprised variously of established experts, self-proclaimed experts, and others of varying competence. The analyses were based on varying assumptions, different methodologies (Bayesian and non-Bayesian), data sets of differing provenances, and so on, as we had suggested that they would. Some of the results were included in the government’s overall deliberations, others broadcast over social and other media. In short, the political decision makers and the public were confronted with a mire of contradictory advice, and no tools or means of comparing the basis and import of each. This inevitably both complicated the political decision-making process and also undermined public confidence. While our experience is based on what happened in the United Kingdom, we are aware of similar confused discussions in other countries.

The broad community of analysts who seek to support societal decisions, be they Bayesians or other members of the wider multiple-criteria decision-making (MCDM) community, need to debate the results of their analyses in a constructive manner, and present a much fairer and wider perspective on the issues than their own analyses alone bring. It is not helpful if, in our case, we ignore other inputs to the decisions, simply because they use methodologies that may contravene Savage’s and other Bayesian principles of rationality. See also S. French, “Reflections on 50 Years of MCDM: Issues and Future Research Needs” (EURO Journal on Decision Processes, 2023).

“Probability Forecasts and Their Combination: A Research Perspective” was written as a reflection on developments related to working with probability forecasts. Our hope was that others might find our summary of such developments, and the connections we drew among them, to be useful. At the time, we anticipated that we would see greater use and aggregation of probability forecasts – given, for instance, developments in machine learning and expert forecasting, and increases in the reporting of probabilities in the media. What we did not fully anticipate was just how much we would witness on this front. With the extensive coverage and use of probability forecasting and ensembles during COVID, the rise of sports analytics and use of probabilities in sports betting, the increased fascination with political forecasting that stems from geopolitical instabilities, and the unstoppable momentum related to tech and AI, probabilities are everywhere. Experts and the general public cannot escape them, and many decision makers find them helpful. Our vision was that increased exposure to and improved visualizations of probability forecasts would enhance the public’s understanding of probabilities and contribute to better decision making. This vision was realized, and these rapid changes are a continuing process, so there is more work to be done. We hope more research and development on probability forecasts and their combination will continue in response.

Cybersecurity is a key issue in modern society, providing many challenges of interest to our discipline, and to operations research and management science at large. It is also worth noting that major risk-analysis standards in the cybersecurity domain are essentially qualitative and based on risk matrices, which is a bit worrisome, given the important stakes at play in some instances (as with critical infrastructures). The motivation for this paper was aligned with this last observation – namely, that objectives and attributes are traditionally considered from a technical (and limited) point of view, with a focus on the criticality/ integrity/availability triad. However, the relevance of cyber systems necessarily leads us to open our minds and think in terms of economic, environmental, health, and reputational impacts, to name but a few. That is what we tried to achieve in this paper – providing a general multiple-objective preference model for cybersecurity risk management adaptable to multiple users, and compatible with standard decision-analytic requirements. The interest in this paper is likely due to the global interest in cybersecurity, and the increasing interest in this community for better-founded approaches, such as those that decision analysis may provide. Let us conclude with a final challenge: Given the growing interest in cybersecurity issues in AI-infused systems, it would be interesting to see if our general cybersecurity tree would need to be modified and updated to take into account the objectives appearing in, e.g., the AI risk-management framework of the U.S. National Institute of Standards and Technology, or the “Proposal for a Regulation laying down harmonised rules for artificial intelligence” of the European Union.

Our work focuses on a hybrid preference system combining two popular preference representation systems (i.e., ranked voting and approval voting), which can interest researchers from both ranked voting and approval voting areas. The hybrid preference system studied in the paper is based on an axiomatic approach, which provides conditions for deriving a distance function that is used to aggregate preferences for group decision making and achieving group consensus. Achieving consensus is a popular problem in decision analysis, negotiation, social choice, group decision making, and other relevant social sciences. The axiomatic distance function also provides a solid theoretical foundation for achieving consensus as a result of a simple minimization problem. Therefore, it is easy to find potential application areas of the method proposed in this paper and build on the current work to develop some further studies.

This paper provides a comprehensive and rigorous study of information security strategies for information-sharing firms facing a strategic hacker. We find that two kinds of security efforts (security investment and security knowledge sharing) act as strategic substitutes when the degree of business information sharing between two firms is low, and act as strategic complements otherwise. The research findings have significant practical implications for firms in understanding the benefits and risks of information sharing, and in making informed security decisions. In writing this paper, we aimed to contribute to the existing knowledge in the field of information security economics, and provide management insights to help firms mitigate security risks associated with business information sharing.

We are honored that our article, “Using Decision Analysis to Determine the Feasibility of a Conservation Translocation,” was among the top downloaded articles in Decision Analysis in 2023. Possible reasons for this are many:

1. As the biodiversity crisis worldwide continues to be urgent and complex while resources are limited, the frequency, taxonomic diversity, and geographic breadth of conservation translocations to save species and restore ecological function continues to escalate;
2. Our application of decision analysis to conduct a feasibility assessment for a conservation translocation is a unique approach that has piqued the interest of those in both the decision analysis and conservation fields; and
3. The open-access format of our publication makes it accessible to practitioners who may not otherwise have easy access to journal articles.

We are grateful for the interest in the article, and hope that it inspires more practical applications of decision analysis in conservation.

Our team at Mississippi State University and the United States Department of Agriculture would like to express sincere thanks to all the readers of Decision Analysis who catapulted our paper into being one of the most downloaded papers of 2023. Our project was inspired by a Soil Science Society of America webinar, in which various producers expressed their experiences and concerns regarding climate-smart agriculture and selling carbon credits. Although we knew our paper would be of interest to a wide range of stakeholders, we suspect that the greater interest was due to two additional factors. First, there is ongoing and accelerating concern about climate change, and our paper comprehensively tests one of the popularly discussed, innovative solutions with unknown environmental, social, and economic consequences. Second, to our knowledge, our paper is among the few applications of structured decision-making to an environmental sustainability problem that combines theoretical, experimental, and economic modeling to consider how an increasing world population can balance trade-offs between food and economic security in the face of climate change. We hope this study provides a template that demonstrates the power of this method, and inspires others to apply structured decision-making to climate and agricultural issues.