Decision analysis

In 2018, the introduction and rapid adoption of once futuristic technologies has become the norm, unfortunately and frequently leaving the numerous hours of design, testing, fabrication and analysis – most importantly decision analysis - without much or any consideration. Prior to becoming realities, these technologies were items of imagination, so much so that they were utilized in movies to add to the “futuristic” element. Going back to “Space Odyssey” (2001), Facetime-like video and Siri-like artificial intelligence were introduced. Facebook took a page out of “The Minority Report” (2002) by using individual’s interests and history to drive personalized advertisements. Perhaps the most important technological advancement of our time, predicted in “The Net” (1995), was online pizza ordering and delivery [1].

These technologies have driven the development of new fields and domains to study, and implicitly with them, experts, models, data and analytics to continuously advance them. Yet, it is rare that a product or system today solely requires the expertise of only one individual or domain. The Encyclopedia of Information Science and Technology expands on this by explaining a core problem is bringing multiple professional communities together: “More specifically, to design effective serious games, the expertise of educators, content experts, and learning experts will need to be utilized on top of all other experts who are typically involved in game design” [2].The complexity inherent with the desired performance of today’s systems requires the combined expertise of many, thus continuously growing the need for collaboration amongst multiple experts when attempting to make decisions.

Project teams frequently struggle with optimally combining the inputs from stakeholders, experts and models to reach decisions. It is typical for the model builders to side with the model results, the experts to believe in their research, and the stakeholders to strive to advance their portion of the project or their leadership’s interests. What is the remedy to effectively managing these conflicting objectives? In addition to the complexities associated with the problem itself, “an effective decision analyst must understand the challenges of decision-making in organizations, the mathematical foundations of decision analysis, and the soft skills required to work with the decision makers, stakeholders, and experts” [3].

What tools can we use as thought leaders, decision-makers, managers and analysts to define these decision frameworks in our own projects? The results of this year’s biennial decision analysis software package survey are meant to assist in providing the tools and numerous capabilities available to us.

The Survey

In line with previous years, the 2018 survey was made available via an online questionnaire sent to vendors who previously participated or those recommended by staff. Vendors who did not respond by the deadline can still provide their software’s information by submitting a questionnaire (https://www.surveymonkey.com/r/D6NNZLH), and it will be added to the online version of the survey. Just as in 2016 and previous years, OR/MS Today provides the vendor responses verbatim and does not intend for the results to imply quality or cost effectiveness. Rather, the list serves to raise awareness of the variety of tools available. Approximately 60 multiple choice and short answer questions made up this year’s questionnaire. The primary focus was to define the current capabilities and any updates or advancements associated with each software package, enabling a high-level perspective on decision analysis software capabilities to our audience.

The primary goal of the software survey was to ask questions focused on the decision analysis capabilities of each tool. These include but are not limited to multiple competing objectives, probabilistic dependencies, stakeholder collaboration, value functions, strategy tables, elicitation and decision algorithms. Moving forward, vendors were asked questions to define the usability of their applications. To do so, they provided feedback on their software interfaces, visualization capabilities, model and data protection, user interfaces, and other usability related features. Types of training offered, as well as the recognized certifications associated with each software tool, were also investigated in the 2018 survey.

2018 Results

This year’s survey includes 28 software packages from 19 different vendors, domestic and international, who responded by the deadline (click here for results). Pricing for licenses varied, ranging from free products to a max of $100,000, depending on the user/license type and elements of the software package. Specific industries and markets for these software applications are widespread, including defense (military, aerospace and shipbuilding) healthcare (pharma, hospitals and clinics), energy-related domains (petroleum and nuclear energy), as well as many others.  

Decision analysis capabilities: Each of the decision analysis-focused capabilities proved to be available by at least one of the software tools reviewed. Uncertainty existed in every tool, yet implementation likely differed within the applications, and it is therefore worth investigating to find the one best fit for an individual project. Unfortunately, there were two minimally available features, an evidential reasoning (e.g., Bayesian belief networks) capability and portfolio decision-making, which do not exist in 79 percent and 48 percent, respectively, of the tools reviewed. Value functions/scores and risk tolerance existed in 75 percent of the tools, while strategy tables were available in 43 percent of them. Seven of the tools offered AHP, while 13 offered MODA/MAUT.

Usability: Usability features were fairly consistent among the vendor responses. XML integration and API (embedded decision support system) proved to be the capabilities most limited. Yet the inclusion of an API has increased since 2016 from 50 percent of the tools to 68 percent this year. Only eight of the tools offered simultaneous data input and viewing.

When considering data visualization, it is difficult to rely solely on the decision analysis software tool. Inherent in the name, these are decision analysis tools and therefore they provide the ability to output decision analysis-related visualizations such as graphical sensitivity analysis, expected value tornado diagrams, decision trees and influence diagrams, but stakeholders and decision-makers are usually not decision analysts. Thus, it is important to consider the exporting capabilities of these tools, particularly related to the export of data which existed in 86 percent of the tools. Exporting the data provides the ability to develop advanced visualizations, customizable to the desires and experience of project stakeholders, which may better convey results than the standard visualization existing within the software tools.

Licensing and training: The majority of the vendors (80 percent) reported that a limited use (run-time) or demo version of the tool is available. Nearly all of the products are available for purchase for educational (96 percent) and/or commercial (100 percent) use. As expected, many of these tools are offered free or at a discounted rate for educational use. The number of tools offering enhanced/high performance capabilities with these software packages showed a significant increase since 2016, from 35 percent to 75 percent this year.

Pricing for these advanced features varies significantly, typically based on the client’s needs and options selected. Multiple options for training were offered in relation to the software tools, mostly offered by the vendors themselves (79 percent). Online resources including training, tutorials and open source collaboration was offered for 75 percent of the tools, a nearly 25 percent increase since 2016.

Conclusion

This year’s decision analysis software survey demonstrates that software capabilities continue to advance, enabling users to conduct decision analysis regardless of industry, interfacing and elicitation needs, and desired visualizations or operating systems. When solving difficult problems, it is important to understand the resources available, interfacing requirements, desired decision analysis techniques to be utilized, required outputs and other problem-specific elements to select the tool that will best fit the analysis.

It is no surprise that every tool reviewed in this survey will not be the “perfect fit” for each decision analysis problem. Thus, the percentages mentioned earlier associated with nonexistent features is not detrimental to a tool’s potential performance for a specific project. A project on a tight timeline may not have the opportunity for iterative analysis; therefore, a tool offering this capability would provide no additional value.

In closing, this year’s software survey includes an impressive list of applications available to decision professionals, and the survey results can assist professionals in selecting the best decision analysis package for their particular needs.

Note: Survey results can be found here.

References

1. Pahle, Rebecca, 2016, “15 Movies That Predicted the Future,” Mental Floss, Sept. 15, mentalfloss.com/article/86080/15-movies-predicted-future.
2. “Gaming,” 2015, Encyclopedia of Information Science and Technology, Information Science Reference, pp. 3,291-3,291.
3. Parnell, Gregory S., and Bresnick, Terry A., 2013, “Introduction to Decision Analytics” in “Handbook of Decision Analysis” (by Parnell, Bresnick, Steven N. Tani and Eric R. Johnson), John Wiley & Sons.