Modeling Complexity Index

Most scientists and engineers understand the difficulty of work within their field of expertise, but the same is not always the case when accessing tools and capabilities used outside of their formal training. In this context, discussions between members of broader communities in an organization and analytics groups often reveal misalignments regarding the difficulty associated with modeling requests. These misalignments represent failures to communicate, which lead to poor resource allocation and lower appreciation of what the modeling groups deliver.

The creation of indexes to improve how we describe difficult-to-quantify ordinal concepts is common. One recent noteworthy example appeared in the December issue of Significance in which the author outlines a new health index [1] and demonstrates how this tool facilitates broader thinking of important issues related to health.

Challenges in gauging the effort required to complete modeling tasks encouraged the development of the Modeling Complexity Index (MCI) detailed herein [2-5]. The MCI should provide a clear basis to communicate the difficulty level of modeling requests, with the scale ranging from 1 (minor) to the more improbable level of 11 (visionary). Descriptions of each level, the approximate time required to complete a task within each level, and examples are provided in Table 1 to enhance clarity, thus facilitating better prioritization of modeling projects and helping communicate the value these bring.

Table 1. Modeling Complexity Index (MCI)

Note. Only odd numbers are used to allow for higher fidelity when assigning intermediate situations (e.g., even numbers) commonly encountered in complex research environments.

Minor (1): Tasks in this category only require well-known methods, currently available technologies and little time to complete. A single modeling team member can typically handle minor requests with very little effort. Examples of minor requests are to 1) create simple visualizations, 2) run basic analyses and 3) build statistical learning models of nominal complexity.

Standard (3): Similar to minor work, the tools required for standard efforts are currently available internally or via a free download. However, the work may require small modifications to default settings, attention to software idiosyncrasies, aggregations of many trivial calculations for further analyses, etc. Two examples of standard efforts include 1) building multivariate models requiring minor data wrangling or 2) extending data-driven models to incorporate new data or diverse data types.

Challenging (5): Although work in this category requires known theory, the tools are not available in-house and need significant resources to bring online. When the tools are easily acquired, the work consists of major code modifications, nontrivial amounts of compute time or combining multiple levels of theory. Three examples of challenging work include 1) modeling extremely large, very noisy, high-dimension and complex data; 2) creating a mathematical framework to model novel systems; and 3) significant recoding of known methods in languages not available in-house (need extensive training or bringing in new hires or consultants).

Very Difficult (7): As with challenging tasks, the theory is known for tasks in the “very difficult” category, but the tools for implementation are mostly underdeveloped or do not exist anywhere. This type of effort may require external resources. Two examples include 1) implementing a very new and untested method or computational paradigms under development or 2) integrating data of very different types that are stored with multiple database technologies broadly distributed.

Herculean (9): Similar to the “very difficult” level, all the basic science is known. However, there are aspects not well understood that require further theoretical development. This category may include expensive and sophisticated solutions that are not easily available. Unless you work in an organization with extensive resources, external help is almost always required for work at the Herculean level. An example of Herculean effort would be the development and integration of multiple modeling paradigms of level 7 working in tandem to be deployed globally.

Visionary (11): The MCI could end at level 9, but experience suggests another level is needed to represent the uncommon, but inevitable, requests to solve problems that are best described as visionary. The science required for tasks in this category is only known at a fundamental level, and many details are not fully developed. The technologies for implementation are generally not well established and therefore constitute long-term endeavors with significant uncertainties. Multiple organizations are likely necessary to complete work in this category (e.g., academics, vendors, government laboratories). The development of a general artificial intelligence agent to solve a wide range of problems is an example of a visionary request commonly encountered. As a fan of the movie “Spinal Tap,” the lure of using 11 for this category could not be helped.

Although many MCI applications are possible, the scale described here will help communication between modeling teams and the communities they support. Specifically, it will assist with resource allocation and expectation management by facilitating a better understanding of the amount of time, tools and people required. In doing so, it establishes reference points for the value analytics and modeling teams bring to their organizations.

References

1. Cockle, S., 2022, “Revolutionizing the Way Health is Measured,” Significance, Vol. 19, No. 6, December.
2. Broekel, T., “Measuring Technological Complexity - Current Approaches and a New Measure of Structural Complexity,” arXiv:1708.07357.
3. Albeaik, S., Kaltenberg, M., Alsaleh, M. and Hidalgo, C. A., 2017, “Improving the Economic Complexity Index,” https://doi.org/10.48550/arXiv.1707.05826.
4. Bonchev, D. and Buck, G., 2005, “Quantitative measures of network complexity,” Complexity in Chemistry, Biology and Ecology, New York: Springer Verlag.
5. West, A., 2009, “NASA Study on Flight Software Complexity,” NASA Office of the Chief Engineer, https://www.nasa.gov/pdf/418878main_FSWC_Final_Report.pdf.