Survey: Forecasting Software Trends in a Challenging World

The extreme social, governmental and environmental events over the past few years have highlighted the vulnerability of the world’s supply chain. Supply shortages and varying demand continue to cause major disruptions to businesses, challenging their forecasting systems. Although there is no forecasting system that guarantees correct, or even half-adequate, forecasts in all situations, the disruptions highlight the importance of having a resilient forecasting system that is capable of being adjusted to appropriately respond to rapidly changing circumstances. Thus, one of the new goals of this biennial survey was to assess the changes introduced by vendors in response to these recent challenges.

About the Survey

For the 2022 Forecasting Software Survey, we cover 24 software vendors, including some of the market leaders in forecasting software development (see Table 1 online for the full list of companies [1]). As with prior surveys, our goal with this article is to help practitioners make the first step in selecting software to improve their forecasting process. In the following, you will see both forecasting and IT aspects of the software. Most of the software solutions are general-purpose software, suitable for a wide range of industries and contexts. However, several software vendors are specialized and offer solutions tailored for organizations such as life science, manufacturing, utilities and higher education. In the online appendix, the interested reader can find the full list of nearly 200 forecasting software vendors that were invited to participate in the survey.

Table 1. Software vendors that participated in the 2022 Forecasting Software Survey (online)

Many companies without large data science teams are in search of automated solutions that provide good forecasts [2]. Software vendors are trying to address this by providing highly automated forecasting and reporting capabilities. Not surprisingly, many of the software’s unique selling propositions center around buzzwords like “ease of use,” “scalability” or its “powerful algorithm.” However, with machine learning (ML) and artificial intelligence (AI) algorithms becoming more popular, typically, the models become substantially more complex. In fact, simple AI implementations seem unlikely to work well, which makes software vendors look to more advanced instruments. At the same time, organizations have an increased need to embed forecasting software into the existing IT landscape and organizational processes. In the following sections, we elaborate on trends that facilitate those improvements in the forecast process and its use.

Algorithm-driven Trends

Continuing the trend observed in the past surveys [3], practitioners’ interest in AI/ML algorithms is very high. Almost all vendors have new algorithms in the latest versions of their software. In particular, trees and boosted trees are becoming increasingly popular among practitioners, whereas several vendors have updated their neural network packages.

With the availability of higher sampling frequencies such as daily or even hourly data, a notable number of products have improved the handling of date schemes and now support a larger range of predefined events, such as regional holidays and 53-week correction. Almost all solutions have some sort of automated outlier detection and can treat missing values. But there may be subtle differences between them when it comes to outlier detection with multiple seasonalities.

Although AI/ML algorithms are natural candidates to address complex seasonal patterns, high-frequency data tends to exhibit intermittency, having zeroes happening at random, especially for granular data at the SKU level. However, only half of the software solutions have algorithms that handle intermittent demand, even though there have been major advancements in algorithms that can handle it [4]. As pointed out by one of the vendors, a reason for this could be that most clients still seem to work with monthly or, at best, weekly data. We think intermittent demand models will become increasingly popular over the next few years because more and more software products offer higher-frequency data handling, whereas client decisions tend to happen on a more and more granular level to respond to challenges in the supply chain.

Amid the strong trend toward more automation, the recent disruptions have demonstrated its limitations. This is why organizations often wish for more interpretability and reproducibility, including model tracking within a forecasting process. Judgmental adjustments are becoming extremely important in situations of rapid change. However, organizations should keep in mind that impaired judgment can lead to biases and a decrease in forecasting accuracy [5]. Software plays a crucial role in facilitating and tracking the outcomes of such adjustments, including reporting the value added. Although most software vendors support adjustments, there is unfortunately little evaluation on how well those functions are designed in order to get the best out of judgment [6].

When it comes to interpretability and ease of use of ML algorithms, there is a move toward providing more interpretable insights. For example, recent algorithms such as Temporal Fusion Transformers (TFTs) aim to make it easier to understand the outputs of neural networks [7], even though they are typically considered to be black-box approaches.

A surprising observation is that rolling origin evaluation is only supported by less than half of the software vendors. Again, this omission might also be driven by the fact that organizations have limited data sets with three to four years of available history. We consider this a serious drawback because it limits model training options and data-based insights, particularly in periods of instability. We argue that in light of increasing interest in ML, rolling origin is essential to make sure that the complex approaches consistently perform better than the simpler ones.

Finally, in the previous survey, Fildes et al. speculated that prediction intervals might become more popular among practitioners [3]. However, they are still widely underused. A software vendor reported that outside of energy forecasting, prediction intervals are a very seldomly requested feature and are thus omitted even by world-leading supply chain companies (see also [8]). This might be surprising in times of high uncertainty but aligns well with the literature that suggests that prediction intervals are often hard to interpret, especially when there is a wide uncertainty range [9].

Process-driven Trends

The trend for more automation does not stop at the algorithm level but also applies to report generation, which has become increasingly important. The democratization of analytics reduces the requirement for coding and allows for a much more visual-driven process. As a consequence, collaborative features and better visual workflow have been some of the most frequently named recent improvements in forecasting software.

This also expands into what-if scenarios and simulations, which only a few solutions provide. In general, forecasts are desired through multiple levels of the organization, highlighting the need for a more coherent process to leverage the forecasting capabilities to the next level. Smartly designed software and probably a more visionary, cross-silo IT infrastructure may have more potential than searching for improved forecasting algorithms.

There are a variety of ways to consume software, ranging from bare metal installation up to Forecasting as a Service. We find that most software solutions now offer cloud connectivity but are also available as standalone solutions. We noted in the past survey that the investment in installation might reduce the switching costs between solutions [3]. However, with the support of cloud-based services, more money might be spent on customization and embedding company-specific process workflows.

IT-driven Trends

Over the past few years, software has become increasingly modular and customizable. Propelled by cloud-native support, more than two-thirds of companies now offer very flexible computational scalability for a range of cloud forms, from public to private and hybrid. The modular software design means vendors can provide tailored forecasting solutions. This allows smaller organizations to afford individual forecasting software without the need to invest in expensive hardware and maintenance or to stick to a generic solution. For larger organizations, the entire software infrastructure is becoming more manageable.

This is further propelled by the increased popularity of no-client-side installation – i.e., when the software runs in-browser or as a progressive web application. In turn, the change in software distribution also makes it more common to have a continuous delivery of features in contrast to the classical waterfall delivery, for which there used to be long waiting times until new features would enter production.

Another trend is that almost two-thirds of all solutions now directly integrate with open-source programming languages such as R and Python. This allows running state-of-the-art approaches to be on the edge of innovative science. However, one hurdle organizations might face with open-source approaches is that they are more difficult to scale to the performance required in a large corporate setting. Moreover, organizations often seem to struggle with the robustness of code and its maintenance. Ultimately, organizations aim to have a resilient IT infrastructure that operates with as little interruption as possible. The direct usage of open-source approaches in practice then becomes less attractive.

Over the years of conducting these software surveys, universal access to databases has become supported by most products. This is important because more and more data is distributed across a variety of servers and often consists of multiple data sources. However, we noticed a shift toward cloud data warehouse solutions such as Google BigQuery or Amazon Redshift. To companies, this allows using powerful business intelligence functionality that can help analyze data and support some calculations prior to loading it into specialized forecasting software.

Finally, data privacy and regulatory compliance can have a substantial impact on software selection. The survey covered questions on data security standards and service-level agreements. It is an area with a variety of options and is important to consider in the software selection process. For some companies, additional requirements might include data protection laws (CCPA and GDPR) or government security standards. We foresee that such standards will become more important, adding them to the list of software requirements. As a consequence, smaller software vendors might struggle to cover all country-specific laws, reducing their potential market.

Conclusions

Problems such as higher rates of natural disasters due to climate change, global chip and material shortages, energy market crisis, and conflicts (including the Russian invasion of Ukraine) will continue to severely disrupt regional and global supply chains. This causes additional challenges for software vendors because it is not clear when a new norm will be achieved. For this reason, we expect the trend in higher customizability of software to continue, allowing a more flexible adoption of forecasting strategies. Moreover, we also expect to see a much heavier focus on data security elements. We hope that in two years, we will be doing the survey in a substantially better economic and political situation.

Acknowledgments

We would like to thank Michele Trovero, Joe Katz, Spiros Potamitis, Catherine Owsik and Simon Spavound for feedback on earlier versions of the survey. We also express our gratitude to Tom Fink and Kara Tucker for helping us reach out to the software vendors and distributing the survey.

References

1. https://pubsonline.informs.org/magazine/orms-today/2022-forecasting-software-survey
2. Gusakov, I., 2021, “The <<Easy Button>> for forecasting,” Business Forecasting: The Emerging Role of Artificial Intelligence and Machine Learning, Hoboken, N.J.: Wiley, pp. 364-367.
3. Fildes, R., Schaer, O., Sventukov, I. and Yusupova, A., 2020, “Survey: What’s new in forecasting software?,” OR/MS Today, Aug. 4, https://doi.org/10.1287/orms.2020.04.05.
4. Boylan, J. and Syntetos, A., 2021, “Intermittent Demand Forecasting. Context, Methods and Applications,” 1st edition, Oxford: Wiley.
5. Fildes, R., Goodwin, P., Lawrence, M. and Nikolopoulos, K., 2009, “Effective forecasting and judgmental adjustments: An empirical evaluation and strategies for improvement in supply-chain planning,” International Journal of Forecasting, Vol. 25, No. 1, pp. 3-23, https://doi.org/10.1016/j.ijforecast.2008.11.010.
6. Fildes, R., 2021, “Statistical algorithms, judgment and forecasting,” Business Forecasting: The Emerging Role of Artificial Intelligence and Machine Learning, Hoboken, N.J.: Wiley, pp. 361-364.
7. Lim, B., Arik, S.Ö., Loeff, N. and Pfister, T., 2021, “Temporal Fusion Transformers for interpretable multi-horizon time series forecasting,” International Journal of Forecasting, Vol. 37, No. 4, pp. 1748-1764, https://doi.org/10.1016/j.ijforecast.2021.03.012.
8. Gilliland, M., 2020, “The value added by machine learning approaches in forecasting,” International Journal of Forecasting, Vol. 36, No. 1, pp. 161-166, https://doi.org/10.1016/j.ijforecast.2019.04.016.
9. Goodwin, P., 2014, “Getting real about uncertainty,” Foresight, No. 33, pp. 4-7.