The Future of Forecasting Competitions: Design Attributes and Principles

References

• Askanazi R, Diebold FX, Schorfheide F, Shin M (2018) On the comparison of interval forecasts. J. Time Series Anal. 39(6):953–965.Google Scholar
• Athanasopoulos G, Hyndman RJ (2011) The value of feedback in forecasting competitions. Internat. J. Forecasting 27(3):845–849.Google Scholar
• Athanasopoulos G, Hyndman RJ, Song H, Wu DC (2011) The tourism forecasting competition. Internat. J. Forecasting 27(3):822–844.Google Scholar
• Bojer CS, Meldgaard JP (2020) Kaggle forecasting competitions: An overlooked learning opportunity. Internat. J. Forecasting 37(2):587–603.Google Scholar
• Boylan JE, Goodwin P, Mohammadipour M, Syntetos AA (2015) Reproducibility in forecasting research. Internat. J. Forecasting 31(1):79–90.Google Scholar
• Chawla V (2020) How much is Kaggle relevant for real-life data science? Accessed January 5, 2021, https://analyticsindiamag.com/how-much-is-kaggle-relevant-for-real-life-data-science/.Google Scholar
• Crone SF, Hibon M, Nikolopoulos K (2011) Advances in forecasting with neural networks? Empirical evidence from the NN3 competition on time series prediction. Internat. J. Forecasting 27(3):635–660.Google Scholar
• Donoho D (2017) 50 years of data science. J. Comput. Graphical Statist. 26(4):745–766.Google Scholar
• Fildes R (2001) Beyond forecasting competitions. Internat. J. Forecasting 17(4):556–560.Google Scholar
• Fry C, Brundage M (2020) The M4 forecasting competition—A practitioner’s view. Internat. J. Forecasting 36(1):156–160.Google Scholar
• Gilliland M (2013) FVA: A reality check on forecasting practices. Foresight: Internat. J. Appl. Forecasting 29:14–18.Google Scholar
• Gilliland M (2019) The value added by machine learning approaches in forecasting. Internat. J. Forecasting 36(1):161–166.Google Scholar
• Goodman SN, Fanelli D, Ioannidis JPA (2016) What does research reproducibility mean? Sci. Translational Medicine 8(341):341ps12.Google Scholar
• Hong T, Pinson P, Fan S (2014) Global energy forecasting competition 2012. Internat. J. Forecasting 30(2):357–363.Google Scholar
• Hong T, Xie J, Black J (2019) Global energy forecasting competition 2017: Hierarchical probabilistic load forecasting. Internat. J. Forecasting 35(4):1389–1399.Google Scholar
• Hong T, Pinson P, Fan S, Zareipour H, Troccoli A, Hyndman RJ (2016) Probabilistic energy forecasting: Global energy forecasting competition 2014 and beyond. Internat. J. Forecasting 32(3):896–913.Google Scholar
• Hyndman RJ (2001) It’s time to move from “what” to “why.” Internat. J. Forecasting 17(4):567–570.Google Scholar
• Hyndman RJ (2020) A brief history of forecasting competitions. Internat. J. Forecasting 36(1):7–14.Google Scholar
• Hyndman RJ, Koehler AB (2006) Another look at measures of forecast accuracy. Internat. J. Forecasting 22(4):679–688.Google Scholar
• Hyndman RJ, Ahmed RA, Athanasopoulos G, Shang HL (2011) Optimal combination forecasts for hierarchical time series. Comput. Statist. Data Anal. 55(9):2579–2589.Google Scholar
• Kang Y, Hyndman RJ, Smith-Miles K (2017) Visualising forecasting algorithm performance using time series instance spaces. Internat. J. Forecasting 33(2):345–358.Google Scholar
• Kerr NL (1998) HARKing: Hypothesizing after the results are known. Personality Soc. Psych. Rev. 2(3):196–217.Google Scholar
• Lishner DA (2021) HARKing: Conceptualizations, harms, and two fundamental remedies. J. Theoretical Philosophical Psych. Forthcoming.Google Scholar
• Makridakis S, Hibon M (1979) Accuracy of forecasting: An empirical investigation. J. Roy. Statist. Soc. Ser. A 142(2):97–145.Google Scholar
• Makridakis S, Hibon M (2000) The M3-competition: Results, conclusions and implications. Internat. J. Forecasting 16(4):451–476.Google Scholar
• Makridakis S, Assimakopoulos V, Spiliotis E (2018) Objectivity, reproducibility and replicability in forecasting research. Internat. J. Forecasting 34(4):835–838.Google Scholar
• Makridakis S, Hyndman RJ, Petropoulos F (2020a) Forecasting in social settings: The state of the art. Internat. J. Forecasting 36(1):15–28.Google Scholar
• Makridakis S, Spiliotis E, Assimakopoulos V (2020b) The M4 competition: 100,000 time series and 61 forecasting methods. Internat. J. Forecasting 36(1):54–74.Google Scholar
• Makridakis S, Spiliotis E, Assimakopoulos V (2020c) The M5 accuracy competition: Results, findings and conclusions. Preprint, submitted October 6, https://www.researchgate.net/publication/344487258_The_M5_Accuracy_competition_Results_findings_and_ conclusions.Google Scholar
• Makridakis S, Spiliotis E, Assimakopoulos V (2020d) Predicting/hypothesizing the findings of the M4 competition. Internat. J. Forecasting 36(1):29–36.Google Scholar
• Makridakis S, Chatfield C, Hibon M, Lawrence M, Mills T, Ord K, Simmons LF (1993) The M2-competition: A real-time judgmentally based forecasting study. Internat. J. Forecasting 9(1):5–22.Google Scholar
• Makridakis S, Andersen A, Carbone R, Fildes R, Hibon M, Lewandowski R, Newton J, Parzen E, Winkler R (1982) The accuracy of extrapolation (time series) methods: Results of a forecasting competition. J. Forecasting 1(2):111–153.Google Scholar
• Makridakis S, Spiliotis E, Assimakopoulos V, Chen Z, Gaba A, Tsetlin I, Winkler RL (2020e) The M5 uncertainty competition: Results, findings and conclusions. Preprint, November 30, https://www.researchgate.net/publication/346493740_The_M5_Uncertainty_competition_Results_findings_and_conclusions.Google Scholar
• Montero-Manso P, Hyndman RJ (2021) Principles and algorithms for forecasting groups of time series: Locality and globality. Internat. J. Forecasting 37(4):1632–1653.Google Scholar
• Montero-Manso P, Athanasopoulos G, Hyndman RJ, Talagala TS (2020) FFORMA: Feature-based forecast model averaging. Internat. J. Forecasting 36(1):86–92.Google Scholar
• Newbold P, Granger CWJ (1974) Experience with forecasting univariate time series and the combination of forecasts. J. Roy. Statist. Soc. Ser. A 137(2):131–165.Google Scholar
• Nikolopoulos K, Petropoulos F (2018) Forecasting for big data: Does suboptimality matter? Comput. Oper. Res. 98:322–329.Google Scholar
• Oreshkin BN, Carpov D, Chapados N, Bengio Y (2019) N-beats: Neural basis expansion analysis for interpretable time series forecasting. Preprint, submitted May 24, https://arxiv.org/abs/1905.10437.Google Scholar
• Petropoulos F, Wang X, Disney SM (2019) The inventory performance of forecasting methods: Evidence from the M3 competition data. Internat. J. Forecasting 35(1):251–265.Google Scholar
• Petropoulos F, Apiletti D, Assimakopoulos V, Babai MZ, Barrow DK, Ben Taieb S, Bergmeir C, et al. (2020) Forecasting: Theory and practice. Preprint, submitted December 4, https://arxiv.org/abs/2012.03854.Google Scholar
• Semenoglou AA, Spiliotis E, Makridakis S, Assimakopoulos V (2020) Investigating the accuracy of cross-learning time series forecasting methods. Internat. J. Forecasting 37(3):1072–1084.Google Scholar
• Smyl S (2020) A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. Internat. J. Forecasting 36(1):75–85.Google Scholar
• Spiliotis E, Kouloumos A, Assimakopoulos V, Makridakis S (2020a) Are forecasting competitions data representative of the reality? Internat. J. Forecasting 36(1):37–53.Google Scholar
• Spiliotis E, Petropoulos F, Kourentzes N, Assimakopoulos V (2020b) Cross-temporal aggregation: Improving the forecast accuracy of hierarchical electricity consumption. Appl. Energy 261:114339.Google Scholar
• Tashman LJ (2000) Out-of-sample tests of forecasting accuracy: An analysis and review. Internat. J. Forecasting 16(4):437–450.Google Scholar
• Winkler RL (1972) A decision-theoretic approach to interval estimation. J. Amer. Statist. Assoc. 67(337):187–191.Google Scholar