Diagnostic Tools for Evaluating and Comparing Simulation-Optimization Algorithms

References

• Ali MM, Khompatraporn C, Zabinsky ZB (2005) A numerical evaluation of several stochastic algorithms on selected continuous global optimization test problems. J. Global Optim. 31(4):635–672.Crossref, Google Scholar
• Amaran S, Sahinidis NV, Sharda B, Bury SJ (2016) Simulation optimization: A review of algorithms and applications. Ann. Oper. Res. 240(1):351–380.Crossref, Google Scholar
• Andradóttir S (2006) An overview of simulation optimization via random search. Henderson SG, Nelson BL, eds. Simulation, Handbooks in Operations Research and Management Science, vol. 13 (Elsevier, North Holland), 617–631.Crossref, Google Scholar
• Andradóttir S (2015) A review of random search methods. Fu MC, ed. Handbook of Simulation Optimization, International Series in Operations Research & Management Science, vol. 216 (Springer, New York), 277–292.Crossref, Google Scholar
• Asmussen S, Glynn PW (2007) Stochastic Simulation: Algorithms and Analysis, Stochastic Modeling and Applied Probability, vol. 57 (Springer, New York).Crossref, Google Scholar
• Barton RR, Ivey JS Jr (1996) Nelder-Mead simplex modifications for simulation optimization. Management Sci. 42(7):954–973.Link, Google Scholar
• Bayraksan G, Morton DP (2006) Assessing solution quality in stochastic programs. Math. Programming 108(2):495–514.Crossref, Google Scholar
• Bayraksan G, Morton DP (2009) Assessing solution quality in stochastic programs via sampling. Informs TutORials in Operations Research, 102–122.Google Scholar
• Beiranvand V, Hare W, Lucet Y (2017) Best practices for comparing optimization algorithms. Optim. Engrg. 18(4):815–848.Crossref, Google Scholar
• Broadie M, Du Y, Moallemi CC (2011) Efficient risk estimation via nested sequential simulation. Management Sci. 57(6):1172–1194.Link, Google Scholar
• Broadie M, Du Y, Moallemi CC (2015) Risk estimation via regression. Oper. Res. 63(5):1077–1097.Link, Google Scholar
• Chang KH (2014) Improving the efficiency and efficacy of stochastic trust-region response-surface method for simulation optimization. IEEE Trans. Automatic Control 60(5):1235–1243.Crossref, Google Scholar
• Chang KH, Hong LJ, Wan H (2013) Stochastic trust-region response-surface method (STRONG)—A new response-surface framework for simulation optimization. INFORMS J. Comput. 25(2):230–243.Link, Google Scholar
• Chau M, Fu MC (2015) An overview of stochastic approximation. Fu MC, ed. Handbook of Simulation Optimization, International Series in Operations Research & Management Science, vol. 216 (Springer, New York), 149–178.Crossref, Google Scholar
• Chia YL, Glynn PW (2013) Limit theorems for simulation-based optimization via random search. ACM Trans. Model. Comput. Simulation 23(3):1–18.Crossref, Google Scholar
• Cooper K, Hunter SR, Nagaraj K (2020) Biobjective simulation optimization on integer lattices using the epsilon-constraint method in retrospective approximation framework. INFORMS J. Comput. 32(4):1080–1100.Abstract, Google Scholar
• Diouf MA, Dufour JM (2005) Improved nonparametric inference for the mean of a bounded random variable with application to poverty measures. Technical report, Université de Montréal, QC, Canada.Google Scholar
• Dolan ED, Moré JJ (2002) Benchmarking optimization software with performance profiles. Math. Programming 91(2):201–213.Crossref, Google Scholar
• Dong N, Eckman DJ, Zhao X, Poloczek M, Henderson SG (2017) Empirically comparing the finite-time performance of simulation-optimization algorithms. Chan WKV, D’Ambrogio A, Zacharewicz G, Mustafee N, Wainer G, Page E, eds. Proc. 2017 Winter Simulation Conf. (Institute of Electrical and Electronics Engineers, Inc., Piscataway, NJ), 2206–2217.Google Scholar
• Eckman DJ, Henderson SG, Pasupathy R (2019) Redesigning a testbed of simulation-optimization problems and solvers for experimental comparisons. Mustafee N, Bae KHG, Lazarova-Molnar S, Rabe M, Szabo C, Haas P, Son YJ, eds. Proc. 2019 Winter Simulation Conf. (Institute of Electrical and Electronics Engineers, Inc., Piscataway, NJ), 3457–3467.Google Scholar
• Eckman DJ, Henderson SG, Shashaani S (2022a) Diagnostic tools for evaluating and comparing simulation-optimization algorithms. Accessed November 18, 2022, http://dx.doi.org/10.5281/zenodo.7329235.Google Scholar
• Eckman DJ, Henderson SG, Shashaani S (2022b) SimOpt: A testbed for simulation-optimization experiments.Google Scholar
• Eckman DJ, Shashaani S, Sanchez SM (2022c) Data farming for simulation optimization. Working paper, Texas A&M University, College Station, TX.Google Scholar
• Eckman DJ, Henderson SG, Shashaani S, Pasupathy R (2020) Simulation optimization library. Accessed May 1, 2020, http://github.com/simopt-admin/simopt.Google Scholar
• Fu MC (2002) Optimization for simulation: Theory vs. practice. INFORMS J. Comput. 14(3):192–215.Link, Google Scholar
• Fu MC (2006) Gradient estimation. Henderson SG, Nelson BL, eds. Simulation, Handbooks in Operations Research and Management Science (Elsevier, North Holland), 575–616.Google Scholar
• Ghadimi S, Lan G (2015) Stochastic approximation methods and their finite-time convergence properties. Fu MC, ed. Handbook of Simulation Optimization, International Series in Operations Research & Management Science, vol. 216 (Springer, New York), 179–206.Crossref, Google Scholar
• Glynn PW (2002) Additional perspectives on simulation for optimization. INFORMS J. Comput. 14(3):220–222.Link, Google Scholar
• Gordy MB, Juneja S (2010) Nested simulation in portfolio risk measurement. Management Sci. 56(10):1833–1848.Link, Google Scholar
• Gould N, Scott J (2016) A note on performance profiles for benchmarking software. ACM Trans. Math. Software 43(2):1–5.Crossref, Google Scholar
• Gould NI, Orban D, Toint PL (2015) CUTEst: A constrained and unconstrained testing environment with safe threads for mathematical optimization. Comput. Optim. Appl. 60(3):545–557.Crossref, Google Scholar
• Homem-de Mello T, Bayraksan G (2014) Monte Carlo sampling-based methods for stochastic optimization. Surveys Oper. Res. Management Sci. 19(1):56–85.Crossref, Google Scholar
• Hong LJ, Juneja S, Liu G (2017) Kernel smoothing for nested estimation with application to portfolio risk measurement. Oper. Res. 65(3):657–673.Link, Google Scholar
• Kushner HJ, Yin GG (2003) Stochastic Approximation and Recursive Algorithms and Applications, 2nd ed. (Springer-Verlag, New York).Google Scholar
• Lan G, Nemirovski A, Shapiro A (2012) Validation analysis of mirror descent stochastic approximation method. Math. Programming 134(2):425–458.Crossref, Google Scholar
• Learned-Miller E, Thomas PS (2019) A new confidence interval for the mean of a bounded random variable. Preprint, submitted May 15, https://arxiv.org/abs/1905.06208v2.Google Scholar
• Lee SH (1998) Monte Carlo computation of conditional expectation quantiles. Unpublished PhD thesis, Stanford University, Stanford, CA.Google Scholar
• Lee SH, Glynn PW (2003) Computing the distribution function of a conditional expectation via Monte Carlo: Discrete conditioning spaces. ACM Trans. Model. Comput. Simulation 13(3):238–258.Crossref, Google Scholar
• Li J, Ryzhov IO (2022) Convergence rates of epsilon-greedy global optimization under radial basis function interpolation. Stochastic Systems, ePub ahead of print August 2, https://doi.org/10.1287/stsy.2022.0096.Google Scholar
• Mak WK, Morton DP, Wood RK (1999) Monte Carlo bounding techniques for determining solution quality in stochastic programs. Oper. Res. Lett. 24(1):47–56.Crossref, Google Scholar
• Matheson JE, Winkler RL (1976) Scoring rules for continuous probability distributions. Management Sci. 22(10):1087–1096.Link, Google Scholar
• Morales JL (2002) A numerical study of limited memory BFGS methods. Appl. Math. Lett. 15(4):481–487.Crossref, Google Scholar
• Moré JJ, Wild SM (2009) Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20(1):172–191.Crossref, Google Scholar
• Nelder JA, Mead R (1965) A simplex method for function minimization. Comput. J. 7(4):308–313.Crossref, Google Scholar
• Nemirovski A, Juditsky A, Lan G, Shapiro A (2009) Robust stochastic approximation approach to stochastic programming. SIAM J. Optim. 19(4):1574–1609.Crossref, Google Scholar
• Netlib (2021) Accessed March 3, 2021, http://netlib.org.Google Scholar
• Pasupathy R, Henderson SG (2006) A testbed of simulation-optimization problems. Perrone LF, Wieland FP, Liu J, Lawson BG, Nicol DM, Fujimoto RM, eds. Proc. 2006 Winter Simulation Conf. (Institute of Electrical and Electronics Engineers, Inc., Piscataway, NJ), 255–263.Google Scholar
• Pasupathy R, Henderson SG (2011) SimOpt: A library of simulation optimization problems. Jain S, Creasey RR, Himmelspach J, White KP, Fu M, eds. Proc. 2011 Winter Simulation Conf., 4075–4085 (Institute of Electrical and Electronics Engineers, Inc., Piscataway, NJ).Google Scholar
• Rosenbrock HH (1960) An automatic method for finding the greatest or least value of a function. Comput. J. 3(3):175–184.Crossref, Google Scholar
• Shashaani S, Hashemi FS, Pasupathy R (2018) ASTRO-DF: A class of adaptive sampling trust-region algorithms for derivative-free stochastic optimization. SIAM J. Optim. 28(4):3145–3176.Crossref, Google Scholar
• Shi HJM, Xuan MQ, Oztoprak F, Nocedal J (2021) On the numerical performance of derivative-free optimization methods based on finite-difference approximations. Preprint, submitted February 19, https://arxiv.org/abs/arXiv:2102.09762v1.Google Scholar
• Sun Y, Apley DW, Staum J (2011) Efficient nested simulation for estimating the variance of a conditional expectation. Oper. Res. 59(4):998–1007.Link, Google Scholar
• Wikipedia (2021) Test functions for optimization. Accessed March 3, 2021, https://en.wikipedia.org/wiki/Test_functions_for_optimization.Google Scholar
• Wild S (2019) Personal communication with authors, October.Google Scholar