Machine Learning-Assisted Stochastic Kriging Metamodel for Offline Simulation Online Application

References

• Adorio EP, Diliman U (2005) MVF—Multivariate test functions library in C for unconstrained global optimization. Working paper, Department of Mathematics, University of the Philippines Diliman, Quezon City, Philippines.Google Scholar
• Ankenman B, Nelson BL, Staum J (2010) Stochastic kriging for simulation metamodeling. Oper. Res. 58(2):371–382.Link, Google Scholar
• Bachoc F (2021) Asymptotic analysis of maximum likelihood estimation of covariance parameters for Gaussian processes: An introduction with proofs. Daouia A, Ruiz-Gazen A, eds. Advances in Contemporary Statistics and Econometrics: Festschrift in Honor of Christine Thomas-Agnan (Springer, Cham, Switzerland), 283–303.Crossref, Google Scholar
• Barton RR (2015) Tutorial: Simulation metamodeling. Yilmaz L, Chan VW, Moon IC, Roeder TM, Macal C, Rossetti MD, eds. Proc. 2015 Winter Simulation Conf. (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 1765–1779.Google Scholar
• Barton RR (2020) Tutorial: Metamodeling for simulation. Bae KH, Feng B, Kim S, Lazarova-Molnar S, Zheng Z, Roeder T, Thiesing R, eds. Proc. 2020 Winter Simulation Conf. (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 1102–1116.Google Scholar
• Barton RR, Nelson BL, Xie W (2014) Quantifying input uncertainty via simulation confidence intervals. INFORMS J. Comput. 26(1):74–87.Link, Google Scholar
• Beckers T, Umlauft J, Hirche S (2018) Mean square prediction error of misspecified Gaussian process models. Proc. 2018 IEEE Conf. Decision Control (CDC) (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 1162–1167.Google Scholar
• Bekki JM, Chen X, Batur D (2014) Steady-state quantile parameter estimation: An empirical comparison of stochastic kriging and quantile regression. Tolk A, Diallo SY, Ryzhov IO, Yilmaz L, Buckley S, Miller JA, eds. Proc. 2014 Winter Simulation Conf. (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 3880–3891.Google Scholar
• Binois M, Gramacy RB, Ludkovski M (2018) Practical heteroscedastic Gaussian process modeling for large simulation experiments. J. Comput. Graphical Statist. 27(4):808–821.Crossref, Google Scholar
• Blundell C, Cornebise J, Kavukcuoglu K, Wierstra D (2015) Weight uncertainty in neural network. Bach F, Blei D, eds. Proc. 32nd Internat. Conf. Machine Learn., Proceedings of Machine Learning Research, vol. 37 (PMLR, New York), 1613–1622.Google Scholar
• Bottou L (2010) Large-scale machine learning with stochastic gradient descent. Lechevallier Y, Saporta G, eds. Proc. COMPSTAT’2010 (Physica-Verlag HD, Heidelberg), 177–186.Google Scholar
• Bowman F (2012) Introduction to Bessel Functions (Dover, New York).Google Scholar
• Buduma N, Locascio N (2017) Fundamentals of Deep Learning: Designing Next-Generation Machine Intelligence Algorithms (O’Reilly Media, Santa Rosa, CA).Google Scholar
• Cai X, Yang Y, Jiang G (2020) Online risk measure estimation via natural gradient boosting. Bae KH, Feng B, Kim S, Lazarova-Molnar S, Zheng Z, Roeder T, Thiesing R, eds. Proc. 2020 Winter Simulation Conf. (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 2341–2352.Google Scholar
• Chen X, Kim KK (2013) Building metamodels for quantile-based measures using sectioning. Pasupathy R, Kim SH, Tolk A, Hill R, Kuhl ME, eds. Proc. 2013 Winter Simulation Conf. (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 521–532.Google Scholar
• Chen X, Kim KK (2014) Stochastic kriging with biased sample estimates. ACM Trans. Model. Comput. Simulation 24(2):8.Crossref, Google Scholar
• Chen X, Kim KK (2016) Efficient VaR and CVaR measurement via stochastic kriging. INFORMS J. Comput. 28(4):629–644.Link, Google Scholar
• Chen H, Lam H (2023) Pseudo-Bayesian optimization. Preprint, submitted October 15, https://arxiv.org/abs/2310.09766.Google Scholar
• Chen H, Huang Z, Lam H, Qian H, Zhang H (2021) Learning prediction intervals for regression: Generalization and calibration. Banerjee A, Fukumizu K, eds. Proc. 24th Internat. Conf. Artificial Intelligence Statist., Proceedings of Machine Learning Research, vol. 130 (PMLR, New York), 820–828.Google Scholar
• Chen X, Ankenman BE, Nelson BL (2012a) The effects of common random numbers on stochastic kriging metamodels. ACM Trans. Model. Comput. Simulation 22(2):7.Crossref, Google Scholar
• Chen X, Ankenman BE, Nelson BL (2013) Enhancing stochastic kriging metamodels with gradient estimators. Oper. Res. 61(2):512–528.Link, Google Scholar
• Chen X, Nelson BL, Kim KK (2012b) Stochastic kriging for conditional value-at-risk and its sensitivities. Laroque C, Himmelspach J, Pasupathy R, Rose O, Uhrmacher AM, eds. Proc. 2012 Winter Simulation Conf. (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 1–12.Google Scholar
• Ding L, Zhang X (2024) Sample and computationally efficient stochastic kriging in high dimensions. Oper. Res. 72(2):660–683.Link, Google Scholar
• Feldman Z, Mandelbaum A, Massey WA, Whitt W (2008) Staffing of time-varying queues to achieve time-stable performance. Management Sci. 54(2):324–338.Link, Google Scholar
• Franke J, Neumann MH (2000) Bootstrapping neural networks. Neural Comput. 12(8):1929–1949.Crossref, Google Scholar
• Gao S, Li C, Du J (2019) Rate analysis for offline simulation online application. Mustafee N, Bae KHG, Lazarova-Molnar S, Rabe M, Szabo C, Haas P, Son YJ, eds. Proc. 2019 Winter Simulation Conf. (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 3468–3479.Google Scholar
• Goodwin T, Xu J, Celik N, Chen CH (2024) Real-time digital twin-based optimization with predictive simulation learning. J. Simulation 18(1):47–64.Crossref, Google Scholar
• Gramacy RB (2020) Surrogates: Gaussian Process Modeling, Design, and Optimization for the Applied Sciences (Chapman and Hall/CRC, Boca Raton, Fl).Crossref, Google Scholar
• Hayashi F (2000) Econometrics (Princeton University Press, Princeton, NJ).Google Scholar
• Hong LJ, Jiang G (2019) Offline simulation online application: A new framework of simulation-based decision making. Asia-Pacific J. Oper. Res. 36(06):1940015.Crossref, Google Scholar
• Hong LJ, Zhang X (2021) Surrogate-based simulation optimization. Emerging Optim. Methods Model. Techniques Appl. 2021(October):287–311.Google Scholar
• Huo H, Zhang X, Zheng Z (2018) A scalable approach to enhancing stochastic kriging with gradients. Rabe M, Juan AA, Mustafee N, Skoogh A, Jain S, Johansson B, eds. Proc. 2018 Winter Simulation Conf. (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 2213–2224.Google Scholar
• Jalali H, Van Nieuwenhuyse I, Picheny V (2017) Comparison of kriging-based algorithms for simulation optimization with heterogeneous noise. Eur. J. Oper. Res. 261(1):279–301.Crossref, Google Scholar
• Jiang G, Hong LJ, Nelson BL (2020) Online risk monitoring using offline simulation. INFORMS J. Comput. 32(2):356–375.Abstract, Google Scholar
• Jiang M, Pedrielli G, Ng SH (2022) Gaussian processes for high-dimensional, large data sets: A review. Feng B, Pedrielli G, Peng Y, Shashaani S, Song E, Corlu CG, Lee LH, Chew EP, Roeder T, Lendermann P, eds. Proc. 2022 Winter Simulation Conf. (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 49–60.Google Scholar
• Kiefer J, Wolfowitz J (1955) On the theory of queues with many servers. Trans. Amer. Math. Soc. 78(1):1–18.Crossref, Google Scholar
• Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. Preprint, submitted December 22, https://doi.org/10.48550/arXiv.1412.6980.Google Scholar
• Kleijnen JPC (2015) Kriging metamodels and their designs. Design and Analysis of Simulation Experiments, International Series in Operations Research & Management Science, vol. 230 (Springer, Cham, Switzerland), 179–239.Crossref, Google Scholar
• Kleijnen JP, Mehdad E (2016) Estimating the variance of the predictor in stochastic kriging. Simulation Model. Practice Theory 66:166–173.Crossref, Google Scholar
• Krivulin NK (1994) A recursive equations based representation for the G/G/M queue. Appl. Math. Lett. 7(3):73–77.Crossref, Google Scholar
• Li C, Gao S, Du J (2023) Convergence analysis of stochastic kriging-assisted simulation with random covariates. INFORMS J. Comput. 35(2):386–402.Link, Google Scholar
• Li F, Chen H, Lin J, Gupta A, Tan X, Zhao H, Xu G, Nevmyvaka Y, Capponi A, Lam H (2024) Prediction-enhanced Monte Carlo: A machine learning view on control variate. Preprint, submitted December 15, https://arxiv.org/abs/2412.11257.Google Scholar
• Lim E, Glynn PW (2023) Simulation-based prediction. Oper. Res. 71(1):47–60.Link, Google Scholar
• Lin Y, Nelson BL (2016) Simulation analytics for virtual statistics via k nearest neighbors. Roeder TMK, Frazier PI, Szechtman R, Zhou E, Huschka T, Chick SE, eds. Proc. 2016 Winter Simulation Conf. (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 448–459.Google Scholar
• Lin H, Chen H, Choromanski KM, Zhang T, Laroche C (2020) Demystifying orthogonal Monte Carlo and beyond. Larochelle H, Ranzato M, Hadsell R, Balcan MF, Lin H, eds. Adv. Neural Inform. Processing Systems, vol. 33 (Curran Associates, Inc., Red Hook, NY), 8030–8041.Google Scholar
• Meng Q, Wang S, Ng SH (2022) Combined global and local search for optimization with Gaussian process models. INFORMS J. Comput. 34(1):622–637.Link, Google Scholar
• Ouyang H, Nelson BL (2017) Simulation-based predictive analytics for dynamic queueing systems. Chan WKV, D’Ambrogio A, Zacharewicz G, Mustafee N, Wainer G, Page E, eds. Proc. 2017 Winter Simulation Conf. (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 1716–1727.Google Scholar
• Paass G (1993) Assessing and improving neural network predictions by the bootstrap algorithm. Hanson SJ, Cowan JD, Giles CL, eds. Adv. Neural Inform. Processing Systems, vol. 5 (Morgan Kaufmann Publishers Inc., San Francisco), 196–203.Google Scholar
• Pearce M, Branke J (2018) Continuous multi-task Bayesian optimisation with correlation. Eur. J. Oper. Res. 270(3):1074–1085.Crossref, Google Scholar
• Peng XB, Andrychowicz M, Zaremba W, Abbeel P (2018) Sim-to-real transfer of robotic control with dynamics randomization. Proc. 2018 IEEE Internat. Conf. Robotics Automation (ICRA) (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 3803–3810.Google Scholar
• Plumlee M (2014) Fast prediction of deterministic functions using sparse grid experimental designs. J. Amer. Statist. Assoc. 109(508):1581–1591.Crossref, Google Scholar
• Qu H, Fu MC (2014) Gradient extrapolated stochastic kriging. ACM Trans. Model. Comput. Simulation 24(4):23.Crossref, Google Scholar
• Quan N, Yin J, Ng SH, Lee LH (2013) Simulation optimization via kriging: A sequential search using expected improvement with computing budget constraints. IIE Trans. 45(7):763–780.Crossref, Google Scholar
• Reed S, Lee H, Anguelov D, Szegedy C, Erhan D, Rabinovich A (2014) Training deep neural networks on noisy labels with bootstrapping. Preprint, submitted December 20, https://doi.org/10.48550/arXiv.1412.6596.Google Scholar
• Salemi PL, Song E, Nelson BL, Staum J (2019) Gaussian Markov random fields for discrete optimization via simulation: Framework and algorithms. Oper. Res. 67(1):250–266.Link, Google Scholar
• Semelhago M, Nelson BL, Song E, Wächter A (2021) Rapid discrete optimization via simulation with Gaussian Markov random fields. INFORMS J. Comput. 33(3):915–930.Link, Google Scholar
• Shen H, Hong LJ, Zhang X (2021) Ranking and selection with covariates for personalized decision making. INFORMS J. Comput. 33(4):1500–1519.Abstract, Google Scholar
• Simon JB, Dickens M, DeWeese MR (2021) Neural tangent kernel eigenvalues accurately predict generalization. Preprint, submitted October 8, https://arxiv.org/abs/2110.03922.Google Scholar
• Sun L, Hong LJ, Hu Z (2014) Balancing exploitation and exploration in discrete optimization via simulation through a Gaussian process-based search. Oper. Res. 62(6):1416–1438.Link, Google Scholar
• van der Vaart AW (2000) Asymptotic Statistics, vol. 3 (Cambridge University Press, Cambridge, UK).Google Scholar
• van der Vaart AW, Wellner J (2013) Weak Convergence and Empirical Processes: With Applications to Statistics (Springer Science & Business Media, New York).Google Scholar
• Wang W, Chen X (2016) The effects of estimation of heteroscedasticity on stochastic kriging. Roeder TMK, Frazier PI, Szechtman R, Zhou E, Huschka T, Chick SE, eds. Proc. 2016 Winter Simulation Conf. (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 326–337.Google Scholar
• Wang S, Ng SH (2020) Enhancing response predictions with a joint Gaussian process model for stochastic simulation models. ACM Trans. Model. Comput. Simulation 30(1):4.Crossref, Google Scholar
• Wang S, Ng SH, Haskell WB (2022) A multilevel simulation optimization approach for quantile functions. INFORMS J. Comput. 34(1):569–585.Link, Google Scholar
• Wang W, Tuo R, Wu CFJ (2020b) On prediction properties of kriging: Uniform error bounds and robustness. J. Amer. Statist. Assoc. 115(530):920–930.Crossref, Google Scholar
• Wang H, Yuan J, Ng SH (2020a) Gaussian process based optimization algorithms with input uncertainty. IISE Trans. 52(4):377–393.Crossref, Google Scholar
• Wang W, Chen N, Chen X, Yang L (2019) A variational inference-based heteroscedastic Gaussian process approach for simulation metamodeling. ACM Trans. Model. Comput. Simulation 29(1):6.Crossref, Google Scholar
• Williams CK, Rasmussen CE (2006) Gaussian Processes for Machine Learning, vol. 2 (MIT Press, Cambridge, MA).Google Scholar
• Wynne G, Briol FX, Girolami M (2021) Convergence guarantees for Gaussian process means with misspecified likelihoods and smoothness. J. Machine Learn. Res. 22(1):5468–5507.Google Scholar
• Xie W, Nelson BL, Barton RR (2014) A Bayesian framework for quantifying uncertainty in stochastic simulation. Oper. Res. 62(6):1439–1452.Link, Google Scholar
• Xie W, Nelson BL, Barton RR (2020) Statistical uncertainty analysis for stochastic simulation. Preprint, submitted November 9, https://arxiv.org/abs/2011.04207.Google Scholar
• Xie W, Yi Y, Zheng H (2021) Global-local metamodel-assisted stochastic programming via simulation. ACM Trans. Model. Comput. Simulation 31(1):2.Crossref, Google Scholar
• Yang F, Ankenman BE, Nelson BL (2008) Estimating cycle time percentile curves for manufacturing systems via simulation. INFORMS J. Comput. 20(4):628–643.Link, Google Scholar
• Yin J, Ng SH, Ng KM (2011) Kriging metamodel with modified nugget-effect: The heteroscedastic variance case. Comput. Indust. Engrg. 61(3):760–777.Crossref, Google Scholar
• Yun X, Hong LJ, Jiang G, Wang S (2019) On gamma estimation via matrix kriging. Naval Res. Logist. 66(5):393–410.Crossref, Google Scholar
• Zhan D, Zhang H, Righter R, Zheng Z, Anderson J (2025) Collaborative Bayesian optimization via Wasserstein barycenters. Proc. 2025 IEEE 64th Conf. Decision Control (CDC) (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 6284–6291.Google Scholar
• Zhang H, Xu J (2025) Robustify simulation uncertainty quantification against input data outlier. Proc. 2025 Annual Modeling and Simulation Conf. (ANNSIM), article 67 (Society for Modeling and Simulation International (SCS), San Diego, CA), 12.Google Scholar
• Zhang H, Zheng Z (2020) Simulating nonstationary spatio-temporal poisson processes using the inversion method. Bae KH, Feng B, Kim S, Lazarova-Molnar S, Zheng Z, Roeder T, Thiesing R, eds. Proc. 2020 Winter Simulation Conf. (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 492–503.Google Scholar
• Zhang H, He J, Righter R, Zheng Z (2024) Language model prompt selection via simulation optimization. Preprint, submitted April 12, https://doi.org/10.48550/arXiv.2404.08164.Google Scholar
• Zhang H, He J, Zhan D, Zheng Z (2021a) Neural network-assisted simulation optimization with covariates. Kim S, Feng B, Smith K, Masoud S, Zheng Z, Szabo C, Loper M, eds. Proc. 2021 Winter Simulation Conf. (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 1–12.Google Scholar
• Zhang H, He J, Righter R, Shen ZJ, Zheng Z (2023) Contextual Gaussian process bandits with neural networks. Oh A, Naumann T, Globerson A, Saenko K, Hardt M, Levine S, eds. Adv. Neural Inform. Processing Systems, vol. 36 (Curran Associates, Inc., Red Hook, NY), 26950–26965.Google Scholar
• Zhang H, He J, Righter R, Shen Z-JM, Zheng Z (2026) Machine learning-assisted stochastic kriging metamodel for offline simulation online application. https://doi.org/10.1287/ijoc.2023.0130.cd, https://github.com/INFORMSJoC/2023.0130.Google Scholar
• Zhang J, Zhang Y, Hong M, Sun R, Luo ZQ (2021b) When expressivity meets trainability: Fewer than n neurons can work. Ranzato M, Beygelzimer A, Dauphin Y, Liang PS, Wortman Vaughan J, eds. Adv. Neural Inform. Processing Systems, vol. 34 (Curran Associates, Inc., Red Hook, NY).Google Scholar
• Zhao W, Queralta JP, Westerlund T (2020) Sim-to-real transfer in deep reinforcement learning for robotics: A survey. Proc. 2020 IEEE Sympos. Ser. Comput. Intelligence (SSCI) (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 737–744.Google Scholar