Help
Cart
Skip main navigation

Available Issues

April 16, 2013 - May 25, 2026

Available Issues

April 16, 2013 - May 25, 2026

Data-Driven Ranking and Selection Under Input Uncertainty

Published Online:18 Oct 2022https://doi.org/10.1287/opre.2022.2375

References

  • Barton RR, Schruben LW (1993) Uniform and bootstrap resampling of empirical distributions. Evans GW, Mollaghasemi M, Russell EC, Biles WE, eds. Proc. 1993 Winter Simulation Conf. (IEEE, Piscataway, NJ), 503–508.Google Scholar
  • Barton RR, Nelson BL, Xie W (2014) Quantifying input uncertainty via simulation confidence intervals. INFORMS J. Comput. 26(1):74–87.Link, Google Scholar
  • Batur D, Kim SH (2006) Fully sequential selection procedures with a parabolic boundary. IIE Trans. 38(9):749–764.Crossref, Google Scholar
  • Bechhofer RE (1954) A single-sample multiple decision procedure for ranking means of normal populations with known variances. Ann. Math. Statist. 25(1):16–39.Crossref, Google Scholar
  • Chang JY, Hsu JC (1992) Optimal designs for multiple comparisons with the best. J. Statist. Planning Inference 30(1):45–62.Crossref, Google Scholar
  • Cheng RC, Holland W (1997) Sensitivity of computer simulation experiments to errors in input data. J. Statist. Comput. Simulation 57(1–4):219–241.Crossref, Google Scholar
  • Chick SE (2001) Input distribution selection for simulation experiments: Accounting for input uncertainty. Oper. Res. 49(5):744–758.Link, Google Scholar
  • Corlu CG, Biller B (2013) A subset selection procedure under input parameter uncertainty. Pasupathy R, Kim S-H, Tolk A, Hill R, Kuhl ME, eds. Proc. 2013 Winter Simulation (IEEE, Piscataway, NJ), 463–473.Google Scholar
  • Corlu CG, Biller B (2015) Subset selection for simulations accounting for input uncertainty. Yilmaz L, Chan WKV, Moon I, Roeder TMK, Macal C, Rossetti MD, eds. Proc. 2015 Winter Simulation Conf. (IEEE, Piscataway, NJ), 437–446.Google Scholar
  • Corlu CG, Akcay A, Xie W (2020) Stochastic simulation under input uncertainty: A review. Oper. Res. Perspect. 7:100162.Crossref, Google Scholar
  • Eckman DJ, Feng MB (2018) Green simulation optimization using likelihood ratio estimators. Rabe M, Juan AA, Mustafee N, Skoogh A, Jain S, Johansson B, eds. Proc. 2018 Winter Simulation Conf. (IEEE, Piscataway, NJ), 2049–2060.Google Scholar
  • Even-Dar E, Mannor S, Mansour Y (2002) PAC bounds for multi-armed bandit and Markov decision processes. Kivinen J, Sloan RH, eds. Internat. Conf. Comput. Learn. Theory (Springer, Berlin), 255–270.Google Scholar
  • Even-Dar E, Mannor S, Mansour Y (2006) Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. J. Machine Learn. Res. 7:1079–1105.Google Scholar
  • Fan W, Hong LJ, Nelson BL (2016) Indifference-zone-free selection of the best. Oper. Res. 64(6):1499–1514.Link, Google Scholar
  • Fan W, Hong LJ, Zhang X (2020) Distributionally robust selection of the best. Management Sci. 66(1):190–208.Link, Google Scholar
  • Feng BM, Song E (2019) Efficient input uncertainty quantification via green simulation using sample path likelihood ratios. Mustafee N, Bae K-HG, Lazarova-Molnar S, Rabe M, Szabo C, Haas P, Son Y-J, eds. Proc. 2019 Winter Simulation Conf. (IEEE, Piscataway, NJ), 3693–3704.Google Scholar
  • Feng M, Staum J (2015) Green simulation designs for repeated experiments. Yilmaz L, Chan WKV, Moon I, Roeder TMK, Macal C, Rossetti MD, eds. Proc. 2015 Winter Simulation Conf. (IEEE, Piscataway, NJ), 403–413.Google Scholar
  • Feng M, Staum J (2017) Green simulation: Reusing the output of repeated experiments. ACM Trans. Model. Comput. Simulation 27(4):1–28.Crossref, Google Scholar
  • Frazier PI (2014) A fully sequential elimination procedure for indifference-zone ranking and selection with tight bounds on probability of correct selection. Oper. Res. 62(4):926–942.Link, Google Scholar
  • Fu MC, ed. (2015) Handbook of Simulation Optimization, vol. 216 (Springer, Berlin).Crossref, Google Scholar
  • Gabillon V, Ghavamzadeh M, Lazaric A (2012) Best arm identification: A unified approach to fixed budget and fixed confidence. Pereira F, Burges CJ, Bottou L, Weinberger KQ, eds. Adv. Neural Inform. Processing Systems, vol. 25 (MIT Press, Cambridge, MA), 3212–3220.Google Scholar
  • Gao S, Xiao H, Zhou E, Chen W (2017) Robust ranking and selection with optimal computing budget allocation. Automatica 81:30–36.Crossref, Google Scholar
  • Garivier A, Kaufmann E (2016) Optimal best arm identification with fixed confidence. Feldman V, Rakhlin A, Shamir O, eds. Conf. Learn. Theory (PMLR, New York), 998–1027.Google Scholar
  • Goldsman D, Kim SH, Marshall WS, Nelson BL (2002) Ranking and selection for steady-state simulation: Procedures and perspectives. INFORMS J. Comput. 14(1):2–19.Link, Google Scholar
  • Ho YC, Sreenivas R, Vakili P (1992) Ordinal optimization of DEDS. Discrete Event Dynamic Systems 2(1):61–88.Crossref, Google Scholar
  • Hong LJ (2006) Fully sequential indifference-zone selection procedures with variance-dependent sampling. Naval Res. Logist. 53(5):464–476.Crossref, Google Scholar
  • Hong LJ (2009) Estimating quantile sensitivities. Oper. Res. 57(1):118–130.Link, Google Scholar
  • Hong LJ, Nelson BL (2005) The tradeoff between sampling and switching: New sequential procedures for indifference-zone selection. IIE Trans. 37(7):623–634.Crossref, Google Scholar
  • Hong LJ, Nelson BL (2007) Selecting the best system when systems are revealed sequentially. IIE Trans. 39(7):723–734.Crossref, Google Scholar
  • Jamieson K, Nowak R (2014) Best-arm identification algorithms for multi-armed bandits in the fixed confidence setting. Czachórski T, Gelenbe E, Lent R, eds. 48th Annual Conf. Inform. Sci. Systems (IEEE, Piscataway, NJ), 1–6.Google Scholar
  • Karnin Z, Koren T, Somekh O (2013) Almost optimal exploration in multi-armed bandits. Dasgupta S, McAllester D, eds. Internat. Conf. Machine Learn. (PMLR, New York), 1238–1246.Google Scholar
  • Kaufmann E, Cappé O, Garivier A (2016) On the complexity of best-arm identification in multi-armed bandit models. J. Machine Learn. Res. 17(1):1–42.Google Scholar
  • Kim SH, Nelson BL (2001) A fully sequential procedure for indifference-zone selection in simulation. ACM Trans. Model. Comput. Simulation 11(3):251–273.Crossref, Google Scholar
  • Kim SH, Nelson BL (2006) On the asymptotic validity of fully sequential selection procedures for steady-state simulation. Oper. Res. 54(3):475–488.Link, Google Scholar
  • Lam H, Qian H (2022) Subsampling to enhance efficiency in input uncertainty quantification. Oper. Res. 70(3):1891–1913.Link, Google Scholar
  • Lam H, Zhou E (2017) The empirical likelihood approach to quantifying uncertainty in sample average approximation. Oper. Res. Lett. 45(4):301–307.Crossref, Google Scholar
  • Lin Y, Song E, Nelson BL (2015) Single-experiment input uncertainty. J. Simulation 9(3):249–259.Crossref, Google Scholar
  • Liu T, Zhou E (2019) Online quantification of input model uncertainty by two-layer importance sampling. Preprint, submitted December 24, https://arxiv.org/abs/1912.11172.Google Scholar
  • Liu T, Zhou E (2020) Simulation optimization by reusing past replications: Don’t be afraid of dependence. Bae K-H, Feng B, Kim S, Lazarova-Molnar S, Zheng Z, Roeder T, Thiesing R, eds. Proc. 2020 Winter Simulation Conf. (IEEE, Piscataway, NJ), 2923–2934.Google Scholar
  • Liu T, Lin Y, Zhou E (2021) A Bayesian approach to online simulation optimization with streaming input data. Kim S, Feng B, Smith K, Masoud S, Zheng Z, Szabo C, Loper M, eds. Proc. 2021 Winter Simulation Conf. (IEEE, Piscataway, NJ), 1–12.Google Scholar
  • Liu T, Lin Y, Zhou E (2022) Bayesian stochastic gradient descent for stochastic optimization with streaming input data. Preprint, submitted February 15, https://arxiv.org/abs/2202.07581.Google Scholar
  • Luo J, Hong LJ, Nelson BL, Wu Y (2015) Fully sequential procedures for large-scale ranking-and-selection problems in parallel computing environments. Oper. Res. 63(5):1177–1194.Link, Google Scholar
  • Mannor S, Tsitsiklis JN (2004) The sample complexity of exploration in the multi-armed bandit problem. J. Machine Learn. Res. 5:623–648.Google Scholar
  • Pichitlamken J, Nelson BL, Hong LJ (2006) A sequential procedure for neighborhood selection-of-the-best in optimization via simulation. Eur. J. Oper. Res. 173(1):283–298.Crossref, Google Scholar
  • Russo D (2016) Simple Bayesian algorithms for best arm identification. Feldman V, Rakhlin A, Shamir O, eds. Conf. Learn. Theory (PMLR, New York), 1417–1418.Google Scholar
  • Song E, Nelson BL (2015) Quickly assessing contributions to input uncertainty. IIE Trans. 47(9):893–909.Crossref, Google Scholar
  • Song E, Nelson BL (2019) Input–output uncertainty comparisons for discrete optimization via simulation. Oper. Res. 67(2):562–576.Abstract, Google Scholar
  • Song E, Shanbhag UV (2019) Stochastic approximation for simulation optimization under input uncertainty with streaming data. Mustafee N, Bae K-HG, Lazarova-Molnar S, Rabe M, Szabo C, Haas P, Son Y-J, eds. Proc. 2019 Winter Simulation Conf. (IEEE, Piscataway, NJ), 3597–3608.Google Scholar
  • Van der Vaart AW (2000) Asymptotic Statistics, vol. 3 (Cambridge University Press, Cambridge, UK).Google Scholar
  • Wu D, Zhou E (2017) Ranking and selection under input uncertainty: A budget allocation formulation. Chan WKV, D’Ambrogio A, Zacharewicz G, Mustafee N, Wainer G, Page E, eds. Proc. 2017 Winter Simulation Conf. (IEEE, Piscataway, NJ), 2245–2256.Google Scholar
  • Wu D, Zhou E (2019) Fixed confidence ranking and selection under input uncertainty. Mustafee N, Bae K-HG, Lazarova-Molnar S, Rabe M, Szabo C, Haas P, Son Y-J, eds. Proc. 2019 Winter Simulation Conf. (IEEE, Piscataway, NJ), 3717–3727.Google Scholar
  • Xiao H, Gao S (2018) Simulation budget allocation for selecting the top-m designs with input uncertainty. IEEE Trans. Automatic Control 63(9):3127–3134.Crossref, Google Scholar
  • Xiao H, Gao F, Lee LH (2020) Optimal computing budget allocation for complete ranking with input uncertainty. IISE Trans. 52(5):489–499.Crossref, Google Scholar
  • Xie W, Nelson BL, Barton RR (2014) A Bayesian framework for quantifying uncertainty in stochastic simulation. Oper. Res. 62(6):1439–1452.Link, Google Scholar
  • Xie W, Nelson BL, Barton RR (2016) Multivariate input uncertainty in output analysis for stochastic simulation. ACM Trans. Model. Comput. Simulation 27(1):1–22.Crossref, Google Scholar
  • Zhou E, Liu T (2018) Online quantification of input uncertainty for parametric models. Proc. 2018 Winter Simulation Conf., 1587–1598.Google Scholar
  • Zhou E, Xie W (2015) Simulation optimization when facing input uncertainty. Yilmaz L, Chan WKV, Moon I, Roeder TMK, Macal C, Rossetti MD, eds. Proc. 2015 Winter Simulation Conf. (IEEE, Piscataway, NJ), 3714–3724.Google Scholar
  • Zhu H, Liu T, Zhou E (2020) Risk quantification in stochastic simulation under input uncertainty. ACM Trans. Model. Comput. Simulation 30(1):1–24.Crossref, Google Scholar
  • Zouaoui F, Wilson JR (2003) Accounting for parameter uncertainty in simulation input modeling. IIE Trans. 35(9):781–792.Crossref, Google Scholar
  • Zouaoui F, Wilson JR (2004) Accounting for input-model and input-parameter uncertainties in simulation. IIE Trans. 36(11):1135–1151.Crossref, Google Scholar
Previous
Back to Top
Next
INFORMS site uses cookies to store information on your computer. Some are essential to make our site work; Others help us improve the user experience. By using this site, you consent to the placement of these cookies. Please read our Privacy Statement to learn more.
Agree