Data-Driven Ranking and Selection Under Input Uncertainty

We consider a simulation-based ranking and selection (R&S) problem with input uncertainty, in which unknown input distributions can be estimated using input data arriving in batches of varying sizes over time. Each time a batch arrives, additional simulations can be run using updated input distribution estimates. The goal is to confidently identify the best design after collecting as few batches as possible. We first introduce a moving average estimator for aggregating simulation outputs generated under heterogenous input distributions. Then, based on a sequential elimination framework, we devise two major R&S procedures by establishing exact and asymptotic confidence bands for the estimator. We also extend our procedures to the indifference zone setting, which helps save simulation effort for practical usage. Numerical results show the effectiveness and necessity of our procedures in controlling error from input uncertainty. Moreover, the efficiency can be further boosted through optimizing the “drop rate” parameter, which is the proportion of past simulation outputs to discard, of the moving average estimator.

Funding: The authors gratefully acknowledge support by the National Science Foundation Division of Civil, Mechanical and Manufacturing Innovation [Grant CMMI-1453934] and Division of Mathematical Sciences [Grant DMS2053489] and the Air Force Office of Scientific Research [Grants FA9550-19-1-0283 and FA9550-22-1-0244].

Supplemental Material: The electronic companion is available at https://doi.org/10.1287/opre.2022.2375.