Two-Stage Multiple Comparisons with the Best for Computer Simulation

We consider the problem of comparing a small number of stochastic systems via computer simulation when the basis for comparison is the expected value of some system performance measure. To solve this problem we develop two-stage sampling procedures that provide confidence intervals for the difference between the expected performance of each system and the best of the others. These confidence intervals are valid under mild conditions, and the procedures allow the experimenter to specify the desired precision in advance. Special cases of our results include standard indifference-zone selection procedures. The paper includes guidelines for experiment design and an illustrative example.