Using Ranking and Selection to “Clean Up” after Simulation Optimization

Justin Boesel
Justin Boesel
[email protected]
The MITRE Corporation, 1820 Dolley Madison Boulevard, McLean, Virginia 22102
Search for more papers by this author
,
Barry L. Nelson
Barry L. Nelson
[email protected]
Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, Illinois 60208-3119
Search for more papers by this author
,
Seong-Hee Kim
Seong-Hee Kim
[email protected]
School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332
Search for more papers by this author

Justin Boesel

[email protected]

The MITRE Corporation, 1820 Dolley Madison Boulevard, McLean, Virginia 22102

Search for more papers by this author

Barry L. Nelson

[email protected]

Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, Illinois 60208-3119

Search for more papers by this author

Seong-Hee Kim

[email protected]

School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332

Search for more papers by this author

Published Online:1 Oct 2003https://doi.org/10.1287/opre.51.5.814.16751

Supplemental Material

opre.51.5.814.16751-sm-companion.pdf (151 KB)

Cited by
- A survey on preference-guided algorithms in surrogate-based multi-objective optimization: Explicit and implicit preferences
  3 June 2026 | Annals of Mathematics and Artificial Intelligence, Vol. 54
- Efficient selection of the best system: Indifference-zone-free procedures with multi-dimensional Brownian motion
  7 May 2026 | IISE Transactions
- A framework for comparing stochastic simulation models against multidimensional data using the Wasserstein distance
  8 April 2025 | Journal of Simulation, Vol. 20, No. 1
- Decision Maker Preferences in Surrogate-Based Multi-objective Optimization: A Survey
  2 January 2026
- TESO: Tabu-Enhanced Simulation Optimization for Noisy Black-Box Problems
- Supercharging Digital Twins with AI
  18 October 2025 | Asia-Pacific Journal of Operational Research, Vol. 42, No. 06
- A Wasserstein distance-based double-bootstrap method for comparing spatial simulation output
  8 May 2025 | SIMULATION, Vol. 101, No. 10
- Bi-objective ranking and selection using stochastic kriging
  European Journal of Operational Research, Vol. 322, No. 2
- Tutorial: Artificial Neural Networks for Discrete-Event Simulation
- Optimized Model Selection for Estimating Treatment Effects from Costly Simulations of the US Opioid Epidemic
- Screening Simulated Systems for Optimization
- Efficient Hybrid Simulation Optimization via Graph Neural Network Metamodeling
- CGCompiler: Automated Coarse-Grained Molecule Parametrization via Noise-Resistant Mixed-Variable Optimization
  16 November 2023 | Journal of Chemical Theory and Computation, Vol. 19, No. 22
- Mixed-integer simulation optimization for multi-echelon inventory problems with lost sales
  3 November 2022 | Journal of the Operational Research Society, Vol. 74, No. 11
- Efficient Sampling Policy for Selecting a Subset With the Best
  IEEE Transactions on Automatic Control, Vol. 68, No. 8
- Metamodel-based simulation optimization considering a single stochastic constraint
  Computers & Operations Research, Vol. 155
- Data-driven stochastic optimization for power grids scheduling under high wind penetration
  3 October 2021 | Energy Systems, Vol. 14, No. 1
- Many-Objective Quality Measures
  29 July 2023
- Plausible Screening Using Functional Properties for Simulations with Large Solution Spaces
  David J. Eckman,
  Matthew Plumlee,
  Barry L. Nelson
  1 February 2022 | Operations Research, Vol. 70, No. 6
- Estimating the true arrival, balking, and reneging processes from censored transactional data: a simulation-based approach
  9 December 2021 | SIMULATION, Vol. 98, No. 7
- Posterior-Based Stopping Rules for Bayesian Ranking-and-Selection Procedures
  David J. Eckman,
  Shane G. Henderson
  19 January 2022 | INFORMS Journal on Computing, Vol. 34, No. 3
- Review on ranking and selection: A new perspective
  29 March 2021 | Frontiers of Engineering Management, Vol. 8, No. 3
- Efficient optimization algorithms for surgical scheduling under uncertainty
  European Journal of Operational Research, Vol. 293, No. 2
- Fixed-Confidence, Fixed-Tolerance Guarantees for Ranking-and-Selection Procedures
  10 February 2021 | ACM Transactions on Modeling and Computer Simulation, Vol. 31, No. 2
- Selection of the best minimal repair systems for Weibull lifetime distribution: A Bayesian approach
  1 February 2019 | Communications in Statistics - Simulation and Computation, Vol. 50, No. 3
- Simulation Optimization and Sensitivity
  27 August 2021
- Mitigating Cascading Outages in Severe Weather Using Simulation-Based Optimization
  IEEE Transactions on Power Systems, Vol. 36, No. 1
- Revisiting Subset Selection
- A simulation optimization method for deep-sea vessel berth planning and feeder arrival scheduling at a container port
  Transportation Research Part B: Methodological, Vol. 142
- A multiobjective stochastic simulation optimization algorithm
  European Journal of Operational Research, Vol. 284, No. 1
- A survey on kriging-based infill algorithms for multiobjective simulation optimization
  Computers & Operations Research, Vol. 116
- Incorporating occupancy data in scheduling building equipment: A simulation optimization framework
  Energy and Buildings, Vol. 209
- A mean-variance approach to transmission network planning
  International Journal of Electrical Power & Energy Systems, Vol. 115
- Multiobjective Ranking and Selection with Correlation and Heteroscedastic Noise
- First-order intrinsic Gaussian Markov random fields for discrete optimisation via simulation
  25 January 2019 | Journal of Simulation, Vol. 13, No. 4
- Simulation-based estimation of the real demand in bike-sharing systems in the presence of censoring
  European Journal of Operational Research, Vol. 277, No. 1
- Simulation optimisation
  13 July 2019
- TD-OCBA: Optimal computing budget allocation and time dilation for simulation optimization of manufacturing systems
  22 February 2019 | IISE Transactions, Vol. 51, No. 3
- An Introduction to Multiobjective Simulation Optimization
  24 January 2019 | ACM Transactions on Modeling and Computer Simulation, Vol. 29, No. 1
- The Local Time Method for Targeting and Selection
  Ilya O. Ryzhov
  30 August 2018 | Operations Research, Vol. 66, No. 5
- Reusing Search Data in Ranking and Selection
  6 July 2018 | ACM Transactions on Modeling and Computer Simulation, Vol. 28, No. 3
- Simulation optimisation
  6 July 2018
- Design and control of manufacturing systems: a discrete event optimisation methodology
  20 December 2017 | International Journal of Production Research, Vol. 56, No. 1-2
- A preference-inspired multi-objective soft scheduling algorithm for the practical steelmaking-continuous casting production
  Computers & Industrial Engineering, Vol. 115
- History of seeking better solutions, AKA simulation optimization
- An Efficient Budget Allocation Approach for Quantifying the Impact of Input Uncertainty in Stochastic Simulation
  27 October 2017 | ACM Transactions on Modeling and Computer Simulation, Vol. 27, No. 4
- Comparison of Kriging-based algorithms for simulation optimization with heterogeneous noise
  European Journal of Operational Research, Vol. 261, No. 1
- Rapid screening algorithms for stochastically constrained problems
  8 March 2017 | Annals of Operations Research, Vol. 254, No. 1-2
- A simulation-based multi-objective optimization framework: A case study on inventory management
  Omega, Vol. 70
- Discrete optimization via simulation to determine reliable network investments
- Efficient Ranking and Selection in Parallel Computing Environments
  Eric C. Ni,
  Dragos F. Ciocan,
  Shane G. Henderson,
  Susan R. Hunter
  7 March 2017 | Operations Research, Vol. 65, No. 3
- Improving power system reliability through optimization via simulation
- A Hybrid Approach of Optimization and Sampling for Robust Portfolio Selection
  8 March 2017
- Parallel Ranking and Selection
  29 August 2017
- Technology transfer of simulation analysis methodology: One man's opinion
- Challenges in applying ranking and selection after search
- Simulation Optimisation
  20 July 2016
- Perspectives of approximate dynamic programming
  7 February 2012 | Annals of Operations Research, Vol. 241, No. 1-2
- A column generation algorithm for the Buffer Allocation Problem approximated by the Time Buffer concept
  IFAC-PapersOnLine, Vol. 49, No. 12
- Improving the Design and Operation of an Integrated Emergency Post via Simulation
- Subset selection for simulations accounting for input uncertainty
- A sequential experiment design for input uncertainty quantification in stochastic simulation
- An introduction to simulation optimization
- Mathematical programming models for joint simulation–optimization applied to closed queueing networks
  11 October 2013 | Annals of Operations Research, Vol. 231, No. 1
- Improving the design and operation of an integrated emergency post via simulation
  19 December 2017 | Journal of Simulation, Vol. 9, No. 2
- Discrete Optimization via Simulation
  18 September 2014
- Genetic-algorithm-based simulation optimization considering a single stochastic constraint
  European Journal of Operational Research, Vol. 236, No. 1
- A subset selection procedure under input parameter uncertainty
- Simulation Optimization: A Review on Theory and Applications
  Acta Automatica Sinica, Vol. 39, No. 11
- Rapid Screening Procedures for Zero-One Optimization via Simulation
  Shing Chih Tsai,
  4 May 2012 | INFORMS Journal on Computing, Vol. 25, No. 2
- Statistical Ranking and Selection
  23 January 2016
- Why Do We Simulate?
  1 December 2012
- Simulation Programming: Quick Start
  1 December 2012
- Examples
  1 December 2012
- Simulation Programming with VBASim
  1 December 2012
- Two Views of Simulation
  1 December 2012
- Simulation Input
  1 December 2012
- Simulation Output
  1 December 2012
- Experiment Design and Analysis
  1 December 2012
- Simulation for Research
  1 December 2012
- Sensitivity analysis of an ICU simulation model
- Screening and selection procedures with control variates and correlation induction techniques
  18 June 2012 | Naval Research Logistics (NRL), Vol. 59, No. 5
- A Framework for Selecting a Selection Procedure
  1 August 2012 | ACM Transactions on Modeling and Computer Simulation, Vol. 22, No. 3
- Selecting the best simulated system with weighted control-variate estimators
  Mathematics and Computers in Simulation, Vol. 82, No. 4
- Optimization for simulation: LAD accelerator
  4 February 2009 | Annals of Operations Research, Vol. 188, No. 1
- Lean-pull strategy in a re-entrant manufacturing environment: a pilot study for TFT-LCD array manufacturing
  International Journal of Production Research, Vol. 49, No. 6
- Statistical Methods for Optimization
  15 February 2011
- Continuous optimization via simulation using Golden Region search
  European Journal of Operational Research, Vol. 208, No. 1
- Testing line optimization based on mathematical modeling from the metamodels obtained from a simulation
- Generating dispatching rules for semiconductor manufacturing to minimize weighted tardiness
- An efficient simulation procedure for point estimation of expected shortfall
- Performance measures for Ranking and Selection procedures
- Nested Simulation in Portfolio Risk Measurement
  Michael B. Gordy,
  Sandeep Juneja,
  20 August 2010 | Management Science, Vol. 56, No. 10
- A Confidence Interval Procedure for Expected Shortfall Risk Measurement via Two-Level Simulation
  Hai Lan,
  Barry L. Nelson,
  Jeremy Staum,
  17 August 2010 | Operations Research, Vol. 58, No. 5
- An improved dynamic structure-based neural networks determination approaches to simulation optimization problems
  2 April 2010 | Neural Computing and Applications, Vol. 19, No. 6
- Fully sequential procedures for comparing constrained systems via simulation
  9 April 2010 | Naval Research Logistics (NRL), Vol. 57, No. 5
- Towards improved dispatching rules for complex shop floor scenarios
  7 July 2010
- A simulation study of the fleet sizing problem arising in offshore anchor handling operations
  European Journal of Operational Research, Vol. 203, No. 1
- Stochastic Kriging for Simulation Metamodeling
  Bruce Ankenman,
  Barry L. Nelson,
  Jeremy Staum,
  8 December 2009 | Operations Research, Vol. 58, No. 2
- Nested Simulation
  15 May 2010
- Merging the Ranking and Selection into ITO Algorithm for Simulation Optimization
- The challenge of optimizing expensive black boxes: a scatter search/rough set theory approach
  21 December 2017 | Journal of the Operational Research Society, Vol. 61, No. 1
- Industrial strength COMPASS
  8 February 2010 | ACM Transactions on Modeling and Computer Simulation, Vol. 20, No. 1
- Simulation optimization using the cross-entropy method with optimal computing budget allocation
  8 February 2010 | ACM Transactions on Modeling and Computer Simulation, Vol. 20, No. 1
- A brief introduction to optimization via simulation
- Simulation optimization with Hybrid Golden Region search
- Statistical analysis and comparison of simulation models of highly dependable systems — An experimental study
- A posterior preference articulation approach to dual-response-surface optimization
  20 November 2009 | IIE Transactions, Vol. 42, No. 2
- Pattern search ranking and selection algorithms for mixed variable simulation-based optimization
  European Journal of Operational Research, Vol. 198, No. 3
- Simulation optimization for an emergency department healthcare unit in Kuwait
  European Journal of Operational Research, Vol. 198, No. 3
- Finding probably best systems quickly via simulations
  11 August 2009 | ACM Transactions on Modeling and Computer Simulation, Vol. 19, No. 3
- Combined Screening and Selection of the Best with Control Variates
  17 July 2009
- Some Personal Thoughts on Robert E. Bechhofer (1919–1996)
  American Journal of Mathematical and Management Sciences, Vol. 29, No. 1-2
- Reserve Design to Maximize Species Persistence
  15 March 2007 | Environmental Modeling & Assessment, Vol. 13, No. 2
- Selecting a Selection Procedure
  Jürgen Branke,
  Stephen E. Chick,
  Christian Schmidt,
  1 December 2007 | Management Science, Vol. 53, No. 12
- New greedy myopic and existing asymptotic sequential selection procedures: preliminary empirical results
- Determining efficient simulation run lengths for real time decision making
- Simulation of Coherent Risk Measures Based on Generalized Scenarios
  Vadim Lesnevski,
  Barry L. Nelson,
  Jeremy Staum,
  1 November 2007 | Management Science, Vol. 53, No. 11
- A chance constraint estimation approach to optimizing resource management under uncertainty
  Canadian Journal of Forest Research, Vol. 37, No. 11
- A framework for locally convergent random-search algorithms for discrete optimization via simulation
  1 September 2007 | ACM Transactions on Modeling and Computer Simulation, Vol. 17, No. 4
- A framework for evaluating remote diagnostics investment decisions for semiconductor equipment suppliers
  European Journal of Operational Research, Vol. 180, No. 3
- Selecting the best system when systems are revealed sequentially
  27 September 2010 | IIE Transactions, Vol. 39, No. 7
- Recent advances in ranking and selection
- Combined Ranking and Selection with Control Variates
- Fully sequential selection procedures with a parabolic boundary
  23 February 2007 | IIE Transactions, Vol. 38, No. 9
- Discrete Optimization via Simulation Using COMPASS
  L. Jeff Hong,
  Barry L. Nelson,
  1 February 2006 | Operations Research, Vol. 54, No. 1
- Integrating Techniques from Statistical Ranking into Evolutionary Algorithms
- Chapter 17 Selecting the Best System
- Chapter 20 An Overview of Simulation Optimization via Random Search
- Chapter 21 Metaheuristics
- Control variates for screening, selection, and estimation of the best
  1 January 2006 | ACM Transactions on Modeling and Computer Simulation, Vol. 16, No. 1
- Simulation of Coherent Risk Measures Based on Generalized Scenarios
  SSRN Electronic Journal, Vol. 9
- The tradeoff between sampling and switching: New sequential procedures for indifference-zone selection
  23 February 2007 | IIE Transactions, Vol. 37, No. 7
- Comparison with a standard via fully sequential procedures
  1 April 2005 | ACM Transactions on Modeling and Computer Simulation, Vol. 15, No. 2
- Simulation Optimization: A Review, New Developments, and Applications
- New Developments in Ranking and Selection: An Empirical Comparison of the Three Main Approaches
- Finding the Best in the Presence of a Stochastic Constraint
- Analysis Methodology: Are We Done?
- Performance of Variance Updating Ranking and Selection Procedures
- Enhancing Evolutionary Algorithms with Statistical Selection Procedures for Simulation Optimization
- Simulation optimization: A comprehensive review on theory and applications
  17 August 2010 | IIE Transactions, Vol. 36, No. 11
- A Framework for Simulating Human Cognitive Behavior and Movement When Predicting Impacts of Catastrophic Events
- A combined procedure for optimization via simulation
  1 April 2003 | ACM Transactions on Modeling and Computer Simulation, Vol. 13, No. 2
- Selecting the best system: theory and methods
- An indifference-zone selection procedure with minimum switching and sequential sampling

Volume 51, Issue 5

September-October 2003

Pages 681-838

Article Information

Supplemental Material

Metrics

Information

Received:August 01, 1999
Accepted:September 01, 2002
Published Online:October 01, 2003

Cite as

Justin Boesel, Barry L. Nelson, Seong-Hee Kim, (2003) Using Ranking and Selection to “Clean Up” after Simulation Optimization. Operations Research 51(5):814-825.

https://doi.org/10.1287/opre.51.5.814.16751

Keywords

PDF download

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Using Ranking and Selection to “Clean Up” after Simulation Optimization

Supplemental Material

Volume 51, Issue 5

Article Information

Supplemental Material

Metrics

Information

Cite as

Keywords

Sign Up for INFORMS Publications Updates and News