State-of-the-Art Review: A User’s Guide to the Brave New World of Designing Simulation Experiments
Published Online:1 Aug 2005https://doi.org/10.1287/ijoc.1050.0136
References
- A comparison of experimental designs in the development of a neural network simulation metamodel. Simulation Modelling: Practice Theory (2004) 12:559–578Crossref, Google Scholar
- To batch or not to batch? ACM Trans. Model. Comput. Simulation (2004) 14:76–114Crossref, Google Scholar
- , Yücesan E., Chen C. H., Snowdon J. L., Charnes J. M. Response surface methodology revisited. Proc. 2002 Winter Simulation Conf. (2002) (Institute of Electrical and Electronics Engineers, Piscataway, NJ) 377–383Crossref, Google Scholar
- Wavelet regression for random or irregular design. Computat. Statist. Data Anal. (1998) 28:353–369Crossref, Google Scholar
- Discrete-event Simulation (2005) 4th ed.(Prentice-Hall, Upper Saddle River, NJ) Google Scholar
- Small response surface designs for metamodel estimation. Eur. J. Oper. Res. (2003) 145:455–470Crossref, Google Scholar
- Measurement scales and resolution IV designs. Amer. J. Math. Management Sci. (1990) 10:309–322Google Scholar
- Searching for important factors in simulation models with many factors: Sequential bifurcation. Eur. J. Oper. Res. (1997) 96:180–194Crossref, Google Scholar
- Empirical Model-Building with Response Surfaces (1987) (Wiley, New York) Google Scholar
- Statistics for Experimenters: An Introduction to Design, Data Analysis and Model Building (1978) (Wiley, New York) Google Scholar
- Operational synthesis: Supporting the maneuver warrior. Phalanx (1999) 32:30–31Google Scholar
- An approach to the design of a manufacturing cell under economic considerations. Internat. J. Production Econom. (2002) 78:223–237Crossref, Google Scholar
- , Saltelli A., Chan K., Scott E. M. Screening methods. Sensitivity Analysis (2000) (Wiley, New York) 65–89Google Scholar
- , Andradottir S., Healy K. J., Withers D. H., Nelson B. L. Searching for important factors: Sequential bifurcation under uncertainty. Proc. 1997 Winter Simulation Conf. (1997) (Institute of Electrical and Electronics Engineers, Piscataway, NJ) 275–280Crossref, Google Scholar
- Improved design of simulation experiments with highly heteroskedastic responses. Oper. Res. (1999) 47:762–777Link, Google Scholar
- New procedures to select the best simulated system usin g common random numbers. Management Sci. (2001) 47:1133–1149Link, Google Scholar
- Efficient nearly orthogonal and space-filling experimental designs for high-dimensional complex models. (2002) . Doctoral dissertation, Operations Research Department, Naval Postgraduate School, Monterey, CA, http://handle.dtic.mil/100.2/ADA406967Google Scholar
- , Ingalls R. G., Rossetti M. D., Smith J. S., Peters B. A. Military applications of agent-based models. Proc. 2004 Winter Simulation Conf. (2004) (Institute of Electrical and Electronics Engineers, Piscataway, NJ) 171–180Crossref, Google Scholar
- Efficient nearly orthogonal and space-filling Latin hypercubes. (2005) . Working paper, Department of Operations Research, Naval Postgraduate School, Monterey, CAGoogle Scholar
- Analysis of support vector regression for approximation of complex engineering analyses. ASME J. Mech. Design (2005) 127Crossref, Google Scholar
- Crary Group (2004) . WebDOE. http://www.webdoe.cc/Google Scholar
- Statistics for Spatial Data (1993) Revised ed.(Wiley, New York) Crossref, Google Scholar
- Screening (2004) (Springer-Verlag, New York) Google Scholar
- Credible uses of the distributed interactive simulation (DIS) system. (1996) . MR-607-A, RAND Corporation, Santa Monica, CA, http://www.rand.org/publications/MR/MR607.pdfGoogle Scholar
- Simulation designs and correlation induction for reducing second-order bias in first-order response surfaces. Oper. Res. (1993) 41:880–902Link, Google Scholar
- Theory of Optimal Experiments (1972) (Academic Press, New York) Google Scholar
- Optimization for simulation: Theory vs. practice. INFORMS J. Comput. (2002) 14:192–215Link, Google Scholar
- Computational Statistics (2002) (Springer, New York) Google Scholar
- Comparison of agent based distillation movement algorithms. Military Oper. Res. (2003) 8:5–16Crossref, Google Scholar
- A comparison of approximating modeling techniques: Polynomial versus interpolating models. 7th AIAA/USAF/NASA/ISSMO Sympos. Multidisciplinary Anal. Optim. (1998) St. Louis, MO:381–391AIAA 98–4758Crossref, Google Scholar
- Ranking and selection for steady-state simulation: Procedures and perspectives. INFORMS J. Comput. (2002) 14:2–19Link, Google Scholar
- A proof of the conjecture that the Tukey-Kramer multiple comparisons procedure is conservative. Ann. Math. Statist. (1984) 12:61–75Crossref, Google Scholar
- Helton J. C., Marietta M. G. Special issue: 1996 performance assessment for the waste isolation pilot plant. Reliability Engrg. Systems Safety (2000) 69:1–454Crossref, Google Scholar
- Six (or so) things you can do with a bad model. Oper. Res. (1991) 39:355–365Link, Google Scholar
- Is it you or your model talking? A framework for model validation. (1992) . Report R-4114-AF/A/OSD, RAND Corporation, Santa Monica, CAGoogle Scholar
- Analysis of supersaturated designs. J. Quality Tech. (2000a) 35:13–27Crossref, Google Scholar
- Some combinatorial aspects, construction methods, and evaluation criteria for supersaturated designs. Quality Reliability Engrg. Internat. (2000b) 18:299–304Crossref, Google Scholar
- Horne G., Johnson S.Maneuver Warfare Science 2002 (2002) (USMC Project Albert, Quantico, VA) Google Scholar
- Horne G., Johnson S.Maneuver Warfare Science 2003 (2003) (USMC Project Albert, Quantico, VA) Google Scholar
- Horne G., Leonardi M.Maneuver Warfare Science 2001 (2001) (Marine Corps Combat Development Command, Defense Automated Printing Service, Quantico, VA) Google Scholar
- Multiple Comparisons; Theory and Methods (1996) (Chapman & Hall, London) Crossref, Google Scholar
- Analyzing transformation-based simulation metamodels. IIE Trans. (2003) 35:271–283Crossref, Google Scholar
- Driving frequency selection for frequency domain simulation experiments. Oper. Res. (1991) 39:917–924Link, Google Scholar
- Comparative studies of metamodeling techniques under multiple modeling criteria. J. Structural Optim. (2001) 23:1–13Crossref, Google Scholar
- On sequential sampling for global metamodeling in engineering design. Proc. DETC ’02, ASME 2002 Design Engrg. Tech. Conf. Comput. Inform. Engrg. Conf. (2002) Montreal, Canada:1–10DETC2002/DAC-34092Crossref, Google Scholar
- , Chick S., Sánchez P. J., Ferrin D., Morrice D. J. Experimental design for simulation. Proc. 2003 Winter Simulation Conf. (2003) (Institute of Electrical and Electronics Engineers, Piscataway, NJ) 59–65Crossref, Google Scholar
- , Ghosh S., Rao C. R. Multiresponse surface methodology. Handbook of Statistics (1996) 13(Elsevier, Amsterdam, The Netherlands) Crossref, Google Scholar
- Optimal experimental designs (with comments). J. Roy. Statist. Soc. Ser. B (1959) 21:272–319Google Scholar
- Statistical Techniques in Simulation (1974–1975) I, II(Marcel Dekker, Inc., New York) Google Scholar
- Statistical Tools for Simulation Practitioners (1987) (Marcel Dekker, Inc., New York) Google Scholar
- Case study: Statistical validation of simulation models. Eur. J. Oper. Res. (1995) 87:21–34Crossref, Google Scholar
- , Banks J. Design for sensitivity analysis, optimization, and validation of simulation models. Handbook of Simulation: Principles, Methodology, Advances, Applications, and Practice (1998) (Wiley, New York) 173–223Crossref, Google Scholar
- An overview of the design and analysis of simulation experiments for sensitivity analysis. Eur. J. Oper. Res. (2005) 164:287–300Crossref, Google Scholar
- Short-term robustness of production management systems: A case study. Eur. J. Oper. Res. (2003) 148:452–465Crossref, Google Scholar
- Maximizing the simulation output: A competition. Simulation (1999) 73:168–173Crossref, Google Scholar
- A methodology for the fitting and validation of metamodels in simulation. Eur. J. Oper. Res. (2000) 120:14–29Crossref, Google Scholar
- Performance metrics in supply chain management. J. Oper. Res. Soc. (2003) 54:507–514Crossref, Google Scholar
- Simulation: A Statistical Perspective (1992) (Wiley, Chichester, UK) Google Scholar
- Application-driven sequential designs for simulation experiments: Kriging metamodeling. J. Oper. Res. Soc. (2004) 55:876–883Crossref, Google Scholar
- Robustness of Kriging when interpolating in random simulation with heterogeneous variances: Some experiments. Eur. J. Oper. Res. (2005) 165:826–834Crossref, Google Scholar
- Robust solutions for supply chain management: Simulation and risk analysis of the Ericsson case study. (2003) . Working paper, Tilburg University, Tilburg, The NetherlandsGoogle Scholar
- , Dean A. M., Lewis S. M. Finding the important factors in large discrete-event simulation: Sequential bifurcation and its applications. Screening (2005) (Springer-Verlag, New York) Google Scholar
- Optimal design of experiments with simulation models of nearly saturated queues. J. Statist. Planning Inference (2000) 85:19–26Crossref, Google Scholar
- , Medeiros D. J., Watson E. F., Carson J. S., Manivannan M. S. Bootstrapping and validation of metamodels in simulation. Proc. 1998 Winter Simulation Conf. (1998) (Institute of Electrical and Electronics Engineers, Piscataway, NJ) 701–706Crossref, Google Scholar
- Generalization of simulation results: Practicality of statistic al methods. Eur. J. Oper. Res. (1979) 3:50–64Crossref, Google Scholar
- , Peters B. A., Smith J. S., Medeiros D. J., Rohrer M. W. Sensitivity analysis of censored output through polynomial, logistic and tobit models: Theory and case study. Proc. 2001 Winter Simulation Conf. (2001) (Institute of Electrical and Electronics Engineers, Piscataway, NJ) 486–491Crossref, Google Scholar
- , Ghosh S., Rao C. R.Handbook of Statistics (1996) 13(Computer experiments, Amsterdam, The Netherlands) 261–308Google Scholar
- Applied Linear Statistical Models (2004) 5th ed.(McGraw-Hill/Irwin, Boston, MA) Google Scholar
- Map-aware non-uniform automata—A New Zealand approach to scenario modelling. J. Battlefield Tech. (2002) 5:27–31Google Scholar
- Simulation Modeling and Analysis (2000) 3rd ed.(McGraw-Hill, New York) Google Scholar
- Generating systematic supersaturated designs. Technometrics (1995) 37:213–225Crossref, Google Scholar
- (2002) . DACE: A Matlab Kriging toolbox, version 2.0.IMM Technical University of Denmark, Lyngby, Denmark, http://www.imm.dtu.dk/~hbn/dace/Google Scholar
- Improving the analytic contribution of advanced warfighting experiments (AWEs). (1997) . Documented Briefing DB-207-A. RAND Corporation, Santa Monica, CAGoogle Scholar
- , Horne G., Johnson S. Better designs for high-dimensional explorations of distillations. Maneuver Warfare Science 2002 (2002) (USMC Project Albert, Quantico, VA) 17–46Google Scholar
- , Horne G., Johnson S. Generating hypotheses on fighting the global war on terrorism. Maneuver Warfare Science 2003 (2003) (USMC Project Albert, Quantico, VA) 117–137Google Scholar
- Computational Statistics Handbook with MATLAB (2002) (Chapman & Hall/CRC, Boca Raton, FL) Google Scholar
- A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics (1979) 21:239–245Google Scholar
- A robust design methodology for Kanban system design. Internat. J. Production Res. (1997) 35:2821–2838Crossref, Google Scholar
- Design and Analysis of Experiments (2000) 5th ed.(Wiley, New York) Google Scholar
- A weighted least squares approach to computer simulation factor screening. Oper. Res. (1995) 43:792–806Link, Google Scholar
- Exploratory designs for computational experiments. J. Statist. Planning Inference (1995) 43:381–402Crossref, Google Scholar
- , Horne G., Leonardi M. Visualization for data farming: A survey of methods. Maneuver Warfare Science 2001 (2001) (Marine Corps Combat Development Command, Quantico, VA) 15–30Google Scholar
- Response Surface Methodology: Process and Product Optimization using Designed Experiments (2002) 2nd ed.(Wiley, New York) Google Scholar
- Response surface alternatives to the Taguchi robust design parameter approach. Amer. Statist. (1992) 46:131–139Crossref, Google Scholar
- , Chick S., Sánchez P. J., Ferrin D., Morrice D. J. Analysis of simulation output. Proc. 2003 Winter Simulation Conf. (2003) (Institute of Electrical and Electronics Engineers, Piscataway, NJ) 49–58Crossref, Google Scholar
- Comparisons with a standard in simulation experiments. Management Sci. (2001) 47:449–463Link, Google Scholar
- Incorporating Information from Replications into Bayesian Surrogate Models. 2003 ASME Design Engrg. Tech. Conf. (2003) Chicago, ILDETC2003/DTM-48644Crossref, Google Scholar
- Optimal Design of Experiments (1993) (Wiley, New York) Google Scholar
- , Nelson B. L., Kelton W. D., Clark G. M. Designing simulation experiments: Taguchi methods and response surface metamodels. Proc. 1991 Winter Simulation Conf. (1991) (Institute of Electrical and Electronics Engineers, Piscataway, NJ) 167–176Crossref, Google Scholar
- Least squares theory using an estimated dispersion matrix and its application to measurement of signals. Proc. 5th Berkeley Sympos. Math. Statist. Probab. (1967) I(University of California Press, Berkeley, CA) 355–372Google Scholar
- Saturated fractions of 2n and 3n factorial designs. Technometrics (1967) 9:569–575Google Scholar
- Design and analysis of computer experiments (includes Comments and Rejoinder). Statist. Sci. (1989) 4:409–435Crossref, Google Scholar
- A quantitative model-independent method for global sensitivity analysis of model output. Technometrics (1999) 41:39–56Crossref, Google Scholar
- , Thesen A., Grant H., Kelton W. D. A model for frequency domain experiments. Proc. 1987 Winter Simulation Conf. (1987) (Institute of Electrical and Electronics Engineers, Piscataway, NJ) 424–427Crossref, Google Scholar
- , Joines J. A., Barton R. R., Kang K., Fishwick P. A. Robust design: Seeking the best of all possible worlds. Proc. 2000 Winter Simulation Conf. (2000) (Institute of Electrical and Electronics Engineers, Piscataway, NJ) 69–76Crossref, Google Scholar
- , Yücesan E., Chen C.-H., Snowdon J. L., Charnes J. Exploring the world of agent-based simulations: Simple models, complex analyses. Proc. 2002 Winter Simulation Conf. (2002) (Institute of Electrical and Electronics Engineers, Piscataway, NJ) 116–126Crossref, Google Scholar
- , Chick S., Sánchez P. J., Ferrin D., Morrice D. J. Frequency-based designs for terminating simulations: A peace-enforcement application. Proc. 2003 Winter Simulation Conf. (2003) (Institute of Electrical and Electronics Engineers, Piscataway, NJ) 952–959Crossref, Google Scholar
- , Wang B. A simulation framework for robust system design. Concurrent Design of Products, Manufacturing Processes and Systems (1998) (Gordon and Breach, New York) 279–314Google Scholar
- Tolerance design revisited: Assessing the impact of correlated noise factors. (2005) . Working paper, Operations Research Department, Naval Postgraduate School, Monterey, CAGoogle Scholar
- Effective engineering design through simulation. Internat. Trans. Oper. Res. (1996) 3:169–185Crossref, Google Scholar
- The Design and Analysis of Computer Experiments (2003) (Springer-Verlag, New York) Crossref, Google Scholar
- SAS Institute, Inc.JMP IN® Version 5.1 for Windows, Macintosh, and Unix (2004) (Duxbury Thompson Learning, Pacific Grove, CA) Google Scholar
- Exploration of metamodeling sampling criteria for constrained global optimization. Engrg. Optim. (2002) 34:263–278Crossref, Google Scholar
- Batch size effects in the analysis of simulation output. Oper. Res. (1982) 30:556–568Link, Google Scholar
- An experimental procedure for simulation response surface model identification. Comm. ACM (1987) 30:716–730Crossref, Google Scholar
- Economical experimentation methods for robust design. Technometrics (1991) 33:415–427Crossref, Google Scholar
- The Sciences of the Artificial (1981) 2nd ed.(MIT Press, Cambridge, MA) Google Scholar
- Sampling strategies for computer experiments: Design and analysis. Internat. J. Reliability Appl. (2001) 2:209–240Google Scholar
- Assessment of business potential at retail sites: Empirical findings from a U. S. supermarket chain. Internat. Rev. Retail, Distribution Consumer Res. (2003) 13:37–58Crossref, Google Scholar
- Introduction to Stochastic Search and Optimization; Estimation, Simulation, and Control (2003) (Wiley, New York) Crossref, Google Scholar
- ASAP3: A batch means procedure for steady-state simulation output analysis. ACM Trans. Model. Comput. Simulation (2005) 15:39–73Crossref, Google Scholar
- @Risk, riskview and bestFit. OR/MS Today (1997) 24:64–66Google Scholar
- System of Experimental Designs (1987) 1, 2(UNIPUB/Krauss International, White Plains, NY) Google Scholar
- , Peters B. A., Smith J. S., Medeiros D. J., Rohrer M. W. An overview of newer, advanced screening methods for the initial phase in an experimental design. Proc. 2001 Winter Simulation Conf. (2001) (Institute of Electrical and Electronics Engineers, Piscataway, NJ) 169–178Crossref, Google Scholar
- Envisioning Information (1990) (Graphics Press, Cheshire, CT) Google Scholar
- Dealing with the least squares regression assumptions in simulation metamodeling. Comput. Indust. Engrg. (2000) 38:307–320Crossref, Google Scholar
- Kriging for interpolation in random simulation. J. Oper. Res. Soc. (2003) 54:255–262Crossref, Google Scholar
- , Ingalls R. G., Rossetti M. D., Smith J. S., Peters B. A. Kriging interpolation in simulation: A survey. Proc. 2004 Winter Simulation Conf. (2004) (Institute of Electrical and Electronics Engineers, Piscataway, NJ) 113–121Crossref, Google Scholar
- Exploring combat models for non-monotonicities and remedies. PHALANX (2002) 35:19–38Google Scholar
- Sensitivity analysis by experimental design and metamodelling: Case study on simulation in national animal disease control. Eur. J. Oper. Res. (2003) 146:433–443Crossref, Google Scholar
- , Chick S., Sánchez P. J., Ferrin D., Morrice D. J. Controlled sequential bifurcation: A new factor-screening method for discrete-event simulation. Proc. 2003 Winter Simulation Conf. (2003) (Institute of Electrical and Electronics Engineers, Piscataway, NJ) 565–573Google Scholar
- An exploratory analysis on the effects of human factors on combat outcomes. (2002) . M. S. thesis, Operations Research Department, Naval Postgraduate School, Monterey, CA, http://handle.dtic.mil/100.2/ADA403526Google Scholar
- Infill sampling criteria to locate extremes. Math. Geology (1995) 27:589–608Crossref, Google Scholar
- Hyperdimensional data analysis using parallel coordinates. J. Amer. Statist. Association (1990) 85:664–674Crossref, Google Scholar
- Using agent-based distillations to explore logistics support to urban, humanitarian assistance/disaster relief operations. (2003) . M.S. thesis, Operations Research Department, Naval Postgraduate School, Monterey, CAGoogle Scholar
- Using agents to model logistics. (2003) . Working paper, Operations Research Department, Naval Postgraduate School, Monterey, CAGoogle Scholar
- Orthogonal column Latin hypercubes and their application in computer experiments. J. Amer. Statist. Association (1998) 93:1430–1439 http://handle.dtic.mil/100.2/ADA418300Crossref, Google Scholar
- Theory of Modeling and Simulation (2000) 2nd ed.(Academic Press, San Diego, CA) Google Scholar

