Free Access

Needed: An Empirical Science of Algorithms

J. N. Hooker
J. N. Hooker
Carnegie Mellon University, Pittsburgh, Pennsylvania
Search for more papers by this author

J. N. Hooker

Carnegie Mellon University, Pittsburgh, Pennsylvania

Search for more papers by this author

Published Online:1 Apr 1994https://doi.org/10.1287/opre.42.2.201

Cited by
- Beyond Smoothed Analysis: Analyzing the Simplex Method By-the-Book
  9 June 2026
- Commentaries on A. Mor, C. Orsenigo and M.G. Speranza “optimization and supervised learning for decision making: competitors or partners?”
  28 April 2026 | Journal of the Operational Research Society, Vol. 77, No. 6
- Machine scheduling instance generation by reverse engineering from instance space analysis
  European Journal of Operational Research, Vol. 330, No. 3
- Using Machine Learning Methods to Assess Module Performance Contribution in Modular Optimization Frameworks
  1 December 2025 | Evolutionary Computation, Vol. 33, No. 4
- First-improvement or best-improvement? An in-depth local search computational study to elucidate a dominance claim
  European Journal of Operational Research, Vol. 326, No. 3
- Exploring module interactions in modular CMA-ES across problem classes
  Swarm and Evolutionary Computation, Vol. 98
- A causal framework for stochastic local search optimization algorithms
  Computers & Operations Research, Vol. 180
- Rethinking Metaheuristics: Unveiling the Myth of “Novelty” in Metaheuristic Algorithms
  1 July 2025 | Mathematics, Vol. 13, No. 13
- Understanding instance hardness for optimisation algorithms: Methodologies, open challenges and post-quantum implications
  Applied Mathematical Modelling, Vol. 142
- Measuring the effectiveness and efficiency of simulation optimization metaheuristic algorithms
  30 January 2025 | Journal of Heuristics, Vol. 31, No. 1
- Systematic Literature Review of Optimization Algorithms for P||Cmax Problem
  24 January 2025 | Symmetry, Vol. 17, No. 2
- Solving the t -Wise Coverage Maximum Problem via Effective and Efficient Local Search-Based Sampling
  31 December 2024 | ACM Transactions on Software Engineering and Methodology, Vol. 34, No. 1
- Instance Space Analysis for Visualization of Algorithmic Trust
  6 August 2025
- Applying instance space analysis for metaheuristic selection to the 0–1 multidemand multidimensional knapsack problem
  Computers & Operations Research, Vol. 170
- Generating linear, semidefinite, and second-order cone optimization problems for numerical experiments
  5 July 2024 | Optimization Methods and Software, Vol. 39, No. 4
- General VNS for asymmetric vehicle routing problem with time and capacity constraints
  Computers & Operations Research, Vol. 167
- Instance space analysis for 2D bin packing mathematical models
  European Journal of Operational Research, Vol. 315, No. 2
- Generating linear programming instances with controllable rank and condition number
  Computers & Operations Research, Vol. 162
- Verifying new instances of the multidemand multidimensional knapsack problem with instance space analysis
  Computers & Operations Research, Vol. 162
- Computational Experimentation
  10 January 2024
- Instance Space Analysis for Algorithm Testing: Methodology and Software Tools
  2 March 2023 | ACM Computing Surveys, Vol. 55, No. 12
- Lessons from the Evolutionary Computation Bestiary
  1 November 2023 | Artificial Life, Vol. 29, No. 4
- Designing New Metaheuristics: Manual Versus Automatic Approaches
  4 December 2023 | Intelligent Computing, Vol. 2
- Classifying Metaheuristics: Towards a unified multi-level classification system
  12 December 2020 | Natural Computing, Vol. 21, No. 2
- Parameter Analysis of Variable Neighborhood Search Applied to Multiprocessor Scheduling with Communication Delays
  30 September 2022
- Augmented intuition: a bridge between theory and practice
  31 January 2021 | Journal of Heuristics, Vol. 27, No. 4
- Design of large-scale metaheuristic component studies
  8 July 2021
- Benchmark Set Reduction for Cheap Empirical Algorithmic Studies
- Revisiting where are the hard knapsack problems? via Instance Space Analysis
  Computers & Operations Research, Vol. 128
- Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances
  5 August 2020 | Journal of Heuristics, Vol. 26, No. 6
- Generation techniques for linear programming instances with controllable properties
  17 August 2019 | Mathematical Programming Computation, Vol. 12, No. 3
- Generating New Space-Filling Test Instances for Continuous Black-Box Optimization
  Evolutionary Computation, Vol. 28, No. 3
- Metaheuristic Design Patterns
- Apriori Rule--Based In-App Ad Selection Online Algorithm for Improving Supply-Side Platform Revenues
  3 July 2017 | ACM Transactions on Management Information Systems, Vol. 8, No. 2-3
- Evaluating the importance of randomization in adaptive large neighborhood search
  2 March 2016 | International Transactions in Operational Research, Vol. 24, No. 5
- An Enhanced tabu search algorithm to minimize a bi-criteria objective in batching and scheduling problems on unrelated-parallel machines with desired lower bounds on batch sizes
  Computers & Operations Research, Vol. 77
- Good Laboratory Practice for optimization research
  21 December 2017 | Journal of the Operational Research Society, Vol. 67, No. 4
- Creativity and Critique: Gap Analysis of Support for Critical Research on Big Data
  SSRN Electronic Journal, Vol. 48
- Generating new test instances by evolving in instance space
  Computers & Operations Research, Vol. 63
- A semi-automated design of instance-based fuzzy parameter tuning for metaheuristics based on decision tree induction
  21 December 2017 | Journal of the Operational Research Society, Vol. 66, No. 5
- A family of composite discrete bivariate distributions with uniform marginals for simulating realistic and challenging optimization-problem instances
  European Journal of Operational Research, Vol. 241, No. 3
- Empirical Computer Science
- Factorial design analysis applied to the performance of parallel evolutionary algorithms
  3 February 2014 | Journal of the Brazilian Computer Society, Vol. 20, No. 1
- Exploring the role of graph spectra in graph coloring algorithm performance
  Discrete Applied Mathematics, Vol. 176
- The executable experimental template pattern for the systematic comparison of metaheuristics
  12 July 2014
- Combinatorial Optimization
  8 May 2014
- Towards objective measures of algorithm performance across instance space
  Computers & Operations Research, Vol. 45
- Getting Started
  26 September 2014
- Semidefinite Bounds
  26 September 2014
- Linear Inequalities and Polyhedra
  26 September 2014
- Valid Inequalities for Structured Integer Programs
  26 September 2014
- Reformulations and Relaxations
  26 September 2014
- Enumeration
  26 September 2014
- Predicting Metaheuristic Performance on Graph Coloring Problems Using Data Mining
- Improving the Performance of Heuristic Algorithms Based on Exploratory Data Analysis
- Progress in Academic Computational Integer Programming
- Multiobjectivization via Helper-Objectives With the Tunable Objectives Problem
  IEEE Transactions on Evolutionary Computation, Vol. 16, No. 3
- Problem reduction heuristic for the 0–1 multidimensional knapsack problem
  Computers & Operations Research, Vol. 39, No. 1
- European Journal of Operational Research, Vol. 214, No. 2
- Helper-objective optimization strategies for the Job-Shop Scheduling Problem
  Applied Soft Computing, Vol. 11, No. 6
- Simulating realistic set covering problems with known optimal solutions
  Computers & Industrial Engineering, Vol. 61, No. 1
- Exploring the performance of spatial stochastic simulation algorithms
  Journal of Computational Physics, Vol. 230, No. 7
- Group-scheduling problems in electronics manufacturing
  17 December 2009 | Journal of Scheduling, Vol. 13, No. 2
- Mixed Integer Programming Computation
  6 November 2009
- The Generation of Experimental Data for Computational Testing in Optimization
  26 October 2010
- AEGIS—attribute experimentation guiding improvement searches
  21 March 2008 | Journal of Heuristics, Vol. 15, No. 5
- Synthetic Optimization Problem Generation: Show Us the Correlations!
  Charles H. Reilly,
  29 June 2009 | INFORMS Journal on Computing, Vol. 21, No. 3
- Testing cut generators for mixed-integer linear programming
  29 May 2009 | Mathematical Programming Computation, Vol. 1, No. 1
- Modelling and solving temporal reasoning as propositional satisfiability
  Artificial Intelligence, Vol. 172, No. 15
- A machine learning approach to algorithm selection for $\mathcal{NP}$ -hard optimization problems: a case study on the MPE problem
  7 August 2007 | Annals of Operations Research, Vol. 156, No. 1
- Analyzing heuristic performance with response surface models
  7 July 2007
- Machine scheduling performance with maintenance and failure
  Mathematical and Computer Modelling, Vol. 45, No. 9-10
- Alternative model representations and computing capacity: Implications for model management
  Decision Support Systems, Vol. 42, No. 3
- Core instances for testing: A case study
  European Journal of Operational Research, Vol. 166, No. 1
- Replicating software engineering experiments: a poisoned chalice or the Holy Grail
  Information and Software Technology, Vol. 47, No. 4
- Learning algorithms for single-instance electronic negotiations using the time-dependent behavioral tactic
  1 February 2005 | ACM Transactions on Internet Technology, Vol. 5, No. 1
- On SAT Instance Classes and a Method for Reliable Performance Experiments with SAT Solvers
  Annals of Mathematics and Artificial Intelligence, Vol. 43, No. 1-4
- The SAT2002 Competition
  Annals of Mathematics and Artificial Intelligence, Vol. 43, No. 1-4
- On SAT instance classes and a method for reliable performance experiments with SAT solvers
  31 December 2004 | Annals of Mathematics and Artificial Intelligence, Vol. 43, No. 1-4
- The SAT2002 competition
  31 December 2004 | Annals of Mathematics and Artificial Intelligence, Vol. 43, No. 1-4
- BIBLIOGRAPHY
- Hard Problem Generation for MKP
- Metaheuristic Applications and Their Solutions Quality
- Pareto Local Optimum Sets in the Biobjective Traveling Salesman Problem: An Experimental Study
- Solving the continuous flow-shop scheduling problem by metaheuristics
  European Journal of Operational Research, Vol. 151, No. 2
- Core Instances for Testing: A Case Study
  13 May 2003
- Nature-inspired cryptography: past, present and future
- An investigation of the relationship between problem characteristics and algorithm performance: a case study of the GAP
  IIE Transactions, Vol. 34, No. 3
- Using Finite Experiments to Study Asymptotic Performance
  16 December 2002
- Experimental Analysis of Algorithms
- Generating Experimental Data for Computational Testing with Machine Scheduling Applications
  Nicholas G. Hall,
  Marc E. Posner,
  1 December 2001 | Operations Research, Vol. 49, No. 6
- SatEx: A Web-based Framework for SAT Experimentation
  Electronic Notes in Discrete Mathematics, Vol. 9
- Experimental Evaluation of Heuristic Optimization Algorithms: A Tutorial
  Journal of Heuristics, Vol. 7, No. 3
- Decomposition methods for large job shops
  Computers & Operations Research, Vol. 28, No. 3
- On the Differences between “Practical” and “Applied”
  24 August 2001
- Asymptotic Complexity from Experiments? A Case Study for Randomized Algorithms
  24 August 2001
- Dijkstra's algorithm on-line
  31 December 2000 | ACM Journal of Experimental Algorithmics, Vol. 5
- Introduction
  Constraints, Vol. 5, No. 4
- Evaluation and comparison of production schedules
  Computers in Industry, Vol. 42, No. 2-3
- References
  30 September 2011
- Local Search Algorithms for SAT: An Empirical Evaluation
  Journal of Automated Reasoning, Vol. 24, No. 4
- The Effects of Coefficient Correlation Structure in Two-Dimensional Knapsack Problems on Solution Procedure Performance
  Raymond R. Hill,
  Charles H. Reilly,
  1 February 2000 | Management Science, Vol. 46, No. 2
- TimeNetManager — A Software Tool for Generating Random Temporal Networks
  15 December 2000
- Scatter Search to Generate Diverse MIP Solutions
- Information Theory and the Finite-Time Behavior of the Simulated Annealing Algorithm: Experimental Results
  Mark Fleischer,
  Sheldon H. Jacobson,
  1 February 1999 | INFORMS Journal on Computing, Vol. 11, No. 1
- Ordinal comparison of heuristic algorithms using stochastic optimization
  IEEE Transactions on Robotics and Automation, Vol. 15, No. 1
- Input models for synthetic optimization problems
- Heuristic sensitivity analysis in a combinatoric environment: An exposition and case study
  European Journal of Operational Research, Vol. 108, No. 3
- On the Performance of Heuristics on Finite and Infinite Fractal Instances of the Euclidean Traveling Salesman Problem
  Pablo Moscato,
  Michael G. Norman,
  1 May 1998 | INFORMS Journal on Computing, Vol. 10, No. 2
- Should computer scientists experiment more?
  Computer, Vol. 31, No. 5
- Properties of synthetic optimization problems
- Integer Programming
  SSRN Electronic Journal, Vol. 44
- A conceptual framework for combining artificial intelligence and optimization in engineering design
  Research in Engineering Design, Vol. 9, No. 1
- Conclusions and Future Directions
- Five pitfalls of empirical scheduling research
  10 June 2005
- Optimal level schedules for mixed-model, multi-level just-in-time assembly systems
  Annals of Operations Research, Vol. 69, No. 0
- On bicriterion minimal spanning trees: An approximation
  Computers & Operations Research, Vol. 23, No. 12
- Metaheuristics: A bibliography
  Annals of Operations Research, Vol. 63, No. 5
- Optimal experimental design for combinatorial problems
  Computational Economics, Vol. 9, No. 3
- Exploratory analysis of speedup learning data using expectation maximization
  Artificial Intelligence, Vol. 85, No. 1-2
- An empirical study of phase transitions in binary constraint satisfaction problems
  Artificial Intelligence, Vol. 81, No. 1-2
- Designing and reporting on computational experiments with heuristic methods
  Journal of Heuristics, Vol. 1, No. 1
- Testing heuristics: We have it all wrong
  Journal of Heuristics, Vol. 1, No. 1
- Readers' corner: The role of experiments in computer science
  Journal of Systems and Software, Vol. 30, No. 1-2
- Scaling effects in the CSP phase transition
  1 June 2005
- Branching rules for satisfiability
  Journal of Automated Reasoning, Vol. 15, No. 3
- Experimental evaluation in computer science: A quantitative study
  Journal of Systems and Software, Vol. 28, No. 1

Volume 42, Issue 2

March-April 1994

Pages 198-385

Article Information

Metrics

Information

Published Online:April 01, 1994

Cite as

J. N. Hooker, (1994) Needed: An Empirical Science of Algorithms. Operations Research 42(2):201-212.

https://doi.org/10.1287/opre.42.2.201

Keywords

analysis of algorithms: empirical methods for analysis of algorithms

PDF download

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Needed: An Empirical Science of Algorithms

Volume 42, Issue 2

Article Information

Metrics

Information

Cite as

Keywords

Sign Up for INFORMS Publications Updates and News