Optimizing over an Ensemble of Trained Neural Networks

References

• Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, et al. (2016) TensorFlow: A System for Large-Scale Machine Learning. Proc. 12th USENIX Conf. on Operating Systems Design and Implementation (USENIX Association, Berkeley, CA), 265–283.Google Scholar
• Anderson R, Huchette J, Ma W, Tjandraatmadja C, Vielma JP (2020) Strong mixed-integer programming formulations for trained neural networks. Math. Programming 183(1–2):1–37.Google Scholar
• Baardman L, Cristian R, Perakis G, Singhvi D, Skali Lami O, Thayaparan L (2022) The role of optimization in some recent advances in data-driven decision-making. Math. Programming. https://doi.org/10.1007/s10107-022-01874-9.Google Scholar
• Bartolini A, Lombardi M, Milano M, Benini L (2011) Neuron constraints to model complex real-world problems. Lee J, ed. Proc. Internat. Conf. on Principles and Practice of Constraint Programming (Springer, Berlin), 115–129.Google Scholar
• Benders JF (1962) Partitioning procedures for solving mixed variables programming problems. Numerical Math. 4(1):238–252.Crossref, Google Scholar
• Bergman D, Huang T, Brooks P, Lodi A, Raghunathan AU (2022) JANOS: An integrated predictive and prescriptive modeling framework. INFORMS J. Comput. 34(2):807–816.Google Scholar
• Bertsimas D, Kallus N (2020) From predictive to prescriptive analytics. Management Sci. 66(3):1025–1044.Link, Google Scholar
• Bertsimas D, O’Hair A, Relyea S, Silberholz J (2016) An analytics approach to designing combination chemotherapy regimens for cancer. Management Sci. 62(5):1511–1531.Link, Google Scholar
• Biggs M, Hariss R, Perakis G (2023) Constrained optimization of objective functions determined from random forests. Production Oper. Manag. 32(2):397–415.Google Scholar
• Biggs M, Sun W, Ettl M (2021) Model distillation for revenue optimization: Interpretable personalized pricing. Meila M, Zhang T, eds. Proc. Internat. Conf. on Machine Learn. (JMLR, Cambridge MA), 946–956.Google Scholar
• Botoeva E, Kouvaros P, Kronqvist J, Lomuscio A, Misener R (2020) Efficient verification of ReLU-based neural networks via dependency analysis. Proc. AAAI Conf. on Artificial Intelligence 34(04):3291–3299.Google Scholar
• Breiman L (1996) Bagging predictors. Machine Learn. 24(2):123–140.Crossref, Google Scholar
• Bunel R, Turkaslan I, Torr PHS, Kohli P, Kumar MP (2018) A unified view of piecewise linear neural network verification. Adv. Neural Inform. Processing Systems 31.Google Scholar
• Cheng CH, Nührenberg G, Ruess H (2017) Maximum resilience of artificial neural networks. D’Souza D, Narayan Kumar K, eds. Proc. Internat. Sympos. on Automated Tech. for Verification and Analysis (Springer, Berlin), 251–268.Google Scholar
• Cortez P, Cerdeira A, Almeida F, Matos T, Reis J (2009) Modeling wine preferences by data mining from physicochemical properties. Decision Support Systems 47(4):547–553.Crossref, Google Scholar
• Dietterich TG (2000) Ensemble methods in machine learning. Kittler J, Roli F, eds. Multiple Classifier Systems (Springer, Berlin), 1–15.Google Scholar
• Dutta S, Jha S, Sankaranarayanan S, Tiwari A (2018) Output range analysis for deep feedforward neural networks. Proc. NASA Formal Methods Sympos. (Springer, Berlin), 121–138.Google Scholar
• Fischetti M, Jo J (2018) Deep neural networks and mixed integer linear optimization. Constraints 23(3):296–309.Crossref, Google Scholar
• Fisher ML (1981) The lagrangian relaxation method for solving integer programming problems. Management Sci. 27(1):1–18.Link, Google Scholar
• Goodfellow I, Bengio Y, Courville A (2016) Deep Learning (MIT Press, Cambridge, MA).Google Scholar
• Grimstad B, Andersson H (2019) ReLU networks as surrogate models in mixed-integer linear programs. Comput. Chemical Engrg. 131:106580.Crossref, Google Scholar
• Gurobi Optimization I (2018) Gurobi optimizer reference manual. Accesed March 23, 2022, http://www.gurobi.com.Google Scholar
• Hansen LK, Salamon P (1990) Neural network ensembles. IEEE Trans. Pattern Anal. Machine Intelligence 12(10):993–1001.Crossref, Google Scholar
• Jamil M, Yang XS (2013) A literature survey of benchmark functions for global optimisation problems. Internat. J. Math. Modelling Numerical Optim. 4(2):150–194.Google Scholar
• Katz G, Barrett C, Dill DL, Julian K, Kochenderfer MJ (2017) Reluplex: An efficient smt solver for verifying deep neural networks. Proc. Internat. Conf. on Comput. Aided Verification (Springer, Berlin), 97–117.Google Scholar
• Li H, Wang X, Ding S (2018) Research and development of neural network ensembles: A survey. Artificial Intelligence Rev. 49(4):455–479.Crossref, Google Scholar
• Liu S, He L, Max Shen ZJ (2020) On-time last-mile delivery: Order assignment with travel-time predictors. Management Sci. 67(7):4095–4119.Google Scholar
• Lombardi M, Gualandi S (2016) A lagrangian propagator for artificial neural networks in constraint programming. Constraints 21(4):435–462.Crossref, Google Scholar
• Lombardi M, Milano M, Bartolini A (2017) Empirical decision model learning. Artificial Intelligence 244:343–367.Crossref, Google Scholar
• Lomuscio A, Maganti L (2017) An approach to reachability analysis for feed-forward ReLU neural networks. Preprint, submitted June 22, https://arxiv.org/abs/1706.07351.Google Scholar
• Lu X, Van Roy B (2017) Ensemble sampling. Adv. Neural Inform. Processing Systems 30.Google Scholar
• Maragno D, Wiberg H, Bertsimas D, Birbil SI, Hertog DD, Fajemisin A (2021) Mixed-integer optimization with constraint learning. Preprint, submitted November 4, https://arxiv.org/abs/2111.04469.Google Scholar
• Mishra SK (2006) Global optimization by differential evolution and particle swarm methods: Evaluation on some benchmark functions. Preprint, submitted October 2, https://dx.doi.org/10.2139/ssrn.933827.Google Scholar
• Mišić VV (2020) Optimization of tree ensembles. Oper. Res. 68(5):1605–1624.Link, Google Scholar
• Mistry M, Letsios D, Krennrich G, Lee RM, Misener R (2021) Mixed-integer convex nonlinear optimization with gradient-boosted trees embedded. INFORMS J. Comput. 33(3):1103–1119.Link, Google Scholar
• Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, et al. (2011) Scikit-learn: Machine learning in python. J. Machine Learn. Res. 12:2825–2830.Google Scholar
• Schweidtmann AM, Mitsos A (2019) Deterministic global optimization with artificial neural networks embedded. J. Optim. Theory Appl. 180(3):925–948.Crossref, Google Scholar
• Schweidtmann AM, Weber JM, Wende C, Netze L, Mitsos A (2022) Obey validity limits of data-driven models through topological data analysis and one-class classification. Optim. Engrg. 23(2):855–876.Crossref, Google Scholar
• Serra T, Kumar A, Ramalingam S (2020) Lossless compression of deep neural networks. Internat. Conf. on Integration of Constraint Programming, Artificial Intelligence, and Oper. Res. (Springer, Berlin), 417–430.Google Scholar
• Serra T, Tjandraatmadja C, Ramalingam S (2018) Bounding and counting linear regions of deep neural networks. Dy J, Krause A, eds. Proc. Internat. Conf. on Machine Learn. (JMLR, Cambridge, MA), 4558–4566.Google Scholar
• Sherali HD, Choi G (1996) Recovery of primal solutions when using subgradient optimization methods to solve lagrangian duals of linear programs. Oper. Res. Lett. 19(3):105–113.Crossref, Google Scholar
• Shi C, Emadikhiav M, Lozano L, Bergman D (2022) Careful! Training relevance is real. Preprint, submitted January 12, https://arxiv.org/abs/2201.04429.Google Scholar
• Smith JE, Winkler RL (2006) The optimizer’s curse: Skepticism and postdecision surprise in decision analysis. Management Sci. 52(3):311–322.Link, Google Scholar
• Thebelt A, Kronqvist J, Mistry M, Lee RM, Sudermann-Merx N, Misener R (2021) Entmoot: A framework for optimization over ensemble tree models. Comput. Chemical Engrg. 151:107343.Crossref, Google Scholar
• Tjeng V, Xiao K, Tedrake R (2017) Evaluating robustness of neural networks with mixed integer programming. Preprint, submitted November 20, https://arxiv.org/abs/1711.07356.Google Scholar
• Tsay C, Kronqvist J, Thebelt A, Misener R (2021) Partition-based formulations for mixed-integer optimization of trained relu neural networks. Preprint, submitted February 8, https://arxiv.org/abs/2102.04373.Google Scholar
• Verwer S, Zhang Y, Ye QC (2017) Auction optimization using regression trees and linear models as integer programs. Artificial Intelligence 244:368–395.Crossref, Google Scholar
• Vielma JP (2015) Mixed integer linear programming formulation techniques. SIAM Rev. 57(1):3–57.Crossref, Google Scholar
• Wang K, Lozano L, Bergman D, Cardonha C (2021) A two-stage exact algorithm for optimization of neural network ensemble. Proc. Internat. Conf. on Integration of Constraint Programming, Artificial Intelligence, and Oper. Res. (Springer, Berlin), 106–114.Google Scholar
• Wolsey LA, Nemhauser GL (1999) Integer and Combinatorial Optimization, vol. 55 (John Wiley & Sons, Hoboken, NJ).Google Scholar
• Xiao KY, Tjeng V, Shafiullah NM, Madry A (2018) Training for faster adversarial robustness verification via inducing ReLU stability. Preprint, submitted September 9, https://arxiv.org/abs/1809.03008.Google Scholar
• Yeh IC (1998) Modeling of strength of high-performance concrete using artificial neural networks. Cement Concrete Res. 28(12):1797–1808.Crossref, Google Scholar
• Zhou ZH, Wu J, Tang W (2002) Ensembling neural networks: many could be better than all. Artificial Intelligence 137(1–2):239–263.Crossref, Google Scholar