Weighted Optimal Classification Forests

This paper introduces weighted optimal classification forests (WOCFs), a new family of classifiers that takes advantage of an optimal ensemble of decision trees to derive accurate and interpretable classifiers. We propose a novel mathematical optimization-based methodology that jointly constructs an ensemble of decision trees, where each tree contributes a class prediction for every observation. The final classification is determined by a weighted majority vote across the trees, with the voting weights and tree structures optimized simultaneously. Exploiting the permutation-invariance property of trees in an ensemble, we introduce novel mixed-integer programming (MIP) strengthening/scaling techniques. We report the results of our computational experiments, from which we conclude that our method has equal or superior performance compared with state-of-the-art tree-based classification methods for small- to medium-sized instances. We also present three real-world case studies showing that our methodology has interesting implications in terms of interpretability. Overall, WOCFs complement existing methods such as CART (Classification and Regression Trees), optimal classification trees, random forests and XGBoost. Beyond improving accuracy without compromising interpretability, we also see unique properties emerging in terms of different trees focusing on different feature variables. This provides nontrivial improvement in interpretability and usability of the trained model in terms of counterfactual explanation. Thus, despite the apparent computational challenge of WOCFs that limit the size of the problems that can be efficiently solved with current MIP, this is an important research direction that can lead to qualitatively different insights for researchers and complement the toolbox of practitioners for high stakes problems.

History: Accepted by Andrea Lodi, Area Editor for Design & Analysis of Algorithms–Discrete.

Funding: This research has been partially supported by AEI [Grants PID2024-156594NB-C21 and PID2024-155024-2] (AEI and EU funds) and Junta de Andalucía [Grant C-EXP-139-UGR23]. The first author was also partially supported by the IMAG-Maria de Maeztu [Grant CEX2020-001105-M]. The second and third authors have been also partially supported by IMUS-Maria de Maeztu [Grant CEX2024-001517-M]. This project is funded in part by Carnegie Mellon University’s Mobility21 National University Transportation Center (US Dept. Transportation).

Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information (https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2025.1475) as well as from the IJOC GitHub software repository (https://github.com/INFORMSJoC/2025.1475). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/.