Offline Multi-Action Policy Learning: Generalization and Optimization

References

• Abbasi-Yadkori Y, Pál D, Szepesvári C (2011) Improved algorithms for linear stochastic bandits. Adv. Neural Inform. Processing Systems 2312–2320.Google Scholar
• Agrawal S, Goyal N (2013) Thompson sampling for contextual bandits with linear payoffs. Proc. Internat. Conf. on Machine Learn. 127–135.Google Scholar
• Aswani A, Shen Z-J, Siddiq A (2018) Inverse optimization with noisy data. Oper. Res. 66(3):870–892.Link, Google Scholar
• Athey S (2017) Beyond prediction: Using big data for policy problems. Science 355(6324):483–485.Crossref, Google Scholar
• Athey S, Wager S (2017) Efficient policy learning. Preprint, submitted February 9, https://arxiv.org/abs/1702.02896.Google Scholar
• Athey S, Imbens GW, Wager S (2016a) Approximate residual balancing: debiased inference of average treatment effects in high dimensions. J. Roy. Statist. Soc. Ser. B Statist. Methodology.Google Scholar
• Athey S, Tibshirani J, Wager S (2016b) Generalized random forests. Preprint, submitted October 5, https://arxiv.org/abs/1610.01271.Google Scholar
• Ban GY, Rudin C (2018) The big data newsvendor: Practical insights from machine learning.Google Scholar
• Bartlett PL, Mendelson S (2002) Rademacher and gaussian complexities: Risk bounds and structural results. J. Machine Learn. Res. 3(Nov):463–482.Google Scholar
• Bastani H, Bayati M (2015) Online decision-making with high-dimensional covariates.Google Scholar
• Bastani H, Bastani O, Kim C (2019) Interpreting predictive models for human-in-the-loop analytics.Google Scholar
• Bastani H, Harsha P, Perakis G, Singhvi D (2018) Sequential learning of product recommendations with customer disengagement. Preprint, submitted August 29, https://dx.doi.org/10.2139/ssrn.3240970.Google Scholar
• Belloni A, Chernozhukov V, Hansen C (2014) Inference on treatment effects after selection among high-dimensional controls. Rev. Econ. Stud. 81(2):608–650.Crossref, Google Scholar
• Belloni A, Chernozhukov V, Fernández-Val I, Hansen C (2017) Program evaluation and causal inference with high-dimensional data. Econometrica 85(1):233–298.Crossref, Google Scholar
• Bertsimas D, Dunn J (2017) Optimal classification trees. Machine Learn. 106(7):1039–1082.Crossref, Google Scholar
• Bertsimas D, Kallus N (2014) From predictive to prescriptive analytics. Preprint, submitted February 22, https://arxiv.org/abs/1402.5481.Google Scholar
• Besbes O, Zeevi A (2009) Dynamic pricing without knowing the demand function: Risk bounds and near-optimal algorithms. Oper. Res. 57(6):1407–1420.Link, Google Scholar
• Bickel P, Klaassen C, Ritov Y, Wellner J (1998) Efficient and Adaptive Estimation for Semiparametric Models (Springer-Verlag).Google Scholar
• Boucheron S, Lugosi G, Massart P (2013) Concentration Inequalities: A Nonasymptotic Theory of Independence (Oxford University Press).Crossref, Google Scholar
• Bousquet O (2002) A bennett concentration inequality and its application to suprema of empirical processes. Comptes Rendus Math. 334(6):495–500.Crossref, Google Scholar
• Breiman L, Friedman J, Stone CJ, Olshen RA (1984) Classification and Regression Trees (CRC Press, Boca Raton, FL).Google Scholar
• Bubeck S, Cesa-Bianchi N, et al. (2012) Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations Trends Machine Learn. 5(1):1–122.Crossref, Google Scholar
• Chan CW, Farias VF, Bambos N, Escobar GJ (2012) Optimizing intensive care unit discharge decisions with patient readmissions. Oper. Res. 60(6):1323–1341.Link, Google Scholar
• Chernozhukov V, Escanciano JC, Ichimura H, Newey WK (2016) Locally robust semiparametric estimation. Preprint, submitted July 29, https://arxiv.org/abs/1608.00033.Google Scholar
• Chernozhukov V, Chetverikov D, Demirer M, Duflo E, Hansen C, Newey W, Robins J (2018) Double/debiased machine learning for treatment and structural parameters. Econom. J. 21(1):C1–C68.Crossref, Google Scholar
• Chu W, Li L, Reyzin L, Schapire R (2011) Contextual bandits with linear payoff functions. Proc. 14th Internat. Conf. on Artificial Intelligence and Statist. 208–214.Google Scholar
• Dudík M, Langford J, Li L (2011) Doubly robust policy evaluation and learning. Proc. 28th Internat. Conf. on Machine Learn. 1097–1104.Google Scholar
• Dudley RM (1967) The sizes of compact subsets of hilbert space and continuity of gaussian processes. J. Functional Anal. 1(3):290–330.Crossref, Google Scholar
• Farias V, Li A (2018) Learning preferences with side information. Management Sci. 65(7):3131–3149.Link, Google Scholar
• Farrell MH (2015) Robust inference on average treatment effects with possibly more covariates than observations. J. Econometrics 189(1):1–23.Crossref, Google Scholar
• Fukuoka Y, Zhou M, Vittinghoff E, Haskell W, Goldberg K, Aswani A (2018) Objectively measured baseline physical activity patterns in women in the mped trial: Cluster analysis. JMIR Public Health Surveillance 4(1):e10.Crossref, Google Scholar
• Gine E, Koltchinskii V (2006) Concentration inequalities and asymptotic results for ratio type empirical processes. Ann. Probability 34(3):1143–1216.Crossref, Google Scholar
• Gurobi Optimization Inc. (2017) Gurobi 7.5 performance benchmarks. http://www.gurobi.com/pdfs/benchmarks.pdf.Google Scholar
• Hahn J (1998) On the role of the propensity score in efficient semiparametric estimation of average treatment effects. Econometrica 66(2):315–331.Crossref, Google Scholar
• Hastie T, Tibshirani R, Friedman J (2013) The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Series in Statistics (Springer, New York).Google Scholar
• Haussler D (1995) Sphere packing numbers for subsets of the Boolean n-cube with bounded Vapnik-Chervonenkis dimension. J. Combinatorial Theory Ser. A 69(2):217–232.Crossref, Google Scholar
• Heckman JJ (1979) Sample selection bias as a specification error. Econometrica 47(1):153–161.Crossref, Google Scholar
• Hirano K, Porter JR (2009) Asymptotics for statistical treatment rules. Econometrica 77(5):1683–1701.Crossref, Google Scholar
• Hirshberg DA, Wager S (2018) Augmented minimax linear estimation. Preprint, submitted April 12, https://arxiv.org/abs/1712.00038.Google Scholar
• IBM ILOG CPLEX (2017) V12.6.0 users manual. https://www.ibm.com/support/knowledgecenter/en/SSSA5P_12.6.0/ilog.odms.studio.help/Optimization_Studio/topics/COS_home.html.Google Scholar
• Imbens GW (2004) Nonparametric estimation of average treatment effects under exogeneity: A review. Rev. Econom. Statist. 86(1):4–29.Crossref, Google Scholar
• Imbens GW, Rubin DB (2015) Causal Inference in Statistics, Social, and Biomedical Sciences (Cambridge University Press).Crossref, Google Scholar
• Joachims T, Swaminathan A, de Rijke M (2018) Deep learning with logged bandit feedback. Proc. Internat. Conf. on Learn. Representations.Google Scholar
• Kallus N (2017) Recursive partitioning for personalization using observational data 1789–1798.Google Scholar
• Kallus N (2018) Balanced policy evaluation and learning. Adv. Neural Inform. Processing Systems.Google Scholar
• Kallus N, Zhou A (2018) Confounding-robust policy improvement. Preprint, submitted May 22, https://arxiv.org/abs/1805.08593.Google Scholar
• Kim ES, Herbst RS, Wistuba II, Lee JJ, Blumenschein GR, Tsao A, Stewart DJ, et al. (2011) The battle trial: Personalizing therapy for lung cancer. Cancer Discovery 1(1):44–53.Crossref, Google Scholar
• Kitagawa T, Tetenov A (2018) Who should be treated? Empirical welfare maximization methods for treatment choice. Econometrica 86(2):591–616.Crossref, Google Scholar
• Lai TL, Robbins H (1985) Asymptotically efficient adaptive allocation rules. Adv. Appl. Math. 6(1):4–22.Crossref, Google Scholar
• Li L, Lu Y, Zhou D (2017) Provably optimal algorithms for generalized linear contextual bandits. Proc. Internat. Conf. on Machine Learn. 2071–2080.Google Scholar
• Li L, Chu W, Langford J, Schapire RE (2010) A contextual-bandit approach to personalized news article recommendation. Proc. 19th Internat. Conf. on World Wide Web (ACM, New York), 661–670.Google Scholar
• Manski CF (2004) Statistical treatment rules for heterogeneous populations. Econometrica 72(4):1221–1246.Crossref, Google Scholar
• Maurer A, Pontil M (2009) Empirical Bernstein bounds and sample variance penalization. Proc. Conf. on Learn. Theory.Google Scholar
• Mendelson S, Neeman J, et al. (2010) Regularization in kernel learning. Ann. Statist. 38(1):526–565.Crossref, Google Scholar
• Mohri M, Rostamizadeh A, Talwalkar A (2012) Foundations of Machine Learning. Adaptive Computation and Machine Learning Series (MIT Press, Cambridge, MA).Google Scholar
• Newey WK (1994) The asymptotic variance of semiparametric estimators. Econometrica 62(6):1349–1382.Crossref, Google Scholar
• Newey WK, Robins JR (2018) Cross-fitting and fast remainder rates for semiparametric estimation. Preprint, submitted January 27, https://arxiv.org/abs/1801.09138.Google Scholar
• Neyman J (1923) Sur les applications de la théorie des probabilités aux experiences agricoles: Essai des principes. Roczniki Nauk Rolniczych 10:1–51.Google Scholar
• Ozanne EM, Howe R, Omer Z, Esserman LJ (2014) Development of a personalized decision aid for breast cancer risk reduction and management. BMC Medical Inform. Decision Making 14(1):4.Crossref, Google Scholar
• Rakhlin A, Sridharan K (2016) BISTRO: An efficient relaxation-based method for contextual bandits. Proc. Internat. Conf. on Machine Learn. 1977–1985.Google Scholar
• Rigollet P, Zeevi A (2010) Nonparametric bandits with covariates. Preprint, submitted March 8, https://arxiv.org/abs/1003.1630.Google Scholar
• Robins JM, Rotnitzky A (1995) Semiparametric efficiency in multivariate regression models with missing data. J. Amer. Statist. Assoc. 90(429):122–129.Crossref, Google Scholar
• Robins JM, Rotnitzky A, Lue PZ (1994) Estimation of regression coefficients when some regressors are not always observed. J. Amer. Statist. Assoc. 89(427):846–866.Crossref, Google Scholar
• Rosenbaum PR, Rubin DB (1983) The central role of the propensity score in observational studies for causal effects. Biometrika 70(1):41–55.Crossref, Google Scholar
• Rubin DB (1974) Estimating causal effects of treatments in randomized and nonrandomized studies. J. Edu. Psych. 66(5):688.Crossref, Google Scholar
• Russo D, Van Roy B (2014) Learning to optimize via posterior sampling. Math. Oper. Res. 39(4):1221–1243.Link, Google Scholar
• Scharfstein DO, Rotnitzky A, Robins JM (1999) Adjusting for nonignorable drop-out using semiparametric nonresponse models. J. Amer. Statist. Assoc. 94(448):1096–1120.Crossref, Google Scholar
• Schick A (1986) On asymptotically efficient estimation in semiparametric models. Ann. Statist. 14(3):1139–1151.Crossref, Google Scholar
• Stoye J (2009) Minimax regret treatment choice with finite samples. J. Econometrics 151(1):70–81.Crossref, Google Scholar
• Sverdrup E, Kanodia A, Zhou Z, Athey S, Wager S (2020) policytree: Policy learning via doubly robust empirical welfare maximization over trees. J. Open Source Software 5(50):2232.Crossref, Google Scholar
• Swaminathan A, Joachims T (2015) Batch learning from logged bandit feedback through counterfactual risk minimization. J. Machine Learn. Res. 16:1731–1755.Google Scholar
• Swaminathan A, Krishnamurthy A, Agarwal A, Dudík M, Langford J, Jose D, Zitouni I (2017) Off-Policy Evaluation for Slate Recommendation (Curran Associates).Google Scholar
• Talagrand M (1994) Sharper bounds for gaussian and empirical processes. Ann. Probability 22(1):28–76.Crossref, Google Scholar
• Vapnik VN, Ya Chervonenkis A (1971) On the uniform convergence of relative frequencies of events to their probabilities. Theory Probability Appl. 16(2):264.Crossref, Google Scholar
• Zhang B, Anastasios A Tsiatis MD, Zhang M, Laber E (2012) Estimating optimal treatment regimes from a classification perspective. Statistics 1(1):103–114.Crossref, Google Scholar
• Zhao Y, Zeng D, Rush AJ, Kosorok MR (2012) Estimating individualized treatment rules using outcome weighted learning. J. Amer. Statist. Assoc. 107(499):1106–1118.Crossref, Google Scholar
• Zhao YQ, Zeng D, Laber EB, Song R, Yuan M, Kosorok MR (2014) Doubly robust learning for estimating individualized treatment with censored data. Biometrika 102(1):151–168.Crossref, Google Scholar
• Zhou M, Fukuoka Y, Mintz Y, Goldberg K, Kaminsky P, Flowers E, Aswani A (2018a) Evaluating machine learning–based automated personalized daily step goals delivered through a mobile phone app: Randomized controlled trial. JMIR Mhealth Uhealth 6(1):e28.Crossref, Google Scholar
• Zhou X, Mayer-Hamblett N, Khan U, Kosorok MR (2017) Residual weighted learning for estimating individualized treatment rules. J. Amer. Statist. Assoc. 112(517):169–187.Crossref, Google Scholar
• Zhou Z, Athey S, Wager S (2018b) Offline multi-action policy learning: Generalization and optimization. Preprint, submitted Ocotber 10, https://arxiv.org/abs/1810.04778.Google Scholar