The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Model

References

• Agarwal A, Kakade S, Yang LF (2020) Model-based reinforcement learning with a generative model is minimax optimal. Abernethy J, Agarwal S, eds. Proc. 33rd Conf. Learn. Theory, Proceedings of Machine Learning Research, vol. 125 (PMLR, New York), 67–83. Google Scholar
• Amico A, Verginer L, Casiraghi G, Vaccario G, Schweitzer F (2024) Adapting to disruptions: Managing supply chain resilience through product rerouting. Sci. Adv. 10(3):eadj1194.Crossref, Google Scholar
• Azar MG, Munos R, Kappen HJ (2013) Minimax PAC bounds on the sample complexity of reinforcement learning with a generative model. Machine Learn. 91(3):325–349.Crossref, Google Scholar
• Badrinath KP, Kalathil D (2021) Robust reinforcement learning using least squares policy iteration with provable performance guarantees. Meila M, Zhang T, eds. Proc. 38th Internat. Conf. Machine Learn., Proceedings of Machine Learning Research (PMLR, New York), 511–520.Google Scholar
• Bäuerle N, Glauner A (2022) Distributionally robust Markov decision processes and their connection to risk measures. Math. Oper. Res. 47(3):1757–1780.Link, Google Scholar
• Beck CL, Srikant R (2012) Error bounds for constant step-size Q-learning. Systems Control Lett. 61(12):1203–1208.Crossref, Google Scholar
• Bertsimas D, Gupta V, Kallus N (2018) Data-driven robust optimization. Math. Programming 167(2):235–292.Crossref, Google Scholar
• Bertsimas D, Sim M, Zhang M (2019) Adaptive distributionally robust optimization. Management Sci. 65(2):604–618.Link, Google Scholar
• Bitran G, Caldentey R (2003) An overview of pricing models for revenue management. Manufacturing Service Oper. Management 5(3):203–229.Link, Google Scholar
• Blanchet J, Murthy K (2019) Quantifying distributional model risk via optimal transport. Math. Oper. Res. 44(2):565–600.Link, Google Scholar
• Blanchet J, Lu M, Zhang T, Zhong H (2023) Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. Adv. Neural Inform. Processing Systems 36:66845–66859.Crossref, Google Scholar
• Chen J, Jiang N (2019) Information-theoretic considerations in batch reinforcement learning. Kamalika Chaudhuri K, Salakhutdinov R, eds. Proc. 36th Internat. Conf. Machine Learn., Proceedings of Machine Learning Research, vol. 97 (PMLR, New York), 1042–1051. Google Scholar
• Chen Z, Sim M, Xu H (2019) Distributionally robust optimization with infinitely constrained ambiguity sets. Oper. Res. 67(5):1328–1344.Link, Google Scholar
• Chen Z, Maguluri ST, Shakkottai S, Shanmugam K (2020) Finite-sample analysis of stochastic approximation using smooth convex envelopes. Preprint, submitted February 3, https://arxiv.org/abs/2002.00874. Google Scholar
• Clavier P, Le Pennec E, Geist M (2024a) Towards minimax optimality of model-based robust reinforcement learning. Kiyavash N, Mooij JM, eds. Proc. 40th Conf. Uncertainty Artificial Intelligence, Proceedings of Machine Learning Research, vol. 244 (PMLR, New York), 820–855. Google Scholar
• Clavier P, Shi L, Le Pennec E, Mazumdar E, Wierman A, Geist M (2024b) Near-optimal distributionally robust reinforcement learning with general lp norms. Globerson A, Mackey L, Belgrave D, Fan A, Paquet U, Tomczak J, Zhang C, eds. Adv. Neural Inform. Processing Systems, vol. 37 (Curran Associates, Inc., Red Hook, NY), 1750–1810. Google Scholar
• de Castro Silva J, Soma N, Maculan N (2003) A greedy search for the three-dimensional bin packing problem: The packing static stability case. Internat. Trans. Oper. Res. 10(2):141–153.Crossref, Google Scholar
• Delage E, Ye Y (2010) Distributionally robust optimization under moment uncertainty with application to data-driven problems. Oper. Res. 58(3):595–612.Link, Google Scholar
• Derman E, Mannor S (2020) Distributional robustness and regularization in reinforcement learning. Preprint, submitted March 5, https://arxiv.org/abs/2003.02894.Google Scholar
• Ding W, Shi L, Chi Y, Zhao D (2023) Seeing is not believing: Robust reinforcement learning against spurious correlation. Oh A, Naumann T, Globerson A, Saenko K, Hardt M, Levine S, eds. Adv. Neural Inform. Processing Systems, vol. 36 (Curran Associates, Inc., Red Hook, NY), 66328–66363.Google Scholar
• Dong J, Li J, Wang B, Zhang J (2022) Online policy optimization for robust MDP. Preprint, submitted September 28, https://arxiv.org/abs/2209.13841.Google Scholar
• Dong K, Wang Y, Chen X, Wang L (2019) Q-learning with UCB exploration is sample efficient for infinite-horizon MDP. Preprint, submitted January 27, https://arxiv.org/abs/1901.09311.Google Scholar
• Duchi JC, Namkoong H (2021) Learning models with uniform performance via distributionally robust optimization. Ann. Statist. 49(3):1378–1406.Crossref, Google Scholar
• Fatemi M, Killian TW, Subramanian J, Ghassemi M (2021) Medical dead-ends and learning to identify high-risk states and treatments. Adv. Neural Inform. Processing Systems 34:4856–4870.Google Scholar
• Federgruen A, Heching A (1999) Combined pricing and inventory control under uncertainty. Oper. Res. 47(3):454–475.Link, Google Scholar
• Gao R (2023) Finite-sample guarantees for Wasserstein distributionally robust optimization: Breaking the curse of dimensionality. Oper. Res. 71(6):2291–2306.Link, Google Scholar
• Goyal V, Grand-Clement J (2023) Robust Markov decision processes: Beyond rectangularity. Math. Oper. Res. 48(1):203–226.Link, Google Scholar
• Han S, Su S, He S, Han S, Yang H, Miao F (2022) What is the solution for state adversarial multi-agent reinforcement learning? Preprint, submitted December 6, https://arxiv.org/abs/2212.02705.Google Scholar
• Ho CP, Petrik M, Wiesemann W (2018) Fast bellman updates for robust MDPs. Dy J, Krause A, eds. Proc. 35th Internat. Conf. Machine Learn., Proceedings of Machine Learning Research, vol. 80 (PMLR, New York), 1979–1988. Google Scholar
• Ho CP, Petrik M, Wiesemann W (2021) Partial policy iteration for l1-robust Markov decision processes. J. Machine Learn. Res. 22(275):1–46.Google Scholar
• Iyengar GN (2005) Robust dynamic programming. Math. Oper. Res. 30(2):257–280.Link, Google Scholar
• Jafarnia-Jahromi M, Wei C-Y, Jain R, Luo H (2020) A model-free learning algorithm for infinite-horizon average-reward MDPs with near-optimal regret. Preprint, submitted June 8, https://arxiv.org/abs/2006.04354.Google Scholar
• Jin Y, Yang Z, Wang Z (2021) Is pessimism provably efficient for offline RL? Meilă M, Zhang T, eds. Proc. 38th Internat. Conf. Machine Learn., Proceedings of Machine Learning Research, vol. 139 (PMLR, New York), 5084–5096. Google Scholar
• Jin C, Allen-Zhu Z, Bubeck S, Jordan MI (2018) Is Q-learning provably efficient? Bengio S, Wallach H, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R, eds. Adv. Neural Inform. Processing Systems (Curran Associates, Inc., Red Hook, NY), 4863–4873.Google Scholar
• Jin C, Krishnamurthy A, Simchowitz M, Yu T (2020a) Reward-free exploration for reinforcement learning. Daumé H III, Singh A, eds. Proc. 37th Internat. Conf. Machine Learn., Proceedings of Machine Learning Research, vol. 119 (PMLR, New York), 4870–4879. Google Scholar
• Jin C, Yang Z, Wang Z, Jordan MI (2020b) Provably efficient reinforcement learning with linear function approximation. Abernethy J, Agarwal S, eds. Proc. 33rd Conf. Learn. Theory, Proceedings of Machine Learning Research, vol. 125 (PMLR, New York), 2137–2143. Google Scholar
• Kaufman DL, Schaefer AJ (2013) Robust modified policy iteration. INFORMS J. Comput. 25(3):396–410.Link, Google Scholar
• Kearns MJ, Singh SP (1999) Finite-sample convergence rates for Q-learning and indirect algorithms. Kearns MJ, Solla SA, Cohn DA, eds. Adv. Neural Inform. Processing Systems (MIT Press, Cambridge, MA), 996–1002.Google Scholar
• Klopp O, Lounici K, Tsybakov AB (2017) Robust matrix completion. Probability Theory Related Fields 169(1–2):523–564.Crossref, Google Scholar
• Kober J, Bagnell JA, Peters J (2013) Reinforcement learning in robotics: A survey. Internat. J. Robotics Res. 32(11):1238–1274.Crossref, Google Scholar
• Kumar N, Derman E, Geist M, Levy K, Mannor S (2023) Policy gradient for s-rectangular robust Markov decision processes. Preprint, submitted January 31, https://arxiv.org/abs/2301.13589.Google Scholar
• Lam H (2019) Recovering best statistical guarantees via the empirical divergence-based distributionally robust optimization. Oper. Res. 67(4):1090–1105.Abstract, Google Scholar
• LeCam L (1973) Convergence of estimates under dimensionality restrictions. Ann. Statist. 1(1):38–53.Crossref, Google Scholar
• Lee J, Jeon W, Lee B, Pineau J, Kim K-E (2021) Optidice: Offline policy optimization via stationary distribution correction estimation. Proc. Internat. Conf. Machine Learn. (PMLR, New York), 6120–6130.Google Scholar
• Li Y, Lan G (2023) First-order policy optimization for robust policy evaluation. Preprint, submitted July 29, https://arxiv.org/abs/2307.15890.Google Scholar
• Li Z, Wang S, Si N (2025) Near-optimal sample complexities of divergence-based s-rectangular distributionally robust reinforcement learning. Preprint, submitted May 18, https://arxiv.org/abs/2505.12202.Google Scholar
• Li Y, Zhao T, Lan G (2022a) First-order policy optimization for robust Markov decision process. Preprint, submitted September 21, https://arxiv.org/abs/2209.10579.Google Scholar
• Li G, Chi Y, Wei Y, Chen Y (2022b) Minimax-optimal multi-agent RL in Markov games with a generative model. Adv. Neural Inform. Processing Systems 35:15353–15367.Crossref, Google Scholar
• Li G, Wei Y, Chi Y, Chen Y (2024a) Breaking the sample size barrier in model-based reinforcement learning with a generative model. Oper. Res. 72(1):203–221.Link, Google Scholar
• Li G, Yan Y, Chen Y, Fan J (2023) Minimax-optimal reward-agnostic exploration in reinforcement learning. Preprint, submitted April 14, https://arxiv.org/abs/2304.07278.Google Scholar
• Li G, Cai C, Chen Y, Wei Y, Chi Y (2024b) Is Q-learning minimax optimal? A tight sample complexity analysis. Oper. Res. 72(1):222–236.Link, Google Scholar
• Li G, Shi L, Chen Y, Chi Y, Wei Y (2024c) Settling the sample complexity of model-based offline reinforcement learning. Ann. Statist. 52(1):233–260.Crossref, Google Scholar
• Li G, Shi L, Chen Y, Gu Y, Chi Y (2021) Breaking the sample complexity barrier to regret-optimal model-free reinforcement learning. Adv. Neural Inform. Processing Systems 34:17762–17776.Google Scholar
• Liang Z, Ma X, Blanchet J, Zhang J, Zhou Z (2023) Single-trajectory distributionally robust reinforcement learning. Preprint, submitted January 27, https://arxiv.org/abs/2301.11721.Google Scholar
• Liao P, Qi Z, Wan R, Klasnja P, Murphy SA (2022) Batch policy learning in average reward Markov decision processes. Ann. Statist. 50(6):3364.Crossref, Google Scholar
• Liu Z, Xu P (2024a) Distributionally robust off-dynamics reinforcement learning: Provable efficiency with linear function approximation. Preprint, submitted February 23, https://arxiv.org/abs/2402.15399.Google Scholar
• Liu Z, Xu P (2024b) Minimax optimal and computationally efficient algorithms for distributionally robust offline reinforcement learning. Preprint, submitted March 14, https://arxiv.org/abs/2403.09621.Google Scholar
• Liu S, Ngiam KY, Feng M (2019) Deep reinforcement learning for clinical decision support: A brief survey. Preprint, submitted July 22, https://arxiv.org/abs/1907.09475.Google Scholar
• Liu Z, Bai Q, Blanchet J, Dong P, Xu W, Zhou Z, Zhou Z (2022) Distributionally robust Q-learning. Chaudhuri K, Jegelka S, Song L, Szepesvári C, Niu G, Sabato S, eds. Proc. 39th Internat. Conf. Machine Learn., Proceedings of Machine Learning Research, vol. 162 (PMLR, New York), 13623–13643. Google Scholar
• Lu M, Zhong H, Zhang T, Blanchet J (2024) Distributionally robust reinforcement learning with interactive data collection: Fundamental hardness and near-optimal algorithm. Preprint, submitted April 4, https://arxiv.org/abs/2404.03578.Google Scholar
• Ma X, Liang Z, Blanchet J, Liu M, Xia L, Zhang J, Zhao Q, Zhou Z (2022) Distributionally robust offline reinforcement learning with linear function approximation. Preprint, submitted September 14, https://arxiv.org/abs/2209.06620.Google Scholar
• Mahmood AR, Korenkevych D, Vasan G, Ma W, Bergstra J (2018) Benchmarking reinforcement learning algorithms on real-world robots. Billard A, Dragan A, Peters J, Morimoto J, eds. Proc. 2nd Conf. Robot Learn., Proceedings of Machine Learning Research, vol. 87 (PMLR, New York), 561–591.Google Scholar
• Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing Atari with deep reinforcement learning. Preprint, submitted December 19, https://arxiv.org/abs/1312.5602.Google Scholar
• Moos J, Hansel K, Abdulsamad H, Stark S, Clever D, Peters J (2022) Robust reinforcement learning: A review of foundations and recent advances. Machine Learn. Knowledge Extraction 4(1):276–315.Crossref, Google Scholar
• Morales-Enciso S, Branke J (2012) Revenue maximization through dynamic pricing under unknown market behaviour. Ravizza S, Holborn P, eds. Proc. 3rd Student Conf. Oper. Res., Open Access Series in Informatics, vol. 22 (Schloss Dagstuhl–Leibniz-Zentrum für Informatik, Dagstuhl, Germany), 11–20. Google Scholar
• Nilim A, El Ghaoui L (2005) Robust control of Markov decision processes with uncertain transition matrices. Oper. Res. 53(5):780–798.Link, Google Scholar
• OpenAI (2023) Gpt-4 technical report (March 15), https://cdn.openai.com/papers/gpt-4.pdf.Google Scholar
• Pan Y, Chen Y, Lin F (2023) Adjustable robust reinforcement learning for online 3D bin packing. Preprint, submitted October 6, https://arxiv.org/abs/2310.04323.Google Scholar
• Panaganti K, Kalathil D (2022) Sample complexity of robust reinforcement learning with a generative model. Camps-Valls G, Ruiz FJR, Valera I, eds. Proc. 25th Internat. Conf. Artificial Intelligence Statist., Proceedings of Machine Learning Research, vol. 151 (PMLR, New York), 9582–9602.Google Scholar
• Panaganti K, Xu Z, Kalathil D, Ghavamzadeh M (2022) Robust reinforcement learning using offline data. Adv. Neural Inform. Processing Systems 35:32211–32224.Crossref, Google Scholar
• Panaganti K, Xu Z, Kalathil D, Ghavamzadeh M (2023) Bridging distributionally robust learning and offline RL: An approach to mitigate distribution shift and partial data coverage. Preprint, submitted October 27, https://arxiv.org/abs/2310.18434.Google Scholar
• Park B, Van Roy B (2015) Adaptive execution: Exploration and learning of price impact. Oper. Res. 63(5):1058–1076.Link, Google Scholar
• Qiaoben Y, Zhou X, Ying C, Zhu J (2021) Strategically-timed state-observation attacks on deep reinforcement learning agents. Proc. ICML 2021 Workshop Adversarial Machine Learn. (OpenReview.net).Google Scholar
• Qu G, Wierman A, Li N (2022) Scalable reinforcement learning for multiagent networked systems. Oper. Res. 70(6):3601–3628.Link, Google Scholar
• Rahimian H, Mehrotra S (2019) Distributionally robust optimization: A review. Preprint, submitted August 13, https://arxiv.org/abs/1908.05659.Google Scholar
• Ramesh SS, Sessa PG, Hu Y, Krause A, Bogunovic I (2023) Distributionally robust model-based reinforcement learning with large state spaces. Preprint, submitted September 5, https://arxiv.org/abs/2309.02236.Google Scholar
• Rashidinejad P, Zhu B, Ma C, Jiao J, Russell S (2021) Bridging offline reinforcement learning and imitation learning: A tale of pessimism. IEEE Trans. Inform. Theory 68(12):8156–8196.Google Scholar
• Roy A, Xu H, Pokutta S (2017) Reinforcement learning under model mismatch. Adv. Neural Inform. Processing Systems 30:3043–3052.Google Scholar
• Shi L, Chi Y (2024) Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. J. Machine Learn. Res. 25(200):1–91.Google Scholar
• Shi L, Mazumdar E, Chi Y, Wierman A (2024) Sample-efficient robust multi-agent reinforcement learning in the face of environmental uncertainty. Salakhutdinov R, Kolter Z, Heller K, Weller A, Oliver N, Scarlett J, Berkenkamp F, eds. Proc. 41st Internat. Conf. Machine Learn., Proceedings of Machine Learning Research, vol. 235 (PMLR, New York), 44909–44959. Google Scholar
• Shi L, Gai J, Mazumdar E, Chi Y, Wierman A (2025) Breaking the curse of multiagency in robust multi-agent reinforcement learning. Singh A, Fazel M, Hsu D, Lacoste-Julien S, Berkenkamp F, Maharaj T, Wagstaff K, Zhu J, eds. Proc. 42nd Internat. Conf. Machine Learn., Proceedings of Machine Learning Research, vol. 267 (PMLR, New York), 54904–54918.Google Scholar
• Shi L, Li G, Wei Y, Chen Y, Chi Y (2022) Pessimistic Q-learning for offline reinforcement learning: Towards optimal sample complexity. Chaudhuri K, Jegelka S, Song L, Szepesvári C, Niu G, Sabato S, eds. Proc. 39th Internat. Conf. Machine Learn., vol. 162 (PMLR, New York), 19967–20025. Google Scholar
• Sidford A, Wang M, Wu X, Yang L, Ye Y (2018) Near-optimal time and sample complexities for solving Markov decision processes with a generative model. Bengio S, Wallach H, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R, eds. Adv. Neural Inform. Processing Systems (Curran Associates, Inc., Red Hook, NY), 5186–5196.Google Scholar
• Smirnova E, Dohmatob E, Mary J (2019) Distributionally robust reinforcement learning. Preprint, submitted February 23, https://arxiv.org/abs/1902.08708.Google Scholar
• Sun K, Liu Y, Zhao Y, Yao H, Jui S, Kong L (2021) Exploring the training robustness of distributional reinforcement learning against noisy state observations. Preprint, submitted September 17, https://arxiv.org/abs/2109.08776.Google Scholar
• Tamar A, Mannor S, Xu H (2014) Scaling up robust MDPs using function approximation. Proc. Internat. Conf. Machine Learn. (PMLR, New York), 181–189.Google Scholar
• Tan KL, Esfandiari Y, Lee XY, Sarkar S (2020) Robustifying reinforcement learning agents via action space adversarial training. Proc. Amer. Control Conf. (IEEE, Piscataway, NJ), 3959–3964.Google Scholar
• Tessler C, Efroni Y, Mannor S (2019) Action robust reinforcement learning and applications in continuous control. Proc. Internat. Conf. Machine Learn. (PMLR, New York), 6215–6224.Google Scholar
• Tsybakov AB (2009) Introduction to Nonparametric Estimation, vol. 11 (Springer, Berlin).Crossref, Google Scholar
• Uehara M, Shi C, Kallus N (2022) A review of off-policy evaluation in reinforcement learning. Preprint, submitted December 13, https://arxiv.org/abs/2212.06355.Google Scholar
• Wainwright MJ (2019) Stochastic approximation with cone-contractive operators: Sharp ℓ∞-bounds for Q-learning. Preprint, submitted May 15, https://arxiv.org/abs/1905.06265.Google Scholar
• Wang Y, Zou S (2021) Online robust reinforcement learning with model uncertainty. Adv. in Neural Inform. Processing Systems 34:7193–7206.Google Scholar
• Wang H, Shi L, Chi Y (2024a) Sample complexity of offline distributionally robust linear Markov decision processes. Reinforcement Learn. J. 3:1467–1510.Google Scholar
• Wang S, Si N, Blanchet J, Zhou Z (2023a) A finite sample complexity bound for distributionally robust Q-learning. Proc. Internat. Conf. Artificial Intelligence Statist. (PMLR, New York), 3370–3398.Google Scholar
• Wang S, Si N, Blanchet J, Zhou Z (2023b) On the foundation of distributionally robust reinforcement learning. Preprint, submitted November 15, https://arxiv.org/abs/2311.09018.Google Scholar
• Wang S, Si N, Blanchet J, Zhou Z (2024b) Sample complexity of variance-reduced distributionally robust Q-learning. J. Machine Learn. Res. 25(341):1–77.Google Scholar
• Wang K, Gadot U, Kumar N, Levy K, Mannor S (2023c) Robust reinforcement learning via adversarial kernel approximation. Preprint, submitted June 9, https://arxiv.org/abs/2306.05859.Google Scholar
• Wiesemann W, Kuhn D, Rustem B (2013) Robust Markov decision processes. Math. Oper. Res. 38(1):153–183.Link, Google Scholar
• Wolff EM, Topcu U, Murray RM (2012) Robust control of uncertain Markov decision processes with temporal logic specifications. Proc. 51st IEEE Conf. Decision Control (IEEE, Piscataway, NJ), 3372–3379.Google Scholar
• Woo J, Joshi G, Chi Y (2025) The blessing of heterogeneity in federated Q-learning: Linear speedup and beyond. J. Machine Learn. Res. 26(26):1–85.Google Scholar
• Woo J, Shi L, Joshi G, Chi Y (2024) Federated offline reinforcement learning: Collaborative single-policy coverage suffices. Proc. 41st Internat. Conf. Machine Learn., Proceedings of Machine Learning Research, vol. 235 (PMLR, New York), 53165–53201. Google Scholar
• Xie T, Jiang N, Wang H, Xiong C, Bai Y (2021) Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Adv. Neural Inform. Processing Systems 34:27395–27407.Google Scholar
• Xiong Z, Eappen J, Zhu H, Jagannathan S (2022) Defending observation attacks in deep reinforcement learning via detection and denoising. Preprint, submitted June 14, https://arxiv.org/abs/2206.07188.Google Scholar
• Xu H, Mannor S (2012) Distributionally robust Markov decision processes. Math. Oper. Res. 37(2):288–300.Link, Google Scholar
• Xu Z, Panaganti K, Kalathil D (2023) Improved sample complexity bounds for distributionally robust reinforcement learning. Ruiz F, Dy J, van de Meent JW, eds. Proc. 26th Internat. Conf. Artificial Intelligence Statist., Proceedings of Machine Learning Research, vol. 206 (PMLR, New York), 9728–9754.Google Scholar
• Yan Y, Li G, Chen Y, Fan J (2023) The efficacy of pessimism in asynchronous Q-learning. IEEE Trans. Inform. Theory 69(11):7185–7219.Crossref, Google Scholar
• Yang K, Yang L, Du S (2021) Q-learning with logarithmic regret. Banerjee A, Fukumizu K, eds. Proc. 24th Internat. Conf. Artificial Intelligence Statist., Proceedings of Machine Learning Research, vol. 130 (PMLR, New York), 1576–1584.Google Scholar
• Yang W, Zhang L, Zhang Z (2022) Toward theoretical understandings of robust Markov decision processes: Sample complexity and asymptotics. Ann. Statist. 50(6):3223–3248.Crossref, Google Scholar
• Yang W, Wang H, Kozuno T, Jordan SM, Zhang Z (2023) Avoiding model estimation in robust Markov decision processes with a generative model. Preprint, submitted February 2, https://arxiv.org/abs/2302.01248.Google Scholar
• Yin M, Bai Y, Wang Y-X (2021) Near-optimal offline reinforcement learning via double variance reduction. Preprint, submitted February 2, https://arxiv.org/abs/2102.01748.Google Scholar
• Zhang R, Hu Y, Li N (2023) Regularized robust MDPs and risk-sensitive MDPs: Equivalence, policy gradient, and sample complexity. Preprint, submitted June 20, https://arxiv.org/abs/2306.11626Google Scholar
• Zhang Z, Zhou Y, Ji X (2020a) Almost optimal model-free reinforcement learning via reference-advantage decomposition. Adv. Neural Inform. Processing Systems, 33:15198–15207.Google Scholar
• Zhang H, Chen H, Boning D, Hsieh C-J (2021) Robust reinforcement learning on state observations with learned optimal adversary. Preprint, submitted January 21, https://arxiv.org/abs/2101.08452.Google Scholar
• Zhang H, Chen H, Xiao C, Li B, Liu M, Boning D, Hsieh C-J (2020b) Robust deep reinforcement learning against adversarial perturbations on state observations. Adv. Neural Inform. Processing Systems 33:21024–21037.Google Scholar
• Zhao H, Yu Y, Xu K (2021) Learning efficient online 3D bin packing on packing configuration trees. Internat. Conf. Learn. Representations ICLR 2022 (OpenReview.net).Google Scholar
• Zhou Z, Bai Q, Zhou Z, Qiu L, Blanchet J, Glynn P (2021) Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. Proc. 24th Internat. Conf. Artificial Intelligence Statist., Proceedings of Machine Learning Research, vol. 130 (PMLR, New York), 3331–3339.Google Scholar
• Ziegler DM, Stiennon N, Wu J, Brown TB, Radford A, Amodei D, Christiano P, Irving G (2019) Fine-tuning language models from human preferences. Preprint, submitted September 18, https://arxiv.org/abs/1909.08593.Google Scholar