Asymptotics of Reinforcement Learning with Neural Networks

References

• Arulkumaran K, Deisenroth MP, Brundage M, Bharath AA (2017) A brief survey of deep reinforcement learning. Preprint, submitted August 19, https://arxiv.org/abs/1708.05866.Google Scholar
• Bertsekas DP, Tsitsiklis J (1996) Neuro-Dynamic Programming (Athena Scientific).Google Scholar
• Borkar VS (1998) Asynchronous stochastic approximations. SIAM J. Control Optim. 36(3):840–851.Google Scholar
• Cai Q, Yang Z, Lee JD, Wang Z (2019) Neural Temporal-Difference Learning Converges to Global Optima (NeurIPS).Google Scholar
• Chizat L, Bach F (2018) On the global convergence of gradient descent for over-parameterized models using optimal transport. Advances in Neural Information Processing Systems, 3040–3050.Google Scholar
• Du S, Zhai X, Poczos B, Singh A (2019a) Gradient Descent Provably Optimizes Over-Parameterized Neural Networks (ICLR).Google Scholar
• Du S, Lee J, Li H, Wang L, Zhai X (2019b) Gradient descent finds global minima of deep neural networks. Proc. 36th Internat. Conf. Machine Learn. (PMLR), 1675–1685.Google Scholar
• Dulac-Arnold G, Evans R, van Hasselt H, Sunehag P, Lillicrap T, Hunt J, Mann T, Weber T, Degris T, Coppin B (2015) Deep reinforcement learning in large discrete action spaces. Preprint, submitted December 24, https://arxiv.org/abs/1512.07679v2.Google Scholar
• Ethier S, Kurtz T (1986) Markov Processes: Characterization and Convergence (Wiley, New York).Google Scholar
• Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. Proc. 13th Internat. Conf. Artificial Intelligence Statist., 249–256.Google Scholar
• Goodfellow I, Bengio Y, Courville A (2016) Deep Learning (MIT Press, Cambridge, MA).Google Scholar
• Hausknecht M, Stone P (2015) Deep recurrent Q-learning for partially observable MDPS. AAAI 2015 Fall Sympos.Google Scholar
• Ito Y (1996) Nonlinearity creates linear independence. Adv. Comput. Math. 5:189–203.Google Scholar
• Jacot A, Gabriel F, Hongler C (2018) Neural tangent kernel: Convergence and generalization in neural networks. 32nd Conf. Neural Inform. Processing Systems.Google Scholar
• Kober J, Peters J (2012) Reinforcement learning in robotics: A survey. Siciliano B, Khatib O, eds. Reinforcement Learning (Springer, Cham, Switzerland), 579–610.Google Scholar
• Kushner HJ, Yin GG (2003) Stochastic Approximation and Recursive Algorithms and Applications (Springer-Verlag, New York).Google Scholar
• Mei S, Montanari A, Nguyen P (2018) A mean field view of the landscape of two-layer neural networks. Proc. Natl Acad. Sci. 115(33):E7665–E7671.Google Scholar
• Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing Atari with deep reinforcement learning. Preprint, submitted December 19, https://arxiv.org/abs/1312.5602.Google Scholar
• Mnih V, Badia AP, Mirza M, Graves A, Lillicrap T, Harley T, Silver D, Kavukcuoglu K (2016) Asynchronous methods for deep reinforcement learning. Internat. Conf. Machine Learn.Google Scholar
• Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, et al. (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533.Google Scholar
• Osband I, Blundell C, Pritzel A, Van Roy B (2016) Deep exploration via bootstrapped DQN. Advances in Neural Information Processing Systems, 4026–4034.Google Scholar
• Rotskoff GM, Vanden-Eijnden E (2018) Neural networks as interacting particle systems: Asymptotic convexity of the loss landscape and universal scaling of the approximation error. Preprint, submitted May 2, https://arxiv.org/abs/1805.00915.Google Scholar
• Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, Hubert T, et al. (2017) Mastering the game of Go without human knowledge. Nature 550(7676):354–359.Google Scholar
• Sirignano J, Spiliopoulos K (2020a) Mean field analysis of neural networks: A central limit theorem. Stochastic Processes Their Appl. 130(3):1820–1852.Google Scholar
• Sirignano J, Spiliopoulos K (2020b) Mean field analysis of neural networks: A law of large numbers. SIAM J. Appl. Math. 80(2):725–752.Google Scholar
• Sirignano J, Spiliopoulos K (2021) Mean field analysis of deep neural networks. Math. Oper. Res. Forthcoming.Google Scholar
• Sutton RS, Barto A (1998) Reinforcement Learning: An Introduction (MIT Press, Cambridge, MA).Google Scholar
• Sutton RS, McAllester DA, Singh SP, Mansour Y (2000) Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, vol. 12 (NeurIPS), 1057–1063.Google Scholar
• Tsitsiklis JN (1994) Asynchronous stochastic approximation and Q-learning. Machine Learn. 16:185–202.Google Scholar
• Van Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with double Q-learning. 30th AAAI Conf. Artificial Intelligence.Google Scholar
• Wang Z, Schaul T, Hessel M, Hasselt H, Lanctot M, Freitas N (2016) Dueling network architectures for deep reinforcement learning. Proc. 33rd Internat. Conf. Machine Learn., vol. 48, 1995–2003.Google Scholar
• Watkins CICH (1989) Learning from delayed rewards. Unpublished PhD thesis, University of Cambridge, UK.Google Scholar
• Watkins CICH, Dayan P (1992) Q-learning. Machine Learn. 8:279–292.Google Scholar
• Zou D, Cao Y, Zhou D, Gu Q (2018) Stochastic gradient descent optimizes over-parameterized deep ReLU networks. Preprint, submitted November 21, https://arxiv.org/abs/1811.08888.Google Scholar