Empirical Q-Value Iteration

References

• Abounadi J, Bertsekas D, Borkar VS (2001) Learning algorithms for Markov decision processes with average cost. SIAM J. Control Optim. 40(3):681–698.Google Scholar
• Abounadi J, Bertsekas DP, Borkar V (2002) Stochastic approximation for nonexpansive maps: Application to q-learning algorithms. SIAM J. Control Optim. 41(1):1–22.Google Scholar
• Bertsekas DP (2012) Dynamic Programming and Optimal Control, vol. 2, 4th ed. (Athena Scientific, Nashua, NH).Google Scholar
• Bertsekas DP, Tsitsiklis JN (1996) Neuro-Dynamic Programming (Athena Scientific, Nashua, NH).Google Scholar
• Blackwell D, Dubins L (1962) Merging of opinions with increasing information. Ann. Math. Statist. 33(3):882–886.Google Scholar
• Borkar VS (1991) Topics in Controlled Markov Chains (Longman Scientific & Technical, Harlow, England).Google Scholar
• Borkar VS (1995) Probability Theory: An Advanced Course (Springer, New York).Google Scholar
• Borkar VS (2008) Stochastic Approximation: A Dynamical Systems Viewpoint (Cambridge University Press).Google Scholar
• Diaconis P, Freedman D (1999) Iterated random functions. SIAM Rev. 41(1):45–76.Google Scholar
• Haskell WB, Jain R, Kalathil D (2013) Empirical dynamic programming. Math. Oper. Res. 41(2):402–429.Google Scholar
• Jaakkola T, Jordan MI, Singh SP (1994) On the convergence of stochastic iterative dynamic programming algorithms. Neural Comput. 6(6):1185–1201.Google Scholar
• Kearns MJ, Singh SP (1999) Finite-sample convergence rates for q-learning and indirect algorithms. Solla SA, Leen TK, Müller KR, eds. Adv. Neural Inf. Processing Systems (MIT Press, Cambridge, MA), 996–1002.Google Scholar
• Konda VR, Borkar VS (1999) Actor-critic–type learning algorithms for Markov decision processes. SIAM J. Control Optim. 38(1):94–123.Google Scholar
• Levin DA, Peres Y, Wilmer EL (2009) Markov Chains and Mixing Times (American Mathematical Society, Providence, RI).Google Scholar
• Powell WB (2007) Approximate Dynamic Programming: Solving the Curses of Dimensionality, vol. 703 (John Wiley & Sons, New York).Google Scholar
• Propp JG, Wilson DB (1996) Exact sampling with coupled Markov chains and applications to statistical mechanics. Random Structures Algorithms 9(1-2):223–252.Google Scholar
• Puterman ML (2005) Markov Decision Processes: Discrete Stochastic Dynamic Programming (John Wiley & Sons, New York).Google Scholar
• Sutton RS, Barto AG (1998) Reinforcement Learning: An Introduction, vol. 1 (MIT Press, Cambridge, MA).Google Scholar
• Szepesvári C (2010) Algorithms for reinforcement learning. Synthesis Lectures Artificial Intelligence Machine Learn. 4(1):1–103.Google Scholar
• Tsitsiklis JN (1994) Asynchronous stochastic approximation and q-learning. Machine Learn. 16(3):185–202.Google Scholar
• Watkins CJ, Dayan P (1992) Q-learning. Machine Learn. 8(3-4):279–292.Google Scholar
• Watkins CJH (1989) Learning from delayed rewards. PhD thesis, University of Cambridge, Cambridge, UK.Google Scholar
• Yu H, Bertsekas DP (2013) On boundedness of Q-learning iterates for stochastic shortest path problems. Math. Oper. Res. 38(2):209–227.Link, Google Scholar