Rates of Convergence in the Central Limit Theorem for Markov Chains, with an Application to TD Learning

The author thanks Professor Siva Theja Maguluri, Georgia Tech, for many stimulating discussions and drawing his attention to several references during the course of this work; Professor Sean Meyn, University of Florida, for pointing out the bias issue with fixed step-size TD learning and why it cannot be eliminated by averaging and other discussions; Professor Mehrdad Moharrami, University of Iowa, for carefully reading an earlier version of the paper and providing many useful comments; and Weichen Wu, CMU, for pointing out a missing factor in Theorem 4 in an earlier version of the paper.