Generalization of White's Method of Successive Approximations to Periodic Markovian Decision Processes

The difficulty in solving a Markovian decision problem with a large number of states by Howard's policy-iteration method is that one has to solve a large system of simultaneous linear equations. Procedures developed by D. J. White and J. MacQueen, which avoid this difficulty, have been widely used to handle large-scale Markovian decision problems. This paper extends these algorithms to the case where the Markovian decision process is periodic. Proofs of the convergence of these new algorithms are sketched. Some computational experiences based on the modeling of the Great Lakes regulation systems are also given.