Optimal Control of a Continuous-Time Markov Chain with Periodic Transition Probabilities

Controlled stationary Markov chains with a finite number of states and possible actions and continuous time have been studied by many authors. It is well known that a control maximizing the expected reward in a finite time interval can be obtained from the equation of dynamic programming, and that there exists a stationary control giving maximal (discounted or average) reward in an infinite time interval. In this paper controlled Markov chains whose transition probabilities are periodic in time are considered. It is shown that there exists a periodic control giving maximal expected future reward, and that it can be approximatively calculated by making a discretization in time. An example of a system that can be described by such a process is a service system or an inventory whose input is, e.g., a Poisson process with an intensity that varies seasonally.