An Approximation Approach for the Deviation Matrix of Continuous-Time Markov Processes with Application to Markov Decision Theory

References

• Altman E., Feinberg E., Shwartz A. Applications of Markov decision processes in communication networks. Handbook of Markov Decision Processes (2002) (Kluwer, Boston) 489–536Crossref, Google Scholar
• Altman E., Hordijk A. Zero-sum Markov games and worst-case optimal control of queueing systems. Queueing Systems (1995) 21:415–447Crossref, Google Scholar
• Altman E., Gaujal B., Hordijk A.Discrete-Event Control of Stochastic Networks: Multimodularity and Regularity (2003) 1829(Springer, New York) Springer Lecture Notes in MathematicsCrossref, Google Scholar
• Altman E., Hordijk A., Spieksma F. Contraction conditions for average and α-discount optimality in countable state Markov games with unbounded rewards. Math. Oper. Res. (1997) 22:588–618Link, Google Scholar
• Arapostathis A., Borkar V., Fernández-Gaucherand E., Ghosh M., Marcus S. Discrete-time controlled Markov processes with average cost criterion: A survey. SIAM J. Control Optim. (1993) 31:282–344Crossref, Google Scholar
• Asmussen S.Applied Probability and Queues (1987) (Springer, New York) Google Scholar
• Bather J. Optimal stationary policies for denumerable Markov chains in continuous time. Adv. Appl. Probab. (1976) 8:144–158Crossref, Google Scholar
• Bertsekas D.Dynamic Programming and Optimal Control (2005) 3rd ed.(Athena Scientific, Nashua, NH) Google Scholar
• Brown L., Gans N., Mandelbaum A., Sakov A., Shen H., Zeltyn S., Zhao L. Statistical analysis of a telephone call center: A queueing-science perspective. J. Amer. Statist. Assoc. (2005) 469:36–50Crossref, Google Scholar
• Cao X.Stochastic Learning and Optimization: A Sensitivity-Based Approach (2007) (Springer, New York) Crossref, Google Scholar
• Chao X., Zhao Y. Analysis of multi-server queues with station and server vacations. Eur. J. Oper. Res. (1997) 110:392–406Crossref, Google Scholar
• Coolen-Schrijner P., van Doorn E. The deviation matrix of a continuous-time Markov chain. Probab. Engrg. Informational Sci. (2002) 16:351–366Crossref, Google Scholar
• Dekker R., Hordijk A. Average, sensitive and Blackwell optimal policies in denumerable Markov decision chains with unbounded rewards. Math. Oper. Res. (1988) 13:395–420Link, Google Scholar
• Dekker R., Hordijk A. Denumerable semi-Markov decision chains with small interest rates. Ann. Oper. Res. (1991) 28:185–212Crossref, Google Scholar
• Dekker R., Hordijk A. Recurrence conditions for average and Blackwell optimality in denumerable state Markov decision chains. Math. Oper. Res. (1992) 17:271–289Link, Google Scholar
• Dekker R., Hordijk A., Spieksma F. On the relation between recurrence and ergodicity properties in denumerable Markov decision chains. Math. Oper. Res. (1994) 19:539–559Link, Google Scholar
• Dynkin E., Yushkevich A.Controlled Markov Processes (1979) (Springer-Verlag, New York) Crossref, Google Scholar
• Feinberg E., Shwartz A.Handbook of Markov Decision Processes: Methods and Applications (2002) (Kluwer, Boston) Crossref, Google Scholar
• Guo X. Continuous-time Markov decision processes with discounted rewards: The case of Polish spaces. Math. Oper. Res. (2007) 32:73–87Link, Google Scholar
• Guo X., Cao X. Optimal control of ergodic continuous-time Markov chains with average sample-path rewards. SIAM J. Control Optim. (2005) 44:29–48Crossref, Google Scholar
• Guo X., Hernández-Lerma O. Continuous-time controlled Markov chains with discounted rewards. Acta Appl. Math. (2003) 79:195–216Crossref, Google Scholar
• Guo X., Liu K. A note on optimality conditions for continuous-time Markov decision processes with average cost criterion. IEEE Trans. Automatic Control (2001) 46:1984–1989Crossref, Google Scholar
• Guo X., Hernández-Lerma O., Prieto-Rumeau T. A survey of recent results on continuous-time Markov decision processes. Sociedad de Estadistica e Investigación Operativa TOP (2006) 14:177–261Google Scholar
• Heidergott B., Hordijk A., Leder N. Series expansions for continuous-time Markov processes. Oper. Res. (2009) . ePub ahead of print October 28, http://or.journal.informs.org/cgi/content/abstract/opre.1090.0738v1Google Scholar
• Heidergott B., Hordijk A., van Uitert M. Series expansions for finite-state Markov chains. Probab. Engrg. Informational Sci. (2007) 21:381–400Crossref, Google Scholar
• Hordijk A., Spieksma F. Constrained admission control to a queueing system. Adv. Appl. Probab. (1989) 21:409–431Crossref, Google Scholar
• Hordijk A., van der Duyn Schouten F. Average optimal policies in Markov decision drift processes with applications to a queueing and a replacement model. Adv. Appl. Probab. (1983) 15:274–303Crossref, Google Scholar
• Hordijk A., Yushkevich A. Blackwell optimality in the class of all policies in Markov decision chains with a Borel state and unbounded rewards. Math. Methods Oper. Res. (1999) 50:421–448Crossref, Google Scholar
• Hordijk A., Yushkevich A., Feinberg E., Shwartz A. Blackwell optimality. Handbook of Markov Decision Processes (2002) (Kluwer, Boston) 231–267Crossref, Google Scholar
• Kakumanu P. Continuous time Markov decision processes with average return criterion. J. Math. Anal. Appl. (1975) 52:173–188Crossref, Google Scholar
• Kijima M.Markov Processes for Stochastic Modeling (1997) (Chapman & Hall, London) Crossref, Google Scholar
• Koole G. The deviation matrix of the M/M/1/∞ and M/M/1/N queue, with application to controlled queueing models. Proc. 37th IEEE CDC (1998) (IEEE Press, Tampa, FL) 56–59Crossref, Google Scholar
• Neuts M.Matrix-Geometric Solutions in Stochastic Models—An Algorithmic Approach (1994) (Constable and Company, London) Google Scholar
• Olsson M. The EMpht-programme. Manual (1998) (Gothenburg University, Gothenburg, Sweden) Google Scholar
• Puterman M.Markov Decision Processes (1994) (John Wiley & Sons, New York) Crossref, Google Scholar
• Riska A., Smirni E. M/G/1-type Markov processes: A tutorial. Performance Evaluation of Complex Systems: Techniques and Tools (2002) 2459:315–325Google Scholar
• Ross S.Applied Probability Models with Optimization Applications (1970) (Holden-Day, San Francisco) Google Scholar
• Schweitzer E. Perturbation theory and finite Markov chains. J. Appl. Probab. (1968) 5:401–413Crossref, Google Scholar
• Serfozo R. An equivalence between continuous and discrete time Markov decision processes. Oper. Res. (1979) 27:616–620Link, Google Scholar
• Stidham S. Optimal control to a queueing system. IEEE Trans. Automatic Control (1985) 30:705–713Crossref, Google Scholar
• Tijms H.Stochastic Models, An Algorithmic Approach (1994) (John Wiley & Sons, New York) Google Scholar
• Yushkevich A. Controlled Markov models with countable state and continuous time. Theory Probab. Its Appl. (1977) 22:215–235Crossref, Google Scholar
• Zhang K., Xu Y., Chen X., Cao X. Policy iteration based feedback control. Automatica (2008) 44:1055–1061Crossref, Google Scholar