Risk-Sensitive Optimal Control for Markov Decision Processes with Monotone Cost

V. S. Borkar
V. S. Borkar
[email protected]
School of Technology and Computer Science, Tata Institute of Fundamental Research, Homi Bhabha Road, Mumbai 400005, India
Search for more papers by this author
,
S. P. Meyn
S. P. Meyn
[email protected]
Coordinated Sciences Laboratory and Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801
Search for more papers by this author

School of Technology and Computer Science, Tata Institute of Fundamental Research, Homi Bhabha Road, Mumbai 400005, India

Search for more papers by this author

S. P. Meyn

[email protected]

Coordinated Sciences Laboratory and Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801

Search for more papers by this author

Published Online:1 Feb 2002https://doi.org/10.1287/moor.27.1.192.334

Abstract

The existence of an optimal feedback law is established for the risk-sensitive optimal control problem with denumerable state space. The main assumptions imposed are irreducibility and a near monotonicity condition on the one-step cost function. A solution can be found constructively using either value iteration or policy iteration under suitable conditions on initial feedback law.

Cited by
- Team variance optimization of n-player stochastic games with separately controlled chains
  5 February 2026 | Discrete Event Dynamic Systems, Vol. 36, No. 1
- On the Convergence of Modified Policy Iteration in Risk-Sensitive Exponential Cost Markov Decision Processes
  Yashaswini Murthy,
  Mehrdad Moharrami,
  Rayadurgam Srikant
  27 November 2025 | Operations Research, Vol. 74, No. 3
- An Actor–Critic Algorithm With Function Approximation for Risk Sensitive Cost Markov Decision Processes
  IEEE Transactions on Automatic Control, Vol. 71, No. 1
- An optimal sequence for sub-Markov decision processes with risk sensitivity
  Systems & Control Letters, Vol. 205
- Stochastic Approximation and Reinforcement Learning: The Interface and a Little Beyond
  Statistical Science, Vol. 40, No. 4
- AoI-aware transmission control in real-time mmwave energy harvesting systems: a risk-sensitive reinforcement learning approach
  Digital Communications and Networks, Vol. 11, No. 3
- A Policy Gradient Algorithm for the Risk-Sensitive Exponential Cost MDP
  Mehrdad Moharrami; ,
  Yashaswini Murthy; ,
  Arghyadip Roy,
  R. Srikant;
  11 March 2024 | Mathematics of Operations Research, Vol. 50, No. 1
- Risk-sensitive discounted Markov decision processes with unbounded reward functions and Borel spaces
  12 March 2024 | Stochastics, Vol. 96, No. 1
- Discrete-time hybrid control with risk-sensitive discounted costs
  21 October 2024 | Discrete Event Dynamic Systems, Vol. 34, No. 4
- Risk-Sensitive Average Markov Decision Processes in General Spaces
  12 July 2024 | SIAM Journal on Control and Optimization, Vol. 62, No. 4
- Data-Driven Direct Adaptive Risk-Sensitive Control of Stochastic Systems
  11 June 2024 | Journal of Systems Science and Complexity, Vol. 37, No. 4
- State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning With Rewards
  IEEE Transactions on Automatic Control, Vol. 69, No. 7
- A Discount Vanishing Approximation for Markov Decision Processes with Risk Sensitivity
  15 April 2024 | Journal of Dynamical and Control Systems, Vol. 30, No. 2
- Markov decision processes under risk sensitivity: A discount vanishing approach
  Journal of Mathematical Analysis and Applications, Vol. 533, No. 2
- Partially observable discrete-time discounted Markov games with general utility
  Operations Research Letters, Vol. 54
- Markov decision processes with risk-sensitive criteria: an overview
  1 April 2024 | Mathematical Methods of Operations Research, Vol. 99, No. 1-2
- Characterization of the optimal average cost in Markov decision chains driven by a risk-seeking controller
  21 July 2023 | Journal of Applied Probability, Vol. 61, No. 1
- Nash equilibria in risk-sensitive Markov stopping games under communication conditions
  AIMS Mathematics, Vol. 9, No. 9
- Average criteria in denumerable semi-Markov decision chains under risk-aversion
  21 August 2023 | Discrete Event Dynamic Systems, Vol. 33, No. 3
- Contractive approximations in average Markov decision chains driven by a risk-seeking controller
  5 July 2023 | Mathematical Methods of Operations Research, Vol. 98, No. 1
- Exponential TD Learning: A Risk-Sensitive Actor-Critic Reinforcement Learning Algorithm
- Discrete-time zero-sum games for Markov chains with risk-sensitive average cost criterion
  Stochastic Processes and their Applications, Vol. 158
- Risk-Sensitive Average Optimality for Discrete-Time Markov Decision Processes
  3 February 2023 | SIAM Journal on Control and Optimization, Vol. 61, No. 1
- Liapunov and Related Systems
  1 February 2024
- Ergodic risk-sensitive control—A survey
  Annual Reviews in Control, Vol. 55
- On the global convergence of relative value iteration for infinite-horizon risk-sensitive control of diffusions
  Systems & Control Letters, Vol. 171
- Risk-sensitive control for a class of diffusions with jumps
  The Annals of Applied Probability, Vol. 32, No. 6
- Risk-averse policy optimization via risk-neutral policy optimization
  Artificial Intelligence, Vol. 311
- Risk-Sensitive Reinforcement Learning via Policy Gradient Search
  15 June 2022 | Foundations and Trends® in Machine Learning, Vol. 15, No. 5
- Risk-sensitive semi-Markov decision problems with discounted cost and general utilities
  Statistics & Probability Letters, Vol. 184
- Zero-sum semi-Markov games with a probability criterion
  22 July 2021 | Stochastics, Vol. 94, No. 3
- Ergodic risk-sensitive control of Markov processes on countable state space revisited
  26 May 2022 | ESAIM: Control, Optimisation and Calculus of Variations, Vol. 28
- Intermittent Communications in Decentralized Shadow Reward Actor-Critic
- Nonzero-sum Risk-Sensitive Average Stochastic Games: The Case of Unbounded Costs
  8 March 2021 | Dynamic Games and Applications, Vol. 11, No. 4
- Risk-sensitive zero-sum stochastic differential game for jump–diffusions
  Systems & Control Letters, Vol. 157
- Convergence of Value Functions for Finite Horizon Markov Decision Processes with Constraints
  4 August 2020 | Applied Mathematics & Optimization, Vol. 84, No. 2
- Risk-sensitive control of branching processes
  25 September 2019 | IISE Transactions, Vol. 53, No. 8
- On the policy improvement algorithm for ergodic risk-sensitive control
  2 September 2020 | Proceedings of the Royal Society of Edinburgh: Section A Mathematics, Vol. 151, No. 4
- On tight bounds for function approximation error in risk-sensitive reinforcement learning
  Systems & Control Letters, Vol. 150
- Ecosystem based multi-species management using Empirical Dynamic Programming
  Ecological Modelling, Vol. 441
- Safe option-critic: learning safety in the option-critic architecture
  7 April 2021 | The Knowledge Engineering Review, Vol. 36
- A Discounted Approach in Communicating Average Markov Decision Chains Under Risk-Aversion
  7 October 2020 | Journal of Optimization Theory and Applications, Vol. 187, No. 2
- Discounted approximations in risk-sensitive average Markov cost chains with finite state space
  5 December 2019 | Mathematical Methods of Operations Research, Vol. 91, No. 2
- Safe Reinforcement Learning in Continuous State Spaces
- Exit time risk-sensitive control for systems of cooperative agents
  1 August 2019 | Mathematics of Control, Signals, and Systems, Vol. 31, No. 3
- Multi-objective safe reinforcement learning: the relationship between multi-objective reinforcement learning and safe reinforcement learning
  8 February 2019 | Artificial Life and Robotics, Vol. 24, No. 3
- The Vanishing Discount Approach in a class of Zero-Sum Finite Games with Risk-Sensitive Average Criterion
  17 January 2019 | SIAM Journal on Control and Optimization, Vol. 57, No. 1
- Risk-Sensitive Discounted Continuous-Time Markov Decision Processes with Unbounded Rates
  26 November 2019 | SIAM Journal on Control and Optimization, Vol. 57, No. 6
- Characterization of the Optimal Risk-Sensitive Average Cost in Denumerable Markov Decision Chains
  Rolando Cavazos-Cadena
  16 March 2018 | Mathematics of Operations Research, Vol. 43, No. 3
- Nonzero-Sum Risk-Sensitive Stochastic Games on a Countable State Space
  Arnab Basu,
  Mrinal K. Ghosh
  31 October 2017 | Mathematics of Operations Research, Vol. 43, No. 2
- Infinite horizon risk-sensitive control of diffusions without any blanket stability assumptions
  Stochastic Processes and their Applications, Vol. 128, No. 5
- Vanishing discount approximations in controlled Markov chains with risk-sensitive average criterion
  20 March 2018 | Advances in Applied Probability, Vol. 50, No. 01
- Linear and dynamic programming approaches to degenerate risk-sensitive reward processes
- Discounted approximations to the risk-sensitive average cost in finite Markov chains
  Journal of Mathematical Analysis and Applications, Vol. 450, No. 2
- Zero-sum risk-sensitive stochastic games
  Stochastic Processes and their Applications, Vol. 127, No. 2
- Discrete Time Homogeneous Markov Processes for the Study of the Basic Risk Processes
  18 July 2014 | Methodology and Computing in Applied Probability, Vol. 17, No. 4
- Analyzing operational risk-reward trade-offs for start-ups
  European Journal of Operational Research, Vol. 247, No. 2
- Finite horizon risk sensitive MDP and linear programming
- Risk-Sensitive Ergodic Control of Continuous Time Markov Processes With Denumerable State Space
  5 August 2015 | Stochastic Analysis and Applications, Vol. 33, No. 5
- A unified framework for risk-sensitive Markov control processes
- Asynchronous Gossip for Averaging and Spectral Ranking
  IEEE Journal of Selected Topics in Signal Processing, Vol. 8, No. 4
- Risk-sensitive control of continuous time Markov chains
  24 February 2014 | Stochastics, Vol. 86, No. 4
- A decision support system for mean–variance analysis in multi-period inventory control
  Decision Support Systems, Vol. 57
- Zero-sum risk-sensitive stochastic games on a countable state space
  Stochastic Processes and their Applications, Vol. 124, No. 1
- Risk-Sensitive Control of Pure Jump Process on Countable Space with Near Monotone Cost
  8 August 2013 | Applied Mathematics & Optimization, Vol. 68, No. 3
- Risk-sensitive control for a class of nonlinear systems with multiplicative noise
  Systems & Control Letters, Vol. 62, No. 10
- Risk-Sensitive Markov Control Processes
  SIAM Journal on Control and Optimization, Vol. 51, No. 5
- Oja’s algorithm for graph clustering, Markov spectral decomposition, and risk sensitive control
  Automatica, Vol. 48, No. 10
- Who is the fairest of them all?
- Zero-Sum Risk-Sensitive Stochastic Differential Games
  Arnab Basu,
  Mrinal K. Ghosh,
  17 May 2012 | Mathematics of Operations Research, Vol. 37, No. 3
- Target-sensitive control of Markov and semi-Markov processes
  12 October 2011 | International Journal of Control, Automation and Systems, Vol. 9, No. 5
- Discounted Approximations for Risk-Sensitive Average Criteria in Markov Decision Chains with Finite State Space
  Rolando Cavazos-Cadena,
  Daniel Hernández-Hernández,
  2 February 2011 | Mathematics of Operations Research, Vol. 36, No. 1
- Reinforcement Learning Algorithms for MDPs
  15 February 2011
- Markov Decision Processes
  8 September 2010 | Jahresbericht der Deutschen Mathematiker-Vereinigung, Vol. 112, No. 4
- Dynamic admission and service rate control of a queue
  22 September 2010 | Queueing Systems, Vol. 66, No. 2
- Manufacturing consent
- Optimality equations and inequalities in a class of risk-sensitive average cost Markov decision chains
  18 February 2009 | Mathematical Methods of Operations Research, Vol. 71, No. 1
- Algorithms for Reinforcement Learning
  11 March 2022
- Necessary and sufficient conditions for a solution to the risk-sensitive Poisson equation on a finite state space
  Systems & Control Letters, Vol. 58, No. 4
- A Learning Algorithm for Risk-Sensitive Cost
  Arnab Basu,
  Tirthankar Bhattacharyya,
  Vivek S. Borkar,
  17 October 2008 | Mathematics of Operations Research, Vol. 33, No. 4
- Average optimality for risk-sensitive control with general state space
  The Annals of Applied Probability, Vol. 17, No. 2
- Infinite Horizon Risk Sensitive Control of Discrete Time Markov Processes under Minorization Property
  SIAM Journal on Control and Optimization, Vol. 46, No. 1
- Remarks on Risk Neutral and Risk Sensitive Portfolio Optimization
- Chapter 11 Rare-Event Simulation Techniques: An Introduction and Recent Advances
- Nonstationary value iteration in controlled Markov chains with risk-sensitive average criterion
  14 July 2016 | Journal of Applied Probability, Vol. 42, No. 4
- The Value Iteration Algorithm in Risk-Sensitive Average Markov Decision Chains with Finite State Space
  Rolando Cavazos-Cadena,
  Raúl Montes-de-Oca,
  1 November 2003 | Mathematics of Operations Research, Vol. 28, No. 4

cover image Mathematics of Operations Research

Volume 27, Issue 1

February 2002

Pages 1-252

Article Information

Metrics

Information

Received:February 11, 1999
Published Online:February 01, 2002

Cite as

V. S. Borkar, S. P. Meyn, (2002) Risk-Sensitive Optimal Control for Markov Decision Processes with Monotone Cost. Mathematics of Operations Research 27(1):192-209.

https://doi.org/10.1287/moor.27.1.192.334

Keywords

PDF download

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Risk-Sensitive Optimal Control for Markov Decision Processes with Monotone Cost

Abstract

Volume 27, Issue 1

Article Information

Metrics

Information

Cite as

Keywords

Sign Up for INFORMS Publications Updates and News