On the Maximization of Long-Run Conditional Value-at-Risk for Markov Decision Processes

References

• Altman E (1999) Constrained Markov Decision Processes: Stochastic Modeling (CRC Press, Boca Raton, FL).Google Scholar
• Arapostathis A, Borkar VS, Fernández-Gaucherand E, Ghosh MK, Marcus SI (1993) Discrete-time controlled Markov processes with average cost criterion: A survey. SIAM J. Control Optim. 31(2):282–344.Crossref, Google Scholar
• Armstrong J, Brigo D (2019) Risk managing tail-risk seekers: VaR and expected shortfall vs S-shaped utility. J. Banking Finance 101:122–135.Crossref, Google Scholar
• Asensio M, Contreras J (2016) Stochastic unit commitment in isolated systems with renewable penetration under CVaR assessment. IEEE Trans. Smart Grid 7(3):1356–1367.Crossref, Google Scholar
• Barron EN (2013) Game Theory: An Introduction, vol. 2 (John Wiley & Sons, Hoboken, NJ).Crossref, Google Scholar
• Bäuerle N, Glauner A (2022) Markov decision processes with recursive risk measures. Eur. J. Oper. Res. 296(3):953–966.Crossref, Google Scholar
• Bäuerle N, Jaśkiewicz A (2024) Markov decision processes with risk-sensitive criteria: An overview. Math. Methods Oper. Res. 99(1):141–178.Crossref, Google Scholar
• Bäuerle N, Ott J (2011) Markov decision processes with average-value-at-risk criteria. Math. Methods Oper. Res. 74(3):361–379.Crossref, Google Scholar
• Bonetti M, Bisi L, Restelli M (2023) Risk-averse optimization of reward-based coherent risk measures. Artificial Intelligence 316:103845.Crossref, Google Scholar
• Boyd S, Vandenberghe L (2004) Convex Optimization (Cambridge University Press, Cambridge, UK).Crossref, Google Scholar
• Çavuş O, Ruszczyński A (2014) Risk-averse control of undiscounted transient Markov models. SIAM J. Control Optim. 52(6):3935–3966.Crossref, Google Scholar
• Chang C, Jiménez-Martín JÁ, McAleer M, Pérez-Amaral T (2011) Risk management of risk under the Basel Accord: Forecasting value-at-risk of VIX futures. Managerial Finance 37(11):1088–1106.Crossref, Google Scholar
• Chow Y, Ghavamzadeh M (2014) Algorithms for CVaR optimization in MDPs. Ghahramani Z, Welling M, Cortes C, Lawrence N, Weinberger K, eds. Adv. Neural Inform. Processing Systems, vol. 27 (Curran Associates, Inc., Red Hook, NY), 3509–3517.Google Scholar
• Chow Y, Tamar A, Mannor S, Pavone M (2015) Risk-sensitive and robust decision-making: A CVaR optimization approach. Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R, eds. Adv. Neural Inform. Processing Systems, vol. 28 (Curran Associates, Inc., Red Hook, NY), 1522–1530.Google Scholar
• Chu S, Zhang Y (2014) Markov decision processes with iterated coherent risk measures. Internat. J. Control 87(11):2286–2293.Google Scholar
• da Costa LC, Thomé FS, Garcia JD, Pereira MVF (2020) Reliability-constrained power system expansion planning: A stochastic risk-averse optimization approach. IEEE Trans. Power Systems 36(1):97–106.Crossref, Google Scholar
• Denardo EV (1970) On linear programming in a Markov decision problem. Management Sci. 16(5):281–288.Link, Google Scholar
• Dilokthanakul N, Shanahan M (2018) Deep reinforcement learning with risk-seeking exploration. Manoonpong P, Larsen JC, Xiong X, Hallam J, Triesch J, eds. Proc. 15th Internat. Conf. Simulation Adaptive Behavior, vol. 10994 (Springer, Cham, Switzerland), 201–211.Google Scholar
• Fernández E, Hinojosa Y, Puerto J, Saldanha-da Gama F (2019) New algorithmic framework for conditional value at risk: Application to stochastic fixed-charge transportation. Eur. J. Oper. Res. 277(1):215–226.Crossref, Google Scholar
• Föllmer H, Schied A (2011) Stochastic Finance: An Introduction in Discrete Time (Walter de Gruyter, Berlin).Crossref, Google Scholar
• García J, Fernández F (2015) A comprehensive survey on safe reinforcement learning. J. Machine Learn. Res. 16(42):1437–1480.Google Scholar
• Guo X, Zhang J (2019) Risk-sensitive continuous-time Markov decision processes with unbounded rates and Borel spaces. Discrete Event Dynamic Systems 29(4):445–471.Crossref, Google Scholar
• Haskell WB, Jain R (2015) A convex analytic approach to risk-aware Markov decision processes. SIAM J. Control Optim. 53(3):1569–1598.Crossref, Google Scholar
• Hernández-Lerma O, Lasserre JB (1996) Discrete-Time Markov Control Processes (Springer Science & Business Media, New York).Crossref, Google Scholar
• Hong LJ, Hu Z, Zhang L (2014) Conditional value-at-risk approximation to value-at-risk constrained programs: A remedy via Monte Carlo. INFORMS J. Comput. 26(2):385–400.Link, Google Scholar
• Howard RA, Matheson JE (1972) Risk-sensitive Markov decision processes. Management Sci. 18(7):356–369.Link, Google Scholar
• Huang Y, Guo X (2016) Minimum average value-at-risk for finite horizon semi-Markov decision processes in continuous time. SIAM J. Optim. 26(1):1–28.Crossref, Google Scholar
• Huang A, Leqi L, Lipton ZC, Azizzadenesheli K (2021) On the convergence and optimality of policy gradient for Markov coherent risk. Preprint, submitted March 4, https://arxiv.org/abs/2103.02827.Google Scholar
• Huo H, Zou X, Guo X (2017) The risk probability criterion for discounted continuous-time Markov decision processes. Discrete Event Dynamic Systems 27(4):675–699.Crossref, Google Scholar
• IESO (2022) Ontario and market demand. Accessed September 23, 2026, https://ieso.ca/Power-Data/Data-Directory.Google Scholar
• Kahneman D, Tversky A (1979) Prospect theory: An analysis of decision under risk. Econometrica 47(2):263–291.Crossref, Google Scholar
• Lim SH, Xu H, Mannor S (2013) Reinforcement learning in robust Markov decision processes. Burges CJ, Bottou L, Welling M, Ghahramani Z, Weinberger K, eds. Adv. Neural Inform. Processing Systems, vol. 26 (Curran Associates, Inc., Red Hook, NY), 701–709.Google Scholar
• Marcus SI, Fernández-Gaucherand E, Hernández-Hernandez D, Coraluppi S, Fard P (1997) Risk sensitive Markov decision processes. Byrnes CI, Datta BN, Martin CF, Gilliam DS, eds. Systems and Control in the Twenty-First Century, Progress in Systems and Control, vol. 22 (Springer, New York), 263–279.Crossref, Google Scholar
• Mihatsch O, Neuneier R (2002) Risk-sensitive reinforcement learning. Machine Learn. 49(2–3):267–290. Crossref, Google Scholar
• Miller CW, Yang I (2017) Optimal control of conditional value-at-risk in continuous time. SIAM J. Control Optim. 55(2):856–884.Crossref, Google Scholar
• Nedić A, Ozdaglar A (2009) Subgradient methods for saddle-point problems. J. Optim. Theory Appl. 142(1):205–228.Crossref, Google Scholar
• Noyan N, Rudolf G (2013) Optimization with multivariate conditional value-at-risk constraints. Oper. Res. 61(4):990–1013.Link, Google Scholar
• NREL (2019) National Wind Technology Center. Accessed September 23, 2026, https://www.nlr.gov/wind/data-tools.Google Scholar
• Ok EA (2007) Real Analysis with Economic Applications (Princeton University Press, Princeton, NJ).Crossref, Google Scholar
• Osogami T (2012) Robustness and risk-sensitivity in Markov decision processes. Pereira F, Burges CJ, Bottou L, Weinberger K, eds. Adv. Neural Inform. Processing Systems, vol. 25 (Curran Associates, Inc., Red Hook, NY), 233–241.Google Scholar
• Ott JT (2010) A Markov decision model for a surveillance application and risk-sensitive Markov decision processes. Ph.D. thesis, Karlsruhe Institute of Technology, Karlsruhe, Germany.Google Scholar
• Philpott A, de Matos V, Finardi E (2013) On solving multistage stochastic programs with coherent risk measures. Oper. Res. 61(4):957–970.Link, Google Scholar
• Prashanth L, Fu MC (2022) Risk-sensitive reinforcement learning via policy gradient search. Foundations Trends Machine Learn. 15(5):537–693.Crossref, Google Scholar
• Puterman ML (1994) Markov Decision Processes: Discrete Stochastic Dynamic Programming (John Wiley & Sons, New York).Crossref, Google Scholar
• Rockafellar RT, Uryasev S (2000) Optimization of conditional value-at-risk. J. Risk 2(3):21–42.Crossref, Google Scholar
• Rockafellar RT, Uryasev S (2002) Conditional value-at-risk for general loss distributions. J. Banking Finance 26(7):1443–1471.Crossref, Google Scholar
• Ruszczyński A (2010) Risk-averse dynamic programming for Markov decision processes. Math. Programming 125(2):235–261.Crossref, Google Scholar
• Shen Y, Stannat W, Obermayer K (2013) Risk-sensitive Markov control processes. SIAM J. Control Optim. 51(5):3652–3672.Crossref, Google Scholar
• Sobel MJ (1982) The variance of discounted Markov decision processes. J. Appl. Probab. 19(4):794–802.Crossref, Google Scholar
• Sobel MJ (1994) Mean-variance tradeoffs in an undiscounted MDP. Oper. Res. 42(1):175–183.Link, Google Scholar
• Stanko S, Macek K (2019) Risk-averse distributional reinforcement learning: A CVaR optimization approach. Merelo JJ, Garibaldi J, Linares-Barranco A, Madani K, Warwick K, eds. Proc. 11th Internat. Joint Conf. Comput. Intelligence, vol. 1 (SciTePress, Setúbal, Portugal), 412–423.Google Scholar
• Tamar A, Glassner Y, Mannor S (2015) Optimizing the CVaR via sampling. Bonet B, Koenig S, eds. Proc. 29th AAAI Conf. Artificial Intelligence, vol. 29 (AAAI Press, Palo Alto, CA), 2993–2999.Google Scholar
• Tamar A, Chow Y, Ghavamzadeh M, Mannor S (2016) Sequential decision making with coherent risk. IEEE Trans. Automatic Control 62(7):3323–3338.Crossref, Google Scholar
• Tavakoli M, Shokridehaki F, Akorede MF, Marzband M, Vechiu I, Pouresmaeil E (2018) CVaR-based energy management scheme for optimal resilience and operational cost in commercial building microgrids. Internat. J. Electr. Power Energy Systems 100:1–9.Crossref, Google Scholar
• Uğurlu K (2017) Controlled Markov decision processes with AVaR criteria for unbounded costs. J. Comput. Appl. Math. 319:24–37.Crossref, Google Scholar
• Uryasev S (2000) Conditional value-at-risk: Optimization algorithms and applications. Proc. IEEE/IAFE/INFORMS 2000 Conf. Comput. Intelligence Financial Engrg. (IEEE, Piscataway, NJ), 49–57.Crossref, Google Scholar
• Wachi A, Sui Y (2020) Safe reinforcement learning in constrained Markov decision processes. Daumé H III, Singh A, eds. Proc. 37th Internat. Conf. Machine Learn., vol. 119 (PMLR, Cambridge, MA), 9797–9806.Google Scholar
• White DJ (1993) Minimizing a threshold probability in discounted Markov decision processes. J. Math. Anal. Appl. 173(2):634–646.Crossref, Google Scholar
• Wu C, Lin Y (1999) Minimizing risk models in Markov decision processes with policies depending on target values. J. Math. Anal. Appl. 231(1):47–67.Crossref, Google Scholar
• Xia L (2016) Optimization of Markov decision processes under the variance criterion. Automatica 73:269–278.Crossref, Google Scholar
• Xia L (2020) Risk-sensitive Markov decision processes with combined metrics of mean and variance. Production Oper. Management 29(12):2808–2827.Crossref, Google Scholar
• Xia L, Zhang L, Glynn PW (2023) Risk-sensitive Markov decision processes with long-run CVaR criterion. Production Oper. Management 32(12):4049–4067.Crossref, Google Scholar
• Xie Y, Wang H, Lu H (2018) Coordination of supply chains with a retailer under the mean-CVaR criterion. IEEE Trans. Systems Man Cybernetics Systems 48(7):1039–1053.Crossref, Google Scholar
• Yu P, Haskell WB, Xu H (2018) Approximate value iteration for risk-aware Markov decision processes. IEEE Trans. Automatic Control 63(9):3135–3142.Crossref, Google Scholar