An MILP-Based Solution Scheme for Factored Markov Decision Processes

References

• Adelman D, Mersereau AJ (2008) Relaxations of weakly coupled stochastic dynamic programs. Oper. Res. 56(3):712–727.Link, Google Scholar
• Angelidakis A, Chalkiadakis G (2015a) Factored MDPs for optimal prosumer decision-making. Proc. 14th Internat. Conf. Autonomous Agents Multiagent Systems (IFAAMAS, Richland, SC), 503–511.Google Scholar
• Angelidakis A, Chalkiadakis G (2015b) Factored MDPs for optimal prosumer decision-making in continuous state spaces. Rovatsos M, Vouros G, Julián V, eds. Multi-Agent Systems and Agreement Technologies (EUMAS 2015), Lecture Notes in Computer Science, vol. 9571 (Springer, Cham, Switzerland), 91–107.Google Scholar
• Bellman R (1952) On the theory of dynamic programming. Proc. Natl. Acad. Sci. USA 38(8):716–719.Crossref, Google Scholar
• Bertele U, Brioschi F (1972) Nonserial Dynamic Programming (Academic Press, New York).Google Scholar
• Bertsekas DP (1995) Dynamic Programming and Optimal Control, Volume I (Athena Scientific, Belmont, MA).Google Scholar
• Bertsekas DP (2009) Convex Optimization Theory (Athena Scientific, Belmont, MA).Google Scholar
• Bertsekas DP, Tsitsiklis JN (1996) Neuro-Dynamic Programming (Athena Scientific, Belmont, MA).Google Scholar
• Bertsimas D, Mišić VV (2016) Decomposable Markov decision processes: A fluid optimization approach. Oper. Res. 64(6):1537–1555.Link, Google Scholar
• Boutilier C, Dearden R, Goldszmidt M (1995) Exploiting structure in policy construction. Proc. Internat. Joint Conf. Artificial Intelligence (Morgan Kaufmann, San Francisco), 1104–1113.Google Scholar
• Boutilier C, Dearden R, Goldszmidt M (2000) Stochastic dynamic programming with factored representations. Artificial Intelligence 121(1–2):49–107.Crossref, Google Scholar
• Brown DB, Zhang J (2022) Dynamic programs with shared resources and signals: Dynamic fluid policies and asymptotic optimality. Oper. Res. 70(5):3015–3033.Link, Google Scholar
• Chen Y, Li L, Wang M (2018) Scalable bilinear π learning using state and action features. Proc. 35th Internat. Conf. Machine Learn., Proceedings of Machine Learning Research, vol. 80 (PMLR, New York), 834–843.Google Scholar
• de Farias DP, Van Roy B (2003) The linear programming approach to approximate dynamic programming. Oper. Res. 51(6):850–865.Link, Google Scholar
• Degris T, Sigaud O, Wuillemin P-H (2006) Learning the structure of factored Markov decision processes in reinforcement learning problems. Proc. 23rd Internat. Conf. Machine Learn. (ACM, New York), 257–264.Google Scholar
• Delgado KV, Sanner S, de Barros LN (2011) Efficient solutions to factored MDPs with imprecise transition probabilities. Artificial Intelligence 175(9–10):1498–1527.Crossref, Google Scholar
• Dolgov D, Durfee E (2006) Resource allocation among agents with preferences induced by factored MDPs. Proc. 5th Internat. Joint Conf. Autonomous Agents Multiagent Systems (ACM, New York), 297–304.Google Scholar
• Guestrin CE, Koller D, Parr R (2001a) Max-norm projections for factored MDPs. Proc. 17th Internat. Joint Conf. Artificial Intelligence (Morgan Kaufmann, San Francisco), 673–680.Google Scholar
• Guestrin CE, Koller D, Parr R (2001b) Multiagent planning with factored MDPs. Advances in Neural Information Processing Systems, vol. 14 (MIT Press, Cambridge, MA), 1523–1530.Google Scholar
• Guestrin CE, Patrascu RE, Schuurmans DE (2002a) Algorithm-directed exploration for model-based reinforcement learning in factored MDPs. Proc. 19th Internat. Conf. Machine Learn. (Morgan Kaufmann, San Francisco), 235–242.Google Scholar
• Guestrin CE, Venkataraman S, Koller D (2002b) Context specific multiagent coordination and planning with factored MDPs. Proc. 18th Natl. Conf. Artificial Intelligence (AAAI Press, Menlo Park, CA), 253–259.Google Scholar
• Guestrin CE, Koller D, Parr R, Venkataraman S (2003) Efficient solution algorithms for factored MDPs. J. Artificial Intelligence Res. 19(1):399–468.Crossref, Google Scholar
• Hawkins JT (2003) A Lagrangian decomposition approach to weakly coupled dynamic optimization problems and its applications. Unpublished PhD thesis, Massachusetts Institute of Technology, Cambridge.Google Scholar
• Hoey J, St-Aubin R, Hu A, Boutilier C (1999) SPUDD: Stochastic planning using decision diagrams. Proc. 15th Conf. Uncertainty Artificial Intelligence (Morgan Kaufmann, San Francisco), 279–288.Google Scholar
• Huang L, Chen J, Zhu Q (2017) A factored MDP approach to optimal mechanism design for resilient large-scale interdependent critical infrastructures. Proc. 2017 Workshop Model. Simulation Cyber-Physical Energy Systems (IEEE, Piscataway, NJ), 1–6.Google Scholar
• Korski J, Pfeuffer F, Klamroth K (2007) Biconvex sets and optimization with biconvex functions: A survey and extensions. Math. Methods Oper. Res. 66(3):373–407.Crossref, Google Scholar
• Osband I, Van Roy B (2014) Near-optimal reinforcement learning in factored MDPs. Advances in Neural Information Processing Systems, vol. 27 (Curran Associates, Red Hook, NY), 604–612.Google Scholar
• Powell WB (2011) Approximate Dynamic Programming: Solving the Curses of Dimensionality, 2nd ed. (John Wiley & Sons, Hoboken, NJ).Crossref, Google Scholar
• Powell WB, Ruszczyński A, Topaloglu H (2004) Learning algorithms for separable approximations of discrete stochastic optimization problems. Math. Oper. Res. 29(4):814–836.Link, Google Scholar
• Puterman ML (1994) Markov Decision Processes: Discrete Stochastic Dynamic Programming (John Wiley & Sons, Hoboken, NJ).Crossref, Google Scholar
• Reyes A, Ibargüengoytia PH, Sucar LE (2004) Power plant operator assistant: An industrial application of factored MDPs. Proc. Third Mexican Internat. Conf. Artificial Intelligence, Lecture Notes in Artificial Intelligence, vol. 2972 (Springer, Berlin), 565–573.Google Scholar
• Reyes A, Spaan MTJ, Sucar LE (2009) An intelligent assistant for power plants based on factored MDPs. Proc. 15th Internat. Conf. Intelligent System Appl. Power Systems (IEEE, Piscataway, NJ), 1–6.Google Scholar
• Sallans B, Hinton GE (2004) Reinforcement learning with factored states and actions. J. Machine Learn. Res. 5:1063–1088.Google Scholar
• Schuurmans DE, Patrascu RE (2001) Direct value-approximation for factored MDPs. Advances in Neural Information Processing Systems, vol. 14 (MIT Press, Cambridge, MA), 1579–1586.Google Scholar
• Schweitzer P, Seidmann A (1985) Generalized polynomial approximations in Markovian decision processes. J. Math. Anal. Appl. 110(2):568–582.Crossref, Google Scholar
• Simon HA (1962) The architecture of complexity. Proc. Amer. Philos. Soc. 106(6):467–482.Google Scholar
• Tavakol M, Brefeld U (2014) Factored MDPs for detecting topics of user sessions. Proc. Eighth ACM Conf. Recommender Systems (ACM, New York), 33–40.Google Scholar
• Tropp JA (2015) An introduction to matrix concentration inequalities. Foundations Trends Machine Learn. 8(1–2):1–230.Google Scholar
• Williams JD, Poupart P, Young S (2005) Factored partially observable Markov decision processes for dialogue management. Proc. Fourth Workshop Knowledge Reasoning Practical Dialog Systems (Edinburgh, UK), 76–90.Google Scholar
• Zhang N, Poole D (1999) On the role of context-specific independence in probabilistic reasoning. Proc. 16th Internat. Joint Conf. Artificial Intelligence (Morgan Kaufmann, San Francisco), 1288–1293.Google Scholar