Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.

Endnotes

¹ We use the DeepMind Control Suite implementation (Tassa et al. [92], Zakka et al. [102]). The image of the Cheetah has been reproduced with permission from Tassa et al. [92].

² In the case of adaptive control, they will form the “reference model” that must be adapted based on feedback (see Section 3.1).

³ Note that the definition of $d_{γ}^{π} (s)$ is slightly different between the episodic and continuing settings in RL. In the continuing setting, $d_{γ}^{π} (s)$ corresponds to the stationary distribution for the policy $π$ , with no dependence on $γ$ . Otherwise, the rest of the gradient remains the same. Because of this similarity, algorithms designed for the episodic setting can often be applied heuristically to the continuing setting, albeit they should account for differences in state weighting. To clarify terminology, the episodic setting corresponds to batch control problems and the continuing setting corresponds to the continuous setting in control. This terminology should not be confused with the fact that in RL the continuous setting usually means the state and/or action are continuous rather than discrete.

References

[1] A. D. Ames, X. Xu, J. W. Grizzle, and P. Tabuada. Control barrier function based quadratic programs for safety critical systems. IEEE Transactions on Automatic Control 62(8):3861–3876, 2017.Crossref, Google Scholar
[2] F. Aminmansour, T. Jafferjee, E. Imani, E. J. Talvitie, M. Bowling, and M. White. Mitigating value hallucination in Dyna-style planning via multistep predecessor models. Journal of Artificial Intelligence Research 80:441–473, 2024.Crossref, Google Scholar
[3] B. D. O. Anderson and J. B. Moore. Linear Optimal Control. Prentice Hall, Upper Saddle River, NJ, 1971.Crossref, Google Scholar
[4] B. D. Anderson, R. R. Bitmead, C. R. Johnson Jr, P. V. Kokotovic, R. L. Kosut, I. M. Mareels, L. Praly, et al. Stability of Adaptive Systems: Passivity and Averaging Analysis. MIT Press, Cambridge, MA, 1986.Google Scholar
[5] A. M. Annaswamy. Adaptive control and intersections with reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems 6:65–93, 2023.Crossref, Google Scholar
[6] A. M. Annaswamy and A. L. Fradkov. A historical perspective of adaptive control and learning. Annual Reviews in Control 52:18–41, 2021.Crossref, Google Scholar
[7] K. Åström. History of adaptive control. Encyclopedia of Systems and Control. Springer, Cham, Switzerland, 902–909, 2021.Crossref, Google Scholar
[8] K. J. Åström and B. Wittenmark. Adaptive Control, 2nd edition. Dover Publications, Mineola, NY, 2013.Google Scholar
[9] P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multi-armed bandit problem. Machine Learning 47:235–256, 2002.Crossref, Google Scholar
[10] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems, volume 29. Curran Associates Inc., Red Hook, NY, 2016.Google Scholar
[11] M. G. Bellemare, S. Candido, P. S. Castro, J. Gong, M. C. Machado, S. Moitra, S. S. Ponda, et al. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature 588(7836):77–82, 2020.Crossref, Google Scholar
[12] R. Bellman. Dynamic Programming. Princeton University Press, Princeton, NJ, 1957.Google Scholar
[13] D. P. Bertsekas. Dynamic Programming and Optimal Control, 4th edition. Athena Scientific, Belmont, MA, 2017.Google Scholar
[14] D. P. Bertsekas and J. N. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, Belmont, MA, 1996.Google Scholar
[15] S. Boyd, L. El Ghaoui, E. Feron, and V. Balakrishnan. Linear Matrix Inequalities in System and Control Theory. SIAM, Philadelphia, 1994.Crossref, Google Scholar
[16] W. I. Caldwell. Control system with automatic response adjustment. US Patent 2517081, 1950.Google Scholar
[17] A. Chan, H. Silva, S. Lim, T. Kozuno, A. R. Mahmood, and M. White. Greedification operators for policy optimization: Investigating forward and reverse KL divergences. Journal of Machine Learning Research 23(253):1–79, 2022.Google Scholar
[18] X. Chen and L. Zhao. On the convergence of continuous single-timescale actor-critic. Proceedings of the International Conference on Machine Learning. 2025.Google Scholar
[19] L. Da, J. Turnau, T. P. Kutralingam, A. Velasquez, P. Shakarian, and H. Wei. A survey of sim-to-real methods in RL: Progress, prospects and challenges with foundation models. Preprint, submitted February 18, https://arxiv.org/abs/2502.13187, 2025.Google Scholar
[20] S. Dean, H. Mania, N. Matni, B. Recht, and S. Tu. Sample complexity of the linear quadratic regulator. Foundations of Computational Mathematics 20:633–679, 2020.Crossref, Google Scholar
[21] J. Degrave, F. Felici, J. Buchli, M. Neunert, B. Tracey, F. Carpanese, T. Ewalds, et al. Magnetic control of Tokamak plasmas through deep reinforcement learning. Nature 602(7897):414–419, 2022.Crossref, Google Scholar
[22] F. Dörfler. Data-driven control: Part two of two: Hot take: Why not go with models? IEEE Control Systems Magazine 43(6):27–31, 2023.Crossref, Google Scholar
[23] J. C. Doyle. Guaranteed Margins for LQG regulators. IEEE Transactions on Automatic Control 23(4):756–757, 1978.Crossref, Google Scholar
[24] G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, T. Hester. Challenges of real-world reinforcement learning: Definitions, benchmarks and analysis. Machine Learning 110(9):2419–2468, 2021.Crossref, Google Scholar
[25] O. Eberhard, M. Muehlebach, and C. Vernade. Partially observable reinforcement learning with memory traces. Proceedings of the International Conference on Machine Learning, volume 267. 14934–14949, 2025.Google Scholar
[26] O. Eberhard, C. Vernade, and M. Muehlebach. Commit to the bit: Reactive reinforcement learning done right. Proceedings of the International Conference on Machine Learning, volume 306. 2026.Google Scholar
[27] O. Eberhard, J. Hollenstein, C. Pinneri, and G. Martius. Pink noise is all you need: Colored noise exploration in deep reinforcement learning. Proceedings of the International Conference on Learning Representations, 2023.Google Scholar
[28] E. Elelimy, A. White, M. Bowling, and M. White. Real-time recurrent learning using trace units in reinforcement learning. Advances in Neural Information Processing Systems, volume 37. Curran Associates Inc., Red Hook, NY, 17006–17043, 2024.Google Scholar
[29] G. Elmkaiel, S. Schmitt, and M. Muehlebach. Embodied intelligence for sustainable flight: A soaring robot with active morphological control. NPJ Robotics, 2026.Crossref, Google Scholar
[30] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. Proceedings of the International Conference on Machine Learning, 1587–1596, 2018.Google Scholar
[31] D. Gabor, W. Wilby, and R. Woodcock. A universal non-linear filter, predictor and simulator which optimizes itself by a learning process. Proceedings of the IEE-Part B: Electronic and Communication Engineering 108(40):422–435, 1961.Crossref, Google Scholar
[32] M. Gaur, A. S. Bedi, D. Wang, and V. Aggarwal. Closing the gap: Achieving global convergence (last iterate) of actor-critic under Markovian sampling with neural network parametrization. Preprint, submitted May 3, https://arxiv.org/abs/2405.01843, 2024.Google Scholar
[33] N. Golowich, A. Moitra, and D. Rohatgi. Planning and learning in partially observable systems via filter stability. Proceedings of the ACM Symposium on Theory of Computing. Association for Computing Machinery, New York, 349–362, 2023.Google Scholar
[34] G. Goodwin and K. Sin. Adaptive Filtering Prediction and Control. Prentice-Hall, Upper Saddle River, NJ, 1984.Google Scholar
[35] E. Graves, E. Imani, R. Kumaraswamy, and M. White. Off-policy actor-critic with emphatic weightings. Journal of Machine Learning Research 24(146):1–63, 2023.Google Scholar
[36] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proceedings of the International Conference on Machine Learning, 1861–1870, 2018.Google Scholar
[37] D. Hafner, W. Yan, and T. Lillicrap. Training agents inside of scalable world models. Preprint, submitted September 29, https://arxiv.org/abs/2509.24527, 2025.Google Scholar
[38] N. Hansen, H. Su, and X. Wang. TD-MPC2: Scalable, robust world models for continuous control. Preprint, submitted October 23, https://arxiv.org/abs/2310.16828, 2023.Google Scholar
[39] Z. He, S. Bolognani, F. Dörfler, and M. Muehlebach. Decision-dependent stochastic optimization: The role of distribution dynamics. Preprint, submitted March 10, https://arxiv.org/abs/2503.07324, 2025.Google Scholar
[40] L. Hewing, J. Kabzan, and M. N. Zeilinger. Cautious model predictive control using Gaussian process regression. IEEE Transactions on Control Systems Technology 28(6):2736–2743, 2020.Crossref, Google Scholar
[41] K. C. Hsu, H. Hu, and J. F. Fisac. The safety filter: A unified view of safety-critical control in autonomous systems. Annual Review of Control, Robotics, and Autonomous Systems 7, 2023.Google Scholar
[42] P. A. Ioannou and J. Sun. Robust Adaptive Control. Dover Publications, 2012.Google Scholar
[43] T. Jaksch, R. Ortner, and P. Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research 11(51):1563–1600, 2010.Google Scholar
[44] M. K. Janjua, H. Shah, M. White, E. Miahi, M. C. Machado, and A. White. GVFs in the real world: Making predictions online for water treatment. Machine Learning 113(8):5151–5181, 2024.Crossref, Google Scholar
[45] S. Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney. Recurrent experience replay in distributed reinforcement learning. Proceedings of the International Conference on Learning Representations, 2018.Google Scholar
[46] I. Karafyllis and M. Krstic. Robust Adaptive Control: Deadzone-Adapted Disturbance Suppression. SIAM, Philadelphia, 2025.Crossref, Google Scholar
[47] P. Karmakar and S. Bhatnagar. Two time-scale stochastic approximation with controlled Markov noise and off-policy temporal-difference learning. Mathematics of Operations Research 43(1):130–151, 2018.Link, Google Scholar
[48] H. K. Khalil. Nonlinear Systems, 3rd edition. Prentice Hall, Upper Saddle River, NJ, 2002.Google Scholar
[49] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. Preprint, submitted December 22, https://arxiv.org/abs/1412.6980, 2014.Google Scholar
[50] L. Kocsis and C. Szepesvári. Bandit based Monte-Carlo planning. Proceedings of the European Conference on Machine Learning, 282–293, 2006.Google Scholar
[51] A. Krishnamurthy, A. Agarwal, and J. Langford. PAC reinforcement learning with rich observations. Advances in Neural Information Processing Systems, volume 29. Curran Associates Inc., Red Hook, NY, 2016.Google Scholar
[52] M. Krstić, I. Kanellakopoulos, and P. V. Kokotović. Nonlinear and Adaptive Control Design. Wiley, New York, 1995.Google Scholar
[53] P. R. Kumar and A. Becker. A new family of optimal adaptive controllers for Markov chains. IEEE Transactions on Automatic Control 27:137–146, 1982.Crossref, Google Scholar
[54] T. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics 6(1):4–22, 1985.Crossref, Google Scholar
[55] T. Lattimore and C. Szepesvári. Sample-based planning and learning with function approximation. Statistical Sciences 40(4):517–545, 2025.Crossref, Google Scholar
[56] M. Lauri, D. Hsu, and J. Pajarinen. Partially observable Markov decision processes in robotics: A survey. IEEE Transactions on Robotics 39(1):21–40, 2022.Crossref, Google Scholar
[57] N. P. Lawrence, T. Banker, and A. Mesbah. MPCritic: A plug-and-play MPC architecture for reinforcement learning. Proceedings of the IEEE Conference on Decision and Control, 1048–1054, 2025.Google Scholar
[58] F. L. Lewis and D. Vrabie. Reinforcement learning and adaptive dynamic programming for feedback control. IEEE Circuits and Systems Magazine 9(3):32–50, 2009.Crossref, Google Scholar
[59] M. L. Littman. Memoryless policies: Theoretical limitations and practical results. Proceedings of the International Conference on Simulation of Adaptive Behavior: From Animals to Animats, volume 3. MIT Press, Cambridge, MA, 238–245, 1994.Google Scholar
[60] Q. Liu, A. Chung, C. Szepesvari, and C. Jin. When is partially observable reinforcement learning not scary? Proceedings of the Conference on Learning Theory, volume 178. PMLR, New York, 5175–5220, 2022.Google Scholar
[61] T. Liu, Y. Song, L. Zhu, and D. J. Hill. Stability and control of power grids. Annual Review of Control, Robotics, and Autonomous Systems 5(1):689–716, 2022.Crossref, Google Scholar
[62] H. Ma, D. Büchler, B. Schölkopf, and M. Muehlebach. Reinforcement learning with model-based feedforward inputs for robotic table tennis. Autonomous Robots 47(8):1387–1403, 2023.Crossref, Google Scholar
[63] G. Martius, R. Der, and N. Ay. Information driven self-organization of complex robotic behaviors. PLoS One 8(5):e63400, 2013.Crossref, Google Scholar
[64] E. Mishkin and L. Braun. Adaptive Control Systems. McGraw-Hill Publishing, New York, 1961.Google Scholar
[65] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, et al. Human-level control through deep reinforcement learning. Nature 518(7540):529–533, 2015.Crossref, Google Scholar
[66] M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of Machine Learning, 2nd edition. MIT Press, Cambridge, MA, 2018.Crossref, Google Scholar
[67] M. Muehlebach, Z. He, and M. I. Jordan. The sample complexity of online reinforcement learning: A multi-model perspective. Proceedings of the International Conference on Learning Representations, 2026.Google Scholar
[68] K. S. Narendra and A. M. Annaswamy. Stable Adaptive Systems, Prentice Hall, Upper Saddle River, NJ, 1989.Google Scholar
[69] I. Osband and B. Van Roy. Why is posterior sampling better than optimism for reinforcement learning? Proceedings of the International Conference on Machine Learning, 2701–2710, 2017.Google Scholar
[70] I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped DQN. Advances in Neural Information Processing Systems, volume 29. Curran Associates Inc., Red Hook, NY, 2016.Google Scholar
[71] Y. Pan, M. Zaheer, A. White, A. Patterson, and M. White. Organizing experience: A deeper look at replay mechanisms for sample-based planning in continuous state domains. Proceedings of the International Joint Conference on Artificial Intelligence. 2018.Google Scholar
[72] C. H. Papadimitriou and J. N. Tsitsiklis. The complexity of Markov decision processes. Mathematics of Operations Research 12(3):441–450, 1987.Link, Google Scholar
[73] A. Patterson, A. White, and M. White. A generalized projected Bellman error for off-policy value estimation in reinforcement learning. Journal of Machine Learning Research 23(145):1–61, 2022.Google Scholar
[74] A. Patterson, S. Neumann, M. White, and A. White. Empirical design in reinforcement learning. Journal of Machine Learning Research 25(318):1–63, 2024.Google Scholar
[75] C. Piazza, G. Grioli, M. G. Catalano, and A. Bicchi. A century of robotic hands. Annual Review of Control, Robotics, and Autonomous Systems 2(1):1–32, 2019.Crossref, Google Scholar
[76] W. B. Powell. Approximate Dynamic Programming: Solving the Curses of Dimensionality. John Wiley & Sons, New York, 2007.Crossref, Google Scholar
[77] F. Rosenblatt. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review 65(6):386, 1958.Crossref, Google Scholar
[78] S. Sastry and M. Bodson. Adaptive Control: Stability, Convergence and Robustness. Prentice Hall, Upper Saddle River, NJ, 1989.Google Scholar
[79] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. Preprint, submitted July 20, https://arxiv.org/abs/1707.06347, 2017.Google Scholar
[80] D. Silver. Mastering chess and Shogi by self-play with a general reinforcement learning algorithm. Preprint, submitted December 5, https://arxiv.org/abs/1712.01815, 2017.Google Scholar
[81] D. Silver. Mastering the game of Go with deep neural networks and tree search. Nature 529:484–489, 2016.Crossref, Google Scholar
[82] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. Proceedings of the International Conference on Machine Learning, 387–395, 2014.Google Scholar
[83] A. Smyshlyaev and M. Krstic. Adaptive Control of Parabolic PDEs. Princeton University Press, Princeton, NJ, 2010.Crossref, Google Scholar
[84] M. J. A. Strens. A Bayesian framework for reinforcement learning. Proceedings of the International Conference on Learning Representations. Morgan Kaufmann Publishers Inc., San Francisco, 943–950, 2000.Google Scholar
[85] B. Sukhija, M. Turchetta, D. Lindner, A. Krause, S. Trimpe, and D. Baumann. GoSafeOpt: Scalable safe exploration for global optimization of dynamical systems. Artificial Intelligence 320, 2023.Crossref, Google Scholar
[86] R. S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning 3(1):9–44, 1988.Crossref, Google Scholar
[87] R. S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. Proceedings of the AAAI Conference on Artificial Intelligence, 216–224, 1991.Google Scholar
[88] R. S. Sutton. The bitter lesson. Accessed August 4, 2026, http://www.incompleteideas.net/IncIdeas/BitterLesson.html. 2019.Google Scholar
[89] R. S. Sutton, A. G. Barto, and R. J. Williams. Reinforcement learning is direct adaptive optimal control. IEEE Control Systems Magazine 12(2):19–22, 2002.Google Scholar
[90] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, volume 12. MIT Press, Cambridge, MA, 1999.Google Scholar
[91] C. Tang, B. Abbatematteo, J. Hu, R. Chandra, R. Martín-Martín, and P. Stone. Deep reinforcement learning for robotics: A survey of real-world successes. Annual Review of Control, Robotics, and Autonomous Systems 8(1):153–188, 2025.Crossref, Google Scholar
[92] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, C. DdL, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, and T. Lillicrap. Deepmind control suite. Preprint, submitted January 2, https://arxiv.org/abs/1801.00690, 2018.Google Scholar
[93] M. Tayal, M. Tayal, A. Singh, S. Kolathaya, and R. Prakash. V-OCBF: Learning safety filters from offline data via value-guided offline control barrier functions. Transactions on Machine Learning, 2026.Google Scholar
[94] P. Thomas. Bias in natural actor-critic algorithms. Proceedings of the International Conference on Machine Learning. PMLR, New York, 441–448, 2014.Google Scholar
[95] E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 5026–5033, 2012.Google Scholar
[96] J. N. Tsitsiklis. Asynchronous stochastic approximation and q-learning. Machine Learning 16(3):185–202, 1994.Crossref, Google Scholar
[97] J. Tsitsiklis and B. Van Roy. Analysis of temporal-diffference learning with function approximation. Advances in Neural Information Processing Systems, volume 9. MIT Press, Cambridge, MA, 1996.Google Scholar
[98] H. P. Van Hasselt, M. Hessel, and J. Aslanides. When to use parametric models in reinforcement learning?. Advances in Neural Information Processing Systems, 32, 2019.Google Scholar
[99] Y. Wang, S. S. Zhan, R. Jiao, Z. Wang, W. Jin, Z. Yang, Z. Wang, et al. Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. Proceedings of the International Conference on Machine Learning. JMLR.org, 2023.Google Scholar
[100] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Advances in Neural Information Processing Systems, volume 5. 1992.Google Scholar
[101] Y. Xie. 2025. A survey of safe reinforcement learning methods in robotics. Proceedings of the ITM Web of Conferences.Google Scholar
[102] K. Zakka, B. Tabanpour, Q. Liao, M. Haiderbhai, S. Holt, J. Y. Luo, A. Allshire, et al. MuJoCo playground. Preprint, submitted February 12, https://arxiv.org/abs/2502.08844, 2025.Google Scholar
[103] S. Zhang, B. Liu, H. Yao, and S. Whiteson. Provably convergent two-timescale off-policy actor-critic with function approximation. Proceedings of the International Conference on Machine Learning. PMLR, New York, 11204–11213, 2020.Google Scholar
[104] Y. Zhao, O. Eberhard, M. Khammassi, A. H. Sayed, and M. Muehlebach. Why linear recurrent memory works in partially observable reinforcement learning. Proceedings of the International Conference on Machine Learning, volume 306. 2026.Google Scholar
[105] Y. Zhu and M. Krstic. Delay-Adaptive Linear Control. Princeton University Press, Princeton, NJ, 2020.Google Scholar

cover image INFORMS TutORials in Operations Research

Operations Research and Artificial Intelligence: Better Together

November 2026

Article Information

Supplemental Material

Metrics

Information

Published Online:November 01, 2026

Cite as

Claire Vernade, Onno Eberhard, Martha White, Florian Dörfler, Csaba Szepesvári, Miroslav Krstic, Michael Muehlebach (2026) Foundations of Reinforcement Learning and Control: Connections and New Perspectives. INFORMS TutORials in Operations Research null(null):187-212.

https://doi.org/10.1287/educ.2026.0310

PDF download