Episodic Bayesian Optimal Control with Unknown Randomness Distributions

Stochastic optimal control with unknown randomness distributions has been studied for a long time, encompassing robust control, distributionally robust control, and adaptive control. We propose a new episodic Bayesian approach that incorporates Bayesian learning with optimal control. In each episode, the approach learns the randomness distribution with a Bayesian posterior and subsequently solves the corresponding Bayesian average estimate of the true problem. The resulting policy is exercised during the episode, whereas additional data/observations of the randomness are collected to update the Bayesian posterior for the next episode. We show that the resulting episodic value functions and policies converge almost surely to their optimal counterparts of the true problem if the parametrized model of the randomness distribution is correctly specified. We also show that the asymptotic convergence rate of the episodic value functions is of the order $O(N^{-1/2})$, where N is the number of episodes, given that only one data point is collected in each episode. We develop an efficient computational method based on stochastic dual dynamic programming (SDDP) for a class of problems that have convex cost functions and linear state dynamics. Our numerical results on a classical inventory control problem verify the theoretical convergence results, and numerical comparison with two other methods demonstrates the effectiveness of the proposed Bayesian approach.

Funding: This work was supported by the Air Force Office of Scientific Research (AFOSR) [Grant FA9550-22-1-0244] to all authors. E. Zhou, Y. Lin, and Y. Wang are also grateful for the support by National Science Foundation (NSF) Division of Electrical, Communications and Cyber Systems [Grant 2419562] and the NSF AI Institute for Advances in Optimization (AI4OPT).

Supplemental Material: All supplemental materials, including the code, data, and files required to reproduce the results, are available at https://doi.org/10.1287/opre.2023.0446.