Scalable Reinforcement Learning for Multiagent Networked Systems

We study reinforcement learning (RL) in a setting with a network of agents whose states and actions interact in a local manner where the objective is to find localized policies such that the (discounted) global reward is maximized. A fundamental challenge in this setting is that the state-action space size scales exponentially in the number of agents, rendering the problem intractable for large networks. In this paper, we propose a scalable actor critic (SAC) framework that exploits the network structure and finds a localized policy that is an $O({\rho }^{\kappa +1})$-approximation of a stationary point of the objective for some $\rho \in (0,1)$, with complexity that scales with the local state-action space size of the largest $\kappa$-hop neighborhood of the network. We illustrate our model and approach using examples from wireless communication, epidemics, and traffic.

Funding: This work was supported by the Caltech Center for Autonomous Systems and Technologies; Office of Naval Research [Grant YIP N00014-19-1-2217]; Air Force Office of Scientific Research [Grant YIP FA9550-18-1-0150]; National Science Foundation [Grants AitF-1637598, CAREER 1553407, and CNS-1518941]; PIMCO [PIMCO Fellowship]; Amazon Web Services [Amazon AI4Science Fellowship]; and Resnick Sustainability Institute for Science, Energy and Sustainability, California Institute of Technology [Postdoctoral Fellowship].