Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Information

Motivated by the human-machine interaction such as recommending videos for improving customer engagement, we study human-guided human-machine interaction for decision making with private information. We model this interaction as a two-player turn-based game, where one player (Bob, a human) guides the other player (Alice, a machine) toward a common goal. Specifically, we focus on offline reinforcement learning (RL) in this game, where the goal is to find a policy pair for Alice and Bob that maximizes their expected total rewards based on an offline data set collected a priori. The offline setting presents two challenges: (i) We cannot collect Bob’s private information, leading to a confounding bias when using standard RL methods, and (ii) there is a distributional mismatch between the behavior policy used to collect data and the desired optimal policy we aim to learn. To tackle the confounding bias, we treat Bob’s previous action as an instrumental variable for Alice’s current decision making to adjust for the unmeasured confounding. We establish a novel identification result and propose a new off-policy evaluation (OPE) method for evaluating policy pairs in this two-player turn-based game. To tackle the distributional mismatch, we leverage the idea of pessimism and use our OPE method to develop an off-policy policy learning algorithm for finding a desirable policy pair for both Alice and Bob. Moreover, we prove that under some technical assumptions, the policy pair obtained through our method converges to the optimal one at a satisfactory rate. Finally, we conduct a simulation study to demonstrate the performance of the proposed method.

This paper was accepted by Nicolas Stier, Special Issue on the Human-Algorithm Connection.

Funding: L. Wang’s research is partially supported by the National Science Foundation [Grant FRGMS-1952373].

Supplemental Material: The online appendices and data files are available at https://doi.org/10.1287/mnsc.2022.04112.