A Tutorial on Reinforcement Learning for LLMs: RLHF and Beyond
Abstract
Large language models require post-training, which takes a foundation model and trains it to follow instructions, behave safely, and perform well across downstream use cases. Reinforcement learning (RL) post-training has become one of the most common post-training approaches, most prominently, RL from human feedback, where the reward reflects human preference judgments. In this tutorial, we formulate RL post-training as a sequential decision problem (a Markov decision process), covering both single-turn settings, where the model produces one response to a prompt, and multi-turn settings, where it interacts with a user or environment over multiple rounds. We also distinguish reward signals by how directly they can be measured or inferred (verifiable, observable, or latent). We then discuss reward overoptimization, a phenomenon in which the policy exploits errors in a learned reward model, motivating a Kullback–Leibler–regularized objective that keeps the policy close to a reference model while still optimizing reward. From this objective, we give precise step-by-step derivations of four central policy optimization algorithms (DPO, REINFORCE, PPO, and GRPO), explicitly distinguishing exact derivations from practical approximations and heuristics. The tutorial also covers deployment paradigms, with a focus on batch-online deployment, in which a policy’s interaction data are periodically collected and used for offline retraining and subsequent redeployment. For multi-turn settings, we cover an extension of GRPO to full trajectories collected from a training environment and a batch-online method that performs approximate policy iteration from logged trajectories. Finally, we discuss open research directions in RL post-training that may benefit from a broad range of research perspectives.
Your Access Options
-
Login Options
Purchase Options
Save for laterOther Options
Token AccessClaim access using a tokenRestore guest accessApplies for purchases made as a guest
