A Tutorial on Reinforcement Learning for LLMs: RLHF and Beyond

Daniel R. Jiang
Daniel R. Jiang
[email protected]
Meta Platforms & University of Pittsburgh, Pittsburgh, Pennsylvania 15217
Search for more papers by this author

Meta Platforms & University of Pittsburgh, Pittsburgh, Pennsylvania 15217

Published Online:1 Nov 2026https://doi.org/10.1287/educ.2026.0311

Acknowledgments

The author thanks four anonymous reviewers, the volume editors, and David Alderson, the series editor, for their helpful comments and in-depth reviews. The author also thanks Mina Lee for multiple rounds of careful proofreading and thoughtful comments that significantly improved this tutorial.

Endnotes

¹ Subword methods like byte pair encoding keep common words as single tokens while splitting rare words into reusable pieces, balancing vocabulary size against sequence length and reducing out-of-vocabulary failures.

² The log-probability gradient $\nabla_{θ} log π_{θ} (a ∣ s)$ plays a central role in policy gradient methods for RL (Williams [113]). Differentiable log-probabilities are what make these methods applicable to LLMs.

³ A causal mask sets $α_{i j} = 0$ for every future position $j > i$ before the softmax, so position i’s output depends only on $j \leq i$ .

⁴ This equivalence relies on each movement direction having a deterministic outcome. If movement were stochastic, the same sequence of directions could produce different paths. The sequence-level formulation would prevent the agent from adapting later moves to earlier outcomes, making it a different decision problem.

⁵ For simplicity, we treat the complete conversation history as a fully observed Markov state. In practice, the underlying environment may be partially observable. In the sales-agent setting (Example 4), the customer’s intent remains latent, whereas in a coding agent (Example 5), relevant parts of the codebase may lie outside the context window (Kausik et al. [48]). These settings are naturally modeled as a partially observed MDP (POMDP), which we leave outside the scope of this tutorial.

⁶ Preference-based feedback also has a long history in the Bayesian learning and bandits literatures (Chu and Ghahramani [18], Yue et al. [119]).

⁷ Other structural assumptions beyond Bradley–Terry also exist; for example, ranking models (e.g., Plackett–Luce) extend it to full orderings over $k \geq 3$ responses, allowing for more information per annotation.

⁸ Best-of-n sampling generates n candidate responses for a prompt and returns the one with the highest reward-model score. This is a simple way to optimize the reward without changing the policy weights.

⁹ The “KL” in (5) is the exact divergence $KL (π_{θ} (\cdot ∣ x) ‖ π_{ref} (\cdot ∣ x)) = E_{y \sim π_{θ}} [log (π_{θ} (y ∣ x) / π_{ref} (y ∣ x))]$ , which averages over all responses. Implementations approximate it from sampled responses, commonly using the log-ratio $log (π_{θ} (y ∣ x) / π_{ref} (y ∣ x))$ or Schulman’s nonnegative $k_{3}$ estimator $π_{ref} (y ∣ x) / π_{θ} (y ∣ x) - log (π_{ref} (y ∣ x) / π_{θ} (y ∣ x)) - 1$ . The KL direction, estimator, and normalization can each affect training behavior (Schulman [88], Tang and Munos [101]).

¹⁰ The identity starts by differentiating the log-probability: $\nabla_{θ} log π_{θ} (y ∣ x) = \nabla_{θ} π_{θ} (y ∣ x) / π_{θ} (y ∣ x)$ . Multiplying by $π_{θ} (y ∣ x)$ gives $\nabla_{θ} π_{θ} (y ∣ x) = π_{θ} (y ∣ x) \nabla_{θ} log π_{θ} (y ∣ x)$ . Hence, for fixed f, $\nabla_{θ} E_{y \sim π_{θ} (\cdot ∣ x)} [f (x, y)] = \sum_{y} \nabla_{θ} π_{θ} (y ∣ x) f (x, y) = \sum_{y} π_{θ} (y ∣ x) f (x, y) \nabla_{θ} log π_{θ} (y ∣ x)$ $= E_{y \sim π_{θ} (\cdot ∣ x)} [f (x, y) \nabla_{θ} log π_{θ} (y ∣ x)]$ (Williams [113]).

¹¹ Simply differentiating the sampled log-ratio does not give the full KL gradient because automatic differentiation treats the sampled response y as fixed and misses how $θ$ changes its sampling probability.

¹² Here “detach” means that the quantity is treated as fixed scalar data during automatic differentiation. Gradients are taken through $log π_{θ} (y ∣ x)$ , but not through the return multiplying it.

¹³ Indeed, $J_{KL, ϕ} (π_{θ}) - {\tilde{J}}_{KL, ϕ}^{-} (θ) = - β E_{x \sim D} [KL (π_{θ} (\cdot | x) ‖ π_{θ^{-}} (\cdot | x))]$ , whose value and gradient are zero at $θ = θ^{-}$ .

¹⁴ Any $b (x)$ independent of the response leaves the expected policy gradient unchanged, because $E_{y \sim π_{θ} (\cdot ∣ x)} [b (x) \nabla_{θ} log π_{θ} (y ∣ x)] = b (x) \nabla_{θ} \sum_{y} π_{θ} (y ∣ x) = 0$ ; see Sutton and Barto [99].

¹⁵ Section 7.2 wrote the corresponding surrogate as ${\tilde{J}}_{KL, ϕ}^{-}$ to make its dependence on the frozen rollout parameter $θ^{-}$ explicit. Here we suppress the analogous dependence on $θ_{old}$ .

¹⁶ The lemma says that improvement over the rollout policy equals the expected sum of rollout-policy advantages along responses sampled from the new policy.

¹⁷ Smaller $λ$ leans on the value predictions and lowers variance when $V_{ψ}$ is accurate; larger $λ$ uses more realized reward and reduces bias from bootstrapping on an imperfect value function. The value function $V_{ψ}$ is trained alongside the policy on the same rollout batch, by regression toward the return estimates $\hat{A} (s_{t}, a_{t}) + V_{ψ} (s_{t})$ .

¹⁸ If all responses in the group receive the same reward, as can happen when all sampled solutions are correct or all are incorrect, the within-group standard deviation is zero. Implementations therefore need a small numerical stabilizer, filtering rule, or resampling/dynamic-sampling scheme.

¹⁹ Strictly speaking, ${\hat{A}}_{i}$ is not an advantage estimate. The group mean is not a valid baseline (it depends on the action being scored, because $r_{ϕ} (x, y_{i})$ appears in it; RLOO instead excludes $y_{i}$ ), and dividing by the group standard deviation rescales the estimate, so ${\hat{A}}_{i}$ no longer estimates $Q^{π} - V^{π}$ . We nonetheless call it an advantage, following common usage.

²⁰ The group standard deviation gives larger updates to prompts with low reward variation, such as groups that are almost all correct or almost all incorrect. The $1 / | y_{i} |$ factor gives longer responses smaller per-token updates, because every token in response i carries the same advantage ${\hat{A}}_{i}$ ; this favors brief correct answers and penalizes long incorrect answers too weakly.

²¹ A verifiable reward (Section 5.1) may coincide with the real environment, such as an execution sandbox for code. In that setting, optimization against the verifier constitutes online RL in the standard sense.

²² FAIR CodeGen Team et al. [28] instead set ${\hat{A}}_{i} = R_{i} - μ$ using the length-weighted baseline $μ = L^{- 1} \sum_{i = 1}^{G} L_{i} R_{i}$ , where $L = \sum_{i} L_{i}$ . This centers the advantages under token weighting because $\sum_{i} L_{i} {\hat{A}}_{i} = 0$ . Their implementation also drops standard-deviation normalization, divides by a fixed maximum token budget rather than trajectory length, and omits the KL penalty.

²³ The derivation parallels the PPO and GRPO derivations in Section 7.3, except that the trajectory law also contains environment transitions and policy terms occur only at agent-generated positions. Because P is independent of $θ$ , its terms vanish when differentiating the log trajectory probability and cancel from the exact cumulative importance ratio; multi-turn GRPO then applies the same local-ratio approximation at the remaining agent-token positions.

²⁴ Note that Samanta et al. [83] operate in a single-turn RLVR setting, where the reasoning trace is broken into multiple discrete “thoughts.” This becomes similar to a multi-turn problem, except that the environment does not produce responses in between turns. Their algorithm performs resets to intermediate reasoning steps, resamples a group, and applies GRPO with the fresh group at the intermediate state.

²⁵ Policy iteration alternates between evaluating the current policy and improving it using the resulting value estimates. In the exact finite-MDP case, the policy-improvement theorem guarantees that the next policy is no worse than the previous one at every state, and repeated exact policy iteration converges to an optimal policy. See Sutton and Barto [99] and Bertsekas and Tsitsiklis [8] for full treatments of these algorithms.

²⁶ Note that $Q^{π_{k}}$ is fit only on data covered by $π_{k}$ , so it can be unreliable for the actions that policy improvement searches over. Jiang et al. [44] note this limitation and discuss exploration or KL-restrained updates to mitigate it.

References

[1] A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers. Association for Computational Linguistics, Kerrville, TX, 12248–12267, 2024. Google Scholar
[2] K. Asadi, J. Han, I. Pipano, X. Xu, D. Perrault-Joncas, S. Sabach, K. Bouyarmane, and M. Ghavamzadeh. C2-DPO: Constrained controlled direct preference optimization. Preprint, submitted February 22, https://arxiv.org/abs/2502.17507, 2025.Google Scholar
[3] S. M. Asghari, C. Chute, V. Dwaracherla, X. Lu, M. Jafarnia, V. Minden, Z. Wen, and B. Van Roy. Efficient exploration at scale. Preprint, submitted March 18, https://arxiv.org/abs/2603.17378, 2026.Google Scholar
[4] Y. Bai, Y. Bao, Y. Charles, C. Chen, G. Chen, H. Chen, H. Chen, et al. Kimi K2: Open agentic intelligence. Preprint, submitted July 28, https://arxiv.org/abs/2507.20534, 2025.Google Scholar
[5] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, et al. Constitutional AI: Harmlessness from AI feedback. Preprint, submitted December 15, https://arxiv.org/abs/2212.08073, 2022.Google Scholar
[6] V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan. $τ^{2}$ -bench: Evaluating conversational agents in a dual-control environment. Preprint, submitted June 9, https://arxiv.org/abs/2506.07982, 2025.Google Scholar
[7] D. P. Bertsekas. Reinforcement Learning and Optimal Control. Athena Scientific, Nashua, NH, 2019.Google Scholar
[8] D. P. Bertsekas and J. N. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, Nashua, NH, 1996.Google Scholar
[9] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, et al. On the opportunities and risks of foundation models. Preprint, submitted August 16, https://arxiv.org/abs/2108.07258, 2021.Google Scholar
[10] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39(3/4):324–345, 1952.Crossref, Google Scholar
[11] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, et al. Language models are few-shot learners. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors. Advances in Neural Information Processing Systems, volume 33. Curran Associates, Red Hook, NY, 1877–1901, 2020.Google Scholar
[12] E. Brynjolfsson, D. Li, and L. Raymond. Generative AI at work. Quarterly Journal of Economics 140(2):889–942, 2025.Crossref, Google Scholar
[13] S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. Preprint, submitted July 27, https://arxiv.org/abs/2307.15217, 2023.Google Scholar
[14] T. Castellani, N. Ye, D. Mittal, T. Yen, E. Koukoumidis, W. Zeng, and H. Namkoong. SynthTools: A framework for scaling synthetic tools for agent development. Preprint, submitted November 11, https://arxiv.org/abs/2511.09572, 2025.Google Scholar
[15] J. Chen and N. Jiang. Information-theoretic considerations in batch reinforcement learning. K. Chaudhuri and R. Salakhutdinov, editors. Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, volume 97. JMLR.org, 1042–1051, 2019.Google Scholar
[16] Z. Chen, Y. Deng, H. Yuan, K. Ji, and Q. Gu. Self-play fine-tuning converts weak language models to strong language models. R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors. Proceedings of the 41st International Conference on Machine Learning. Proceedings of Machine Learning Research, volume 235. JMLR.org, 6621–6642, 2024.Google Scholar
[17] P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors. Advances in Neural Information Processing Systems, volume 30. Curran Associates, Red Hook, NY, 4299–4307, 2017.Google Scholar
[18] W. Chu and Z. Ghahramani. Preference learning with Gaussian processes. Proceedings of the 22nd International Conference on Machine Learning. Association for Computing Machinery, New York, 137–144, 2005.Google Scholar
[19] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors. Advances in Neural Information Processing Systems, volume 31. Curran Associates, Red Hook, NY, 4754–4765, 2018.Google Scholar
[20] A. T. H. Chung, B. Zhang, L. C. Kung, H. Bastani, and O. Bastani. Effective personalized AI tutors via LLM-guided reinforcement learning. Preprint, submitted March 31, https://doi.org/10.2139/ssrn.6423358, 2026.Google Scholar
[21] T. Coste, U. Anwar, R. Kirk, and D. Krueger. Reward model ensembles help mitigate overoptimization. The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, 2024.Google Scholar
[22] D. P. de Farias and B. Van Roy. On the existence of fixed points for approximate value iteration and temporal-difference learning. Journal of Optimization Theory and Applications 105(3):589–608, 2000.Crossref, Google Scholar
[23] DeepSeek-AI, A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, et al. DeepSeek-V3 technical report. Preprint, submitted December 27, https://arxiv.org/abs/2412.19437, 2024.Google Scholar
[24] V. Dwaracherla, S. M. Asghari, B. Hao, and B. Van Roy. Efficient exploration for LLMs. R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors. Proceedings of the 41st International Conference on Machine Learning. Proceedings of Machine Learning Research, volume 235. JMLR.org, 12215–12227, 2024.Google Scholar
[25] J. Eisenstein, C. Nagpal, A. Agarwal, A. Beirami, A. D’Amour, D. Dvijotham, A. Fisch, et al. Helping or herding? Reward model ensembles mitigate but do not eliminate reward hacking. First Conference on Language Modeling, COLM 2024, Philadelphia, 2024.Google Scholar
[26] D. Ernst, P. Geurts, and L. Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research 6:503–556, 2005.Google Scholar
[27] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Model alignment as prospect theoretic optimization. R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors. Proceedings of the 41st International Conference on Machine Learning. Proceedings of Machine Learning Research, volume 235. JMLR.org, 12634–12651, 2024.Google Scholar
[28] FAIR CodeGen Team, J. Copet, Q. Carbonneaux, G. Cohen, J. Gehring, J. Kahn, J. Kossen, et al. CWM: An open-weights LLM for research on code generation with world models. Preprint, submitted September 30, https://arxiv.org/abs/2510.02387, 2025.Google Scholar
[29] A. M. Farahmand, R. Munos, and C. Szepesvári. Error propagation for approximate policy and value iteration. J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, editors. Advances in Neural Information Processing Systems, volume 23. Curran Associates, Red Hook, NY, 568–576, 2010.Google Scholar
[30] L. Feng, Z. Xue, T. Liu, and B. An. Group-in-group policy optimization for LLM agent training. Preprint, submitted May 16, https://arxiv.org/abs/2505.10978, 2025.Google Scholar
[31] L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors. Proceedings of the 40th International Conference on Machine Learning. Proceedings of Machine Learning Research, volume 202. JMLR.org, 10835–10866, 2023.Google Scholar
[32] M. Gheshlaghi Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. S. Dasgupta, S. Mandt, and Y. Li, editors. Proceedings of the 27th International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, volume 238. JMLR.org, 4447–4455, 2024.Google Scholar
[33] G. J. Gordon. Approximate solutions to Markov decision processes. PhD thesis, Carnegie Mellon University, Pittsburgh, 1999.Google Scholar
[34] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, et al. The llama 3 herd of models. Preprint, submitted July 31, https://arxiv.org/abs/2407.21783, 2024.Google Scholar
[35] D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645(8081):633–638, 2025. Crossref, Google Scholar
[36] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, 2020.Google Scholar
[37] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. K. Chaudhuri and R. Salakhutdinov, editors. Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, volume 97. JMLR.org, 2555–2565, 2019.Google Scholar
[38] Z. Hu, Y. Feng, A. T. Luu, B. Hooi, and A. Lipani. Unlocking the potential of user feedback: Leveraging large language model as user simulator to enhance dialogue system. Preprint, submitted June 16, https://arxiv.org/abs/2306.09821, 2023.Google Scholar
[39] Hugging Face. GRPO trainer. Accessed July 24, https://huggingface.co/docs/trl/grpo_trainer, 2026.Google Scholar
[40] H. Ivison, Y. Wang, J. Liu, Z. Wu, V. Pyatkin, N. Lambert, N. A. Smith, Y. Choi, and H. Hajishirzi. Unpacking DPO and PPO: Disentangling best practices for learning from preference feedback. Preprint, submitted June 13, https://arxiv.org/abs/2406.09279, 2024.Google Scholar
[41] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett, editors. Advances in Neural Information Processing Systems, volume 32. Curran Associates, Red Hook, NY, 12498–12509, 2019.Google Scholar
[42] Y. Ji, Z. Ma, Y. Wang, G. Chen, X. Chu, and L. Wu. Tree search for LLM agent reinforcement learning. The Fourteenth International Conference on Learning Representations, ICLR 2026, Rio de Janeiro, Brazil, 2026.Google Scholar
[43] D. R. Jiang, A. Nikulkov, Y. C. Chen, Y. Bai, and Z. Zhu. Improving generative ad text on Facebook using reinforcement learning. Preprint, submitted July 29, https://arxiv.org/abs/2507.21983, 2025.Google Scholar
[44] D. R. Jiang, A. Samanta, J. Bhandari, A. Yang, R. Munos, and T. Lu. Aligning LLMs toward multi-turn conversational outcomes using iterative RLHF. Lifelong Agents: Learning, Aligning, Evolving, Workshop at the Fourteenth International Conference on Learning Representations, ICLR 2026, Rio de Janeiro, Brazil, 2026.Google Scholar
[45] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. P, O. Press, and K. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, 2024.Google Scholar
[46] S. Kakade and J. Langford. Approximately optimal approximate reinforcement learning. Proceedings of the 19th International Conference on Machine Learning. Morgan Kaufmann Publishers, San Francisco, 267–274, 2002.Google Scholar
[47] T. Kaufmann, P. Weng, V. Bengs, and E. Hüllermeier. A survey of reinforcement learning from human feedback. Transactions on Machine Learning Research 2025, 2025.Google Scholar
[48] C. Kausik, A. Swaminathan, and N. Kallus. The context gathering decision process: A POMDP framework for agentic search. Preprint, submitted May 7, https://arxiv.org/abs/2605.07042, 2026.Google Scholar
[49] P. Laban, H. Hayashi, Y. Zhou, and J. Neville. LLMs get lost in multi-turn conversation. Preprint, submitted May 9, https://arxiv.org/abs/2505.06120, 2025.Google Scholar
[50] N. Lambert. Reinforcement Learning from Human Feedback. Manning Publications, Shelter Island, NY, 2026.Google Scholar
[51] N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, et al. Tulu 3: Pushing frontiers in open language model post-training. Preprint, submitted November 22, https://arxiv.org/abs/2411.15124, 2024.Google Scholar
[52] S. Lange, T. Gabel, and M. Riedmiller. Batch reinforcement learning. M. Wiering and M. van Otterlo, editors. Reinforcement Learning. Adaptation, Learning, and Optimization, volume 12. Springer, Berlin, 45–73, 2012.Crossref, Google Scholar
[53] Y. Lee, R. Nair, Q. Zhang, K. Lee, O. Khattab, and C. Finn. Meta-harness: End-to-end optimization of model harnesses. Preprint, submitted March 30, https://arxiv.org/abs/2603.28052, 2026.Google Scholar
[54] M. Lee, K. I. Gero, J. J. Y. Chung, S. Buckingham Shum, V. Raheja, H. Shen, S. Venugopalan, et al. A design space for intelligent and interactive writing assistants. Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, 1054, 2024. Google Scholar
[55] H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. R. Lu, C. Bishop, et al. RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with AI feedback. R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors. Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, volume 235. JMLR.org, 26874–26901, 2024.Google Scholar
[56] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. Preprint, submitted May 4, https://arxiv.org/abs/2005.01643, 2020.Google Scholar
[57] A. Li, H. Chen, H. Namkoong, and T. Peng. LLM generated persona is a promise with a catch. Preprint, submitted March 18, https://arxiv.org/abs/2503.16527, 2025.Google Scholar
[58] J. Li, P. Zhou, R. Meng, M. P. Vadera, L. Li, and Y. Li. Turn-PPO: Turn-level advantage estimation with PPO for improved multi-turn RL in agentic LLMs. Preprint, submitted December 18, https://arxiv.org/abs/2512.17008, 2025.Google Scholar
[59] K. Liu, J. K. Liu, M. Chen, and Y. Liu. Rethinking KL regularization in RLHF: From value estimation to gradient optimization. Preprint, submitted October 2, https://arxiv.org/abs/2510.01555, 2025.Google Scholar
[60] Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin. Understanding R1-zero-like training: A critical perspective. Preprint, submitted March 26, https://arxiv.org/abs/2503.20783, 2025.Google Scholar
[61] J. Lu, J. Li, S. An, M. Zhao, Y. He, D. Yin, and X. Sun. Eliminating biased length reliance of direct preference optimization via down-sampled KL divergence. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Kerrville, TX, 1047–1067, 2024.Google Scholar
[62] X. Luo, Z. Tang, J. Wang, and X. Zhang. DuetSim: Building user simulator with dual large language models for task-oriented dialogues. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. ELRA Language Resource Association, Paris, 5414–5424, 2024.Google Scholar
[63] C. Ma, J. Zhang, Z. Zhu, C. Yang, Y. Yang, Y. Jin, Z. Lan, L. Kong, and J. He. AgentBoard: An analytical evaluation board of multi-turn LLM agents. A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors. Advances in Neural Information Processing Systems, volume 37. Curran Associates, Red Hook, NY, 74325–74362, 2024.Crossref, Google Scholar
[64] Y. Meng, M. Xia, and D. Chen. SimPO: Simple preference optimization with a reference-free reward. A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors. Advances in Neural Information Processing Systems, volume 37. Curran Associates, Red Hook, NY, 124198–124235, 2024.Crossref, Google Scholar
[65] Microsoft AI Team. MAI-Thinking-1: Building a hill-climbing machine. https://microsoft.ai/pdf/mai-thinking-1.pdf, 2026.Google Scholar
[66] T. Moskovitz, A. K. Singh, D. Strouse, T. Sandholm, R. Salakhutdinov, A. D. Dragan, and S. McAleer. Confronting reward model overoptimization with constrained RLHF. The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, 2024.Google Scholar
[67] R. Munos. Error bounds for approximate policy iteration. Proceedings of the 20th International Conference on Machine Learning. Association for the Advancement of Artificial Intelligence, Washington, DC, 560–567, 2003.Google Scholar
[68] R. Munos. Error bounds for approximate value iteration. Proceedings of the AAAI Conference on Artificial Intelligence. Association for the Advancement of Artificial Intelligence, Washington, DC, 1006–1011, 2005.Google Scholar
[69] R. Munos. Performance bounds in $L_{p}$ -norm for approximate value iteration. SIAM Journal on Control and Optimization 46(2):541–561, 2007.Crossref, Google Scholar
[70] R. Munos and C. Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research 9(27):815–857, 2008.Google Scholar
[71] H. Nam, O. Gottesman, A. Zhang, D. Foster, E. Brunskill, and L. Ungar. Efficient RL for optimizing conversation level outcomes with an LLM-based tutor. Preprint, submitted July 22, https://arxiv.org/abs/2507.16252, 2025.Google Scholar
[72] OpenAI. Introducing ChatGPT. (November 30), https://openai.com/index/chatgpt/, 2022.Google Scholar
[73] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, et al. Training language models to follow instructions with human feedback. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors. Advances in Neural Information Processing Systems, volume 35. Curran Associates, Red Hook, NY, 27730–27744, 2022.Crossref, Google Scholar
[74] A. Pal, D. Karkhanis, S. Dooley, M. Roberts, S. Naidu, and C. White. Smaug: Fixing failure modes of preference optimisation with DPO-positive. Preprint, submitted February 20, https://arxiv.org/abs/2402.13228, 2024.Google Scholar
[75] R. Park, R. Rafailov, S. Ermon, and C. Finn. Disentangling length from quality in direct preference optimization. Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, Kerrville, TX, 4998–5017, 2024.Google Scholar
[76] W. B. Powell. Approximate Dynamic Programming: Solving the Curses of Dimensionality, 2nd edition. John Wiley & Sons, Hoboken, NJ, 2011.Crossref, Google Scholar
[77] M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Hoboken, NJ, 1994.Crossref, Google Scholar
[78] Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, et al. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. Preprint, submitted July 31, https://arxiv.org/abs/2307.16789, 2023.Google Scholar
[79] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. Technical report, OpenAI, San Francisco, 2018.Google Scholar
[80] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors. Advances in Neural Information Processing Systems, volume 36. Curran Associates, Red Hook, NY, 53728–53741, 2023.Crossref, Google Scholar
[81] P. Rajasekaran, E. Dixon, C. Ryan, and J. Hadfield. Effective context engineering for AI agents. Engineering at Anthropic (blog) (September 29), https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents, 2025.Google Scholar
[82] N. Razin, S. Malladi, A. Bhaskar, D. Chen, S. Arora, and B. Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization. Preprint, submitted October 11, https://arxiv.org/abs/2410.08847, 2024.Google Scholar
[83] A. Samanta, A. Magesh, A. Jain, Y. Yu, D. Jiang, K. Asadi, K. Hassani, P. Sajda, J. Bhandari, and Y. Efroni. Credit assignment with resets in language model reasoning. Preprint, submitted May 25, https://arxiv.org/abs/2605.25507, 2026.Google Scholar
[84] A. Samanta, A. Magesh, R. Wu, A. Jain, Y. Yu, D. Jiang, B. Vidolov, P. Sajda, Y. Efroni, and K. Hassani. Self-improvement of language models by post-training on multi-agent debate. Preprint, submitted September 18, https://arxiv.org/abs/2509.15172, 2025.Google Scholar
[85] B. Scherrer, M. Ghavamzadeh, V. Gabillon, B. Lesner, and M. Geist. Approximate modified policy iteration and its application to the game of Tetris. Journal of Machine Learning Research 16(49):1629–1676, 2015.Google Scholar
[86] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, et al. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature 588(7839):604–609, 2020.Crossref, Google Scholar
[87] J. Schulman. Approximating KL divergence. http://joschu.net/blog/kl-approx.html, 2020.Google Scholar
[88] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. F. Bach and D. Blei, editors. Proceedings of the 32nd International Conference on Machine Learning, Proceedings of Machine Learning Research, volume 37. JMLR.org, 1889–1897, 2015.Google Scholar
[89] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. Y. Bengio and Y. LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, 2016.Google Scholar
[90] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. Preprint, submitted July 20, https://arxiv.org/abs/1707.06347, 2017.Google Scholar
[91] R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Kerrville, TX, 1715–1725, 2016.Google Scholar
[92] O. Shaikh, S. Sapkota, S. Rizvi, E. Horvitz, J. S. Park, D. Yang, and M. S. Bernstein. Creating general user models from computer use. Preprint, submitted May 16, https://arxiv.org/abs/2505.10831, 2025.Google Scholar
[93] L. Shani, A. Rosenberg, A. Cassel, O. Lang, D. Calandriello, A. Zipori, H. Noga, et al. Multi-turn reinforcement learning with preference human feedback. A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors. Advances in Neural Information Processing Systems, volume 37. Curran Associates, Red Hook, NY, 118953–118993, 2024.Crossref, Google Scholar
[94] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. Preprint, submitted February 5, https://arxiv.org/abs/2402.03300, 2024.Google Scholar
[95] W. Shi, M. Yuan, J. Wu, Q. Wang, and F. Feng. Direct multi-turn preference optimization for language agents. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Kerrville, TX, 2312–2324. 2024.Google Scholar
[96] Y. Song, G. Swamy, A. Singh, J. A. Bagnell, and W. Sun. The importance of online data: Understanding preference fine-tuning via coverage. Preprint, submitted June 3, https://arxiv.org/abs/2406.01462, 2024.Google Scholar
[97] N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. Christiano. Learning to summarize with human feedback. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors. Advances in Neural Information Processing Systems, volume 33. Curran Associates, Red Hook, NY, 3008–3021, 2020.Google Scholar
[98] R. S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin 2(4):160–163, 1991.Crossref, Google Scholar
[99] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction, 2nd edition. MIT Press, Cambridge, MA, 2018.Google Scholar
[100] C. Szepesvári and R. Munos. Finite time bounds for sampling based fitted value iteration. Proceedings of the 22nd International Conference on Machine Learning. Association for Computing Machinery, New York, 880–887, 2005.Google Scholar
[101] Y. Tang and R. Munos. On a few pitfalls in KL divergence gradient estimation for RL. Preprint, submitted June 11, https://arxiv.org/abs/2506.09477, 2025.Google Scholar
[102] Y. Tang, D. Z. Guo, Z. Zheng, D. Calandriello, Y. Cao, E. Tarassov, R. Munos, et al. Understanding the performance gap between online and offline alignment algorithms. Preprint, submitted May 14, https://arxiv.org/abs/2405.08448, 2024.Google Scholar
[103] G. Tennenholtz, O. Meshi, A. Globerson, U. Shalit, J. Jeong, and C. Boutilier. Controllable user simulation. Preprint, submitted May 12, https://arxiv.org/abs/2605.11519, 2026.Google Scholar
[104] B. Van Roy. Performance loss bounds for approximate value iteration with state aggregation. Mathematics of Operations Research 31(2):234–244, 2006.Link, Google Scholar
[105] M. Vandergrift, E. Elelimy, and M. White. Position: RL researchers need to distinguish between solving simulators and using simulators as a proxy. Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, volume 306. JMLR.org, 2026.Google Scholar
[106] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors. Advances in Neural Information Processing Systems, volume 30. Curran Associates, Red Hook, NY, 2017.Google Scholar
[107] X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, 2023.Google Scholar
[108] Z. Wang, K. Ramnath, B. Bi, S. K. Pentyala, S. Chaudhuri, S. Mehrotra, Z. J. Zhu, X. B. Mao, S. Asur, and N. C. Cheng. Reinforcement learning for LLM post-training: A survey. Preprint, submitted July 23, https://arxiv.org/abs/2407.16216, 2024.Google Scholar
[109] Z. Wang, K. Wang, Q. Wang, P. Zhang, L. Li, Z. Yang, X. Jin, et al. RAGEN: Understanding self-evolution in LLM agents via multi-turn reinforcement learning. Preprint, submitted April 24, https://arxiv.org/abs/2504.20073, 2025.Google Scholar
[110] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors. Advances in Neural Information Processing Systems, volume 35. Curran Associates, Red Hook, NY, 24824–24837, 2022.Crossref, Google Scholar
[111] Z. Wei, W. Yao, Y. Liu, W. Zhang, Q. Lu, L. Qiu, C. Yu, et al. WebAgent-R1: Training web agents via end-to-end multi-turn reinforcement learning. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Kerrville, TX, 7920–7939. 2025.Google Scholar
[112] Q. Wei, S. Zeng, C. Li, W. Brown, O. Frunza, W. Deng, A. Schneider, et al. Reinforcing multi-turn reasoning in LLM agents via turn-level reward design. Preprint, submitted May 17, https://arxiv.org/abs/2505.11821, 2025.Google Scholar
[113] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8(3–4):229–256, 1992.Crossref, Google Scholar
[114] S. Wu, E. Choi, A. Khatua, Z. Wang, J. He-Yueya, T. C. Weerasooriya, W. Wei, D. Yang, J. Leskovec, and J. Zou. HumanLM: Simulating users with state alignment beats response imitation. Preprint, submitted February 7, https://arxiv.org/abs/2603.03303, 2026.Google Scholar
[115] T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, et al. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Preprint, submitted April 11, https://arxiv.org/abs/2404.07972, 2024.Google Scholar
[116] S. Yao, N. Shinn, P. Razavi, and K. Narasimhan. $τ$ -bench: A benchmark for tool-agent-user interaction in real-world domains. Preprint, submitted June 17, https://arxiv.org/abs/2406.12045, 2024.Google Scholar
[117] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma. MOPO: Model-based offline policy optimization. Advances in Neural Information Processing Systems, volume 33 (Curran Associates Inc., Red Hook, NY), 14129–14142, 2020.Google Scholar
[118] Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, et al. DAPO: An open-source LLM reinforcement learning system at scale. Preprint, submitted March 18, https://arxiv.org/abs/2503.14476, 2025.Google Scholar
[119] Y. Yue, J. Broder, R. Kleinberg, and T. Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences 78(5):1538–1556, 2012.Crossref, Google Scholar
[120] Z.ai. GLM-5.2: Built for long-horizon tasks. (June 16), https://z.ai/blog/glm-5.2, 2026.Google Scholar
[121] Y. Zhang, C. Ye, S. Jin, C. Yu, W. Xiong, S. Sahu, and N. Jiang. Rethinking importance sampling in LLM policy optimization: A cumulative token perspective. Preprint, submitted May 8, https://arxiv.org/abs/2605.07331, 2026.Google Scholar
[122] L. Zheng, W. L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, et al. Judging LLM-as-a-judge with MT-bench and chatbot arena. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors. Advances in Neural Information Processing Systems, volume 36. Curran Associates, Red Hook, NY, 46595–46623, 2023.Crossref, Google Scholar
[123] R. Zheng, S. Dou, S. Gao, Y. Hua, W. Shen, B. Wang, Y. Liu, et al. Secrets of RLHF in large language models part I: PPO. Preprint, submitted July 11, https://arxiv.org/abs/2307.04964, 2023.Google Scholar
[124] C. Zheng, S. Liu, M. Li, X. H. Chen, B. Yu, C. Gao, K. Dang, et al. Group sequence policy optimization. Preprint, submitted July 24, https://arxiv.org/abs/2507.18071, 2025.Google Scholar
[125] Y. Zhou, A. Zanette, J. Pan, S. Levine, and A. Kumar. ArCHer: Training language model agents via hierarchical multi-turn RL. R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors. Proceedings of the 41st International Conference on Machine Learning. Proceedings of Machine Learning Research, volume 235. JMLR.org, 62178–62209, 2024.Google Scholar
[126] S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, et al. WebArena: A realistic web environment for building autonomous agents. Preprint, submitted July 25, https://arxiv.org/abs/2307.13854, 2023.Google Scholar
[127] S. Zhu, C. Yu, R. Yang, Z. Liu, J. Hu, Q. Chen, and Y. Zhang. GAGPO: Generalized advantage grouped policy optimization. Preprint, submitted May 13, https://arxiv.org/abs/2605.13217, 2026.Google Scholar
[128] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. Preprint, submitted September 18, https://arxiv.org/abs/1909.08593, 2019.Google Scholar
[129] Y. Zuo, K. Zhang, L. Sheng, S. Qu, G. Cui, X. Zhu, H. Li, et al. TTRL: Test-time reinforcement learning. Preprint, submitted April 22, https://arxiv.org/abs/2504.16084, 2025.Google Scholar

cover image INFORMS TutORials in Operations Research

Operations Research and Artificial Intelligence: Better Together

November 2026

Article Information

Metrics

Information

Published Online:November 01, 2026

Cite as

Daniel R. Jiang (2026) A Tutorial on Reinforcement Learning for LLMs: RLHF and Beyond. INFORMS TutORials in Operations Research null(null):154-186.

https://doi.org/10.1287/educ.2026.0311

Keywords

PDF download