Bias and Variance Approximation in Value Function Estimates

Shie Mannor
Shie Mannor
[email protected]
Department of Electrical and Computer Engineering, McGill University, Montreal, Quebec, Canada H3A 2A7
Search for more papers by this author
,
Duncan Simester
Duncan Simester
[email protected]
Sloan School of Management, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139
Search for more papers by this author
,
Peng Sun
Peng Sun
[email protected]
Fuqua School of Business, Duke University, Durham, North Carolina 27708
Search for more papers by this author
,
John N. Tsitsiklis
John N. Tsitsiklis
[email protected]
Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139
Search for more papers by this author

Shie Mannor

[email protected]

Department of Electrical and Computer Engineering, McGill University, Montreal, Quebec, Canada H3A 2A7

Search for more papers by this author

Duncan Simester

[email protected]

Sloan School of Management, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139

Search for more papers by this author

Peng Sun

[email protected]

Fuqua School of Business, Duke University, Durham, North Carolina 27708

Search for more papers by this author

John N. Tsitsiklis

[email protected]

Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139

Search for more papers by this author

Published Online:1 Feb 2007https://doi.org/10.1287/mnsc.1060.0614

Supplemental Material

mnsc.1060.0614-sm-ec.pdf (67 KB)

Cited by
- A data-driven robust approach to a problem of optimal replacement in maintenance
  27 April 2026 | Annals of Operations Research, Vol. 37
- Robust screening in a pandemic
  5 January 2026 | IISE Transactions
- Divide-and-conquer offline policy evaluation for contextual bandits
  Physica A: Statistical Mechanics and its Applications, Vol. 676
- Sequential decision-making under uncertainty: a robust MDPs review
  18 August 2025 | Annals of Operations Research, Vol. 353, No. 3
- Convex Approximations of Random Constrained Markov Decision Processes
  7 August 2025 | SIAM Journal on Optimization, Vol. 35, No. 3
- On Practical Robust Reinforcement Learning: Adjacent Uncertainty Set and Double-Agent Algorithm
  IEEE Transactions on Neural Networks and Learning Systems, Vol. 36, No. 4
- Tools at the Frontiers of Quantitative Verification
  1 November 2024
- Robust Markov Decision Processes: A Place Where AI and Formal Methods Meet
  18 November 2024
- Approximate Information States for Worst Case Control and Learning in Uncertain Systems
  IEEE Transactions on Automatic Control, Vol. 70, No. 1
- Explaining the performance impact of opportunity costs approximation in integrated demand management and vehicle routing
  EURO Journal on Transportation and Logistics, Vol. 14
- Adaptive moving average Q-learning
  12 August 2024 | Knowledge and Information Systems, Vol. 66, No. 12
- A Meta-Learning Approach to Mitigating the Estimation Bias of Q-Learning
  12 November 2024 | ACM Transactions on Knowledge Discovery from Data, Vol. 18, No. 9
- Is Separately Modeling Subpopulations Beneficial for Sequential Decision-Making?
  Ilbin Lee
  18 May 2023 | Operations Research, Vol. 72, No. 6
- Stochastic control with distributionally robust constraints for cyber–physical systems vulnerable to attacks
  European Journal of Control, Vol. 80
- Percentile optimization in multi-armed bandit problems
  19 July 2024 | Annals of Operations Research, Vol. 340, No. 2-3
- Distributionally Robust Chance-Constrained Markov Decision Processes with Random Payoff
  26 July 2024 | Applied Mathematics & Optimization, Vol. 90, No. 1
- Uncertainty Quantification and Exploration for Reinforcement Learning
  Yi Zhu,
  Jing Dong,
  Henry Lam
  2 March 2023 | Operations Research, Vol. 72, No. 4
- Constrained Markov decision processes with uncertain transition probabilities
- Rank-1 transition uncertainties in constrained Markov decision processes
  European Journal of Operational Research, Vol. 46
- Off-policy evaluation for tabular reinforcement learning with synthetic trajectories
  17 November 2023 | Statistics and Computing, Vol. 34, No. 1
- A Data-Driven Framework for Clinical Decision Support Systems in Positive Airway Pressure and Oxygen Titration
  28 January 2024 | Journal of Clinical Medicine, Vol. 13, No. 3
- Worst-Case Control and Learning Using Partial Observations Over an Infinite Time Horizon
- Runtime Verification of Learning Properties for Reinforcement Learning Algorithms
  15 November 2023 | Electronic Proceedings in Theoretical Computer Science, Vol. 395
- Optimizing User Engagement Through Adaptive Ad Sequencing
  Omid Rafieian
  29 December 2022 | Marketing Science, Vol. 42, No. 5
- Robustness of Proactive Intensive Care Unit Transfer Policies
  Julien Grand-Clément,
  Carri W. Chan,
  Vineet Goyal,
  Gabriel Escobar
  22 November 2022 | Operations Research, Vol. 71, No. 5
- Time-aware deep reinforcement learning with multi-temporal abstraction
  25 March 2023 | Applied Intelligence, Vol. 53, No. 17
- Joint Recurrent Actor-Critic Model for Partially Observable Control
- QVDDPG: QV Learning with Balanced Constraint in Actor-Critic Framework
- Decision-making under uncertainty: beyond probabilities
  30 May 2023 | International Journal on Software Tools for Technology Transfer, Vol. 25, No. 3
- On Robust Control of Partially Observed Uncertain Systems with Additive Costs
- Compromise policy for multi-stage stochastic linear programming: Variance and bias reduction
  Computers & Operations Research, Vol. 153
- Joint chance-constrained Markov decision processes
  27 October 2022 | Annals of Operations Research, Vol. 322, No. 2
- An internal supplemental action measurement to increase the gap of action values and reduce the sensitivity to overestimation error
  26 July 2021 | Journal of Experimental & Theoretical Artificial Intelligence, Vol. 34, No. 6
- Risk-Sensitive Reinforcement Learning Via Entropic- VaR Optimization
- Deterministic policies based on maximum regrets in MDPs with imprecise rewards
  AI Communications, Vol. 34, No. 4
- Constrained Markov decision processes with uncertain costs
  Operations Research Letters, Vol. 50, No. 2
- Dual Discounting in Renewable Resource Planning under Risk
  26 October 2021 | Environmental Management, Vol. 69, No. 2
- PRAG: Periodic Regularized Action Gradient for Efficient Continuous Control
  4 November 2022
- On Robust and Adaptive Fidelity Selection for Human-in-the-loop Queues
- Towards Safe Motion Planning in Human Workspaces: A Robust Multi-agent Approach
- Multi-model Markov decision processes
  10 May 2021 | IISE Transactions, Vol. 59
- Policy-based branch-and-bound for infinite-horizon Multi-model Markov decision processes
  Computers & Operations Research, Vol. 126
- Stochastic and Distributionally Robust Load Ensemble Control
  IEEE Transactions on Power Systems, Vol. 35, No. 6
- A TD3-based multi-agent deep reinforcement learning method in mixed cooperation-competition environment
  Neurocomputing, Vol. 411
- Risk aversion to parameter uncertainty in Markov decision processes with an application to slow-onset disaster relief
  11 November 2019 | IISE Transactions, Vol. 52, No. 8
- Dynamic Programming Models for Maximizing Customer Lifetime Value: An Overview
  24 August 2019
- A Configurable off-Policy Evaluation with Key State-Based Bias Constraints in AI Reinforcement Learning
  22 September 2020
- Distributionally robust optimization for sequential decision-making
  18 August 2019 | Optimization, Vol. 68, No. 12
- Data-Driven Methods for Markov Decision Problems with Parameter Uncertainty
  Shie Mannor,
  Huan Xu
  2 October 2019
- Data-Driven Percentile Optimization for Multiclass Queueing Systems with Model Ambiguity: Theory and Application
  Austin Bren,
  Soroush Saghafian
  29 April 2019 | INFORMS Journal on Optimization, Vol. 1, No. 4
- An active-set strategy to solve Markov decision processes with good-deal risk measure
  25 February 2019 | Optimization Letters, Vol. 13, No. 6
- Probabilistic sensitivity analysis on Markov models with uncertain transition probabilities: an application in evaluating treatment decisions for type 2 diabetes
  27 October 2017 | Health Care Management Science, Vol. 22, No. 1
- Optimization of Sequential Decision Making for Chronic Diseases: From Data to Decisions
  Brian T. Denton
  19 October 2018
- Multi-agent Robust Time Differential Reinforcement Learning Over Communicated Networks
- A Knowledge Management model for corporate water responsibility
  12 January 2018
- Harvesting Randomness to Optimize Distributed Systems
  30 November 2017
- Distributionally robust games with an application to supply chain
  Journal of Intelligent & Fuzzy Systems, Vol. 33, No. 5
- Robust Evaluation Matrix
  12 April 2017
- Living-Donor Liver Transplantation Timing under Ambiguous Health State Transition Probabilities
  SSRN Electronic Journal, Vol. 42
- Field Experiments in Marketing
- Reinforcement Learning in Robust Markov Decision Processes
  Shiau Hong Lim,
  Huan Xu,
  Shie Mannor
  21 September 2016 | Mathematics of Operations Research, Vol. 41, No. 4
- Robust MDPs with k-Rectangular Uncertainty
  Shie Mannor,
  Ofir Mebel,
  Huan Xu
  23 September 2016 | Mathematics of Operations Research, Vol. 41, No. 4
- Distributionally Robust Counterpart in Markov Decision Processes
  IEEE Transactions on Automatic Control, Vol. 61, No. 9
- Planning with Information-Processing Constraints and Model Uncertainty in Markov Decision Processes
  4 September 2016
- Bayesian Reinforcement Learning: A Survey
  26 November 2015 | Foundations and Trends® in Machine Learning, Vol. 8, No. 5-6
- Risk aversion in belief-space planning under measurement acquisition uncertainty
- Data-Driven Cyber-Vulnerability Maintenance Policies
  21 November 2017 | Journal of Quality Technology, Vol. 46, No. 3
- Robust Modified Policy Iteration
  David L. Kaufman,
  Andrew J. Schaefer,
  6 June 2012 | INFORMS Journal on Computing, Vol. 25, No. 3
- Robust Markov Decision Processes
  Wolfram Wiesemann,
  Daniel Kuhn,
  Berç Rustem,
  14 November 2012 | Mathematics of Operations Research, Vol. 38, No. 1
- Performance Guarantees for Empirical Markov Decision Processes with Applications to Multiperiod Inventory Models
  William L. Cooper,
  Bharath Rangarajan,
  9 October 2012 | Operations Research, Vol. 60, No. 5
- Robust Adaptive Markov Decision Processes: Planning with Model Uncertainty
  IEEE Control Systems, Vol. 32, No. 5
- Distributionally Robust Markov Decision Processes
  Huan Xu,
  Shie Mannor,
  1 May 2012 | Mathematics of Operations Research, Vol. 37, No. 2
- Bayesian Reinforcement Learning
- Spectral and graph-theoretic bounds on steady-state-probability estimation performance for an ergodic Markov chain
  Journal of the Franklin Institute, Vol. 348, No. 9
- The Impact of Tariff Structure on Customer Retention, Usage, and Profitability of Access Services
  Raghuram Iyengar,
  Kamel Jedidi,
  Skander Essegaier,
  Peter J. Danaher,
  15 July 2011 | Marketing Science, Vol. 30, No. 5
- Informing sequential clinical decision-making through reinforcement learning: an empirical study
  22 December 2010 | Machine Learning, Vol. 84, No. 1-2
- The optimal unbiased value estimator and its relation to LSTD, TD and MC
  29 October 2010 | Machine Learning, Vol. 83, No. 3
- Spectral and graph-theoretic bounds on steady-state-probability estimation performance for an ergodic Markov chain
- Robust Performance Evaluation of POMDP-Based Dialogue Systems
  IEEE Transactions on Audio, Speech, and Language Processing, Vol. 19, No. 4
- Optimal learning of transition probabilities in the two-agent newsvendor problem
- Dynamic Allocation of Pharmaceutical Detailing and Sampling for Long-Term Profitability
  Ricardo Montoya,
  Oded Netzer,
  Kamel Jedidi,
  27 May 2010 | Marketing Science, Vol. 29, No. 5
- Approximate dynamic programming with correlated Bayesian beliefs
- Percentile Optimization for Markov Decision Processes with Parameter Uncertainty
  Erick Delage,
  Shie Mannor,
  17 August 2009 | Operations Research, Vol. 58, No. 1
- Robust adaptive Markov Decision Processes in multi-vehicle applications
- Robust Markov Decision Processes using Sigma Point sampling
- Estimation of non-stationary Markov Chain transition models
- Percentile optimization in uncertain Markov decision processes with application to efficient exploration
  20 June 2007

Volume 53, Issue 2

February 2007

Pages iv-355

Article Information

Supplemental Material

Metrics

Information

Received:July 14, 2004
Published Online:February 01, 2007

Cite as

Shie Mannor, Duncan Simester, Peng Sun, John N. Tsitsiklis, (2007) Bias and Variance Approximation in Value Function Estimates. Management Science 53(2):308-322.

https://doi.org/10.1287/mnsc.1060.0614

Keywords

PDF download

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Bias and Variance Approximation in Value Function Estimates

Supplemental Material

Volume 53, Issue 2

Article Information

Supplemental Material

Metrics

Information

Cite as

Keywords

Sign Up for INFORMS Publications Updates and News