Free Access

Technical Note—An Equivalence Between Continuous and Discrete Time Markov Decision Processes

Richard F. Serfozo
Richard F. Serfozo
Bell Laboratories, Holmdel, New Jersey
Search for more papers by this author

Richard F. Serfozo

Bell Laboratories, Holmdel, New Jersey

Search for more papers by this author

Published Online:1 Jun 1979https://doi.org/10.1287/opre.27.3.616

Cited by
- A non-parametric dynamic degradation-based inspection and replacement model for single-unit continuously deteriorating systems
  European Journal of Operational Research, Vol. 334, No. 2
- Dynamic repair and maintenance of heterogeneous machines dispersed on a network: A rollout method for online reinforcement learning
  Computers & Operations Research, Vol. 193
- Control-limit policies for coordinated dispatching and preventive maintenance in multi-product manufacturing systems
  European Journal of Operational Research, Vol. 333, No. 2
- An analysis of energy-efficient production control strategies in serial systems with two nonidentical machines
  21 May 2026 | Flexible Services and Manufacturing Journal, Vol. 18
- Stochastic dynamic job scheduling with interruptible setup and processing times: An approach based on queueing control
  European Journal of Operational Research, Vol. 329, No. 3
- SDN and BBR Based Friendly Congestion Control for MPTCP
- Optimizing production and remanufacturing decisions for make-to-stock hybrid manufacturing systems using real-time information on products in use
  29 September 2025 | International Journal of Production Research, Vol. 64, No. 3
- Optimal purchasing and production control for a circular production system
  20 January 2026 | IISE Transactions
- Purchasing and remanufacturing decisions with different quality returned material and finished goods
  21 January 2025 | International Journal of Production Research, Vol. 63, No. 12
- Superposed semi-Markov decision process with application to optimal maintenance systems
  31 March 2025 | Journal of Combinatorial Optimization, Vol. 49, No. 3
- Logarithmic Regret Bounds for Continuous-Time Average-Reward Markov Decision Processes
  10 September 2024 | SIAM Journal on Control and Optimization, Vol. 62, No. 5
- Discounted cost exponential semi-Markov decision processes with unbounded transition rates: a service rate control problem with impatient customers
  18 April 2024 | Probability in the Engineering and Informational Sciences, Vol. 38, No. 4
- A General Boolean Semantic Modelling Approach for Complex and Intelligent Industrial Systems in the Framework of DES
  Journal of Systems Engineering and Electronics, Vol. 35, No. 5
- Decision synthesis framework approach for dynamic dispatching and preventive maintenance in large flexible manufacturing systems
  Advanced Engineering Informatics, Vol. 61
- Age-minimal CPU Scheduling
- On the computational complexity of ethics: moral tractability for minds and machines
  31 March 2024 | Artificial Intelligence Review, Vol. 57, No. 4
- Integrated optimisation of pricing, manufacturing, and procurement decisions of a make-to-stock system operating in a fluctuating environment
  5 December 2022 | International Journal of Production Research, Vol. 61, No. 24
- Joint Dynamic Dispatching and Preventive Maintenance for Unrelated Parallel Machines With Equipment Health Considerations
  IEEE Transactions on Semiconductor Manufacturing, Vol. 36, No. 4
- Deep Learning Agents for Efficient Dynamic Production Control in Semiconductor Manufacturing
- Two‐class constrained optimization with applications to queueing control
  29 February 2020 | Naval Research Logistics (NRL), Vol. 70, No. 5
- Discrete-time switching control in random walks
  1 February 2022 | International Journal of Control, Vol. 96, No. 4
- Optimal Capacity Rationing Policy for a Container Leasing System with Multiple Kinds of Customers and Substitutable Containers
  Xufeng Yang,
  Juliang Zhang,
  Wen Jiao,
  Hong Yan
  19 May 2022 | Management Science, Vol. 69, No. 3
- Structure-Aware Reinforcement Learning for Node-Overload Protection in Mobile Edge Computing
  IEEE Transactions on Cognitive Communications and Networking, Vol. 8, No. 4
- K competing queues with customer abandonment: optimality of a generalised $$c \mu $$-rule by the Smoothed Rate Truncation method
  11 January 2019 | Annals of Operations Research, Vol. 317, No. 2
- OPTIMAL ADMISSION AND ROUTING WITH CONGESTION-SENSITIVE CUSTOMER CLASSES
  1 March 2021 | Probability in the Engineering and Informational Sciences, Vol. 36, No. 3
- Queueing Network Controls via Deep Reinforcement Learning
  J. G. Dai,
  Mark Gluzman
  3 December 2021 | Stochastic Systems, Vol. 12, No. 1
- ECN-based shared bottleneck detection for multi-path TCP
  Computer Communications, Vol. 186
- Kolmogorov's Equations for Jump Markov Processes and Their Applications to Control Problems
  3 February 2022 | Theory of Probability & Its Applications, Vol. 66, No. 4
- A queueing model for customer rescheduling and no-shows in service systems
  Operations Research Letters, Vol. 49, No. 6
- Imperfect Maintenance Optimization of Multi-State Rolling Stocks Based on Deep Reinforcement Learning
- AAC: Adaptively Adjusting Concurrency by Exploiting Path Diversity in Datacenter Networks
  5 March 2021 | Journal of Network and Systems Management, Vol. 29, No. 3
- A multi-objective approach for PH-graphs with applications to stochastic shortest paths
  24 October 2020 | Mathematical Methods of Operations Research, Vol. 93, No. 1
- A useful technique for piecewise deterministic Markov decision processes
  Operations Research Letters, Vol. 49, No. 1
- Уравнения Колмогорова для скачкообразных марковских процессов и их применения в задачах управления
  22 October 2021 | Теория вероятностей и ее применения, Vol. 66, No. 4
- A conservative index heuristic for routing problems with multiple heterogeneous service facilities
  22 July 2020 | Mathematical Methods of Operations Research, Vol. 92, No. 3
- Model Synthesis and Stochastic Automated Verification of Systems-of-Systems Dynamic Architectures
- Concurrent MDPs with Finite Markovian Policies
  9 March 2020
- Dynamic dispatching and preventive maintenance for parallel machines with dispatching-dependent deterioration
  Computers & Operations Research, Vol. 113
- Optimal server assignment in a two-stage tandem queueing system
  Operations Research Letters, Vol. 48, No. 1
- A dynamic ambulance routing model with multiple response
  Transportation Research Part E: Logistics and Transportation Review, Vol. 133
- Light robustness in the optimization of Markov decision processes with uncertain parameters
  Computers & Operations Research, Vol. 108
- EMPTCP: An ECN Based Approach to Detect Shared Bottleneck in MPTCP
- Optimal Dynamic Allocation of Collaborative Servers in Two Station Tandem Systems
  IEEE Transactions on Automatic Control, Vol. 64, No. 4
- Computation of weighted sums of rewards for concurrent MDPs
  31 October 2018 | Mathematical Methods of Operations Research, Vol. 89, No. 1
- Protocol invariance and the timing of decisions in dynamic games
  1 January 2019 | Theoretical Economics, Vol. 14, No. 2
- A novel use of value iteration for deriving bounds for threshold and switching curve optimal policies
  28 December 2018 | Naval Research Logistics (NRL), Vol. 65, No. 8
- Assigning multiple job types to parallel specialized servers
  17 May 2018 | Discrete Event Dynamic Systems, Vol. 28, No. 4
- Optimization of Sequential Decision Making for Chronic Diseases: From Data to Decisions
  Brian T. Denton
  19 October 2018
- Harvesting Short-Lived White Spaces via Opportunistic Traffic Offloading Between Mobile Service Providers
  IEEE Transactions on Cognitive Communications and Networking, Vol. 4, No. 3
- Power Management Policy for Heterogeneous Data Center Based on Histogram and Discrete-Time MDP
  Electronic Notes in Theoretical Computer Science, Vol. 337
- Customizing exponential semi-Markov decision processes under the discounted cost criterion
  European Journal of Operational Research, Vol. 266, No. 1
- TWO-CLASS ROUTING WITH ADMISSION CONTROL AND STRICT PRIORITIES
  13 June 2017 | Probability in the Engineering and Informational Sciences, Vol. 32, No. 2
- Scheduling Markovian PERT networks to maximize the net present value: New results
  Operations Research Letters, Vol. 46, No. 2
- Detecting anomalous patterns in time-series data using sparse hierarchically parameterized transition matrices
  21 April 2016 | Pattern Analysis and Applications, Vol. 20, No. 4
- Algorithm 972
  9 January 2017 | ACM Transactions on Mathematical Software, Vol. 43, No. 3
- Joint dynamic pricing and capacity control for hotels and rentals with advanced demand information
  Operations Research Letters, Vol. 45, No. 5
- Constrained total undiscounted continuous-time Markov decision processes
  Bernoulli, Vol. 23, No. 3
- Value Function Approximation in Complex Queueing Systems
  11 March 2017
- Structures of Optimal Policies in MDPs with Unbounded Jumps: The State of Our Art
  11 March 2017
- Analysis of Markov Decision Processes Under Parameter Uncertainty
  13 August 2017
- Bounded Aggregation for Continuous Time Markov Decision Processes
  13 August 2017
- Optimal decisions for continuous time Markov decision processes over finite planning horizons
  Computers & Operations Research, Vol. 77
- Scheduling Markovian PERT Networks to Maximize the Net Present Value: New Results
  SSRN Electronic Journal, Vol. 45
- Optimality of the fastest available server policy
  1 October 2016 | Queueing Systems, Vol. 84, No. 3-4
- Determining the conditions for reverse triage in emergency medical services using queuing theory
  8 November 2015 | International Journal of Production Research, Vol. 54, No. 11
- Optimal control of a multiclass queueing system when customers can change types
  28 November 2015 | Queueing Systems, Vol. 82, No. 3-4
- Containment of socially optimal policies in multiple-facility Markovian queueing systems
  21 December 2017 | Journal of the Operational Research Society, Vol. 67, No. 4
- Activity imputation for trip-chains elicited from smart-card data using a continuous hidden Markov model
  Transportation Research Part B: Methodological, Vol. 83
- Optimality of the Fastest Available Server Policy
  SSRN Electronic Journal, Vol. 61
- On the Control of a Queueing System with Aging State Information
  4 August 2015 | Stochastic Models, Vol. 31, No. 4
- A congestion control scheme based on probabilistic packet acceptance and drop in delay‐tolerant networks
  27 January 2015 | Concurrency and Computation: Practice and Experience, Vol. 27, No. 13
- PH-graphs for analyzing shortest path problems with correlated traveling times
  Computers & Operations Research, Vol. 59
- Property-Driven Design for Robot Swarms
  19 December 2014 | ACM Transactions on Autonomous and Adaptive Systems, Vol. 9, No. 4
- Optimal control of noncollaborative servers in two‐stage tandem queueing systems
  31 July 2014 | Naval Research Logistics (NRL), Vol. 61, No. 6
- Multi-period marketing-mix optimization with response spike forecasting
  IBM Journal of Research and Development, Vol. 58, No. 5/6
- On structural properties of the value function for an unbounded jump Markov process with an application to a processor sharing retrial queue
  14 July 2013 | Queueing Systems, Vol. 76, No. 4
- Daily activity pattern recognition by using support vector machines with multiple classes
  Transportation Research Part B: Methodological, Vol. 58
- Flexible Server Allocation and Customer Routing Policies for Two Parallel Queues When Service Rates Are Not Additive
  Hyun-Soo Ahn,
  Mark E. Lewis,
  1 April 2013 | Operations Research, Vol. 61, No. 2
- Strong polynomiality of the Gass-Saaty shadow-vertex pivoting rule for controlled random walks
  1 August 2012 | Annals of Operations Research, Vol. 201, No. 1
- A Queueing Model to Evaluate the Impact of Patient “Batching” on Throughput and Flow Time in a Medical Teaching Facility
  Gregory Dobson,
  Hsiao-Hui Lee,
  Arvind Sainathan,
  Vera Tilson,
  13 April 2012 | Manufacturing & Service Operations Management, Vol. 14, No. 4
- The Transformation Method for Continuous-Time Markov Decision Processes
  2 March 2012 | Journal of Optimization Theory and Applications, Vol. 154, No. 2
- Finite horizon analysis of infinite CTMDPs
- Reduction of Discounted Continuous-Time MDPs with Unbounded Jump and Reward Rates to Discrete-Time Total-Reward MDPs
  12 July 2012
- Numerical analysis of continuous time Markov decision processes over finite horizons
  Computers & Operations Research, Vol. 38, No. 3
- Structured Optimal Policies for M arkov Decision Processes: Lattice Programming Techniques
  15 February 2011
- Uniformization in M arkov Decision Processes
  15 February 2011
- Dynamic control of a single-server system with abandonments
  16 November 2010 | Queueing Systems, Vol. 67, No. 1
- Discounted Continuous-Time Markov Decision Processes with Unbounded Rates: The Convex Analytic Approach
  SIAM Journal on Control and Optimization, Vol. 49, No. 5
- Time-Bounded Reachability Probabilities in Continuous-Time Markov Decision Processes
- An Approximation Approach for the Deviation Matrix of Continuous-Time Markov Processes with Application to Markov Decision Theory
  Nicole Leder,
  Bernd Heidergott,
  Arie Hordijk,
  26 February 2010 | Operations Research, Vol. 58, No. 4-part-1
- Control of Admission and Routing in Loss Networks: Hybrid Dynamic Programming Equations
  IEEE Transactions on Automatic Control, Vol. 55, No. 2
- Average Continuous Control of Piecewise Deterministic Markov Processes
  SIAM Journal on Control and Optimization, Vol. 48, No. 7
- A Strongly Polynomial Algorithm for Controlled Queues
  Alexander Zadorojniy,
  Guy Even,
  Adam Shwartz,
  7 October 2009 | Mathematics of Operations Research, Vol. 34, No. 4
- Scheduling projects with stochastic activity duration to maximize expected net present value
  European Journal of Operational Research, Vol. 198, No. 3
- OPTIMAL PRICING AND PRODUCTION POLICIES OF A MAKE-TO-STOCK SYSTEM WITH FLUCTUATING DEMAND
  16 February 2009 | Probability in the Engineering and Informational Sciences, Vol. 23, No. 2
- Heuristics for allocation of reconfigurable resources in a serial line with reliability considerations
  7 April 2008 | IIE Transactions, Vol. 40, No. 6
- Optimal Dynamic Production and Inventory Transshipment Policies for a Two-Location Make-to-Stock System
  Hui Zhao,
  Jennifer K. Ryan,
  Vinayak Deshpande,
  1 April 2008 | Operations Research, Vol. 56, No. 2
- OPTIMAL CONTROL OF FLEXIBLE SERVERS IN TWO TANDEM QUEUES WITH OPERATING COSTS
  18 December 2007 | Probability in the Engineering and Informational Sciences, Vol. 22, No. 1
- Stability Criteria for Multi-class Queueing Networks with Re-entrant Lines
  27 July 2007 | Methodology and Computing in Applied Probability, Vol. 9, No. 3
- Optimal use of excess capacity in two interconnected queues
  23 September 2006 | Mathematical Methods of Operations Research, Vol. 65, No. 1
- Dynamic vs static pricing in a make-to-stock queue with partially controlled production
  18 August 2006 | OR Spectrum, Vol. 29, No. 2
- A survey of recent results on continuous-time Markov decision processes
  TOP, Vol. 14, No. 2
- Emergency transshipment in decentralized dealer networks: When to send and accept transshipment requests
  3 May 2006 | Naval Research Logistics (NRL), Vol. 53, No. 6
- Optimal Service Policy for a Workstation Subject to Breakdown
- Stability criteria for controlled queueing systems
  Queueing Systems, Vol. 52, No. 3
- On the control of a truncated general immigration process through the introduction of a predator
  Journal of Applied Mathematics and Decision Sciences, Vol. 2006
- Optimal Server Allocation in Wireless Networks: The Use of Index Policies
- Capacity Management in Rental Businesses with Two Customer Bases
  Sergei V. Savin,
  Morris A. Cohen,
  Noah Gans,
  Ziv Katalan,
  1 August 2005 | Operations Research, Vol. 53, No. 4
- Markov decision Processes with fractional costs
  IEEE Transactions on Automatic Control, Vol. 50, No. 5
- An Optimal Production and Shutdown Strategy when a Supplier Offers an Incentive Program
  Xiuli Chao,
  Frank Y. Chen,
  1 April 2005 | Manufacturing & Service Operations Management, Vol. 7, No. 2
- Computation of the Optimal Policy for the Control of a Compound Immigration Process through Total Catastrophes
  Methodology and Computing in Applied Probability, Vol. 7, No. 1
- Modeling Medical Treatment Using Markov Decision Processes
- Optimal Control of Ergodic Continuous-Time Markov Chains with Average Sample-Path Rewards
  SIAM Journal on Control and Optimization, Vol. 44, No. 1
- Continuous Time Discounted Jump Markov Decision Processes: A Discrete-Event Approach
  Eugene A. Feinberg,
  1 August 2004 | Mathematics of Operations Research, Vol. 29, No. 3
- Optimal control and performance analysis of an M X /M/1 queue with batches of negative customers
  15 April 2004 | RAIRO - Operations Research, Vol. 38, No. 2
- Neural Approximation of Resource Allocation Functions in Multiservice Access Multiplexers
- On the control of a compound immigration process through total catastrophes
  European Journal of Operational Research, Vol. 147, No. 3
- Analysis, Design, and Control of Queueing Systems
  Shaler Stidham,
  1 February 2002 | Operations Research, Vol. 50, No. 1
- Optimal Maintenance Problems for Markovian Deteriorating Systems
- Optimal control of average reward constrained continuous-time finite Markov decision processes
- Optimal Operating Policies for Multiplant Stochastic Manufacturing Systems in a Changing Environment
  Lode Li,
  Evan L. Porteus,
  Hongtao Zhang,
  1 November 2001 | Management Science, Vol. 47, No. 11
- State dependent pricing with a queue
  IIE Transactions, Vol. 33, No. 10
- On optimal call admission control in resource-sharing system
  IEEE Transactions on Communications, Vol. 49, No. 9
- A note on optimality conditions for continuous-time Markov decision processes with average cost criterion
  IEEE Transactions on Automatic Control, Vol. 46, No. 12
- An optimal maintenance policy for a server with decreasing arrival rate
  Mathematical and Computer Modelling, Vol. 31, No. 10-12
- Optimal Control of Markov Chains
- Optimal control of a truncated general immigration process through total catastrophes
  14 July 2016 | Journal of Applied Probability, Vol. 36, No. 2
- Airline Yield Management with Overbooking, Cancellations, and No-Shows
  Janakiram Subramanian,
  Shaler Stidham,
  Conrad J. Lautenbacher,
  1 May 1999 | Transportation Science, Vol. 33, No. 2
- M/M/1 Queues with Switching Costs and Hysteretic Optimal Control
  M. Yu. Kitaev,
  Richard F. Serfozo,
  1 April 1999 | Operations Research, Vol. 47, No. 2
- Monotone Optimal Policies for Left-Skip-Free Markov Decision Processes
- Structural results for the control of queueing systems using event-based dynamic programming
  Queueing Systems, Vol. 30, No. 3-4
- An optimal maintenance policy for a server with decreasing arrival rate and non‐cancelable customer
  Journal of Quality in Maintenance Engineering, Vol. 4, No. 4
- Bibliography
  27 May 2008
- Continuous time markov decision processes with nonuniformly bounded transition rate: expected total rewards
  Optimization, Vol. 43, No. 3
- On submodular value functions and complex dynamic programming
  Communications in Statistics. Stochastic Models, Vol. 14, No. 5
- An adaptive neural network admission controller for dynamic bandwidth allocation
  IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics), Vol. 28, No. 4
- The deviation matrix of the M/M/1/∞ and M/M/1/N queue, with applications to controlled queueing models
- Assigning a single server to inhomogeneous queues with switching costs
  Theoretical Computer Science, Vol. 182, No. 1-2
- Control of multirate synchronous streams in hybrid TDM access networks
  IEEE/ACM Transactions on Networking, Vol. 5, No. 2
- Queues with switchover - A review and critique
  Annals of Operations Research, Vol. 69, No. 0
- Scheduling a repairman in a finite source system
  Mathematical Methods of Operations Research, Vol. 44, No. 3
- Optimal control of polling models for transportation applications
  Mathematical and Computer Modelling, Vol. 23, No. 11-12
- Coordinated production and inspection in a tandem system
  Queueing Systems, Vol. 24, No. 1-4
- A Congestion Control Mechanism for Connectionless Services offered by ATM Networks
- Optimal control for dynamic bandwidth allocation in communication networks: a neural approach
- An assignment problem for a parallel queueing system with two heterogeneous servers
  Mathematical and Computer Modelling, Vol. 22, No. 10-12
- Call admission control and bandwidth allocation in a multiservice DQDB network
  Computer Communications, Vol. 18, No. 8
- On arrival driven queueing models: Admission control, traffic policing, abandonments, and correlated arrivals
  Applied Mathematics and Computation, Vol. 70, No. 1
- Modeling and control of isochronous and asynchronous traffic flows in a metropolitan area network
- Optimal buffer sharing
  IEEE Journal on Selected Areas in Communications, Vol. 13, No. 7
- Neural approximations of optimal allocation policies for hybrid multiplexing
- Control and scheduling in a two-station queueing network: Optimal policies and heuristics
  Queueing Systems, Vol. 18, No. 3-4
- Look-back policies for two-stage, pull-type production/inventory systems
  Annals of Operations Research, Vol. 48, No. 4
- Bibliography
  27 May 2008
- Discrete-Time Controlled Markov Processes with Average Cost Criterion: A Survey
  SIAM Journal on Control and Optimization, Vol. 31, No. 2
- Optimal Transient Service Strategies for Adaptive Heterogeneous Queuing Systems
- Optimal use of an extra server in a two station tandem queueing network
  IEEE Transactions on Automatic Control, Vol. 38, No. 8
- Optimal service control of a station connected with two parallel substations
  IEEE Transactions on Automatic Control, Vol. 38, No. 10
- On arrival driven queueing models: correlated arrivals, abandonments, traffic policing and admission control
- A graph formulation of some supervisory control problems
- Approximate uniformization for continuous-time Markov chains with an application to performability analysis
  Stochastic Processes and their Applications, Vol. 40, No. 2
- Monotonicity properties of cost functions in queueing networks
  Discrete Event Dynamic Systems, Vol. 1, No. 3
- Impulsive and continuously acting control of jump processes-time discretization
  4 April 2007 | Stochastics and Stochastic Reports, Vol. 36, No. 3-4
- Lock and no-lock mortgage plans: Is it only a matter of risk shifting?
  Operations Research Letters, Vol. 10, No. 4
- Simultaneous optimization of flow control and scheduling in a single server queue with two job classes
  Operations Research Letters, Vol. 10, No. 2
- Monotonicity of performance measures in a processor sharing queue
  Performance Evaluation, Vol. 12, No. 1
- Simultaneous optimization of flow-control and scheduling in a single server queue with two job classes: Numerical results and approximation
  Computers & Operations Research, Vol. 18, No. 4
- Some models of queueing control with switchover
  Computers & Mathematics with Applications, Vol. 21, No. 11-12
- Optimal allocation of customers in a two server queue with resequencing
  IEEE Transactions on Automatic Control, Vol. 36, No. 11
- Throughput maximization in a loss queueing system with heterogeneous servers
  14 July 2016 | Journal of Applied Probability, Vol. 27, No. 03
- Throughput maximization in a loss queueing system with heterogeneous servers
  14 July 2016 | Journal of Applied Probability, Vol. 27, No. 3
- Optimal Control and Trunk Reservation in Loss Networks
  27 July 2009 | Probability in the Engineering and Informational Sciences, Vol. 4, No. 2
- Chapter 11 Queueing networks
- Constrained admission control to a queueing system
  1 July 2016 | Advances in Applied Probability, Vol. 21, No. 2
- Modeling and dynamic scheduling of a queueing system with blocking and starvation
  IEEE Transactions on Communications, Vol. 37, No. 12
- Optimal service assignment in a finite-source queue
  IEEE Transactions on Automatic Control, Vol. 34, No. 1
- Optimal maintenance policies for constantly monitored systems
  Naval Research Logistics, Vol. 35, No. 4
- Scheduling, Routing, and Flow Control in Stochastic Networks
- Adaptive Markov and semi-Markov decision processes: a weak contrast function approach
- Uniformization for semi-Markov decision processes under stationary policies
  14 July 2016 | Journal of Applied Probability, Vol. 24, No. 3
- Optimal control of service rates in networks of queues
  1 July 2016 | Advances in Applied Probability, Vol. 19, No. 1
- Optimal control of a queueing system with an exponential and an Erlangian server and renewal input stream
  27 June 2007 | Optimization, Vol. 18, No. 6
- A new condition for the existence of optimal stationary policies in average cost Markov decision processes
  Operations Research Letters, Vol. 5, No. 1
- Optimal dynamic routing in Markov queueing networks
  Automatica, Vol. 22, No. 3
- A Second bibliography on semi-Markov processes
- Control of arrivals to two queues in series
  European Journal of Operational Research, Vol. 21, No. 3
- Optimal control of admission to a quenching system
  IEEE Transactions on Automatic Control, Vol. 30, No. 8
- Numerical Computation of Sojourn-Time Distributions in Queuing Networks
  20 September 1984 | Journal of the ACM, Vol. 31, No. 4
- Optimal control of a queueing system with two heterogeneous servers
  IEEE Transactions on Automatic Control, Vol. 29, No. 8
- Continuous‐time markov decision processes with nonzero terminal reward
  10 October 2006 | Naval Research Logistics Quarterly, Vol. 31, No. 2
- Optimal control of two interacting service stations
  IEEE Transactions on Automatic Control, Vol. 29, No. 6
- Optimal service-rate control of M/G /1 queueing systems using phase methods
  1 July 2016 | Advances in Applied Probability, Vol. 15, No. 3
- Optimal admission pricing and service rate control of an M[x] /M/ s queue with reneging
  21 November 2006 | Naval Research Logistics Quarterly, Vol. 30, No. 2
- Applying the method of phases in the optimization of queuing systems
  1 July 2016 | Advances in Applied Probability, Vol. 14, No. 1
- Dominant Strategies in Stochastic Allocation and Scheduling Problems
- Computing the discounted return in markov and semi‐markov chains
  21 November 2006 | Naval Research Logistics Quarterly, Vol. 28, No. 4
- Monotone optimal preventive maintenance policies for stochastically failing equipment
  5 July 2007 | Naval Research Logistics Quarterly, Vol. 28, No. 3
- Optimal control of price through restricted production
  5 July 2007 | Naval Research Logistics Quarterly, Vol. 28, No. 3
- Action-dependent stopping times and Markov decision process with unbounded rewards
  1 September 1981 | Operations-Research-Spektrum, Vol. 3, No. 3
- Optimal control of random walks, birth and death processes, and queues
  1 July 2016 | Advances in Applied Probability, Vol. 13, No. 1
- Optimal admission pricing policies for M/E k /1 queues
  21 November 2006 | Naval Research Logistics Quarterly, Vol. 27, No. 1
- On theory and algorithms for Markov decision problems with the total reward criterion
  8 May 1979 | Operations-Research-Spektrum, Vol. 1, No. 1

Volume 27, Issue 3

May-June 1979

Pages 431-627

Article Information

Metrics

Information

Published Online:June 01, 1979

Cite as

Richard F. Serfozo, (1979) Technical Note—An Equivalence Between Continuous and Discrete Time Markov Decision Processes. Operations Research 27(3):616-620.

https://doi.org/10.1287/opre.27.3.616

PDF download

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Technical Note—An Equivalence Between Continuous and Discrete Time Markov Decision Processes

Volume 27, Issue 3

Article Information

Metrics

Information

Cite as

Sign Up for INFORMS Publications Updates and News