Optimal Control of Partially Observable Semi-Markovian Failing Systems: An Analysis Using a Phase Methodology

We formulate a maintenance control model as an optimal stopping problem under partial observations. The key challenge in our formulation is that the underlying state process is not restricted to be Markovian but rather is allowed to follow a semi-Markov process, which is more realistic in practice. Consequently, the stopping problem is not representable as a partially observable Markov decision process (POMDP) with finite state space, a commonly adopted modeling framework in the maintenance optimization literature; it constitutes a partially observable semi-Markov decision process (POSMDP), a problem class that in general is both computationally intractable and not amenable to structural analysis. In this paper, we develop a new analysis approach based on a phase methodology where the idea is to view the intractable POSMDP as the limiting problem of a sequence of tractable POMDPs. We show how this approach allows us to (i) characterize the structural form of the optimal policy and (ii) efficiently compute the optimal policy and associated optimal value via successive approximation. In particular, we show that the optimal control policy can be represented as a control limit policy that monitors the estimated conditional reliability at each decision epoch, and, by exploiting this structure, we develop an efficient computational approach to solve for the optimal control limit and corresponding optimal value. Numerical comparisons are provided that show substantial improvement over existing policies.