An Informal Look at the Principle of Optimality

The Principle of Optimality is examined informally in the context of discounted Markov decision processes. Our purpose is to illustrate that one should be invoking the optimality equations and/or the optimality criterion, rather than the Principle of Optimality in analyzing dynamic models. A counterexample to one interpretation of the Principle is given. It involves a foolish action at the second stage from a state that can be reached, but with probability zero. Redefining optimality as in Hinderer [Hinderer, K. 1970. Foundations of Non-stationary Dynamic Programming with Discrete Time Parameter. Springer-Verlag, New York.], restores the Principle, at the cost of a weaker notion of optimality.