Decision Problems with Expected Utility Criteria, II: Stationarity

The study of sequential decision problems with expected utility criteria is continued. Stationary problems are defined similarly to stationary problems with additive or separable utility, except that an additional requirement, stationarity of the preference structure, is imposed. The optimality of stationary and memoryless strategies is investigated for problems with upper and lower convergent utility or which are upper or lower transient. By example, the definition of stationarity is seen to be inadequate for generalizations of “average reward” criteria. Simplifications in the strategy iteration algorithm which result from stationarity are discussed.