Stochastic Recursive Inclusions in Two Timescales with Nonadditive Iterate-Dependent Markov Noise

In this paper, we study the asymptotic behavior of a stochastic approximation scheme on two timescales with set-valued drift functions and in the presence of nonadditive iterate-dependent Markov noise. We show that the recursion on each timescale tracks the flow of a differential inclusion obtained by averaging the set-valued drift function in the recursion with respect to a set of measures accounting for both averaging with respect to the stationary distributions of the Markov noise terms and the interdependence between the two recursions on different timescales. The framework studied in this paper builds on a recent work by Ramaswamy and Bhatnagar, by allowing for the presence of nonadditive iterate-dependent Markov noise. As an application, we consider the problem of computing the optimum in a constrained convex optimization problem, where the objective function and the constraints are averaged with respect to the stationary distribution of an underlying Markov chain. Further, the proposed scheme neither requires the differentiability of the objective function nor the knowledge of the averaging measure.