Quasi-feller Markov Chains

We consider the class of Markov kernels for which the weak or strong Feller property fails to hold at some discontinuity set. We provide a simple necessary and sufficient condition for existence of an invariant probability measure as well as a Foster-Lyapunov sufficient condition. We also characterize a subclass, the quasi (weak or strong) Feller kernels, for which the sequences of expected occupation measures share the same asymptotic properties as for (weak or strong) Feller kernels. In particular, it is shown that the sequences of expected occupation measures of strong and quasi strong-Feller kernels with an invariant probability measure converge set-wise to an invariant measure. Measures.


Introduction
We consider a Markov chain on a locally compact separable metric space.A common assumption when studying a Markov chain is that the stochastic kernel of transition probabilities is (weak) Feller, i.e., it maps the space of bounded continuous functions into itself (whereas the strong-Feller kernels map the bounded measurable functions into bounded continuous functions).Indeed, under such an (easy to check) assumption, various properties can be derived for the long-run behavior of the Markov chain.In particular, for the existence of invariant probability measures, simple necessary and sufficient conditions are available (for instance of.[9,7]).In addition, a nice property of (weak) Feller kernels is that every weak* limit point of the expected occupation measures is a (possibly trivial) invariant measure.
However, it may happen that the transition kernel fails to have the (weak or strong) Feller property at some points in an exceptional set.In some cases, this pathology is serious in that it prevents the kernel from having the above mentioned 1part of this work was done during a visit at Stanford University.author wishes to thank P. Glynn and A.F. Veinott for their kind hospitality.properties of (weak) Feller kernels.The example in Section 2 illustrates the dramatic consequences if the (weak) Feller property fails at a single point only, even on a com- pact metric space.In some other cases, the kernel behaves practically as a (weak) Feller kernel, i.e., the above pathology is not serious.Therefore, having a means to distinguish between those two types of kernels is highly desirable.
A practical example of interest, which motivated this work, is the important class of Generalized Semi-Markov Processes (GSMP), which permits modeling of the essen- tial dynamical structure of a discrete-event system (cf.[6]).Indeed, a time-homogen- eous GSMP can be studied via Markov chain techniques, particularly its long-run be- havior via ergodic theorems (cf.[6]).However, the (discrete-time) associated Markov kernel is not (weak) Feller as discontinuities occur when (at least) two "clocks" run out of time simultaneously.See also the threshold models in Tong [10].It is thus ne- cessary to provide conditions of existence of an invariant probability measure for such pathological kernels.
In the present paper, we propose such conditions which are in fact a simple exten- sion of the ones in [7] for (weak) Feller kernels.In addition, we characterize a class of kernels, the quasi-Feller kernels, i.e., those kernels with a discontinuity set but with the same properties as (weak) Feller kernels.
We also prove that strong-Feller kernels enjoy an additional nice property; name- ly, if the transition kernel has an invariant probability measure #, then #-a.e., the se- quence of expected occupation measures converges setwise to an invariant probability measure, in contrast to the (only)weak convergence for (weak) Feller kernels.The corresponding class of quasi strong-Feller kernels also has this property.Finally, the necessary and sufficient condition for existence of a unique invariant probability mea- sure proposed in [8] also applies to a quasi-Feller kernel.

Notation and Definitions
Let (X,%) be a measurable space with X a locally compact separable metric space and % its usual Borel r-field.Denote by M(X), the Banach space of signed Borel measures on (X, %) endowed with the total variation form.
Cb(X), the Banach space of bounded continuous functions on X, endowed with the sup-norm.
Co(X C Cb(X), the Banach space of continuous functions on X that vanish at infinity, endowed with the sup-norm.By "vanish at infinity", we mean that f E Co(X if SUPx 6 K f(x) -O whenever K n compact and KnTX.
B(X), the Banach space of bounded measurable functions on X, endowed with the sup-norm.
Let P be the transition probability kernel of a Markov chain with values in X, i.e., P(x,.) is a probability measure on (X,%) for every x e X, and P(.,B)is a mea- surable function on X for every B %.
P is a (weak) Feller kernel if PIe Cb(X whenever I e Cb(X).Note that, as X is a locally compact separable metric space, this condition is in fact equivalent to the apparently weaker condition PI or_ Cb(X whenever f C: Co(X).P is (strong) Feller if Pf e Cb(X whenever f e B(X).
A measure # E M(X) is invariant if and only if #(B)-/P(x,B)#(dx) BE%. (1) 3. Quasi-Feller Markov Kernels Let P be a transition kernel for which the (weak) Feller property fails at some points x G D C X.It is important to note that even if this property fails at only a single point, it can have dramatic consequences, as shown in the following elementary exam- ple on a compact metric space" X: [0, 1], P(x, {x/2}) lVx :/= 0, P(0, {1}) 1.
It is trivial to check that the above kernel has no invariant probability measure, despite X being compact, and the kernel is "almost Feller."In addition to not being Feller, the necessary and sufficient condition of existence stated in [7], namely, t--0 for some z X and some arbitrary 0 < fo Co(X), is not valid (and similarly, equivalent conditions that use compact sets in Beneg [1] and Foguel [5]).Indeed, for a fixed arbitrary 0 < fo Co(X), we have n-1 nlirn n 1Ex E fo(X, -nlkrnf0 (2 nx) f0 (0) > 0, t--O so that (2) holds, but there is no i.p.m.This example contradicts the conjecture in [3] that if the property fails at a finite number of points, then the kernel is still well-behaved.Another example is the oscil- where {n} and {n} are two unrelated sequences of independent equally distributed random variables.
The corresponding transition kernel is (weakly) continuous except at x-0.How- ever, if for instance Prob({ 1 V/)-pl 1-Prob( -1)and Prob( 1= -)-P2 1-Prob( 1), the n-step probability distribution converges weakly to an invariant uniform probability distribution.In this case, the discontinuity at zero does not prevent the existence of an i.p.m.
Let D be the set of discontinuity of the transition kernel, i.e., x DC=Pf(xn)-Pf(x whenever f Cb(X and xn--x and let Y: X-D be the subspace of X with the usual induced topology and %' its usual Borel e-field.In many cases of interest, D E % and P(x,D) 0Vx E X-D and P(x, X-D) > 0Vx D. This can be checked easily on the above two examples and also in the GSMP models in [6].
3.1 Existence of an Invariant Probability Measure We now state a necessary and sufficient condition for existence of an invariant proba- bility measure for Markov kernels with a discontinuity set D.
Theorem 3.1: Assume that D G % is closed, P(x,D) O Vx G X-D and Pn(x,D) < I Vx G D for some n >_ l.Let O <_ fo E Co(X) be fixed arbitrary and such that f o vanishes on D and is strictly positive elsewhere.Then, P has an invar- iant probability measure (i.p.m) if and only if n-1 linm__.suopExn -1 E fo(Xt > 0 (3) t=0 for some x X-D.
Proof: Let Y: =X-D.As asubset of X, X-Dis open and thus Y, with the topology induced by X, is a locally compact separable metric space and its usual Borel r-field coincides with %': {% fl ctJl% %}.The Banach space Co(Y of continuous functions that vanish at "infinity" is the subset of functions in Co(X that vanish on D. In addition, M(Y), the Banach space of finite Borel sign measures on (Y, N'), is the topological dual of Co(Y).Now, let P' be the restriction of P on Y, i.e., P'(x,B)" P(x,B) whenever x e Y, B e %'.
It is trivial to check that P' is weak Feller.Moreover, from every initial state x E Y, the Markov chain stays in Y with probability 1.Therefore, the Markov chain induced by P' coincides with the original chain for every initial state x Y.For every x Y, one may use indifferently P'(x,.or P(x,.).
Let # be an invariant probability measure for P. From P(x,D)-O Vx Y, pn(x, Y) > 0 Vx D for some n >_ 1, and the invariance of #, we also have #(D) 0, i.e., # E M(Y) and is an invariant probability measure for P', it is also invariant for P.
Therefore, one may apply directly to P' the necessary and sufficient condition for existence on an i.p.m, given in [7] which is (3) with "lim" instead of "limsup" and the operator E x instead of E x.However, since with initial state x Y the Markov chain induced by P' coincides with the one induced by P, one may replace E by E x.
Also a simple examination of the proof of Theorem 2.1 in [7] shows that one may use indifferently "lim", "liminf" or "limsup".
We now give other properties, using limits of the expected occupation measures.Let Y), e Y.
x For every x X, fixed arbitrary, {#} is a sequence of probability measures on (X,%) and for every x Y, fixed arbitrary, {Ur} is a sequence of probability measures on (Y, ').
It is important to note that the weak* convergence in M(Y) is not the same as the weak* convergence in M(X).
Lemma 3.2: Let n be closed with P(x,D)-0 Vx X-D and Pn(x,n)<1 Vx D for some n _ l.Then, (a) For every x G Y, fixed arbitrary, every weak* accumulation point of the sequence {Ux n} in M(Y) is a (possibly trivial) invarianl measure u x e M(Y). (b) If P has an invariant probability measure #, then # i(Y) and u x is an i.p m for P, #-a e in Y.In addition, x x Un=U #-i.e.(c) Let #xE M(X) be a weak* accumulation point in M(X) of the sequence x {#n}, x Y. Then ux, the restriction of #x to (Y,%'), is a weak* accumu- lation point in M(Y) of {u}, and therefore, an invariant measure for P. Hence, #x is an invariant measure if and only if #Z(D)-O.
(a) As Y is a locally compact separable metric space, the unit ball in Hence, consider an arbitrary (weak*) From uP u,x + n ((P')' 6) (with Proof: M(Y) is weak* sequentially compact.
convergent subsequence u x nk---u x M(Y). 6x the Dirac at x), we conclude that i x, lim f d(UnkP )-f du x for every f e Co(Y).(6) On the other hand, as P' is weak Feller, P'f Cb(Y and for every 0 _ f Co(Y I "' I J S lim f d(unkP lim (P'f )duk > (P'f)du x fd(uXP').
(b) Assume that # is an invariant probability measure for P, hence for P', i.e., # E M(Y).From the Birkhoff Individual and Mean Ergodic Theorem (cf. [11]), we have, for every f LI (Y %', #), #-a.e.In addition, f f'd# f fd#.On the other hand, f*(x) =nlirn ] fdu so that for every f C0(Y), and an arbitrary weak* accumulation point u x of {u}, we have #-a.e.
f*(x) f f dux. (9 This in urn implies V f e Co(Y).
As It is a probability measure, this implies that It-a.e. uz(Y) 1, i.e., u z is a probabi- lity measure It-a.e.By the Portmanteau Theorem (cf.[2]), we also conclude that It- a.e.every weak* accumulation point of {ur} is also a weak accumulation point.
As Co(Y is separable, it contains a countable dense subset F: {fl,'",} C Co(Y).For each f F there is a set N l with It(N l) 0 such that, from (9), Hence, as It( U ] FNI) 0 and r is dense in Co(Y), As (10) holds for every weak* (hence weak) accumulation point and every f E Co(Y), all the weak limit points ux are identical It-a.e., i.e.It-a.e., It,ux .
(c) Let Itz M(X) be a weak* accumulation point in M(X)of {ItS}, i.e., there is some subsequence {It,k } such that lim / x / f V f Co(X)" (11) k--, f dItnk Now, for every x Y and f e Co(Y f fdItk is just f fduZn.Therefore, since Co(Y) C Co(X), to every weak* accumulation point Itx E M(X) of {It,} corresponds to a weak* accumulation point M(Y) of {}.From ffdItf fdu for every f Co(Y), we conclude that the restriction of It to (Y,%') is just x, and, from (b), is an invariant measure.In addition, as P(x,D)=O Yx Y and Pn(x,D)<I VxED for some n>l, #z is invariant only if f DPn(y,D)#X(dy)< pX(D), i.e.only if #X(D)-O.On the other hand, if #X(D)-0 then as the restriction of # to X D is an invariant measure, and P(x, D) 0 Vx X-D, then so is #z.
The last statement suggests the following, definition of a quasi-Feller Markov kernel with discontinuity set D.
Definition: If P has a closed (weak) discontinuity set D % with P(x,D)-0 Vx X-D and P'(x,D)< 1 Yx D for some n > 1, then P is said to be quasi- Feller if every weak* accumulation point # M(X) of the sequence of expected occupation measures {#}, x E X-D, satisfies #(D) 0.
For such kernels, every weak* accumulation point #x of {#} (x (possibly trivial) invariant measure (with #X(D)-0) as for (weak) Feller kernels, which justifies the label "quasi-Feller." In the first example, one may easily check that P is not quasi-Feller since for every x X, #x_ 50 and thus, #x({0})-1.On the other hand, in the second example, #x({0})-0 so that P is quasi-Feller.A sufficient condition for P to be quasi-Feller is as follows" Corollary 3.3: Let D% be closed, with P(x,D)-O VxX-D and pn(x,X-D) > 0 x D for some n > 1.Let D e % be open and DD as s--O.
Then, P is quasi-Feller if for some scalar K and for every sufficiently small s > O, liminf#(D) < Ks (12) for every x X-D.
Proof: Assume that (12) holds and consider a subsequence {#k } that converges weakly* to some #z M(X).As D e is open, for sufficiently small s, we have (e.g.cf. [4]).
Letting s0 yields the desired result.Note that similar condition was given in [3] (cf. condition III(a), p. 546).

A Lyapunov Condition of Existence
A sufficient condition of existence, although stronger than necessary and sufficient condition, is also useful and sometimes easier to manipulate.The condition below is a Lyapunov-type condition for non-Feller kernels with a discontinuity set D.
Corollary 3.4: Assume that D % is closed, with P(x,D)-0 ktx X-D and pn(x,X-D) > 0 Vx D for some n > 1.Let 0 < fo Co(Y) be fixed arbitrary.
(a) If there exists a nonnegative measurable finite function f and a scalar 0 < ) such that Pf(x) < f(x)-l + $fo(x,) x e Y, (13) then there exists an i.p.m. # M(Y).
If there exist a nonnegative measurable finite function f, a compact set K C Y and a scalar 0 < such that Printed in the U.S.A. (C)2000 by North Atlantic Science Publishing Company 15