Subgeometric Ergodicity Analysis of Continuous-Time Markov Chains under Random-Time State-Dependent Lyapunov Drift Conditions

The ergodic theory of Markov processes in recent years has received quite substantial attention. With our focus on subgeometric ergodicity, which, loosely speaking, is a kind of convergence that is faster than ordinary ergodicity but slower than geometric ergodicity, much study is needed especially for continuous Markov processes [1]. It is for this reason that interest in this study developed. Meyn and Tweedie [2] pioneered the study of statedependent drift for Ψ-irreducible Markov processes. Then Connor and Fort [3] found that state-dependent FosterLyapunov criteria can be employed not only to determine the ergodic properties of theMarkov processΦ t but also to infer a greater range of convergence rates for Φ t . The study focused on the process Φ t subsampled at some deterministic times. In [3] the results of the study were used for the classification of tame chains. Ergodicity in the context of state-dependent drift conditions on a deterministic time index was then studied by many authors, amongst them [4, 5]. Yüksel and Meyn [4] applied their results to “stochastic stabilization over an erasure channel.”Themotivation for Zurkowski’s [5] study was its possible applications in network control and event triggered control systems. Although we cannot ignore the practical applications of our study in areas such as stability of stochastic networks, presenting such applications is beyond the scope of our work. Connor and Fort [3] and Yüksel and Meyn [4] studied ergodicity with a drift condition taking the form Pn(x)V(x) ≤ βV(x) + b1 C (x) for some deterministic function n : X → [1,∞) and a constantβ ∈ (0, 1). According toTheorem 2.1(ii) of [6] a deterministic sequence of functions V n exists, V : X → [1,∞), and satisfies a Foster-Lyapunov drift condition:


Introduction
The ergodic theory of Markov processes in recent years has received quite substantial attention.With our focus on subgeometric ergodicity, which, loosely speaking, is a kind of convergence that is faster than ordinary ergodicity but slower than geometric ergodicity, much study is needed especially for continuous Markov processes [1].It is for this reason that interest in this study developed.
Meyn and Tweedie [2] pioneered the study of statedependent drift for Ψ-irreducible Markov processes.Then Connor and Fort [3] found that state-dependent Foster-Lyapunov criteria can be employed not only to determine the ergodic properties of the Markov process Φ  but also to infer a greater range of convergence rates for Φ  .The study focused on the process Φ  subsampled at some deterministic times.In [3] the results of the study were used for the classification of tame chains.Ergodicity in the context of state-dependent drift conditions on a deterministic time index was then studied by many authors, amongst them [4,5].Yüksel and Meyn [4] applied their results to "stochastic stabilization over an erasure channel." The motivation for Zurkowski's [5] study was its possible applications in network control and event triggered control systems.Although we cannot ignore the practical applications of our study in areas such as stability of stochastic networks, presenting such applications is beyond the scope of our work.
It has been proved that this Foster-Lyapunov condition holds not just for every  ∈ Z + but also for a sequence of stopping times {T  },  ∈ Z + for some discrete time Markov chain (Φ  ) ∈Z + [5].The results of Zurkowski [5] relied heavily on the work of Connor and Fort [3], Meyn and Tweedie [7], and Tuominen and Tweedie [6].
This study follows on the work of previous studies to investigate random-time state-dependent drift conditions results on the subgeometric ergodicity for CTMCs in an easy and reader-friendly manner.Amongst the previous authors Connor and Fort [3] studied state-dependent geometric Lyapunov drift condition for a general Markov chain subsampled at some deterministic times.Random-time statedependent drift studies were then done by Yüksel and Meyn [4] and Zurkowski [5] for general Markov chains and discrete chains, respectively.The work in this study utilizes the results of the above mentioned studies with focus on subgeometric ergodicity for countable state space -processes.
The paper is organized as follows.In Section 2, we have the Preliminaries section which introduces basic notations, definitions, and theorems.The main results are availed in Section 3 which is mostly concerned with subgeometric ergodicity at rate , in the -norm.Section 4 is about the conclusions of this study.

Preliminaries
Let Z + = {0, 1, 2, . ..} and let R + = [0,∞).We let (Φ  ) ∈R + denote a continuous-time Markov chain (CTMC) on a countable state space (X, B(X)).The transition function of the continuous-time Markov process is denoted by , where here and hereafter 1  is the indicator function of set  and   and   , respectively, denote the probability and expectation of the chain under the condition that Φ 0 = .
2.1.First Hitting Times.Given a subset  ⊆ X we let be the first hitting time on  delayed by a constant  > 0. In the case when  = 0 we have  0  =   .We also have as the first hitting time on the set  after the first jump  1 of the process.We also note that  +  =   if Φ 0 ∉ .If  is a singleton consisting only of state , then we write    for    and equivalently  +  for  +  .

ℓ-Ergodicity
< ∞ for some (and then for all) finite nonempty  ⊆ X.For more studies on ergodic degrees, ℓ-ergodicity, and related topics, the interested reader may consult Mao [8] and references therein.
The properties of  ∈ Λ 0 which follow from ( 4) and are to be used frequently in this study are 2.5.Modulated Moments.Sufficient conditions for -ergodicity involve the modulated moments of the first hitting times.Having the subgeometric rate function  ∈ Λ given by ( 4), a function  : X → [1, ∞), a subset  ⊆ , and a constant  ≥ 0, we define the modulated moment of    by Analogous to the modulated moments of the first hitting times    given as (7) above, the modulated moments of the first hitting times  +  , namely,  +  (, , ), are defined as follows.For a subgeometric rate function  ∈ Λ, a function  : X → [1, ∞), and a subset  ⊆ , then 2.6.Subgeometric Rate Ergodicity.Let  ∈ Λ; then the ergodic chain Φ  is said to be subgeometric ergodic of order  in the -norm or simply (, )-ergodic if for all  ∈ X; then lim where  is the unique invariant distribution of the process for a (signed) measure , where It is known that (9) holds if and only if the Foster-Lyapunov (or just Lyapunov) drift condition holds.It is for this reason that the Lyapunov drift conditions play a very crucial role in our study.The Lyapunov drift conditions provide bounds on the return time to accessible sets thereby availing some control on the Markov process dynamics by focusing on the hitting times on a particular set.
We are aware that techniques that rely on small sets in some of the previous studies may become unavailable in the random-time drift setting because a small set for Φ  may not necessarily be small for Φ T  .However, we also know that, for a Ψ-irreducible process, all finite subsets of X are petite and that an accessible closed petite set always exists [10], suggesting our work will be confined to working with petite sets.Then by Theorem 5.5.7 in Meyn and Tweedie [7] we know that if the Markov chain is Ψ-irreducible and aperiodic, then every petite set is small.Therefore all the petite sets in this study are assumed to be small because all chains are assumed to be Ψ-irreducible and aperiodic.
In light of the foregoing definitions and notations we are ready to state the following subgeometric rate ergodicity propositions about (, 1)-ergodicity and (1, )-ergodicity.
Proposition 1 (see Proposition 2.11 in [11]).Let Φ  be an irreducible Markov process, and suppose that, for some  > 0 and some function  :  → [1,∞) on X, one has   [(Φ  )] < (), for all  ∈ X and 0 ≤  ≤ .If there exist a petite set  ⊆ X and some constant  > 0 such that Proposition 2 (see Theorem 3.3 in Liu et al. [12]).An irreducible Markov chain Φ  is subgeometrically ergodic at rate  in the total variation norm if and only if for some (and then for all) finite subset  ⊂ X one has Proposition 3 (Theorem 3.2 in Liu et al. [12]).For some finite nonempty set , sup ∈  +  (, 1, ) < ∞ if and only if for any finite set  and any  ∈ X, then  +  (, 1, ) < ∞.
Throughout this study the nondecreasing sequence of stopping times {T  :  ∈ Z + } with T 0 = 0 is assumed to be similar to the sequence in Assumption 2.1 of Spieksma [13] , which is restated as follows.Recursively we define T 0 = 0 and T +1 = inf{ > T  : Φ  ̸ = Φ − }, if Φ T  is not an absorbing state (i.e., Φ T  ̸ = 0).We put T  ,  >  if Φ T  is an absorbing state; then, we have Φ T  = Φ T  .In this sense the sequence {T  } ∈Z + is a nondecreasing sequence of stopping times representing successive jump times.

Main Results
The content presented in this section is on the subgeometric rate ergodicity of Φ  , where Φ  is assumed to be an irreducible and aperiodic CTMC.We have results presented as Theorems 4, 6, and 7. Theorem 4 is a modified version of Theorem 3.1 in [12], while Theorem 6 is a continuous counterpart of Proposition 5.1.2in Zurkowski [5].Finally we have Theorem 7 which follows from Theorem 2.16 of [11].Theorem 4. Let Φ  be an irreducible and aperiodic chain.Further suppose that for some (and then for any) finite nonempty set  ⊆ X there exist a function  :  → (0, ∞) and a constant  ∈ R such that, for an increasing sequence of stopping times {T  ,  ∈ N + }, then for any  ∈ Λ the chain Φ  is (1, )-ergodic.
Proof.It is worthwhile to realize that the chain Φ T  is also aperiodic and irreducible and that  is petite and hence small and that  is bounded on .Let   = min{ ≥ 0 : Φ T  ∈ }.
We know that   ≤   ∀ ∈ B + (X), because the chain Φ T  can miss some of the visits of the chain Φ  to the set , so that, for any  ∈ Λ, we have sup which implies that sup for any  ∈ .We get (14) owing to the equivalence of Λ 0 and Λ and supposing that  ∈ Λ 0 and hence satisfies the submultiplicative property (5).By employing Proposition 3 we get that ( 14) implies  +  (, 1, ) < ∞ for any  ∈ X.Then, by Proposition 2, the set  is (1, )-regular; hence we conclude that the chain is (1, )-ergodic.
It is worth noting that, in practice, the drift condition in Theorem 4 above is not easy to verify because it involves the unknown information  Φ T  [(  )].The practically more favorable drift condition is stated in Corollary 5.It was Mao [8] who investigated ℓ-ergodicity when ℓ is restricted to Z + ; then Liu et al. [12] extended the results to the case when ℓ ∈ R + .Under random-time state-dependent drift function, we state the following corollary which is a modified version of Corollary 2.1 of Liu et al. [12].
Proof.The proof of this corollary is analogous to the proof of Corollary 2.1 of Liu et al. [12] and follows naturally therefrom.
Remark.In a particular case of Corollary 5, the chain is said to be subgeometrically ergodic at rate () =  ℓ ∨ 1, for some ℓ ∈ Z + , in the total variation norm if and only if we have nonnegative functions  (0) , . . .,  (+1) on X, with  ⊂ X and  ∈ R such that  (0) ≥ 1, and for 0 ≤  ≤  and any Φ T  ∈ .Also the Lyapunov drift in Theorem 4 and Corollary 5 which is of the where ,  : → (0, ∞) and  ∈ R, is used here and in the rest of this paper for convenience.We note that for the matrix  = [  ] and the real-valued function  on X we have ()() := ∑ ∈   (), ∀ ∈ X. Theorem 6 which follows deals with subgeometric rate convergence of -ergodic chains, in the case when  is not a total variation norm.In a nutshell we will be dealing with (, )-regularity of the petite subset  ⊂ X. Theorem 6. Suppose that Φ  is a -irreducible and aperiodic Markov chain.Further suppose that there are functions  : X → (0, ∞) and ,  : X → [1, ∞), where  is bounded on the small set , constant  > 0,  ∈ R, and  ∈ Λ such that, for an increasing sequence of stopping times {T  ,  ∈ Z Proof.For all Φ T  ∈ X and for all stopping time T, by Dynkin's inequality, we get because  is bounded on .Again as in Theorem 4 we have that   ≤   , for all  ∈ B + (X), where    = min{ >  : Φ T  ∈ } is the sampled hitting times for the chain Φ T  .Then for all  ∈ is taken from formula (15) in [14, Section 5] , with the obvious notational changes of course.
The following theorem generalizes the results of (, )ergodicity in Theorem 6 which can be established by combining the results of Proposition 1 (i.e., (, 1)-ergodicity) with Theorem 4 (i.e., (1, )-ergodicity).This theorem is similar to Proposition 5.1.4in [5] and Theorem 2.8 in [15].Theorem 7 (borrowed from Theorem 2.16 in [11]).Let Φ  be an irreducible and aperiodic chain.Further suppose that for some (and then for any) finite nonempty set  ⊆ X there exist a function  :  → (1, ∞) and constants  and  such that, for an increasing sequence of stopping times {T  ,  ∈ Z + },  Φ T  [(Φ () )] ≤ (Φ T  ), for all Φ T  ∈ X and 0 ≤  ≤ .Then, for some  ∈ Λ and a constant  > 0,  The drift condition for the chain in Theorem 7 is not easy to find but an easier way to verify drift is given in [11,Corollary 2.17] .It is restated as the following corollary.