Stability and Linear Quadratic Differential Games of Discrete-Time Markovian Jump Linear Systems with State-Dependent Noise

We mainly consider the stability of discrete-time Markovian jump linear systems with state-dependent noise as well as its linear quadratic (LQ) differential games. A necessary and sufficient condition involved with the connection between stochastic T n stability of Markovian jump linear systems with state-dependent noise and Lyapunov equation is proposed. And using the theory of stochastic T n -stability, we give the optimal strategies and the optimal cost values for infinite horizon LQ stochastic differential games. It is demonstrated that the solutions of infinite horizon LQ stochastic differential games are concerned with four coupled generalized algebraic Riccati equations (GAREs). Finally, an iterative algorithm is presented to solve the four coupled GAREs and a simulation example is given to illustrate the effectiveness of it.


Introduction
In this paper we discuss linear systems with Markovian jump and state-dependent noise.Here the discrete-time stochastic linear systems subject to abrupt parameter changes can be modeled by a discrete-time finite-state Markov chain.They are a special sort of hybrid systems with both modes and state variables.Since the class of systems was firstly introduced in early 1960s, the hybrid systems driven by continuous-time Markov chains have been broadly employed to model many practical systems which may experience abrupt changes in system structure and parameters such as solar-powered systems, power systems, battle management in command, and control and communication systems [1][2][3][4].In the past several decades, considerable attention has been focused on the analysis and synthesis of linear systems with Markovian jump, including stability analysis, state feedback, and output feedback controller design, filter design, and so forth [5][6][7][8][9][10][11].
The stability theory of linear systems with Markovian jump and state-dependent noise, here we also say Markovian jump stochastic linear systems (MJSLS for short), is rather complex in that there exist some stability concepts.Particularly, the study of stability about these systems has attracted the attention of many researchers [12][13][14][15][16][17].The very important stability notions are mean-square stability, moment stability, and almost sure stability.Mean-square stability deals with the asymptotic convergence to zero of the second moment of the state norm.There are some necessary and sufficient conditions for mean-square stability involving either the solution of the coupled Lyapunov equations or the location in the complex plane of the eigenvalues of suitable augmented matrices [12,13].Moment stability, -moment stability, requires the convergence to zero of -moment of the state norm (mean-square stability is just a particular case for  = 2).Although there exist some practical sufficient conditions, a simple necessary and sufficient condition testing -moment stability is not available (except for  = 2).Almost sure stability holds if the sample path of the state converges to zero with probability one.The checking about almost sure stability involves the determination of the sign of the top Lyapunov exponent, which is usually a rather difficult task [14,15].Contrary to deterministic systems, for which all 2 Mathematical Problems in Engineering moments are stable whenever the sample path is stable, the moment stability for stochastic systems implies almost sure stability, but not vice versa, as pointed out by [16].
On the other hand, the LQ differential games have many applications in the economy, military, and intelligent robots.Since the book [18], entitled "Differential Games, " written by Dr. Isaacs came out, theory and application of differential games have been developed greatly.A differential game is a mathematical model that represents a conflict between different agents which control a dynamical system and each of them is trying to minimize his individual objective function by giving a control to the system.In fact, many situations in industry, economies, management, and elsewhere are characterized by multiple decision makers and enduring consequences of decisions which can be treated as dynamic games.Particularly applications of differential games are widely researched in LQ control problem [19][20][21][22][23].By solving the LQ control problems, players can avoid most of the additional cost incurred by this perturbation.The authors consider the zero-sum, infinite-horizon, and LQ differential games in [19].A sufficient condition for the LQ differential games is applied to the  ∞ optimization problem in [20].The authors in [21] study the problem of designing suboptimal LQ differential games with multiple players.In [22], the authors study the LQ nonzero sum stochastic differential games problem with random jump.A leader-follower stochastic differential game is considered with the state equation being a linear Itô-type stochastic differential equation and the cost functionals being quadratic [23].
As far as we know, there are few researchers paying attention to the discrete-time stochastic LQ differential games, especially MJSLS.Thus, it is significant to consider these systems.In [24], we have considered stochastic differential games in infinite-time horizon.By introducing stochastic exact observability and stochastic exact detectability, the optimal strategies (Nash equilibrium strategies) and the optimal cost values have been given.In [25], we have considered LQ differential games in finite horizon for discrete-time stochastic systems with Markovian jump parameters and multiplicative noise.Furthermore, a suboptimal solution of the LQ differential games is proposed based on a convex optimization approach.On the basis of [26], in the paper we further investigate the LQ differential games for discretetime MJSLS with a finite number of jump times.It gives the optimal strategies and the optimal cost values for infinite horizon LQ stochastic differential games which are associated with the four coupled GAREs.Generally, it is difficult to solve the four coupled GAREs analytically.Here, we will solve the LQ differential games by means of stochastic   -stability and we will employ a recursive procedure to solve the coupled GAREs.
The paper is organized as follows.In Section 2, some basic definitions are recalled.A necessary and sufficient condition in relation to the connection between the   -stability for MJSLS and Lyapunov equation is presented.In Section 3, we formulate LQ differential games with quadratic cost function for the MJSLS and propose the stochastic   -stability of discrete-time MJSLS with a finite number of jump times, which is essential to obtain the main results.An iterative algorithm for solving the four coupled GAREs is put forward, and an illustrative example is also displayed in Section 4. Section 5 ends this paper with some concluding remarks.
For convenience, the paper adopts the following basic notations.It uses R  to denote the linear space of all dimensional real vectors.R × denotes the linear space of all  ×  real matrices.N = {0, 1, 2, . ..}.   indicates the transpose of matrix  and  ≥ 0 ( > 0) represents a nonnegative definite matrix (positive definite).The standard vector norm in R  is indicated by ‖ ⋅ ‖ and the corresponding induced norm of matrix  by ‖‖.
where {(),   ;  ∈ N} are the states of process with values in R  × X; {  ;  ∈ N} is a time homogeneous Markov chain taking values in a finite set X = {1, 2, . . ., }, with initial distribution  and transition probability matrix P = [  ], where The set X comprises the various operation modes of the system (1).For each   =  ∈ X, the matrices    ∈ R × and    ∈ R × (associated with "th" mode) will be assigned as    :=   and    :=   in someplace of the paper.{(),  ∈ N} is a sequence of real random variables, which is also a wide sense stationary, second-order process with [()] = 0 and [()()] =   , where   refers to a Kronecker function; that is,   = 1 if  =  and   = 0 if  ̸ = .The MJSLS as defined is trivially a strong Markov process.We assume that   is independent of ().To make the operation more convenient, let () =   .When system (1) is stable, we also say (  ,   ) stable for short.
Although, several concepts of stochastic stability can be found in the literature, in this paper, the stochastic stability concept associated with the stopping times for MJSLS is researched.The stopping times in relation to jump times are defined as follows: Mathematical Problems in Engineering 3 The stopping time may represent interesting situations from the point of view of applications.For instance, it can be the accumulated nth failure and repair of the system.In another situation, the stopping time can represent the occurrence of a "crucial failure" (which may happen after a random number of failures).This class of stochastic systems is associated with systems subject to failures in their components or connections according to a Markov chain.The situation that we are interested in arises when one wishes to study the stability of such a system until the occurrence of a fixed number  of failures and repairs.The paper recognizes the sequence of the stopping times containing the successive times of the occurrence of such failures and then it studies the stability of system (1) according to these stopping times.
Next, we will give an important theorem that will be used later.

Theorem 4. The MJSLS (1) is 𝑇 𝑛 -stable if and only if, for any
given set of matrices   > 0, there exists a unique set of matrices   > 0, satisfying the Lyapunov equations Proof.Sufficiency.In the proof we employ an induction argument on the stopping times   .First, define the function where   > 0, and   > 0 is the solution of The existence of such   > 0 relies on (7).Hence, to functional   (, ) in the following operation, we can derive We know that [()] = 0 and [()()] =   , where   = 1 if  =  and   = 0 if  ̸ = .Calculating the expected values above, we can obtain that The above relation can be rewritten as where Mathematical Problems in Engineering Now, let us observe that By applying (12) and considering that   > 0,   > 0 for each initial condition  0 and initial distribution , then we have for some  > 0. (5).Finally, it is easy to verify that, for any  1 , lim sup holds from (15).Therefore, for any  1 , MJSLS ( 1) is stable according to (i) of Definition 1. Now, using an induction argument, we assume that for some  the inequality lim sup holds and thus, by setting  ≡ , [‖   ‖ 2  {  ≥} ] < ∞.However, lim sup Notice that using the strong Markov property and the homogeneity property, the second term conditioned to the knowledge of (   ,    ) can be written as lim sup So one can conclude from ( 18) and ( 19) that lim sup Therefore, for any   , lim sup indicate that the MJSLS ( 1) is   -stable.
Necessity.As in the previous part define the functional for all ( 0 ,  0 ) ∈ R  × X.Therefore, The right-hand side of ( 22) can be expressed as In addition, Thus, based on the strong Markov property, applying homogeneity in (25) and introducing it in (24), we arrive at Since  0 is arbitrary, and calculating the expected values above, (26) implies that using the fact that ( 1 =  + 1 |   = ) = 1 −   and ( 1 >  + 1 |   = ) =   .Thus, from the Lyapunov stability theory, the existence of the set   > 0 satisfying ( 7) is guaranteed, completing the proof for  = 1.Now, for the general case, from the stochastically  stable of the system we can obtain that And from the strong Markov property, we can deduce that for  = 0, 1, . . ., −1.By the homogeneity property, it follows that ( 29) is equivalent to (22) with  0 =    and  0 =    , and the existence of a set of matrices   > 0 satisfying ( 7) is assured.Then, the proof of Theorem 4 is completed.

LQ Differential Games for MJSLS with a Finite Number of Jump Times
3.1.Problem Formulation.Now we study the LQ differential games for discrete-time MJSLS.Comparing with system (1), consider the following discrete-time MJSLS with a finite number of jump times: () ∈ R  are the measurement outputs for each player.
Here ((), V()) ∈ R  × R  represent the system control inputs.The matrices associated with "th" mode) will be assigned as (  ,   ,   ,   ,    ) for each   =  ∈ X.Throughout this paper, we choose the infinite horizon quadratic cost functions associated with each player: The weighting matrices     =    ≥ 0,     =    > 0 ∈ R × , and     =    > 0 ∈ R × .So we are looking for actions that satisfy simultaneously where To ensure the finiteness of the infinite-time cost function, we restrain the admissible control set to the constant linear feedback strategies; that is, () = We say that the optimization problem is well posed and the () and V() have the following two additional properties: The optimal strategies  * and V * determined by (32) are also called the Nash equilibrium strategies ( * , V * ).In order to guarantee the unique global Nash game solutions, both the players are only allowed to take constant feedback controls.Next we focus on finding the optimal strategies.

Main Results
. First, we give an important lemma that will be used later.If the system (1) is   -stable, we can obtain the following result for the discrete-time MJSLS (30).
Proof.Sufficiency.The proof employs an induction argument on the stopping times   .First, define the function where   > 0, and   > 0 is the solution of 6
Theorem 6.For system (30), suppose the following coupled equations admit the solutions where (ii) the problem of infinite horizon stochastic differential games admits a pair of solutions ( * (), V * ()) with  * () =  1  (), V * () =  2  (); (iii) the optimal cost functions incurred by playing strategies Proof.In the deduction of Lemma 5, we can prove that (i) is correct.Next what we have to do is to prove (ii) and (iii).

Iterative Algorithm and Simulation
4.1.An Iterative Algorithm.In this section, an iterative algorithm is proposed to solve the four coupled GAREs (40)-(43).Infinite horizon Riccati equations are hard to be solved; hence the particular problems can be solved via finite horizon equations. represents the finite number of iterations in the following equations: where   As in [28], under the assumptions of stabilizability, for any    Because it is the same as the above process (  = 1), we do not introduce it again due to space limitations.

Conclusions
In this paper we have discussed the   -stability for the discrete-time MJSLS with a finite number of jump times and its infinite horizon LQ differential games.Based on the relations between the Lyapunov equation and the stability of discrete-time MJSLS, we have obtained some useful theorems on finding the solutions of the LQ differential games.Moreover, an iterative algorithm has been presented for the solvability of the four coupled equations.Finally, a numerical example is offered to demonstrate the efficiency of the algorithm.Exact observability and -observability for discrete-time MJSLS are investigated by [29,30].On the basis of exact observability and -observability, infinite horizon stochastic differential games should be discussed and we will do further research in the future.

1 𝑖
(e) Then,  :=  − 1. Repeat steps (b)-(d) until the number of iterations is  + 1.We can finally obtain the numerical values of