Consensus of Noisy Multiagent Systems with Markovian Switching Topologies and Time-Varying Delays

Stochastic multiagent systems have attractedmuch attention during the past few decades.This paper concerns the continuous-time consensus of a network of agents under directed switching communication topologies governed by a time-homogeneousMarkovian process. The agent dynamics are described by linear time-invariant systems, with random noises as well as time-varying delays. Two types of network-induced delays are considered, namely, delays affecting only the output of the agents’ neighbors and delays affecting both the agents’ own output and the output of their neighbors. We present necessary and sufficient consensus conditions for these two classes of multiagent systems, respectively. The design method of consensus gains allows for decoupling the design problem from the graph properties. Numerical simulations are implemented to test the effectiveness of our obtained results as well as the tightness of necessary/sufficient conditions.


Introduction
In the past few years, distributed coordination of multiagent systems has been considered by many researchers due to its broad applications in such areas as swarming of animals/ robots, cooperative unmanned aerial/underwater vehicles, distributed computation, air traffic control, and distributed sensor networks.One of the important issues in coordinated control is network based consensus protocol design.In this setting, consensus refers to every agent achieving agreement about some common or shared quantity by exchanging information according to a set of rules.
For the purpose of reaching consensus, important interaction details of agents in a system are mostly encoded by the communication graph of the system, which gives a general setting to study consensus and allows for the application of graph-theoretical notations and tools.Distributed computation over networks has been studied in the pioneering work of Tsitsiklis [1] and Chatterjee and Seneta [2] in systems and ergodicity theory.More recently, analytical frameworks for solving consensus problems were introduced by Olfati-Saber and Murray [3] and Jadbabaie et al. [4] based on graph and matrix theory.Since then numerous consensus protocols have been proposed, mostly for simple single-and doubleintegrator dynamics (see, e.g., [5][6][7][8][9] and references therein).It is pointed out that [10] design of consensus protocols for agent dynamics delineated, more generally, by linear timeinvariant systems is more challenging due to the possible existence of strictly unstable eigenvalues (poles) in the open-loop matrix.Necessary and sufficient consensus conditions for linear time-invariant systems were explored in [11][12][13][14] recently.
In many cases, the communication between agents is subject to stochastic perturbation-the connections change with time due to packet drops, agent failure, and various external disturbances.Therefore, the communication graphs underpinning the physical systems are better characterized by random switching networks.Stochastic consensus with singleand double-integrator dynamics has been well researched [15][16][17][18][19][20][21].For example, asymptotic agreement of continuoustime single-integrator agent dynamics over Poisson random graphs is considered in [15].The results are further extended in [16] to solve mean square consensus under directed and weighted independently switching random graphs.When the communication topology is described by a strictly stationary ergodic graph process, a necessary and sufficient condition for almost sure consensus of single-integrator agents is shown 2 Mathematical Problems in Engineering to be the connectivity of the mean topology with respect to a stationary distribution of the process [17].For both discretetime and continuous-time multiagent systems with singleintegrator dynamics and balanced communication graphs, it has been shown in [21] that the ergodic Markov jump linear system achieves average consensus almost surely if and only if the union of topologies corresponding to the states of the Markov process is strongly connected.Similarly, for second-order discrete systems with (not necessarily ergodic) Markovian switching topologies, the necessary and sufficient condition for mean square consensus becomes that each union of graphs corresponding to the closed sets of positive recurrent states has a spanning tree [20].Recently, this result is extended to linear time-invariant agent dynamics in [10] for both discrete-and continuous-time consensus.
It is well documented in the literature [22] that unmodeled delay effects in a feedback mechanism may destabilize an otherwise stable system.In multiagent systems, time-varying delays may arise naturally due to the asymmetry of interactions, the congestion of the communication channels, and the finite transmission speed.Moreover, noise/uncertainty frequently occurs to agents through, for example, communication errors and spurious measurements in communication systems.Therefore, it would be desirable to understand consensus problems in the setting of Markovian switching topologies with interactions affected by both time-varying delays and random noises.
In this paper, we investigate consensus problems for continuous-time multiagent systems with linear time-invariant agent dynamics under Markovian switching topologies, timevarying delays, and stochastic noises.In particular, we consider two types of communication delays: delays affecting both the state of the agents and that of their neighbors and delays affecting only the state of the agents' neighbors.A unified framework that considers these delays in continuous-time multiagent systems with fixed topology is first established in [23].It is worth noting that although other interesting delay-dependent robustness results are reported in, for example, [24][25][26], the communication topologies considered are either static or switch deterministically.
This work deals with a group of identical agent dynamics, each of which follows a linear time-invariant system with white noises input.The information flow between agents is modeled by a time-homogenous Markov process, whose state space corresponds to all the possible communication patterns (directed graphs).We establish necessary and sufficient conditions to guarantee all agents asymptotically achieve an agreement in the mean square sense and in the almost sure sense for both types of time delays, respectively.When each graph corresponding to a state of the Markov process contains a spanning tree or is -regular for a fixed  ≥ 1, we show that the agents can reach consensus for suitable time-varying delays in terms of -matrices if the agent dynamics is stabilizable.Conversely, if the consensus is achieved, the agent dynamics must be stabilizable and each union of graphs corresponding to the closed sets of positive recurrent states of the Markov process contains a spanning tree.The main mathematical techniques used here are based on the stability analysis of Markovian jump linear systems, stochastic differential delay equations, and graph and matrix theory.
The rest of the paper is organized as follows.Section 2 contains the problem formulation.Section 3 presents the main results.A couple of numerical examples are given in Section 4. The conclusion is drawn in Section 5.
Notation.Let 1  and 0  be the -dimensional column vectors of all ones and all zeros, respectively.  represents an  ×  identity matrix.If the dimension is clear from the context, we sometimes suppress the subscript .The sets of real and complex numbers are denoted by R and C, respectively.The closed right half plane is signified by C + .We say  >  ( ≥ ) if  −  is positive definite (semidefinite), where  and  are symmetric matrices of the same dimensions.  means the transpose of matrix , while   means its conjugate transpose.For a vector , ‖‖ refers to its Euclidean norm; for a matrix , ‖‖ = √ trace(  ) represents its trace norm.Let ‖⋅‖ max represent the max norm of a matrix, namely, the maximum of the absolute values of elements.For a matrix  ∈ R × , its null space is designated by Null() = { ∈ R  :  = 0}.By  ⊗  we denote the Kronecker product of matrices  and , which admits the following properties: =1 the eigenvalues of a matrix  ∈ C × .Throughout this paper, we will order them in a nondecreasing order according to their modules:

Graph and Consensus
Properties.Let G = (V, E, A) represent a directed graph of order , where V = {V 1 , V 2 , . . ., V  } is the set of nodes (agents) and E ⊆ V × V is the set of directed edges.The ordered pair (V  , V  ) ∈ E denotes a directed edge from node V  to node V  , indicating that the information can be sent from agent V  to agent V  .The weighted adjacency matrix A = (  ) ∈ R × is defined by   > 0 if (V  , V  ) ∈ E and   = 0 otherwise.Define the indegree matrix as a diagonal matrix D = diag( in 1 , . . .,  in  ), with  in  = ∑  =1   being the in-degree of agent V  .Similarly, the out-degree of agent V  is defined by for all  = 1, . . .,  [3].Define the graph Laplacian matrix as L = (  ) = D − A, which has all row sums equal to zero.
A sequence of edges We say that G contains a spanning tree if there is an agent (called root) such that every other agent can be connected by a directed path starting from the root.By Lemma 3.3 of [6], G contains a spanning tree if and only if 0 =  1 (L) < | 2 (L)|.Let  be a positive integer.The union of For  ≥ 0, let () ≥ 0 be a differentiable function which will stand for the time-varying communication delay.
For  = 1, . . ., , the dynamics of each agent V  in continuous time takes the following two different forms: (i) with self-delay: (ii) without self-delay: where   () ∈ R  represents the state of agent V  at time ,    (), û  (),    (), û  () ∈ R  are control inputs of agent V  given by respectively, , Â ∈ R × , , B ∈ R × are system matrices, and {  (),   () : ,  = 1, 2, . . ., } are independent standard white noises.Here, , K ∈ R × are common consensus gains to be designed later, and   are referred to as the intensity of noise.To highlight the presence of noise, it is natural to define a noise graph Ĝ = (V, E, Â) with the adjacency matrix Â = (  ) ∈ R × satisfying   > 0 if (V  , V  ) ∈ E and   = 0 otherwise.By definition, if viewing G and Ĝ as unweighted graphs, that is, the adjacency matrices are taken as binary ones, we have G = Ĝ.Likewise, the corresponding degree and Laplacian matrices are denoted by D = diag( 1 , . . .,   ) with   = ∑  =1   and L = ( l ) = D − Â, respectively.
Remark 1.The above dynamical models characterize system uncertainties with Gaussian white noise appearing as an exogenous input (similar treatment can be found in, e.g., [27][28][29][30]).To see this, take Â =  1  and B =  2  ( 1 ,  2 > 0).System (1), for example, can be recast as The perturbations are represented by a linear combination of gain matrices  and K to be determined.Moreover, if  1 = 1, we take K = .(Although other choices are theoretically allowed as per Theorem 8 below, we make them equal in practice since one usually is not able to separate out the noise from the rest of the state.)Thus, the uncertainty reduces to the conventional form   () +  2   () ẇ  ().We mention that other commonly studied uncertainties pertaining to the consensus problems include the measurement noises which only affect the received neighbors' states (e.g., [18]) and the additive plant noises (e.g., [31]).
Multiagent system (2) with consensus protocols ( 5) and ( 6) considers only propagation delays for information transmitted from agent V  to agent V  on the communication network.Propagation delay has been addressed previously, for example, in works [24,[32][33][34][35]. Multiagent system (1) with consensus protocols (3) and ( 4) models both self-delay and neighboring delay.This scheme is relevant for dynamic agents with computation or reaction delays; see, for example, [3,25,26,36,37].Although it would be more realistic to explore heterogeneous delays, we consider the uniform delay () as a first step, and this simplifies the derivation.
We say matrix  in ( 1) and ( 2) is Hurwitz (or stable) if every eigenvalue of  has strictly negative real part; that is, it belongs to C \ C + .The pair (, ) is called stabilizable if there exists  ∈ R × such that  +  is Hurwitz [38].Item (a) in Assumption 3 is also used in [10,21].Item (b) is meant to eliminate the triviality, since consensus can be reached by setting zero consensus gains if  is Hurwitz.In this paper, we assume a special sort of noise-possibly due to the homogeneity of the communication channels between each agent and its neighbors-in which   () is independent of  as item (c) indicated.The consensus in Definition 2 is defined in the sense of mean square convergence.This implies that the consensus can also be achieved in the almost sure sense in view of item (d) and the homogeneity of the Markov process (see Corollary 3.46 of [39,40]).

Exponential Stability for Delay Markovian Jump Systems.
Denote by (Ω, F, P) the underlying common probability space for the Markov process and Brownian motions discussed above.The homogeneous continuous-time Markov process () with generator  = (  ) ∈ R × is formally given by where ℎ > 0 and ℎ → 0.Here   is the transition rate from  to  if  ̸ = , while   = − ∑  ̸ =   .As is known, the state space  = {1, 2, . . ., } of () can be decomposed uniquely into the form  = { ∪  1 ∪ ⋅ ⋅ ⋅ ∪   }, where each   ( = 1, . . ., ) is a closed communication class (i.e., closed set in the Markov process) of positive recurrent states and  is a set of transient states [41].
The objective of this paper is to reveal how stability analysis of differential delay equations, together with techniques used in matrix, Markov chain, and graph theory, can be applied to investigate stochastic consensus problems (1) and (2).

Main Results
In this section, we derive necessary and sufficient conditions for reaching consensus of noisy linear systems (1) and (2) under Markovian switching topologies and time-varying delays.

Consensus Conditions for Systems with
Self-Delay.We first consider multiagent system (1) with protocols (3) and ( 4), where both self-delay and neighboring delay are factored in.
Note that the assumption Â =   A  ( = 1, . . ., ) is only used in the proof of statement (a), while the assumptions Â =  1  and B =  2  are only used in the proof of statement (b).
Proof.The idea is to apply Lemma 5 to the error dynamics (13).It suffices to check Assumption 4 holds.
To verify item (c), note that Therefore, similarly as in the proof of ( 21) and ( 22) we obtain where we have taken K = η( B P B) −1 B P Â with η ≥ 1/min 2≤≤, 1≤≤ |  ( L )|.Since the fact that G  contains a spanning tree implies that Ĝ also contains a spanning tree, η is well defined with the same reason as above.Hence, we take   = 0 and  =   =   ( P)/ 1 ( P) for all  ∈ .
Remark 9. (a) The design of consensus gains  and K splits the design problem from the underlying communication topology.For example,  is constructed based on the system matrices ( 18) and a multiplicative coefficient  depending only on the graphs.Such a design procedure decouples the effects of agent dynamics and the network topologies, which simplifies the consensus design for the cases where the number of agents is large (see also [10,13]).(b) When  = ,  = 0, and  =   , we reproduce singleintegrator agent dynamics, and ( 18) and ( 19) always hold true.This can be viewed as a generalization of results in [21] by introducing random noise and time delay.
(c) The assumption in Theorem 8 about  being a nonsingular -matrix and ( + ) −1 1  −  −1 1  having positive entries is easy to verify.Indeed, all the off-diagonal entries are nonpositive by the definition of generator .Thus, it suffices to show that  −1 is nonnegative (this always holds if  = 1) and ensure that every row sum of it is less than ( + ) −1 .
(d) There is a gap pertaining to graph connectivity between sufficient conditions (Theorem 8) and necessary conditions (Theorem 7).Comparing with the previous work [10] for noise-free and delay-free systems, we understand that the stronger connectivity requirement-each graph G  contains a spanning tree-is introduced to accommodate the added noises and time-varying delays.Notice that the results are based on Lemma 5, which is about nonlinear systems.This also suggests the conditions derived here could be conservative.Notwithstanding, the study of weaker sufficient condition (e.g., using some algebraic methods) comparable to that of the necessary condition is an interesting future research.

Consensus Conditions for Systems without Self-Delay.
Next, we study multiagent system (2) with protocols ( 5) and ( 6), where only neighboring delay is considered.
As is noted below Theorem 7, the assumption Â =   A  ( = 1, . . ., ) is only used in the proof of statement (a), while the assumptions Â =  1  and B =  2  are only used in the proof of statement (b).
Recall that if (, ) is stabilizable, there exists a  ∈ R × such that  +  is Hurwitz.Therefore, by the Lyapunov stability theorem, there exists an  ×  matrix  > 0 such that Define a symmetric matrix  ∈ R × by  = ( + )   +  ( + ) .
Proof.As in the proof of Theorem 8, we will apply Lemma 5 to the error dynamics (26).
For item (b), again by applying the Rayleigh quotient inequality we derive Therefore, we take To show (c), we recall the simple norm inequality ‖ + ‖ 2 ≤ 2(‖‖ 2 + ‖‖ 2 ).It suffices to find suitable   and   so that the following two inequalities hold for  ∈ : Remark 12. (a) Similar comments in Remark 9 can be applied here.In addition, we note that the requirement in Theorem 11-G  are -regular graphs for all -is somewhat restrictive.If this condition is violated, we might use Weyl's inequality (see, e.g., [50]) to bound the maximum eigenvalue in (31), which nonetheless will lead to a more cumbersome expression.
(b) Interestingly, sufficient conditions in Theorem 11 do not explicitly mention connectivity assumptions, whereas necessary conditions in Theorem 10 clearly state certain connectivity assumption.To see how some connectivity is implicitly required in Theorem 11, we consider a special case with  =  = 1,  = {1, 2}, and G  ( ∈ ) being not connected and having two connected components, out of which one is a complete graph with  + 1 nodes.We can show that matrix  in Theorem 11 is not a nonsingular -matrix, violating the assumption of Theorem 11.Indeed, it is straightforward to check that  = 2,  1 =  2 = ||, and  = 2( + ) < 0 with  ≥ 0. Setting  = ( −   − ) with ,  > 0, we obtain the 2 × 2 matrix . being a nonsingular -matrix is equivalent to the fact that all its leading principal minors are positive [41], which in turn yields −2( + ) − 2 − || > 0. However, this inequality does not hold for any  ∈ R when  +  ≥ −1.
The program stops if all components of   ( = 1, 2, . . ., 10) are less than 10 −5 ; if the program does not stop before  = 10 4 , we regard that the consensus is not achieved.For each given , we collect 500 samples (1 sample consists of 10 graphs) to check whether the system finally achieves consensus.The fraction of samples that reach consensus is shown in Figure 5 as a function of .The curve displays a sigmoidal variation with respect to , saturating at 1 when  is just over 3.1 × 10 −3 .It is well known that [52] the random graphs are not connected with high probability if  < (ln )/ (here, about 6.9 × 10 −3 ) in the large  limit.Figure 5 reveals that consensus can still be achieved with a much smaller  than the connectivity threshold indicating that the sufficient condition regarding connectivity in Theorem 8 can be weakened.

Conclusion
This paper has studied the continuous-time consensus problem of linear multiagent systems under Markovian switching interaction topologies, random noises, and time-varying delays.The agent dynamics are described by linear timeinvariant systems, in which two types of network-induced delays are considered, namely, delays affecting only the output of the agents' neighbors and delays affecting both the agents' own output and the output of their neighbors.Necessary and sufficient consensus conditions have been derived, respectively, for these two classes of multiagent systems.The design of consensus gains has a computationally advantageous decoupling feature.Numerical examples are given to demonstrate the effectiveness of the proposed methods.Although there is a gap between necessary and sufficient conditions, the necessary one seems to be tighter in view of the simulations.Future research worth investigation could be the heterogeneous time delays, general uncertainties, and systems with different dynamics (see, e.g., [53]).

Figure 5 :
Figure 5: Probability of reaching consensus for multiagent system (1) as a function of  over Markovian switching random graphs G(, ).
) achieves consensus.Furthermore, assume that there exist   > 0 for  = 1, . . ., , and  1 ,  2 > 0 such that Â =   A  , Â =  1 , and B =  2 .Then (a) each union of the graphs G  ( = 1, . . ., ) corresponding to all the states in the closed set   for  = 1, . . .,  has a spanning tree; To prove (a), let G  1 = ⋃ ∈ 1 G  be the union graph corresponding to the closed set  1 .Without loss of generality, assume that G  1 does not contain a spanning tree.Denote by L  1 its Laplacian matrix.By Assumption 3(a), L   1 is also a Laplacian matrix.It follows from Corollary 4.2 of (b) (, ) is stabilizable.Proof.