Modeling of Random Delays in Networked Control Systems

In networked control systems (NCSs), the presence of communication networks in control loops causes many imperfections such as random delays, packet losses, multipacket transmission, and packet disordering. In fact, random delays are usually the most important problems and challenges in NCSs because, to some extent, other problems are often caused by random delays. In order to compensate for random delays which may lead to performance degradation and instability of NCSs, it is necessary to establish the mathematical model of random delays before compensation. In this paper, four major delay models are surveyed including constant delay model, mutually independent stochastic delay model, Markov chain model, and hidden Markov model. In each delay model, some promising compensation methods of delays are also addressed.


Introduction
Communication technology and digital control have shown remarkable progress in recent years, which prompts the emergence and development of networked control systems (NCSs) [1][2][3].NCSs are completely distributed feedback control systems, in which a control loop is closed via a communication network to connect sensors, controller, and actuators.Compared with traditional point-to-point control systems, NCSs reduce system wiring and cost, facilitate system diagnosis and maintenance, and increase system flexibility and reliability [4].Thus, NCSs are becoming increasingly popular in various contexts, such as automobile, aircraft, manufacturing plant, and remote surgery [5][6][7][8].
However, the finite bandwidth and limited service in communication networks cause some new problems and challenges, such as random delays, packet losses, multipacket transmission, and packet disordering [9].In general, packet losses mean that the delay of the packet transmission over networks is infinite.Multipacket transmission happens when the information to be transmitted is too widely scattered; that is, the delay for packing all the scattered information into one packet is too long to guarantee the packing efficiency.Packet disordering occurs when the delays of the packets transmission over networks are different, which results in that the packet sent earlier arrives at the destination node later or vice versa.Thus it can be seen that packet losses, multipacket transmission, and packet disordering are mainly caused by the existence of random delays.So, random delays are the main problem and challenge in NCSs, and the randomness of delays is generally affected by many stochastic factors (e.g., network load, nodes competition, and network congestion).
There are mainly two kinds of random delays in NCSs.One is the sensor-to-controller delay (S-C delay) in the backward network channel, and the other is the controller-toactuator delay (C-A delay) in the forward network channel.Random delays are the major reasons for the performance deterioration and potential instability of NCSs.Conventional control theories with ideal assumptions, such as nondelayed sensing and actuation, must be reevaluated before they are applied to NCSs, which makes the analysis and design of NCSs very complex.In order to compensate for random delays in NCSs, it is necessary to establish the mathematical model of random delays before compensation.This problem has attracted strong research interests in NCSs within the control community.Generally speaking, four types of modeling methods have been proposed for the random delays in NCSs.This paper provides an overview of these methods including constant delay model in Section 2, mutually independent stochastic delay model in Section 3, Markov chain model in Section 4, and hidden Markov model in Section 5. Finally, we conclude the work in Section 6.

Constant Delay Model
During the early stage of NCS-related research, when the distribution characteristics of random delays are difficult to obtain, the most direct approach is to model the random delays as a constant.In this model, a receiver buffer is introduced at the controller (or actuator) node, and the buffer size is equal to the maximum S-C (or C-A) delay [10,11].With the constant delay model, the NCS can be treated as a deterministic system, and many deterministic control methods can be applied to the NCS.For instance, a deterministic predictorbased delay compensation methodology was proposed in [12,13], where an observer was used to estimate the plant states and a predictor was used to compute the predictive control inputs based on past output measurements.Along this line, a first-in-first-out buffer was set for the controller to store the past output measurements, and another first-in-first-out buffer was set for the actuator to store the control inputs.All network nodes were time driven and acted synchronously.The sizes of these two buffers were, respectively, determined based on the upper bounds of S-C and C-A delays.Thus, both S-C and C-A delay were transformed to two constant values.Also, the NCS turns into a deterministic system, which is much easier to control than random delay systems.Similarly, Yu et al. [14] designed a multistep delay compensator for the NCS with dynamic noises and measurement noises.
For the NCS with constant S-C delays, Montestruque and Antsaklis [15] proposed both state-feedback and outputfeedback control methodologies based on a plant model that was incorporated in the controller/actuator and was updated using the actual plant state and derived the necessary and sufficient conditions for the system stability.For the multiinput multioutput (MIMO) NCS, Lian et al. [16] established the discrete model of the NCS and designed an optimal controller, in which both S-C delay and C-A delay were constant and their sum was less than one sampling period.For the asymmetric NCS, in which sensor measurements and control inputs are transmitted by using UDP protocol and TCP protocol, respectively, Kim et al. [17] modeled such an NCS as a switched system with constant input delays and derived the sufficient conditions for the system stability by using piecewise continuous Lyapunov methods.Besides, the augmented state model methodology can also be used to analyze and design the NCS with constant delays [18,19].
The constant delay model makes it feasible to apply conventional deterministic control methodologies to NCSs with random delays.Particularly for switched Ethernet, the random delay is relatively short and changes little, so it can be regarded as a constant delay and the deterministic control methodologies can be directly applied to the NCS closed over switched Ethernet.However, the constant delay model tends to take the upper bound of the random delay as the constant value.As a result, the random delay is artificially enlarged and then the system performance becomes degraded.Or, even worse, the system stability margin decreases so much that the system becomes unstable, which has been proved in [20,21].In view of this, an increasing number of researchers began to investigate stochastic control methodologies for NCSs with random delays.

Mutually Independent Stochastic Delay Model
Network delay tends to be stochastic since it is affected by many stochastic factors (e.g., network load, nodes competition, and network congestion), so the constant delay model and the corresponding deterministic control methodologies could hardly meet the requirement of the system performance.Thus, many people have started to study the stochastic delay model for NCSs.The stochastic delay model can be divided into two categories: the model in which delays are mutually independent and the model in which delays are probabilistically dependent.When the probabilistic dependence is unknown, the mutually independent stochastic delay model is often applied to the modeling and control of NCSs with random delays.Three main control strategies (including stochastic control, robust control, and predictive control) are surveyed in this section.

Stochastic Control.
In the mutually independent stochastic delay model, each delay is treated as a mutually independent stochastic variable and its distribution can be described by a stochastic function.With this model, stochastic (optimal) control approaches are often used to investigate NCSs.
Early in the year of 1998, Nilsson et al. [22] designed an LQG-optimal stochastic controller for an NCS with mutually independent stochastic delays.In [22], the NCS was modeled as a stochastic system and the distribution of random delays was assumed to be known in advance.This assumption was relaxed by Wei et al. [23] who designed an average delay window to achieve online delay prediction for an NCS with unknown delay distribution and improved the LQG-optimal control performance of the NCS.
The delays considered in [22,23] are all less than one sampling period.But, in real NCSs, random delays often exceed one sampling period.In view of this situation, Lincoln and Bemhardsson [24] and Hu and Zhu [25] designed the stochastic optimal controllers to guarantee the mean square exponential stability of the NCS with full (or partial) state feedback.Moreover, when the delay is infinite, Zhu et al. [26] also derived the stochastic optimal controller through solving an algebraic Riccati equation.A novel control method was presented in [27], and the maximum allowed delay bound was given based on the stochastic stability conditions.
In [28], the random delays were modeled as a linear function of the stochastic variable satisfying Bernoulli random binary distribution, and the prescribed H ∞ disturbance attenuation performance was achieved by designing an observer-based controller to guarantee the stochastically exponential stability of the closed-loop NCS.It was assumed that the construction of the observer was based on the knowledge of all the control inputs at the actuator node.The assumption was relaxed in [29] to meet the actual engineering needs.When the random delays are independent and identically distributed stochastic variables, Yu et al. [30] designed an optimal stochastic controller to guarantee the mean square exponential stability of the NCS with time-driven sensor nodes and event-driven controller/actuator nodes.For the MIMO NCS with multiple independent stochastic delays, a delayed state-variable model was formulated and a linear quadratic regulator (LQR) optimal controller was designed to compensate for the multiple time delays in [31].

Robust Control.
The stochastic delay can be transformed into the uncertainty (or disturbance) of an NCS, and then a robust controller can be designed to guarantee the robust stability and robust performance of the NCS.Different from stochastic control, robust control does not need the prior knowledge about the distribution characteristics of random delays.For instance, in [32], the random delay (S-C or C-A delay) was divided into two parts.The first part was a constant delay and the second part was an uncertain delay which was treated as the multiplicative perturbation of the NCS.And a continuous-time robust controller was designed using synthesis.In [33], the NCS with asymmetric path-delays over arbitrary communication networks was investigated under the criteria of minimum H ∞ norm, and the delay-dependent switching controllers were designed via a piecewise Lyapunov function approach as well as a common Lyapunov function approach.In [34], a new discrete-time switched system model was proposed to describe the NCS with random delays, and a sufficient condition was derived for the NCS to be exponentially stable and ensure a prescribed H ∞ performance.Moreover, the obtained condition established the relations among the delay length, the delay variation frequency, and the system performances.
In [35], the H ∞ performance analysis and the H ∞ control problem of uncertain NCSs with both delays and packet losses were investigated based on a Lyapunov-Krasovskii functional method.The H ∞ performance was analyzed by introducing some slack matrix variables and employing the information of the lower bound of delays, and the H ∞ controller was designed to be of memoryless type by solving a set of linear matrix inequalities.A new Lyapunov-Krasovskii functional method, which made use of the information of both the lower and upper bounds of delays, was proposed in [36] to derive a new delay-dependent H ∞ controller that is much less conservative than the controller derived in [35] because it avoided employing model transforming and bounding technique for some cross-terms in the Lyapunov-Krasovskii function and avoided introducing slack matrix variables.
The H ∞ control problem of NCSs with both delays and packet disordering was investigated in [37], where the actuator always uses the latest arrival control law.This problem was then comprehensively investigated in [38], where the NCS was modeled as a parameter-uncertain discrete-time system with multistep delays based on matrix theory.An improved Lyapunov-Krasovskii functional method was proposed in [38] to design a less conservative H ∞ stabilization controller by solving a minimization problem based on linear matrix inequalities.
In [39], the stochastic delay was treated as a kind of uncertainty of NCSs and an H ∞ controller was designed via -synthesis.When the stochastic delay is unknown, bounded, and time-varying, the method to compute the maximum allowable delay bound was given in [40] and the robust stability bound was also derived there.In [41], both the deterministic information from the current timestamp and the stochastic information from the past timestamps were used to design a network-traffic-dependent H ∞ controller via introducing two algorithms: pattern generation (PG) algorithm and pattern identification (PI) algorithm.The PG algorithm classified off-line the network traffic into some separate patterns to derive a less conservative NCS, and the PI algorithm searched on-line for a pattern representing the recent network traffic to operate the designed controller.In [42], the robust control theory was combined with the network calculus theory for the delay compensation.The minimal network service curves and the maximal network traffic curves were used in the network calculus theory to estimate bounded end-to-end delays, which were translated into uncertainties in the robust controller design.For the networked cascade control system with disturbances, the criterion of its robust asymptotic stability was derived and the -suboptimal state feedback H ∞ controller was designed in [43].

Predictive Control.
Predictive control of NCSs with random delays was investigated in [44][45][46][47].Predictive control is suitable for complex dynamic systems with uncertain models since it is based on multistep test, rolling optimization, and feedback tuning control.In general, the setup of a networked predictive control system consists of two parts: the control prediction generator and the network delay compensator.The former generates a sequence of future control predictions to satisfy the system performance requirements using conventional control methods (e.g., PID, LQG).The latter compensates for the random network delay by choosing the latest control value from the control prediction sequences available on the plant side.In practice, predictive control has been applied into networked Hammerstein systems [48] and networked Wiener systems [49].
In [44][45][46][47], only delayed controlling data are used to derive the control predictions.However, in real systems, it is difficult to obtain the controlling data due to the existence of delays.So, an improved predictive controller, designed by using delayed sensing data, was proposed in [50].Compared with previous results, the predictive control-based approach in [50] is easy to be implemented in practice.Besides, at any future time the predictive controller in [46] cannot know the control input of the plant in future time due to the stochasticity of network delays.So, constructing the state predictor in [46] cannot use the real but the estimative control input of the future time, which causes the problem that the error of the estimation affects the precision of the predictor and even destroys its stability.In order to decrease the conservativeness, a new design method of networked predictive control was proposed in [51] by packing the current predictive control input with history predictive inputs.By this means the future plant input is known, and the state predictor can be designed.Thus its performance and stability are not affected by the future input of the plant.
The networked predictive control approach transforms the closed-loop system to a switched system, and the stability of the switched system is guaranteed based on the condition that all subsystems of the switched system have to possess a common Lyapunov function under arbitrary switching.However, this condition is too strict in practice because some delay values may be not admissible considering the system stability.In order to overcome this limitation, an improved networked predictive control strategy was proposed in [52] by appropriately assigning the subsystem or designing the switching signal.Average dwell time method was given as an effective tool for finding such switching signal that a common Lyapunov function was not necessary for the overall system still to be stable even though there existed unstable subsystems.Furthermore, an improved predictive control design strategy, combined with the switched Lyapunov function technique, was proposed in [53], where the controller gain varied with the random delay to make the corresponding closed-loop system asymptotically stable with an H ∞ -norm bound.Besides, a new predictive control scheme based on multirate Kalman Filtering was presented in [54] to compensate for the random delays and packet losses in the feedback channel of NCSs.

Markov Chain Model
Stochastic delays are not always mutually independent.Sometimes, there are some probabilistic dependency relationships among the delays, such as Bernoulli distribution, Markov chain.For the past few years, a lot of effort has been put on the research about Markov-based modeling and compensating methods for stochastic delays in NCSs.Since there are mainly two kinds of random delays: S-C delays and C-A delays, most existing methods can be divided into two categories: one considering S-C delays (or C-A delays) and the other one considering both S-C delays and C-A delays.The second category can be further divided into two subcategories: the method using one Markov chain to model the sum of S-C delay and C-A delay and the method using two Markov chains to model S-C delay and C-A delay, respectively.

Considering S-C Delays or C-A Delays.
In [55], network only existed between the sensor and the controller, and the stochastic S-C delay was modeled as a finite-state Markov chain.Based on the Markov delay model, the NCS was modeled as a discrete-time jump linear system and a V-K iteration algorithm was proposed to design switching and nonswitching output feedback stabilizing controllers.When the S-C delay was longer than one sampling period [56], it was divided into two parts: the first part was integer times of the sampling period and the second part was less than one sampling period.The second part was modeled as a finite-state Markov chain.Thus the NCS was modeled as a discrete-time Markovian jump linear system (MJLS) and an H ∞ optimal controller was designed.
In the aforementioned two papers, the Markov states were assumed to be completely observed.However, the access to the Markov states may not be possible in some circumstances, which makes is quite possible that the associated Markov states are inaccessible to controller.For this case, to design mode-independent controllers, a mode-dependent Lyapunov function was used to derive output-feedback H 2 /H ∞ controller for the NCS subject to random failures and stochastic Markovian S-C delays in [57].In practical systems, it is difficult to obtain the exact transition probability matrix of the Markov chain of the S-C delays.To solve this problem, a polytopic-type uncertainty in the transition probability matrix of the Markov chain was taken into account in [58], where both S-C delays and packet losses were simultaneously considered.The robust fault detection problem addressed there was converted into an auxiliary robust filtering problem, and a sufficient condition for the existence of the desired robust fault detection filter was established in terms of linear matrix inequalities.
A more general NCS framework was considered in [59], where Markovian C-A delays, packet losses, and uncertainty of NCSs were simultaneously investigated.Packet losses were compensated and a sufficient condition on the existence of robust H ∞ disturbance attenuation level was given.In [60], the C-A delays were also modeled as a Markov chain.Thus, the NCS was modeled as a discrete-time MJLS and a stabilizing controller was designed by using the delayquantization and augmented controller vector methods.

Considering Both S-C Delays and C-A Delays. Generally, S-C delays and C-A delays coexist in NCSs.
In [61], both S-C delay and C-A delay were lumped together as a single delay governed by a Markov chain, and an optimal controller was designed to satisfy the prespecified performance.Besides, a method was proposed to calculate the state transition matrix of the Markov chain.This calculating method was reused in [62].The delay from the sensor to the actuator, which is equal to the sum of S-C delay and C-A delay because the calculating time at controller is usually very short, was also modeled as a Markov chain in [63].And a T-S model was employed to represent an NCS with all possible delays considered there.As an application of the T-S based modeling method, a parityequation approach and a fuzzy-observer based approach were developed for the fault detection of the NCS.Under some geometric conditions, the T-S model was transformed to an output-feedback form and then an observer-based fault detection method was presented for a class of nonlinear NCSs with Markovian delays from sensor to actuator [64].Based on the same delay model, an uncertain NCS with stochastic but bounded delays was modeled as a discrete-time jump linear system governed by a finite-state Markov chain in [65].In order to make the system asymptotically mean square stable under a serials of stochastic disturbance such as random delays and actuator failures, an improved V-K iteration algorithm was used to successfully designed a class of reliable controllers.For the NCS with multiple faults and Markovian delays from senor to actuator, a robust fault isolation filter was developed with its parameters satisfying H ∞ disturbance attenuation and poles assignment constraints in the frame of MJLS [66].When the delay is modeled as a Markov chain, a fixed state transition matrix is usually assigned.However, the controller designed based on the fix transition matrix may not stabilize NCSs when delays change along with network load.In order to solve the problem, an unknown and time-varying Markov chain with multiple possible transition matrices was used to model the time-varying part of delays, and then a stochastic switching controller was constituted in [67].The switching rules were found through an intelligent algorithm based on the greedy algorithm.
In aforementioned papers (i.e., [61][62][63][64][65][66][67]), S-C delay and C-A delay are lumped together as a single delay and then modeled as a single Markov chain.But it is not always feasible for the two delays to be lumped together.So it is quite general to model S-C delay and C-A delay as two Markov chains.In [68], the two delays were modeled as two different Markov chains, and the resulting closed-loop system was jump linear system with two modes characterized by two Markov chains.A mode-dependent state-feedback controller was designed with its gain depending on both the current S-C delay and the previous C-A delay.However, practically the previous C-A delay is not always available because the transmission of the C-A delay value will also suffer from the S-C delay when it is transmitted through the S-C network before reaching the controller.In order to overcome the problem, the most recent available C-A delay and the current S-C delay were used to derive a two-modedependent controller in [69].It is worthwhile noting that the most recent available C-A delay may not be the previous C-A delay.Therefore, the resulting closed-loop system is not a standard MJLS.Actually, it is a special MJLS with multistep delay mode jump as shown in [69].Through calculating the transition probability matrix for the multistep delay mode jump, an output feedback controller was derived and the sufficient and necessary conditions for the stochastic stability were established.Furthermore, the mixed H 2 /H ∞ control problem of NCSs with S-C delays and C-A delays governed by two Markov chains was investigated in [70], where a twomode-dependent output feedback controller was designed based on the available S-C and C-A delay values.In [71], the S-C and C-A delays were also modeled as two Markov chains and a state-feedback controller based on both the plant mode and the delay mode was designed to guarantee the stochastic stability of NCSs.
Networked predictive control strategy has been discussed in the previous section, where the random delays are mutually independent.But, when the random delays have Markovian properties, how to achieve the predictive control of such NCSs?This problem was investigated in [72,73], where the resulting closed-loop system eventually turned into a jump linear system with two modes, and a state-feedback model predictive controller was designed to guarantee the stochastic stability of the closed-loop system.

Hidden Markov Model
As mentioned in Section 1, the network-induced delay is random because it is affected by many stochastic factors (e.g., network load, nodes competition, and network congestion).All of these factors can be combined into an abstract and hidden variable which is defined as the network state.So, the distribution of delays is governed by the network state.
With every packet transmission over network, the network state will jump from one mode to another within a finite state space.In general, the jump of modes satisfies Markovian property, which makes it possible to model the network state as a Markov chain.Moreover, it should be a hidden Markov chain because the network state cannot be observed directly but can be estimated through observing network delays.Therefore, the relation between the network state and the network delay is modeled as a hidden Markov model (HMM).In HMM, the network state (i.e., the state of the hidden Markov chain) determines the stochastic distribution of delays, while delays can be used as a kind of probes to infer the HMM and estimate the hidden network states [74].Different from the Markov chain model in which the current delay is governed by the previous delay, the current delay is governed by the current network state in the HMM.From this point of view, the HMM reveals the essential generation mechanism of random delays.
To the best of our knowledge, the HMM was first introduced to NCSs by Nilsson [75].In [75], the network state was simplified as the network load with three states ("L" for low network load, "M" for medium network load, and "H" for high network load).The transitions between different states were modeled as a Markov chain.With every state in the Markov chain, there was a corresponding delay distribution modeling the delay for the state, which was referred to as an HMM.Based on the HMM, the NCS was treated as a jump linear system and an LQG optimal controller was designed to guarantee the system stability.Following the same line as in [75], the sensor-to-actuator delays were considered as interval variables governed by a Markov chain in [76].Based on the intervals of different mode, the corresponding plant was converted to a discretetime Markov jump system with norm-bounded uncertainties, and an H ∞ state feedback controller was constructed via a set of linear matrix inequalities.
When S-C delays and C-A delays coexisted in an NCS, two hidden Markov chains were used in [77] to model the sensor-to-controller network state and the controllerto-actuator network state, respectively.So, there are two HMMs: one is for the relation between the S-C delay and the sensor-to-controller network state, and the other is for the relation between the C-A delay and the controller-toactuator network state.Based on the Lyapunov-Razumikhin method, a mode-dependent state feedback controller was designed to stabilize this class of systems by solving bilinear matrix inequalities.Based on the same delay modeling method as that in [77], a dynamic output feedback controller was designed to achieve both robust stability and prescribed disturbance attenuation performance for a class of uncertain NCSs with random delays in [78], and a new stochastic Lyapunov-Krasovskii function was proposed to develop a delay-dependent criterion for determining a mode-dependent state-feedback H ∞ controller for a class of continuous-time NCSs with random delays in [79].In order to obtain less conservative results than those in [77], free weighting matrices were introduced in [79] by using the Newton-Leibniz formula to avoid estimating some crossterms in the Lyapunov-Krasovskii function.
Generally, the Baum-Welch algorithm (i.e., expectation maximization algorithm) can be used to estimate the parameters (e.g., initial network state distribution, network state transition probability matrix, and delay observation matrix) of the HMM of delays in NCSs (see [80,81]).Based on these parameters, the current C-A delay can be predicted before designing the current controller [82], and the prediction can be used to design a state-feedback controller or an optimal controller to compensate for the effect of the current C-A delay on NCSs [83,84].In order to estimate the parameters of the HMM, the historical C-A delays were uniformly quantized to obtain the discrete observations which were used as the inputs of the estimating process [80][81][82][83][84].However, in practical systems, delays seldom uniformly distribute over a delay interval.So, when quantizing the delays, some relatively centralized delays should be quantized as the same observation value, while those relatively decentralized delays should be quantized into different observation values.Based on this rule, K-means clustering was proposed in [85] to quantize the past C-A delays, which leads to more precise prediction and better compensation of the current C-A delay.A composite hidden Markov model was presented in [86] to model both S-C delays and C-A delays and the model illustrated the inner relationship between these two kinds of delays.Based on the composite HMM, the C-A delay was estimated by using S-C delays and a single modedependent state feedback controller was designed via solving linear/bilinear matrix inequalities.

Conclusions
NCS is a very important research area with wide applications.Random delay is one of the fundamental problems in NCSs.In this paper, various approaches for the modeling of random delays in NCSs are surveyed.The earliest and most simple approach is to model the random delay as a constant, and the corresponding delay compensation is usually based on deterministic control methods.Since the real network-induced delay is often timevarying or even random, an increasing number of literatures tend to adopt the stochastic delay model without or with probabilistic dependency relationships among delays.The one without probabilistic dependency mainly deals with mutually independent stochastic delays and utilizes stochastic control, robust control, predictive control, or other intelligent control strategies to compensate for the random delays.The one with probabilistic dependency mainly uses one or two Markov chains-based modeling and compensation methods for random delays.When the stochastic characteristics of delays are caused by many stochastic factors in networks (such as network load, nodes competition, and network congestion), all these factors are integrally defined as the network state, and the HMM is introduced to model the random delays.Even though the HMM is, to some extent, superior to the Markov chain model, it is imprudent to consider the HMM as the best and ultimate model for the random delays in NCSs because of, for example, the poor real-time performance of the HMM.There is still much work to do to sufficiently and completely describe the stochastic distribution of random delays in NCSs.In summary, the modeling of random delays in NCSs is still in their infancy and should deserve a lot of attention in the years to come.