Robust State Estimation for Delayed Neural Networks with Stochastic Parameter Uncertainties

1School of Electrical Engineering, Chungbuk National University, 52 Naesudong-ro, Cheongju 361-763, Republic of Korea 2Nonlinear Dynamics Group, Department of Electrical Engineering, Yeungnam University, 280 Daehak-ro, Gyeongsan 712-749, Republic of Korea 3School of Electronic Engineering, Daegu University, Gyeongsan 712-714, Republic of Korea 4Department of Biomedical Engineering, School of Medicine, Chungbuk National University, 52 Naesudong-ro, Cheongju 361-763, Republic of Korea


Introduction
Neural networks are generally recognized as one of the simplified models of neural processing in the human brain [1].Due to their strong capability of information processing, neural networks have been applied in many areas such as signal and image processing, pattern recognition, fault diagnosis, associative memories, fixed-point computations, optimization, power systems, and other scientific areas [2][3][4][5][6][7][8][9].Since the application of neural networks is heavily dependent on the dynamic behavior of their equilibrium points, it is important and a prerequisite job to analyze the stability of neural networks.Moreover, the neural networks are the network of mutual elements that behave like biological neurons.These neurons can be mathematically described by difference or differential equations.Each single neuron has a simple structure.Before handling this network, because most systems are exposed to nonideal environment such as a random change and a limit of the realization of the real situation, we need to pay keen attention to the three following considerations.
(i) State estimation problem is important in both control theory and practice applications because the system states, particularly in large scale systems, are not completely available in the system outputs in real applications.
(ii) During the implementation of many practical systems such as aircraft and electric circuits, there are stochastic perturbations occasionally.Also, the stochastic perturbations are as significant as the system parameter uncertainties as a factor affecting dynamics in the applications of science and engineering.
(iii) Time delay is inevitable in real-world neural networks.In practical implementations by electrical circuits, time delays occur in the signal transmissions among neurons and the finite switching speed amplifiers.It is well known that the existence of time delays causes oscillation, poor performance, and instability of the concerned networks.
Naturally, state estimation problem [10][11][12][13][14] and stability analysis for time delays [15][16][17][18][19] were issued in the various dynamic systems.In addition, in stability analysis for the time delays, delay-dependent methods have been paid more attentions than delay-independent ones because of the fact that the sufficient conditions with delay-dependent method provide maximum allowable delay bounds for guaranteeing the asymptotic stability of the concerned system and are generally less conservative than delay-independent ones.The main concern in delay-dependent stability analysis is to find a maximum value of the admissible delay to guarantee the asymptotic stability of the concerned networks.The maximum allowable delay bounds are recognized as one of the indexes for checking the conservatism of stability criteria.
In this regard, state estimation is a prerequisite and important job to the applications of many practical systems with time-delay and parameter uncertainties.In [11][12][13][14], state estimation analysis for the delayed network with the uncertainties in both the original system and estimator parts has investigated, but, in [10], the estimation with the uncertainties in only the original system part was addressed.Here, since the original system part is always exposed in the environment change, its states are not fully available in the system outputs.Hence, through the state estimator design, the estimated states are utilized in many applications.In view of this, the model with the uncertainties in only the original system proposed in [10] is more like a real environment.
On the other hand, in very recent times, after the introduction of the Bernoulli sequence to engineering, it has been applied in many situations such as random delays [20] and sensors fault [21].Also, various forms of randomly occurring concept, for example, randomly occurring nonlinearities, randomly occurring delays, randomly occurring sensors saturations, and so on, are represented by the Bernoulli sequence [22,23].One regret in the aforementioned literature is that the works have been mainly addressed by the Bernoulli sequence without the variance information.Thus, it is more realistic to assume that the system parameter uncertainties are the stochastic sequence with the variance information.Moreover, the stability problem for various delayed neural networks with the stochastic property such as Markovian jump [24][25][26][27][28][29][30][31][32][33][34] is investigated.However, to the best of authors' knowledge, this problem of state estimation for delayed neural networks with the randomly occurring parameter uncertainties utilizing the variance information has not been tackled in any other literature yet.
With this motivation mentioned above, in this paper, a new model of neural networks with stochastic parameter uncertainties in only the original system part is constructed.Then, by construction of a newly augmented Lyapunov-Krasovskii functional and utilization of reciprocally convex approach [35] with some added decision variables, a new state estimator design method for the new proposed model is derived in Theorem 8 with the framework of LMIs, which can be formulated as convex optimization algorithms amenable to computer solution [36].Here, the information about not only mean but also variance of stochastic parameter uncertainties is utilized in Theorem 8 when designing a gain of state estimator.Next, based on the result of Theorem 8, to show the less conservatism of main ideas such as a proposed Lyapunov-Krasovskii functional and some utilized techniques, a state estimation criterion for the nominal form of neural networks with time-varying delays will be introduced in Theorem 9. Through two numerical examples, it will be shown that the proposed methods can give their effectiveness and usefulness.
Notation.R  is the -dimensional Euclidean space, and R × denotes the set of all  ×  real matrices.For symmetric matrices  and ,  >  means that the matrix  −  is positive definite, whereas  ≥  means that the matrix  −  is nonnegative.  , 0  , and 0 ⋅ denote  ×  identity matrix, ×, and × zero matrices, respectively.E{⋅} stands for the mathematical expectation operator.‖ ⋅ ‖ refers to the Euclidean vector norm or the induced matrix norm.diag{⋅ ⋅ ⋅ } denotes the block diagonal matrix.For square matrix , sym{} means the sum of  and its symmetric matrix   ; that is, sym{} =  +   .For any vectors . Pr{} means the occurrence probability of the event .
Assumption 1.The functions  and g are assumed to be nondecreasing, bounded, and globally Lipschitz as follows: where () = () 1 () + (1 − ()) 2 () and the information of the parameter uncertainties is changed with Bernoulli sequence () defined by where () satisfies Pr{() = 1} = E{()} =  0 and  1 () and  2 () satisfy E{  ()} =   and E{(  () −   ) 2 } =  2  ( = 1, 2).In addition, it is assumed that the random variables () and   () ( = 1, 2) are mutually independent.Remark 3. In this work, the system parameter uncertainties are described by the general stochastic sequence with its information of variance and mean.After the introduction of the Bernoulli sequence to engineering, it has been applied in many situations such as random delays [20] and sensors fault [21].In very recent times, various forms of randomly occurring concept, for example, randomly occurring uncertainties, randomly occurring nonlinearities, randomly occurring delays, and so on, are represented by the Bernoulli sequence [22,23].Besides, the Markovian sequence, which is favorite stochastic sequence, is used to describe the unexpected changes of parameters in hybrid systems [37][38][39][40][41].It should be noted that the previous results utilizing Bernoulli and Markovian sequences have not utilized the information about the variance.In this paper, by defining () which includes  1 () and  2 (), the variance values of  1 () and  2 () will be considered in designing a state estimator of system (11).The effectiveness of these considerations will be explained in Example 1 of Section 4.
According to Bernoulli sequence (), two stochastic sequences  1 () and  2 () occurred in Figure 1(c).Thus, when () = 1, the  2 () is estranged from the systems, and the converse is also true.Among a lot of factors that influence the system parameter uncertainties, two or three dominant factors can be represented as the major ones.Then, the concept of Assumption 2 is introduced to describe the total effect of the dominant factors on the uncertainties which is one of the main ideas in this work.From Figure 1(c), the stochastic information, that is, the mean and the variance, can affect the dominant factors.Since the effects of the dominant factors cannot always coexist together, the effect degree of the dominant factors is different in the system.Therefore, to analyze this problem mentioned above, in this work, the state estimation for neural works with stochastic parameter uncertainties is dealt with by adopting the property of the stochastic sequence, which contain the information for mean and variance instead of studying the problem of stochastic state estimation of neural networks including (−  () +  0 (()) +  1 (( − ℎ())))(), where () is Wiener process.Moreover, by utilizing the proposed model, the dynamic behavior of practical problem nearer to the random change of real environment will become accessible.
By consideration of Remarks 3 and 4, in this paper, a model of system (3) with randomly occurring parameter uncertainties is considered as where () was defined in Assumption 2.
In order to estimate the neuron state of system (7), the following full-order state estimator is constructed as follows: where ŷ() is the estimate of the neuron state and  ∈ R × is the estimator gain matrix to be designed.Remark 5. Figure 2, which is cited from ETAP Co. website (http://www.etap.com)and drawn to explain clearly, shows state estimators telemetry data such as power measurements to obtain an estimate of the magnitudes and phase angles of bus voltages in the actual power systems as an example for randomly occurring parameter uncertainties.
In more detail, the system is affected with both uncertainties shown in Figure 3(a).Sometimes, the system parameter uncertainties are changed with randomly occurring situation shown in Figure 3(b).For details, from Figure 3, the uncertainties can affect multiply the system.The occurrence degree of uncertainties is different because the effects of uncertainties do not always occur in the same time.Therefore, to analyze this problem mentioned above, in this work, the randomly occurring parameter uncertainties are dealt with by adopting the property of the stochastic sequence.
Here, it is assumed that this situation obeys the stochastic rule in Remark 4.
The aim of this paper is to design a gain  which guarantees the asymptotic state estimation for the neural networks with time-varying delays and parameter uncertainties influenced by randomly occurring situations; in other words, to design the stabilization controller gain  for the error system (9).Therefore, the estimator gain  will be obtained through a design of the stabilization controller gain  for the error system (9).For simplicity of system representation, let us define the following vectors: The augmented system can be formulated as follows:  where Also, before deriving our main results, the following lemmas will be used in main results.

Main Results
In this section, a new estimator design method for the system (11) will be proposed in Theorem 8. Based on the results of Theorem 8, an estimator design method for the nominal form of the error dynamic system (9), that is,  = 0 ⋅ in (9), will be derived in Theorem 9.
Proof.Let us consider the following Lyapunov-Krasovskii functional candidate as where where  1 () = [ 1 , By the weak infinitesimal operator L in [44], the L 1 , L 2 , and L 3 are calculated as Before calculating the estimation of L 4 and L 5 estimates, inspired by the work of [45], the following zero equalities with any symmetric matrices   ( = 1, 2, 3, 4) are considered as a tool of reducing the conservatism of criterion which rearrange where Here,  1 () and  2 () will be used in L 4 and L 5 , respectively.By Lemma 6 and the  1 () in ( 23), the L 4 can be bounded as where  3 (, ) = col{(), ẋ (), (), ∫   ẋ ()} and  1 () = ℎ()ℎ −1  .Here, when 0 < ℎ() < ℎ  , since  1 () satisfies 0 <  1 () < 1, the following inequality for any matrix M 1 holds by reciprocally convex approach in [35] as Also, when ℎ() = 0 or ℎ() = ℎ  , we get −ℎ   3 () = 0 4×1 , respectively.Thus, if Ψ 1 ≥ 0 holds, then an upper bound of the L 4 can be rebounded as By similar process obtaining L 4 with  2 () in ( 23) and any matrix M 2 , the upper bound of L 5 is calculated as where . By Lemma 6, the upper bound of L 6 is estimated as Mathematical Problems in Engineering Lastly, the upper bound of L 7 is estimated as The condition of the activation functions (()) and () in (11) For any scalar  1 > 0 and Assumption 1, the following inequality holds: and summarizing (31) and ( 32) leads to In succession, since the relational expression between () and (), that is,   ()() ≤   ()(), holds from the system (11), there exists any scalar  2 > 0 satisfying the following inequality: where and, from (11), Moreover, to design the gain , we add the following zero equality with the matrix X = diag{ 1 ,  2 }, where  1 and  2 are any matrices, to be chosen as Then, by defining  =  2 , the equality just above leads to Therefore, from ( 21)- (38) and by application of the S-procedure [36], a sufficient condition guaranteeing stability for the system (11) can be Furthermore, by Lemma 7, condition ( 39) is equivalent to the following condition with any matrices ϝ 1 and ϝ 2 : In conclusion, condition ( 40) is equivalent to the LMIs ( 16) and ( 17).This completes the proof.
Theorem 8 provides robust estimator design method for the system (11) in the LMI framework.Based on the results of Theorem 8, we will propose an estimator design method for the nominal form of the system (11) which will be introduced as Theorem 9.For the sake of simplicity on matrix representation, ẽ ∈ R (16+)× ( = 1, 2, . . ., 17) are defined as block entry matrices and the notations of several matrices are defined as where Ω[ℎ()] ( = 1, 2) and Π, are substituted as Ω [ℎ()] , with replacing  and  by  and , and Π , with replacing   by ẽ and  = 0, respectively.Now, the following theorem is given by the second result.
Proof.Based on the Lyapunov-Krasovskii functional (19), the deriving process of its new upper bound given by is very similar to the proof of Theorem 8; so, it is omitted.At this time, the augmented vector Unlike the previous results [10][11][12][13][14], these state vectors have not been utilized as an element of augmented vector.Furthermore,  1 ,  2 ,  3 ,  4 , and  5 in Theorems 8 and 9 have not been proposed yet in designing a gain matrix of estimator for neural networks with time-varying delays.Thus, some new cross terms which may play a role to reduce the conservatism of criteria were considered in designing a gain matrix of (11).In Section 4, it will be shown that the proposed Lyapunov-Krasovskii functionals and some utilized techniques can reduce the conservatism by the comparison of maximum delay bounds with the previous results.
Remark 11.It should be noted that the zero equalities (23) are added in the results of L() as shown in (39).Inspired by the work of [45], the first two zero equalities at (22) are proposed and utilized in Theorems 8 and 9 to enhance the feasible region of stability criterion.Furthermore, the last two zero equalities at (22) are proposed for the first time in designing a gain matrix of estimator for neural networks with time-varying delays.By adding the integral terms  1 () and  2 () into L 4 and L 5 , respectively, and utilizing reciprocally convex optimization method [35], it may lead to less conservative results of the proposed theorems.

Numerical Examples
In this section, two numerical examples will be provided to illustrate the effectiveness of the proposed criteria.
By using the gain  c2 obtained with C.2 and ℎ() = 0.5 + 0.5 sin((0.3/0.5)), the results are drawn in Figure 6.Comparing with Figure 5, it can be confirmed that it is necessary to consider the information for the mean and variance of the stochastic sequences to investigate the state estimation in the real environment like the situation explained in Remarks 3 and 4.However, the performance is substantially improved by gain  c2 , but the maximum allowable delay bound has fallen off to ℎ  = 1.Moreover, the results with the gain  c3 under C.3 and ℎ() = 1 + sin(0.3)show that the mean information of the stochastic sequence is only considered as the simulation condition.The gain  c3 , although promising, has the maximum allowable delay bound, that is, ℎ  = 2,  that outweighs ℎ  = 1 of the case with C.2.At this time, the related simulation results are shown in Figure 7.
Finally, in order to have a favorable consideration, the sum of square of errors, that is, SOS = ∑ 3 =1  2  (), under  c1 ,  c2 , and  c3 with C.2 is drawn in Figure 8.   (46) For the above system, the results of maximum allowable delay bounds for various ℎ  are listed in Table 2.By applying Theorem 9, it can be guaranteed that the maximum allowable delay bounds under the same conditions are larger than the ones in the previous works, which supports the fact that the proposed Lyapunov-Krasovskii functional and some utilized techniques effectively reduce the conservatism in designing an estimator gain.Remark 12.In the field of delay-dependent analysis, one of the major issues is to obtain a delay bound for guaranteeing asymptotic stability of time-delay systems as large as possible and to use the decision variables while keeping the same delay bound as few as possible.In general, the former has taken priority over the latter.The number of decision variables used in Theorem 9 is larger than the existing results [10, 12, 14] since the utilized Lyapunov-Krasovskii functional in Theorem 9 is of the newly form and the development process of Theorem 9 uses Lemma 7 (see (40)).But, at the expense of larger number of decision variables comparing with the existing works, meaningful improvements on the feasible region of Theorem 9 were shown in Table 2 by comparing maximum delay bounds.Thus, by applying the main idea to various and important issues such as controller design and synchronization, the feasible region can be enhanced.

Conclusion
In this paper, the delay-dependent state estimation problem for the neural networks with time-varying delays and randomly occurring parameter uncertainties was studied.In Theorem 8, the robust state estimation criterion for delayed neural networks with stochastic parameter uncertainties was proposed by introducing the augmented Lyapunov-Krasovskii functional and proposing new idea mentioned in Remark 3. In Theorem 9, based on the result of Theorem 8, the improved state estimation criterion for the nominal form of neural networks with time-varying delays was presented.At this time, the new model of neural networks with stochastic parameter uncertainties was proposed and the uncertainties were assumed to be in only the original system part.Two illustrative examples have been given to show the effectiveness and usefulness of the presented criteria.

0
SOS underK c1 with C.2 SOS under K c2 with C.2 SOS under K c3 with C.2

Table 1 :
Estimator gains and maximum allowable delay bounds with fixed  0 = 0.3 and ℎ  = 0.3 (Example 1).This value guarantees stability of the concerned system, but the maximum allowable delay bound is larger than this value.