Without Diagonal Nonlinear Requirements : The More General P-Critical Dynamical Analysis for UPPAM Recurrent Neural Networks

Continuous-time recurrent neural networks (RNNs) play an important part in practical applications. Recently, due to the ability of assuring the convergence of the equilibriums on the boundary line between stable and unstable, the study on the critical dynamics behaviors of RNNs has drawn especial attentions. In this paper, a new asymptotical stable theorem and two corollaries are presented for the unified RNNs, that is, the UPPAM RNNs. The analysis results given in this paper are under the generally P-critical conditions, which improve substantially upon the existing relevant critical convergence and stability results, and most important, the compulsory requirement of diagonally nonlinear activation mapping in most recent researches is removed. As a result, the theory in this paper can be applied more generally.


Introduction
Neural networks have been rapidly developing into an important technology for about 30 years, and they can be grouped into two main types according to the networks' structure, that is, feed-forward neural networks (FNNs) and recurrent neural networks (RNNs).RNNs are the neural networks with feedback loops and by which the output of each neuron in the monolayer would feedback to the input of other neurons.RNNs are the dynamic systems that their states will vary with time pasting by.The crucial foundation of the RNNs consists in their dynamical properties, such as the global convergence, asymptotic stability, and exponential stability.Therefore, the analysis of such dynamical behaviors is the first and necessary step for any practical design and application of RNNs, such as recognition classification, adaptive control, and optimization.
In recent years, for different model individuals, considerable efforts have been devoted to the analysis on the stability of RNNs without and with delay (see, e.g., [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18] and the references therein).In order to generalize those results, the authors of [19] point out that most of the exponential stability results are given under the condition that one discriminate matrix defined by the RNNs, which is denoted by (, Γ), is positive.Here, (, Γ) =  −1 Γ − (Γ +   Γ)/2, and  = diag{ 1 ,  2 , . . .,   } with each   > 0 being the Lipschitz constant of   , () = ( 1 (),  2 (), . . .,   ())  ,  ∈ R  is the activation operator of the network, Γ is an arbitrary positive define diagonal matrix, and  is the weight matrix of the network.When another discriminate matrix which has the similar form as (, Γ) is negative, the RNNs are of exponential unstability.Further, they present the concept of the special critical condition and some special critical convergence analysis for two kinds of RNN models.In [20,21], the general critical condition is defined.From which, we can see that the critical condition is really an essential gap between stable and unstable for the given RNNs; that is, on one side of this gap, the RNNs are sure to be stable and on the other side, the RNNs are certainly unstable.And, when the RNNs fall into this gap, there exist stable trajectories as well as unstable ones.Studying the dynamics behaviors of RNNs under the critical condition is called as the critical analysis.The goal of the critical analysis is to find the least restrictions to assure the most stability of RNNs, which is usually corresponding to the design of where  = ( 1 ,  2 , . . .,   )  is the neural state vector,  = ( 1 ,  2 , . . .,   )  is the local field vector,  = (  ) × is the synaptic weight matrix,  is a positive constant,  is a fixed external bias vector, and  : R  → R  is the nonlinear activation operator.We now recall some notion and notations (taking system (1) as an example).A constant vector  * is said to be an equilibrium state of system (1), if  * is a fixed point of the operator () := ( + ), for all  ∈ R  . * is said to be stable if any trajectory of system (1) can stay within a small neighborhood of  * whenever the initial state  0 is close to  * , and it is said to be attractive if there is a neighborhood Δ( * ), called the attraction basin of  * , such that any trajectory of system (1) initialized from a state in Δ( * ) will approach to  * as time goes to infinity. * is said to be globally asymptotically stable on Δ( * ) if it is both stable and attractive, with the attraction basin Δ( * ).System (1) is said to be globally convergent on Θ if for every initial point  0 ∈ Θ, (,  0 ) converges to an equilibrium state of system (1) (the limit of (,  0 ) may not be the same for different  0 ).
Here we give some definitions about the activation operator  : R  → R  and the nonlinear operator .Denote the range of  by R().
Definition 1 (see [22]).The nonlinear activation operator  : R  → R  is said to be an -uniformly antimonotonous operator if there is a constant positive number  such that In [22], it is shown that most of the common nonlinear activation operators have the uniformly antimonotonous properties, such as the nearest point projection operator, the linear saturating operator, the signum operator, and the pseudoprojection operator.
Obviously, all the projection operators are pseudoprojection operators (here  = , and  refers to the identity matrix), and most of the concrete activation operators diversely appeared in RNNs are pseudoprojection operators.
In [22], when the operator  has both the pseudoprojection and uniformly antimonotonous properties, it is called as uniformly pseudoprojection, antimonotonous (UPPAM) operator.Specially, we say  is a (, )-UPPAM whenever it is a -projection and -uniformly antimonotonous operator.It is worthwhile to note that the UPPAM operator provides a very appropriate, unified framework within which most of the known RNN models can be embedded and uniformly studied [21][22][23].
The following definition of the nonlinear norm is similar to that of the matrix norm.

The Global Convergence Theorems of RNNs
In this section, the global convergence and asymptotic stability theorem and corollaries for RNNs with UPPAM operators of both systems (1) and (2) will be established under the critical condition.We consider the networks of form (1) first.
Suppose that  : R  → R  is the nonlinear activation operator.For any V ∈ R(), define (V) = (V + ) and Fix() as being the fixed point set of (V).Then by Brouwer's fixed point theorem,  has at least one fixed point V * ∈ Fix().As a result, the equilibrium state set of ( 1) is not empty.
Theorem 5 gives the global convergence and asymptotic stability result of UPPAM RNNs without diagonal nonlinear requirement under the -critical condition, while it is not quite easy to judge the condition that  ‖⋅‖ 2 (, , V * , Θ) ≤ 1.
Correspondingly, we can deduce the critical global convergence and asymptotical stability conclusions for RNN system (2).

Corollary 7.
Assume that  : R  → Θ is a (, )-UPPAM operator with each   being monotonically increasing and continuous.If there exists a nonnegative diagonal matrix  such that (L, ) +  ≥ 0 and one of the following conditions holds, then RNN model ( 2) is globally convergent on (Θ + ) when Fix() is disconnected.Moreover, when  * is the unique equilibrium point of (2), then  * is globally asymptotically stable on (Θ + ).
Proof.For any trajectory () of ( 2) starting from  0 ∈ (Θ) + , let  0 =  0 +  with  0 ∈ Θ and let () be the solution of (1) with initial value (0) =  0 ; then by the uniqueness of solution of differential equations, it is easy to verify that () ≡ () +  and there exists an equilibrium state of (1), denoted by V * , such that  * = V * + .Thus, by [25], the convergence of () to an equilibrium state of ( 2) can be shown by studying the asymptotic behavior of ().Then, the conclusion of Corollary 7 readily follows from Theorem 5 and Corollary 6.
Remark 8.In this section, we have presented the -critical convergence as well as the stability results for the uniformly pseudoprojection antimonotone RNNs.In detail, we obtain the global convergence and the asymptotic stability for the static UPPAM RNNs under the conditions that either the nonlinear norm defined by the networks is less than 1, or the matrix norms given by the networks satisfy one bounded requirement.And the corresponding results for the local field UPPAM RNNs are discussed.
It should be noticed that in the achieved theorem as well as the two corollaries here, the diagonally nonlinear requirement of the activation operators has been removed directly, which is a basic hypothesis in nearly all of the dynamics analysis obtained before.Thus, we improve most of the existing results for RNNs, either for those ones under the noncritical conditions or for those under the critical conditions (see, e.g., [6,13,19,21,[25][26][27][28] and the references therein).In addition of this, we know from [20] that the -critical analysis of RNNs can give the basic argument between stability and unstability of RNNs, so the discussion of the -critical dynamics analysis for RNNs without diagonally nonlinear requirement is quite meaningful both in theory and in applications.Further, since the UPPAM RNNs can formalize most of the existing RNNs individuals, thus the analysis results of dynamics behaviors for UPPAM RNNs may achieve the unified conclusions for RNNs, and which can discriminate the similarity and redundant of the dynamics results among the known RNNs individuals.In particular, the achieved results here can be applied directly to many RNNs models and can improve deeply the main results of those models, for example, the Cellular Neural Networks (CNNs) [7][8][9][10][11], the Brain-State-in-Box Neural Networks (BSB NNs) [29,30], the BCOp-type RNNs [31], and other commonly used specific individuals.

Illustrative Examples
In this section, we provide several illustrative examples to demonstrate the validity of the convergence and stability results formulated in the previous section.
Example 1.Consider the following RNN: where each In this example, it is easy to find that the activation operator  : R 4 → Θ = [−1,1] 4 and that the minimum Lipschitz constant of   = 1, ( = 1, 2, 3, 4).So we get that L = .In addition, the unique equilibrium state is For any positive diagonal matrix Γ, it is easy to verify that (L, Γ) = Γ − (Γ +   Γ)/2 is not positive and, further, not nonnegative.That is, all of the noncritical and critical conclusions in the literature (see, e.g., [19,27,32]) cannot be used here.But Theorem 5 can be applied to this example.Actually, the projection operator  is a (, 1)-UPPAM operator.Letting  = diag{3, 3, 3, 3}, we have  (L, ) +  = ( 0.5 1 1 0.5 1 1 0.5 1 1 0.5 It is obvious that (L, ) +  > 0 and  = Deco( On noting that for any  = ( 1 , From ( 29) and (30), it shows that      − V *     ≥ 4 (( So the inequality always holds.According to Theorem 5, system ( 25) is globally asymptotically stable on Θ.The following Figure 1 depicts the time responses of state variables of the system with random initial point starting from Θ, which confirm that the proposed condition in Theorem 5 ensures the globally asymptotical stability of the RNNs.Obviously, this example is established on a general projection operator, and all the diagonally nonlinear conclusions in the literature [20,28,32] cannot be used here.But the Corollary 7 established in Section 3 can be applied to Example 2. Actually, in this example  =  and  = 1.And it could be proofed that L = , that is because when Θ is the unit sphere in R 3 and  = ( 1 ,  2 ,  3 )  ,   is defined as follows: Let V ̸ =  be two arbitrary points in R 3 , we know that | And then we get L = .

Conclusion
Two basic dynamics behaviors, global convergence and asymptotical stability of both static and local field RNNs with UPPAM operators, have been studied under the critical condition.It has been proved that when the nonlinear norm determined by the network is bounded, then RNN with UPPAM operator possesses convergent and stable properties in the sense that a discriminant matrix (L, ) +  is nonnegative definite, where (L, ) is a matrix related to the network and  is an arbitrary nonnegative definite matrix.Compared with the existing dynamics analysis, the results in this paper extend most of the dynamics conclusions achieved.The requirements of net type, critical form and activation operator's character in the available literatures have been dearly relaxed.Some typical RNNs with UPPAM activation operators, such as most of the CNNs, BSB, and BCOp-type networks, can apply the theory obtained here to judge the dynamical behavior directly.The significance of the results obtained here not only lies in providing some further cognizance on the essentially dynamical behavior of RNNs, but also in enlarging the application field of them.
In this paper, we only achieved the global convergence and asymptotical stability result for UPPAM RNNs and did not discuss another important dynamics behavior, that is, the exponential stability for UPPAM RNNs, and this is under our current investigation.