Robust Linear Neural Network for Constrained Quadratic Optimization

Based on the feature of projection operator under box constraint, by using convex analysismethod, this paper proposed three robust linear systems to solve a class of quadratic optimization problems. Utilizing linear matrix inequality (LMI) technique, eigenvalue perturbation theory, Lyapunov-Razumikhin method, and LaSalle’s invariance principle, some stable criteria for the related models are also established. Compared with previous criteria derived in the literature cited herein, the stable criteria established in this paper are less conservative andmore practicable. Finally, a numerical simulation example and an application example in compressed sensing problem are also given to illustrate the validity of the criteria established in this paper.


Introduction
Quadratic optimization problem is a simple but very important and basic problem in convex optimization theory.It is widely applied in many scientific and engineering applications, such as regression analysis, data fusion, system identification, filter design, and compressed sensing [1][2][3][4][5].Among these applications, the real-time solutions of such quadratic optimization problems are often required.In order to solve quadratic optimization problem, many different algorithms are provided in previous reference, such as proximal point algorithm (PPA), extended PPA, and splitting methods [6][7][8].However, in many practical optimization problems, the numbers of decision variables and constraints are usually very large.When a large-scale quadratic optimization problem has to be performed in practical problem, computation complexity becomes more challenging.For such applications, classical optimization techniques may not be competent due to the problem dimension and stringent requirement on computational time [9].One promising method for solving these problems is to employ artificial recurrent neural networks, since neural network has parallel computing capacity [10].
Mathematically, the optimization problem to be solved is mapped into a dynamical system so that its state output can give the optimal solution and the optimal solution is then obtained by tracking the state trajectory of the designed dynamical system based on the numerical ordinary differential equation technique [11].From the pioneering work of McCulloch and Pittes, numerous neural network models have been developed.Compared with conventional numerical optimization algorithm, neural network has a low model complexity and parallel computing capacity, it is more suitable for engineering applications, and it has a weaker global convergence condition.
In the recent decades, all kinds of different neural network models were established to solve variant constrained optimization problems.These optimization problems include game theory, linear programming problems, linear complement problems, projection equation problems, variational inequality problems, nonlinear optimization problems, general convex optimization problems, nonconvex optimization problems, and nonsmooth optimization problems.
Neural network for solving linear programming problem perhaps stemmed back from Pyne's work [12] and Tank and Hopfield's work [13].Their seminal work has inspired other researchers to develop recurrent neural networks for nonlinear optimization.Zhang and Constantinides derived a Lagrangian neural network for solving nonlinear convex optimization problems with linear equality constraints in [14].In [15], Zhang researched the exponential stability of quadratic optimization problem on neural network and established a discrete-time neural network model to solve quadratic optimization problem with convex constraint only.Tan et al. studied the global exponential stability of discretetime neural network for constrained quadratic optimization problem in [16].In order to solve more general optimization problem, Yashtini and Malek investigated a discrete-time neural network model for solving nonlinear convex problems with hybrid constraints in [17].Bouzerdoum and Pattison presented a neural network for solving quadratic convex optimization problems with bounded variables in [18].Xia proposed primal-dual neural networks for solving linear and quadratic programming problems in [19], researched a dual neural network for solving strictly convex quadratic programming problems in [20], and proposed a Bi-projection neural network for solving constrained quadratic optimization problems in [21].To solve quadratic minimax optimization problems, Liu and Wang [22] proposed a projection neural network (PNN) for constrained quadratic minimax optimization problem.To solve nonsmooth optimization problems, Liu and Wang [23] proposed a one-layer PNN for solving a class of pseudoconvex and nonsmooth nonlinear optimization problems.
It is worth pointing out that most of above established neural network models are nonlinear forms.And the stability criteria derived in the literature are based on Lyapunov stable theory.However, when the constrained conditions of quadratic optimization are box constrained, nonlinear projection operator satisfies section constraint; in this case, nonlinear projection operator can be expressed by a linear form with uncertain term.This means that nonlinear projection neural network can be rewritten as a robust linear system.Thus, utilizing eigenvalue perturbation theory, we can give out some new system stable criteria.This idea inspires this work.In this paper, using convex analysis tool, we first establish some new robust linear neural network for box constrained quadratic optimization; then, by using LMI technique and eigenvalue perturbation theory, some exponentially stable criteria are established.When time delays are considered, by using Lyapunov-Razumikhin method and LaSalle's invariance principle, we further derive some asymptotical stability criteria for the established time-delayed linear robust neural network.To illustrate the efficiencies and validity of the derived stability criteria in this paper, a numerical example and an application example in compressed sensing problem are also given.
The remainder of this paper is organized as follows.In Section 2, a constrained quadratic optimization problem and related neural network models are described.In Section 3, the global stability and convergence of the proposed neural network are analyzed.In Section 4, a simulation numerical example and an application to compressed sensing are given.Finally, conclusions are drawn in Section 5.

Problem and Neural Network Model
Consider the following constrained quadratic optimization problem: where  = ( In what follows, by using projection theory and equivalent transform, some neural network models for solving problem (1) will be first introduced.

Neural Network without Time Delays.
As is well known, without considering time delays, the optimal solution  * for problem (1) is equivalent to the equilibrium point of the following project neural network: where  > 0 is an arbitrary constant and  Ω (⋅) is a projection operator defined by where , the display expression of  Ω (  ) is as follows: ( Denote  * as an equilibrium point of system (5), and let () = () −  * ; system (5) can be rewritten in the following form: where  Ω (()) =  Ω (() +  * ) −  Ω ( * ).Notice the expression of  Ω (⋅) in ( 4); one can obtain that Denote Set And system (6) can be rewritten in the following linear robust neural network model form:

Time-Delayed Neural Network Type II.
Another timedelayed neural network, which can be regarded as another improvement form for model (2), can be suggested for solving (1) as follows: where time delay  > 0 is a constant.Set ( − ) = ( − ) − (−)−, and substitute (−) into (20); similar to the technique used above, system (20) can be transformed into the following equivalent time-delayed linear robust neural network form:

Stability Analysis
Since systems (11), (19), and ( 21) are all linear differential equations with uncertain term, the stability criteria for these systems can be derived by using eigenvalue perturbation theory, Lyapunov-Razumikhin method, and LMI technique.
To derive the stability criteria, we first introduce the following lemma.
Lemma 1 (see [24]).Suppose that where  is a constant matrix satisfying  −   > 0, and () is an uncertain matrix satisfying   ()() ≤ ,   = , , and  of appropriate dimensions; the inequality holds if and only if, for some  > 0, By using Lemma 1, the following stability criterion can be derived.
Theorem 2. The equilibrium point  * of system ( 2) is exponentially stable if there exists a positive constant  such that the following linear matrix inequality holds: and the exponential convergence rate is min  {  }, where   is the eigenvalue of matrix  = ( +   )/2,  = − + ( − )Δ().
Proof.From Section 2.1, it follows that the equilibrium point's exponential stability of system (2) is equivalent to the exponential stability of the trivial solution of system (11).Notice that system (11) is a linear system structure; thus the exponential stability of the trivial solution of system ( 11) is equivalent to Re() < 0, where  is an arbitrary eigenvalue of matrix = 0 be an identity eigenvector belonging to eigenvalue   ; namely, Since  is a normal matrix, there exists a unitary matrix  such that this means that Re (  ) = ⟨, ⟩ = ⟨,   ⟩ =     .(29) Set  =   ; one can obtain that thus Notice that  1 ≥  2 ≥ ⋅ ⋅ ⋅ ≥   ; it yields which means that   ≤ Re(  ) ≤  1 .Obviously, if   < 0 ( = 1, 2, . . ., ) it means that Re(  ) < 0. Because   is eigenvalue of matrix  and matrix  is a symmetric real matrix, this implies that if matrix  < 0, then Re(  ) < 0. On the other hand,  < 0 is equivalent to Since Δ  ()Δ() ≤ , by Lemma 1, if there is a positive constant  such that then Re(  ) < 0,  = 1, 2, . . ., , which means that the trivial solution is exponentially stable; this completes the proof.
Remark 3. Obviously, the stability condition established in Theorem 2 is a LMI form.Using Matlab LMI tool box, it can be easily solved.However, when a large-scale quadratic optimization problem has to be performed, the computation complexity becomes challenging.In order to overcome this flaw, by using eigenvalue perturbation theory, another more simple and practical result can be derived as follows.Before continuing, the following lemma is needed.
Obviously, ‖ − ‖ ∞ < 1 means that arbitrary eigenvalue  of matrix −+(−)Δ() is negative; this means that the equilibrium point of system (2) is exponentially stable, which completes the proof.Remark 6. Usually, matrix infinite norm is larger than matrix 2-norm; thus the result established in Theorem 5 can be rewritten by matrix 2-norm form further.And notice the special property of identity matrix ; by using improved eigenvalue perturbation theory, another exponentially stable criterion can be derived as follows.Before continuing, the following lemma is needed.

Remark 9.
As is well known, Lyapunov-Razumikhin method is a powerful stability analysis tool for linear time-delayed system.When time delays are considered, by using Lyapunov-Razumikhin method, a similar asymptotically stable result with Theorem 8 for systems (12) and ( 19) can be derived as follows.Before continuing, the following Razumikhin condition is needed.

Proof. Construct a positive definite
Obviously, from Razumikhin condition, if ‖ − ‖ 2 < 1, then the equilibrium point  * of system ( 12) is asymptotically stable, which completes the proof.
Remark 12. Similar to the proof of Theorem 11, by using Razumikhin condition, a delay-dependent stable result for systems (20) and ( 21) can be derived as follows.
Theorem 13.The equilibrium point  * of system (21) is Proof.
Remark 14.If  = 0, system (21) degenerates into system (11); Theorem 13 becomes the same as in Theorem 8; thus the criterion derived in Theorem 8 can be regarded as a special case of Theorem 13.Theorem 15.If the symbol < in Theorems 2, 5, 8, and 11 becomes ≤, then, for any initial value  0 ∈ Ω, the solutions of systems (2) and (12) converge to a related equilibrium point  * .In particular, these systems are asymptotically stable when their equilibrium set contains exactly one point.
Proof.Since Ω is a bounded and closed convex set, as is well known, the state vectors of systems ( 2) and ( 12) are bounded, and Ω is an invariance set.Utilizing LaSalle's invariance principle (see [28]), similar to the proofs in [20][21][22][23], one can easily obtain the results described in Theorem 15; this complete the proof.
Remark 16.Theorems 2, 5, 8, and 11 show that if the inequality signs are strict, then, for any initial  0 whether it is in feasible set Ω or not, the state vectors are exponentially or asymptotically convergent.When the inequality signs are not strict, Theorem 15 shows that the sufficient condition ensuring system being asymptotically stable is  0 ∈ Ω.On the other hand, notice that if  is sufficiently small, nonstrict inequality sign in Theorems 2, 5, 8, and 11 can be guaranteed; this means that, for arbitrary initial value  0 ∈ Ω with appropriate dimension, systems (2) and ( 12) are asymptotically stable.

Numerical Simulation
and  = − = [20, −20, −20]  .For quadratic optimization (1), the author in [29] constructed a discrete-time neural network to solve this problem and gave out a globally convergent criterion.In order to further illustrate the globally exponential convergence of the neural networks constructed in [29], Tan et al. gave out some new improvement stable criteria in [16].In [30], the authors constructed a continuous neural network to solve problem (1) and derived a globally exponential convergent criterion.In order to reduce the conservation of the criterion established in [30], literature [31] further gave out an improved stable result as ‖ − ‖ 2 < 1.
If we construct model (11) to solve problem (1), by direct computation, it follows that ‖ − ‖ 2 = 0.9984 < 1; from Theorem 8, it yields that the state vector of system (11) is exponentially convergent to the optimization value of problem (1).Simulation result is illustrated in Figure 1 with initial value [2, −1, 1.5]  , from which one can see that the state vector of system ( 11) is convergent to the equilibrium point exponentially.Obviously, ‖ − ‖ 2 = 0.9984 < ‖ − ‖ ∞ = 1.0696 < ‖ − ‖ 2 = 1.5338; this means that the criterion established in Theorem 8 is less conservative than some earlier results.
If we adopt model (19) to solve problem (1), by direct computing, it follows that ‖ − ‖ 2 = 0.9984 < 1; from Theorem 11, it means that the state vector of system ( 19) is exponentially stable.Simulation result is illustrated in Figure 2 with  = 4.If we use model (21) to solve problem (1), from the result established in Theorem 13, one can see that the state vector stable condition is stringently required for time delay.For example, in system (21), let  = 0.01; simulation  result in Figure 3 shows that the state vector of system ( 21) is divergent.This means that even small time delay can cause the state vector of system (21) to be instable.Hence, when time delays must be considered, model (19) has more superiority than model (21).

Application to Compressed
Sensing.The sparsest solution of an undermined linear system of equations can be found by solving the so-called ℓ 0 -norm minimization problem; that is, where  ∈ R × ,  < ,  ∈ R  , and ‖‖ 0 denotes the number of nonzero components in .It is well known that Obviously, problem (57) is a typical quadratic optimization problem.From ( 1) and ( 2), the optimal solutions of (57) are equivalent to the equilibrium points of the following projective neural network: direct computing shows that ‖ − ‖ 2 = 1; from Theorem 15, if we adopt initial  0 in feasible set Ω, then the state vector of system (61) converges to optimizing solution of problem (55).
Remark 17.In addition to the application to compressed sensing, projection neural networks can also be applied to the motion generation and control of redundant robot manipulators.In [33], based on control perspective and projection neural networks, the authors researched distributed task allocation problem of multiple robots.Utilizing projection neural networks, literature [34] researches manipulability optimization problem of redundant manipulators.These new applications in robotics extend new application fields for projection neural network.How to use the technique derived in this paper to robotics field and solve related optimization problem will be a meaningful work, and this is also our future work direction.

Conclusions
This paper proposed three linear robust systems to solve a class of quadratic optimization problems.Utilizing LMI technique, eigenvalue perturbation theory, Lyapunov-Razumikhin method, and LaSalle's invariance principle, some stable criteria for related models are also established.Compared with previous criteria derived in the literature cited herein, the stable criteria established in this paper are less conservative and more practicable.Meanwhile, simulation results show that time-delay linear robust system type II is more sensitive to time delay than type I.This means that, in practical engineering problems, when time delay is needed to consider, model (19) has more superiority than model (21).Simulation example and application example in compressed sensing show that the results derived in this paper are valid.

Figure 1 :
Figure 1: State vector of system (2) with parameters given in Example 1.
The equilibrium point  * of system (2) is exponentially stable if ‖−‖ ∞ < 1, and the exponential convergence rate is min  {  }.