Neural Network for Sparse Reconstruction

We construct a neural network based on smoothing approximation techniques and projected gradient method to solve a kind of sparse reconstruction problems.Neural network can be implemented by circuits and can be seen as an importantmethod for solving optimization problems, especially large scale problems. Smoothing approximation is an efficient technique for solving nonsmooth optimization problems. We combine these two techniques to overcome the difficulties of the choices of the step size in discrete algorithms and the item in the set-valuedmap of differential inclusion. In theory, the proposed network can converge to the optimal solution set of the given problem. Furthermore, some numerical experiments show the effectiveness of the proposed network in this paper.


Introduction
Sparse reconstruction is the term used to describe the process of extracting some underlying original source signals from a number of observed mixture signals, where the mixing model is either unknown or the knowledge about the mixing process is limited.The problem of recovering a sparse signal from noisy linear observation arises in many real world sensing applications [1][2][3][4][5].Mathematically, a signal recovery problem can be formulated as estimating the original signal based on noisy linear observations, which can be expressed as where  ∈ R × is the mixing matrix,  ∈ R  is the original signal,  ∈ R  is the observed signal, and  ∈ R  is the noise.In many cases,  is a matrix of block Toeplitz with Toeplitz blocks (BTTB) when zero boundary conditions are applied and block Toeplitz-plus-Hankel with Toeplitzplus-Hankel blocks (BTHTHB) when Neumann boundary conditions are used [6].Then, this problem can be viewed as a linear inverse problem.A standard approach to solve linear inverse problems is to define a suitable objective function and to minimize it.It is often divided into two steps to solve this problem, which are estimation of mixture matrix  and recovery of original signal .In this paper, we focus on the study of the second step, where we assume that we have known mixture matrix .
Generally, finding a solution with few nonzero entries for an underdetermined linear system with noise is often modeled as the regularization problem: where  > 0 and ‖‖ 0 is defined by the number of nonzero entries in .However, the  2 −  0 regularized problem ( 2) is difficult to deal with because of the discrete structure of the  0 norm, which derives researchers to pay attention to the continuous  2 −  1 minimization problem: The first term in (3) is often called as date fitting term, which forces the solutions of (3) closeness to the data, and the second term in it is often called as regularization or potential term, which is to push the solutions to exhibit some prior expected features.Under certain conditions,  2 −  1 problem and  2 −  0 problem have the same solution sets [7].The  2 −  1 problem is a continuous convex optimization problem and can be efficiently solved, which is known as Lasso [8].

Mathematical Problems in Engineering
A class of signal recovery problems can be formulated as where  is a linear operator,  is the regularization parameter that controls the trade-off between the regularization term and the data-fitting term, and constraint set Ω is a closed convex subset of R  .Optimization problems arise in a variety of scientific and engineering applications and they really need real time solutions.Since the computing time greatly depends on the dimension and the structure of the optimization problems, numerical algorithms are usually less effective in large scale or real time optimization problems.In many applications, real time optimal solutions are usually imperative, such as on-board signal processing and robot motion planning and control.One promising approach to handle these problems is to employ artificial neural network.During recent decades, neural dynamical method for solving optimization problems has been a major area in neural network research based on circuit implementation [9][10][11][12].First, the structure of a neural network can be implemented physically by designated hardware such as application-specific integrated circuits where the computational procedure is distributed and parallel.This lets the neural network approach solve optimization problems in running time at the order of magnitude much faster than conventional optimization algorithms executed on generalpurpose digital computers.Second, neural networks can solve many optimization problems with time-varying parameters.Third, the dynamical and ODE techniques can be applied to the continuous-time neural networks.And recent reports have shown that the global convergence can be obtained by the neural network approach under some weaker conditions.
Since the neural network was first proposed for solving linear [13,14] and nonlinear [15] programming problems, many researchers were inspired to develop neural networks for optimization.Many types of neural networks have been proposed to solve various optimization problems, for example, the recurrent neural network, the Lagrangian network, the deterministic annealing network, the projection-type neural network, a generalized neural network, and so forth.In [16], Chong et al. proposed a neural network for linear programming problem with finite time convergence.In [17], a generalized neural network was presented for solving a class of nonsmooth convex optimization problems.In [18], a neural network was defined by using the penalty function method and differential inclusion for solving a class of nonsmooth convex optimization problems.In fact, in many important applications, neural network built by a differential inclusion is an important method to solve a class of nonsmooth optimization problems.One has to mention that the optimization problems are not differentiable in many important applications.Moreover, the neural networks for smooth optimization problems required the gradients of the objective and constrained functions in such neural networks.So these networks cannot solve nonsmooth optimization problems.Using smoothing techniques in neural network is an effective method for solving nonsmooth optimization problems [19,20].The main feature of smoothing method is to approximate the nonsmooth functions by parameterized smooth functions [21,22].By smoothing approximations, we can give a class of smooth functions, which converge to the original nonsmooth function and whose gradients converge to the subgradient of nonsmooth function.For solving many constrained optimization problems, projection is a simple and effective method for solving the constraints.In [23,24], projection had been used in neural networks for solving some kind of constrained optimization problems.
Basing on the advantages of the neural networks, in this paper, we will propose a neural network and use some mathematical techniques to solve optimization problem (4).The problem (4) is nonsmooth.Many neural networks are modeled by differential inclusions, which have the difficulty in the choice of the right set-valued map.In this paper, we will introduce a smoothing function to overcome this problem.Using smoothing techniques into neural network is an interesting and promise method for solving (4).

Preliminary Results
In this section, we will introduce several basic definitions and lemmas, which are used in the development.
Furthermore, the Clarke generalized gradient of  at  is defined as Moreover, if  : R  → R is a convex function, then it has the following properties as well.
Proposition 2. If  : R  → R is a convex function, the following property holds: Since the constraint set of ( 4) is a closed convex subset of R  , then we use the projection operator to handle the constraint.The projection operator of  to the closed convex subset Ω is defined by The projection operator has the following properties.
Proposition 3. Consider the following: Definition 4. Let ℎ : R  → R be a locally Lipschitz function.
We call h : R  × [0, +∞) → R a smoothing function of ℎ, if h satisfies the following conditions.
Next, we will give some analysis on the proposed neural network (14).Theorem 6.For any initial point  0 ∈ Ω, there is a global and uniformly bounded solution of (14).
Theorem 7.For any initial point  0 ∈ Ω, the solution of ( 14) is unique and satisfies the following: (ii) the solution of ( 14) is convergent to the optimal solution set of (4).
From (20), we obtain that ‖() − ‖ 2 +  ∑  =1 (   (), ()) is nonincreasing and bounded form below on [0, +∞); therefore we have that lim Using Proposition 5 to the above result, we obtain that lim Moreover, we have that lim Combining ( 20) and (33), we confirm that lim Since () is uniformly bounded on the global interval, there is a cluster point of it, denoted as  * , which follows that there exists an increasing sequence   such that lim Using the expression of ( 14) and lim  → +∞ ẋ () = 0, we have Using Proposition 3 in the above equation, we have From Proposition 5, there exists Therefore,  * is a Clarke stationary point of (4).Since ( 4) is a convex programming,  * is an optimal solution of (4).Owning to the random cluster point of (), we know that any cluster point of () is an optimal solution of (4), which means that the solution of ( 14) converges to the optimal solution set of (4).

Numerical Experiments
In this section, we will give two numerical experiments to validate the theoretical results obtained in this paper and the good performance of the proposed neural network in solving the sparse reconstruction problems.
Example 1.In this experiment, we will test an experiment for the signal recovered with noise.Every original signal with sparsity 1 means that there is only one sound at time point.We use the following MATLAB codes to generate a 100 length original signal  ∈ R 5×100 , mixing matrix  ∈ R 5×4 , observed signal  ∈ R 4×100 , and noise  ∈ R 4×100 : s=zeros(5,100); We denote  * as the recovered signal using our method.Figures 1(a)-2(a) show the original, observed, and recovered signals using (14).Figure 2(b) presents the convergence of signal-to-noise ratio (SNR) along the solution of the proposed neural network.From this figure, we see that our method recovers this random original effectively.And we should state that the SNR of the recovered signal is 22.15 dB, where Example 2. In this experiment, we perform the proposed network ( 14) on the restoration of 20 × 20 circle image.The observed image is distorted from the unknown true image mainly by two factors: the blurring and the random noise.The blurring is a 2 Gaussian function: which is truncated such that the function has a support of 7 × 7. A Gaussian noise with zero mean and standard derivation of 0.05 dB is added to the blurred image.

Conclusion
Basing on the smoothing approximation technique and projected gradient method, we construct a neural network modeled by a differential equation to solve a class of constrained nonsmooth convex optimization problems, which have wide applications in sparse reconstruction.The proposed network has a unique and bounded solution with any initial point in the feasible region.Moreover, the solution of proposed network converges to the solution set of the optimization problem.Simulation results on numerical examples are elaborated upon to substantiate the effectiveness and performance of the neural network.