Multivariate Spectral Gradient Algorithm for Nonsmooth Convex Optimization Problems

We propose an extended multivariate spectral gradient algorithm to solve the nonsmooth convex optimization problem. First, by using Moreau-Yosida regularization, we convert the original objective function to a continuously differentiable function; then we use approximate function and gradient values of the Moreau-Yosida regularization to substitute the corresponding exact values in the algorithm. The global convergence is proved under suitable assumptions. Numerical experiments are presented to show the effectiveness of this algorithm.


Introduction
Consider the unconstrained minimization problem min ∈R   () , where  : R  → R is a nonsmooth convex function.The Moreau-Yosida regularization [1] of  at  ∈ R  associated with  ∈ R  is defined by where ‖⋅‖ is the Euclidean norm and  is a positive parameter.
The function minimized on the right-hand side is strongly convex and differentiable, so it has a unique minimizer for every  ∈ R  .Under some reasonable conditions, the gradient function of () can be proved to be semismooth [2,3], though generally () is not twice differentiable.It is widely known that the problem min ∈R   () and the original problem (1) are equivalent in the sense that the two corresponding solution sets coincidentally are the same.The following proposition shows some properties of the Moreau-Yosida regularization function ().
This proposition shows that the gradient function  : R  → R  is Lipschitz continuous with modulus 1/.In this case, the gradient function  is differentiable almost everywhere by the Rademacher theorem; then the Bsubdifferential [4] of  at  ∈ R  is defined by (7) where   = { ∈ R  :  is differentiable at }, and the next property of BD-regularity holds [4][5][6].
Proposition 2. If  is BD-regular at , then (i) all matrices  ∈   () are nonsingular; (ii) there exists a neighborhood N of  ∈ R  ,  1 > 0, and  2 > 0; for all  ∈ N, one has Instead of the corresponding exact values, we often use the approximate value of function () and gradient () in the practical computation, because () is difficult and sometimes impossible to be solved precisely.Suppose that, for any  > 0 and for each  ∈ R  , there exists an approximate vector   (, ) ∈ R  of the unique minimizer () in ( 2) such that The implementable algorithms to find such approximate vector   (, ) ∈ R  can be found, for example, in [7,8].
The existence theorem of the approximate vector   (, ) is presented as follows.
Proposition 3 (see Lemma 2.1 in [7]).Let {  } be generated according to the formula where   > 0 is a stepsize and   is an approximate subgradient at   ; that is, then (11) holds with (ii) Conversely, if (11) holds with   given by (13), then (12) holds: We use the approximate vector   (, ) to define approximation function and gradient values of the Moreau-Yosida regularization, respectively, by The following proposition is crucial in the convergence analysis.The proof of this proposition can be found in [2].
Multivariate spectral gradient (MSG) method was first proposed by Han et al. [16] for optimization problems.This method has a nice property that it converges quadratically for objective function with positive definite diagonal Hessian matrix [16].Further studies on such method for nonlinear equations and bound constrained optimization can be found, for instance, in [17,18].By using nonmonotone technique, some effective spectral gradient methods are presented in [13,16,17,19].In this paper, we extend the multivariate spectral gradient method by combining with a nonmonotone line search technique as well as the Moreau-Yosida regulation function to solve the nonsmooth problem (1) and do some numerical experiments to test its efficiency.
The rest of this paper is organized as follows.In Section 2, we propose multivariate spectral gradient algorithm to solve (1).In Section 3, we prove the global convergence of the proposed algorithm; then some numerical results are presented in Section 4. Finally, we have a conclusion section.

Algorithm
In this section, we present the multivariate spectral gradient algorithm to solve the nonsmooth convex unconstrained optimization problem (1).Our approach is using the tool of the Moreau-Yosida regularization to smoothen the nonsmooth function and then make use of the approximate values of function  and gradient  in multivariate spectral gradient algorithm.

Remarks. (i)
The definition of  +1 = (‖  (  ,   )‖ 2 ) in Algorithm 5, together with (15) and Proposition 3, deduces that then, with the decreasing property of  +1 , the assumed condition   = ( 2  ‖  ‖ 2 ) in Lemma 7 holds.(ii) From the nonmonotone line search technique (22), we can see that  +1 is a convex combination of the function value   ( +1 ,  +1 ) and   .Also   is a convex combination of the function values   (  ,   ), . ..,   ( 1 ,  1 ),   ( 0 ,  0 ) as  0 =   ( 0 ,  0 ). is a positive value that plays an important role in manipulating the degree of nonmonotonicity in the nonmonotone line search technique, with  = 0 yielding a strictly monotone scheme and with  = 1 yielding   =   , where which shows that the proposed multivariate spectral gradient algorithm possesses the sufficient descent property.

Global Convergence
In this section, we provide a global convergence analysis for the multivariate spectral gradient algorithm.To begin with, we make the following assumptions which have been given in [5,[12][13][14].
Assumption A. (i)  is bounded from below.
The following two lemmas play crucial roles in establishing the convergence theorem for the proposed algorithm.By using (26) and (27) and Assumption A, similar to Lemma 1.1 in [20], we can get the next lemma which shows that Algorithm 5 is well defined.The proof ideas of this lemma and Lemma 1.1 in [20] are similar, hence omitted.Lemma 6.Let {  (  ,   )} be the sequence generated by Algorithm 5. Suppose that Assumption A holds and   is defined by (25).Then one has   (  ,   ) ≤   ≤   for all .Also, there exists a stepsize   satisfying the nonmonotone line search condition.Lemma 7. Let {(  ,   )} be the sequence generated by Algorithm 5. Suppose that Assumption A and   = ( 2  ‖  ‖ 2 ) hold.Then, for all , one has where  0 > 0 is a constant.

Proof (Proof by Contradiction
where the second inequality follows from (26), Part 3 in Proposition 4, and  +1 ≤   , the equality follows from   = ( 2  ‖  ‖ 2 ), and the last inequality follows from (27).Dividing each side by   and letting  → ∞ in the above inequality, we can deduce that which is impossible, so the conclusion is obtained.
By using the above lemmas, we are now ready to prove the global convergence of Algorithm 5. Proof.Suppose that there exist  0 > 0 and  0 > 0 such that       (  ,   )     ≥  0 , ∀ >  0 .
From ( 22), (26), and (29), we get Therefore, it follows from the definition of  +1 and (23) that By Assumption A,  is bounded from below.Further by Proposition 4, (  ) ≤   (  ,   ) for all , we see that   (  ,   ) is bounded from below.Together with   (  ,   ) ≤   for all  from Lemma 6, it shows that   is also bounded from below.By (38), we obtain On the other hand, the definition of  +1 implies that  +1 ≤  + 2, and it follows that

Numerical Results
This section presents some numerical results from experiments using our multivariate spectral gradient algorithm for the given test nonsmooth problems which come from [21].We also list the results of [14] (modified Polak-Ribière-Polyak gradient method, MPRP) and [22] (proximal bundle method, PBL) to make a comparison with the result of Algorithm 5.All codes were written in MATLAB R2010a and were implemented on a PC with 2.8 GHz CPU, 2 GB of memory, and Windows 8. We set  =  = 1,  = 0.9,  = 10 −10 , and  = 0.01, and the parameter  is chosen as  2-3, where "Nr." denotes the name of the tested problem, "NF" denotes the number of function evaluations, "NI" denotes the number of iterations, and "()" denotes the function value at the final iteration.The value of  controls the nonmonotonicity of line search which may affect the performance of the MSG algorithm.Table 2 shows the results for different parameter , as well as different values of the parameter   ranging from 1/6( + 2) 6 to 1/2 2 on problem Rosenbrock, respectively.We can conclude from the table that the proposed algorithm works reasonably well for all the test cases.This table also illustrates that the value of  can influence the performance of the algorithm significantly if the value of  is within a certain range, and the choice  = 0.75 is better than  = 0.
Then, we compare the performance of MSG to that of the algorithms MPRP and PBL.In this test, we fix   = 1/2 2 and  = 0.75.To illustrate the performance of each algorithm more specifically, we present three comparison results in terms of number of iterations, number of function evaluations, and the final objective function value in Table 3.
The numerical results indicate that Algorithm 5 can successfully solve the test problems.From the number of iterations in Table 3, we see that Algorithm 5 performs best among these three methods, and the final function value obtained by Algorithm 5 is closer to the optimal function value than those obtained by MPRP and PBL.In a word, the numerical experiments show that the proposed algorithm provides an efficient approach to solve nonsmooth problems.

Conclusions
We extend the multivariate spectral gradient algorithm to solve nonsmooth convex optimization problems.The proposed algorithm combines a nonmonotone line search technique and the idea of Moreau-Yosida regularization.The algorithm satisfies the sufficient descent property and its global convergence can be established.Numerical results show the efficiency of the proposed algorithm.

Table 2 :
Results on Rosenbrock with different  and .