Scaled Diagonal Gradient-Type Method with Extra Update for Large-Scale Unconstrained Optimization

and Applied Analysis 3 This scaling resembles the Al-Baali [18] scaling that is applied within the Broyden family. Because the value of γ k is always < 1, then by incorporating the scaling to B k , it decreases the large eigenvalues of B k constantly, and consequently we can keep positive definiteness of B k+1 (since sT k y k > 0), which is an important property in descent method. In this case, the following updating: B k+1 = γ k B k + (sT k y k − γ k sT k B k s k )


Introduction
In this paper, we consider the unconstrained optimization problem min  () ,  ∈   , where () is a continuously differentiable function from   to .Given a starting point  0 , using notations   = (  ) = ∇(  ) and   as an approximation to the Hessian   = [∇ 2 (  )], the quasi-Newton-based methods for solving (1) are defined by the iteration  +1 =   −    −1    ,  = 0, 1, 2, . . ., where the stepsize   is determined through an appropriate selection.The updating matrix   is usually required to satisfy the quasi-Newton equation where  −1 =   −  −1 and  −1 =   −  −1 .One of the widely used quasi-Newton method to solve general nonlinear minimization is the BFGS method, which uses the following updating formula: On the numerical aspect, this method supersedes most of the optimization methods; however, it needs ( 2 ) storage which makes it unsuitable for large-scale problems.
On the other hand, an ingenious stepsizes selection for gradient method was proposed by Barzilai and Borwein [1] in which the updating scheme is defined by where   = (1/  ) and   =   −1  −1 /  −1  −1 .Since that, the study of new effective methods in the frame of BB-like gradient methods becomes an interesting research topic for a wide range of mathematical programming; for example, see [2][3][4][5][6][7][8][9][10].However, it is well known that BB method cannot guarantee a descent in the objective function at each iteration and the extent of the nonmonotonicity depends in some way on the size of the condition number of objective function [11].Therefore, the performance of BB method is greatly influenced by the condition of the problem (particularly, condition number of the Hessian matrix).Some new fixed stepsizes gradient-type methods of BB kind are proposed by [12][13][14][15][16] to overcome these difficulties.In contrast with the BB approach in which the stepsize is computed by means of a simple approximation of the Hessian in the form of scalar multiple of identity, these proposed methods consider approximation of the Hessian and its inverse in diagonal matrix form based on the weak secant equation and quasi-cauchy relation, respectively (for more details see [15,16]).Though these diagonal updating methods are efficient, their performance can be greatly affected by solving illconditioned problems.Thus, there is room for improve on the quality of the diagonal updates formulation.Since methods as described in [15,16] have useful theoretical and numerical properties, it is desirable to derive a new and more efficient updating frame for general functions.Therefore our aim is to improve the quality of diagonal updating when it is poor in approximating Hessian.
This paper is organized as follows.In the next section, we describe our motivation and propose our new-gradient type method.The global convergence of the method under mild assumption will be established in Section 3. Numerical evidence of the vast improvements due to the new approach is given in Section 4. Finally, conclusion is made in the last section.

Scaling and Extra Updating
Assume that   is positive definite, and let {  } and {  } be two sequences of -vectors such that      > 0 for all .Because it is usually difficult to satisfy the quasi-Newton equation (3) with a nonsingular  +1 of the diagonal form, one can consider satisfying it in some directions.If we project the quasi-Newton equation (3) (also called the secant equation), in a direction  such that     ̸ = 0, then it gives If  =   is chosen, it leads to the so-called weak-secant relation, Under this weak-secant equation, [15,16] employ variational technique to derive updating matrix that approximates the Hessian matrix diagonally.The resulting update is derived to be the solution of the following variational problem: and gives the corresponding solution  +1 as follows: where   = diag( 2 ,1 ,  2 ,2 , . . .,  2 , ),  , is the th component of the vector   , and tr denotes the trace operator.
Note that when      <        , the resulting  +1 is not necessarily positive definite and it is not appropriate for use within a quasi-Newton-based algorithm.Thus, it is desirable to propose a technique to measure the quality of   in terms of its Rayleigh quotient and try to find a way to improve "poor" quality   before calculating  +1 .For this purpose, it will be useful to propose, at first quality a criterion to distinguish between poor, and acceptable quality of   .
Let us begin by considering the curvature of an objective function,  in direction   , which is represented by where is the average Hessian matrix along   .Since it is not practical to compute the eigenvalue of the Hessian matrix in each iteration, we can estimate its relative size on the basis of the scalar If   > 1, it implies that the eigenvalues of   approximated by its Rayleigh are relatively small compared to those of the local Hessian matrix at   .In this condition, we find that the strategy of extra update [17] seems to be useful for improving the quality of   by rapidly increasing its eigenvalues up to those of the actual Hessian relatively.This is done by updating   twice to obtain B+1,2 : and use it to obtain, finally, the updated  +1 : On the other hand, when   < 1, it implies that the eigenvalue of   represented by its Rayleigh is relatively large and we have      −        < 0. In this case, we should suggest a useful strategy to encounter this drawback.As we reviewed before, the updating scheme may generate nonpositive definite  +1 when   has large eigenvalues relative to those possible values of   , that is, when      <        .On the contrary, this argument disappears when the eigenvalues of   are small (i.e.,      >        ).This suggests that the scaling should be made to scale down   , that is, choosing   < 1 only when      −       < 0 and take   = 1, whenever      >        .Combining these two arguments, we choose the scaling parameter   such that This scaling resembles the Al-Baali [18] scaling that is applied within the Broyden family.Because the value of   is always < 1, then by incorporating the scaling to   , it decreases the large eigenvalues of   constantly, and consequently we can keep positive definiteness of  +1 (since      > 0), which is an important property in descent method.In this case, the following updating: will be used.To this end, we have the following general updating scheme for  +1 : where B+1,2 and   are given by ( 13) and ( 15), respectively.An advantage of using ( 17) is that the positive definiteness of  +1 can be guaranteed in all iterations.This property is not exhibited in the other diagonal updating formula such as those in [15,16].Note that there is no extra storage required to impose our strategy and the cost of computing is also not increased significantly throughout the entire iteration.Now we can state the steps of our new diagonal-gradient method algorithm with the safeguarding strategy for monotonicity as follows.
In Step 4, we employ the nonmonotone line search of [19,20] to ensure the convergence of the algorithm.However, some other line search strategies may also be used.

Convergence Analysis
This section is devoted to study the convergence behavior of ESDG method.We will establish the convergence of the ESDG algorithm when applied to the minimization of a strictly convex function.To begin, we give the convergence result, which is due to Grippo et al. [21] for the step generated by the nonmonotone line search algorithm.Here and elsewhere, ‖ ⋅ ‖ denotes the Euclidean norm.
Theorem 1. Assume that  is a strictly convex function and its gradient  satisfies the Lipschitz condition.Suppose that the nonmonotone line search algorithm is employed in a case that the steplength,   , satisfies where  max  = max{(  ), ( −1 ), . . ., ( − )}, with  ≤  and  ∈ (0, 1), and the search direction   is chosen to obey the following conditions.There exist positive constants  1 and  2 such that for all sufficiently large .Then the iterates   generated by the nonmonotone line search algorithm have the property that To prove that the ESDG algorithm is globally convergent, it is sufficient to show that the sequence {‖  ‖} generated by ( 17) is bounded both above and below, for all finite  so that its associated search direction satisfies condition (19).Since   is diagonal, it is enough to show that each element of   , say  ()   ,  = 1, . . ., , is bounded above and below by some positive constants.The following theorem gives the boundedness of {‖  ‖}.Theorem 2. Assume that  is strictly convex function where there exist positive constants  and  such that for all ,  ∈   .Let {‖  ‖} be a sequence generated by the ESDG method.Then ‖  ‖ is bounded above and below for all finite , by some positive constants.

Numerical Results
In this section we present the results of numerical investigation for ESDG method on different test problems.We also compare the performance of our new method with that of the BB method and that of MDGRAD method which is implemented using SMDQN of [22] with a same nonmonotone strategy as the ESDG method.Our experiments are  performed on a set of 20 nonlinear unconstrained problems with dimensions ranging from 10 to 10 4 (Table 1).
These test problems are taken from [23,24].The codes are developed with Matlab 7.0.All runs are performed on a PC with Core Duo CPU.For each test, the termination condition is ‖(  )‖ ≤ 10 −4 .The maximum number of iterations is set to 1000.
Figures 1 and 2 show the efficiency of ESDG method when compared to MDGRAD and BB methods.Note that ESDG method increases the efficiency of Hessian approximation devoid of increasing the number of storages.Figure 2 also shows the implementation of the ESDG method with BB and MDGRAD methods using the CPU time as a measure.This figure shows that ESDG method is again faster than MDGRAD method in most problems and requires reasonable time to solve large-scale problems when compares to the BB method.Finally, we can conclude that our experimental comparisons indicate that our extension is very beneficial to the performance.

Conclusion
We have presented a new diagonal gradient method for unconstrained optimization.Numerical study of the proposed method when compared with BB and MDGRAD methods is also performed.Based on our numerical experiments, we can conclude that ESDG method is significantly preferable compared to the BB and MDGRAD methods.Particularly, the ESDG method is proven to be a good option for large-scale problems when high-memory locations are required.In view of the remarkable performance of ESDG method, globally converged and with only () storage, we can expect that our proposed method would be useful for unconstrained large-scale optimization problems.

Figure 1 :
Figure 1: Performance profile based on iterations.

Figure 2 :
Figure 2: Performance profile based on CPU time per iteration.

Table 1 :
Test problem and its dimension.