A New Method with Sufficient Descent Property for Unconstrained Optimization

and Applied Analysis 3 Theorem 2. Let sequences {d k } and {x k } be generated by (13) and (2); then gT k d k ≤ − 󵄩󵄩󵄩gk 󵄩󵄩󵄩 2 , (16) for all k ≥ 0. Proof. Obviously, the conclusion is true for k = 0. If k ≥ 1, multiplying (13) by gT k , we have gT k d k = − 󵄩󵄩󵄩gk 󵄩󵄩󵄩 2 + gT k (β k g k−1 − θ k y k−1 ) = − 󵄩󵄩󵄩gk 󵄩󵄩󵄩 2 + gT k y k−1 󵄩󵄩󵄩gk−1 󵄩󵄩󵄩 2 gT k g k−1 − 󵄩󵄩󵄩gk 󵄩󵄩󵄩 2 󵄩󵄩󵄩gk−1 󵄩󵄩󵄩 2 gT k y k−1 = − 󵄩󵄩󵄩gk 󵄩󵄩󵄩 2 + gT k y k−1 󵄩󵄩󵄩gk−1 󵄩󵄩󵄩 2 (gT k g k−1 − 󵄩󵄩󵄩gk 󵄩󵄩󵄩 2 ) = − 󵄩󵄩󵄩gk 󵄩󵄩󵄩 2 + gT k y k−1 󵄩󵄩󵄩gk−1 󵄩󵄩󵄩 2 gT k (g k−1 − g k ) = − 󵄩󵄩󵄩gk 󵄩󵄩󵄩 2 − (gT k y k−1 ) 2


Introduction
Consider the unconstrained optimization problem min ∈   () , where  :   →  is a continuously differentiable function.
For solving (1), the following iterative formula is often used: where   is the current iterative point,   > 0 is a step size which is determined by some line search, and   is a search direction.Different search directions correspond to different iterative methods [1][2][3][4].Throughout this paper,   = ∇(  ) is an -dimensional column vector,  −1 =   − −1 , ‖⋅‖ and  are defined as the Euclidian norm and transpose of vectors, respectively.Generally, if there exists a positive constant  > 0, such that then the search direction   possesses sufficient descent property.This property may be crucial for the iterative methods to be global convergence [5], and some numerical experiments have shown that sufficient descent methods are efficient [6].
However, not all iterative methods can satisfy sufficient descent condition (3) under some inexact linear search conditions, such as the conjugate gradient method proposed by Wei et al. [7] or the gradient method presented in [8].In order to make the search direction   satisfy the condition (3) at each step, much effort has been done [9][10][11][12].
In [9], Cheng proposed a modified PRP conjugate gradient method in which the search direction   is determined by where   =  PRP  =     −1 /‖ −1 ‖ 2 ,      is a × matrix and  is an identity matrix.
In [10], Zhang et al. derived a simple sufficient descent method; the search direction   is given by Recently, Zhang et al. [11] presented a three-term modified PRP conjugate gradient method; the search direction   is generated by where We note that (4), (5), and ( 6) can be written as a linear combination of the steepest descent direction and the projection of the original direction; that is, where   is an original direction,   is a scalar, and   ∈   is any vector such that      ̸ = 0 holds.Indeed, if   =  PRP  ,   =   , and   =  −1 , then (8) reduces to the method (4).Let   = 1,   =   , and   =  −1 ; then (8) reduces to the method (5).When   =  PRP  ,   =  −1 , and   =  −1 , it is easy to deduce that (8) reduces to the method (6).From (8), we can easily obtain Thus, one has      = −‖  ‖ 2 for all .It implies that the sufficient descent condition (3) holds with  = 1.But the method (5) does not possess a restart feature which can avoid the jamming phenomenon.In addition, the methods ( 4) and ( 6) may not always be globally convergent under some inexact linear search [13], such as the standard Armijo-type line search which is given as follows: where  ∈ (0, 1) and  ∈ (0, 1/2).Motivated by ( 8) and ( 9), our purpose is to design a direction in the subspace { ∈   |     = −  }, where   ≥ 0 is a parameter.This direction can be written as where V  ∈   is any vector such that V     ̸ = 0 holds.Let It is clear that (8) can be regarded as a special case of (12) with   = 0. Therefore, (12) will have a wider application than (8).
If we take   =  PRP  ,   =  −1 , V  =  −1 ,   =  −1 , and   = (    −1 ) 2 /‖ −1 ‖ 2 in (12), then a new search direction is given as follows: where In this paper, we present a new iterative method for unconstrained optimization problems; the search direction is defined by ( 13) and (14).We prove that   satisfies      ≤ −‖  ‖ 2 without any line search.It means that the sufficient descent condition (3) holds with  = 1.Furthermore, we prove that the proposed method is globally convergent under the standard Armijo-type line search or the modified Armijotype line search.From ( 13) and ( 14), we can see that the proposed method has a restart feature that directly addresses the jamming problem.In fact, when the step   −  −1 is small, then the factor  −1 tends to zero vector.Therefore, the direction   generated by ( 13) is very close to the steepest descent direction −  .
The rest of this paper is organized as follows.In Section 2, we propose a new algorithm and discuss its sufficient descent property.In Section 3, the global convergence of the proposed method is proved under the modified Armijo-type line search or the standard Armijo line search.Some numerical results are given to test the performance of the proposed method in Section 4. Finally, we have some conclusions about the proposed method.

New Algorithm
In this section, the specific iterative steps of the proposed algorithm are listed as follows.
Algorithm 1.Consider the following.
Step 2. If ‖  ‖ = 0, then stop; otherwise go to the next step.
Proof.Obviously, the conclusion is true for  = 0.
Theorem 2 shows that the search direction   given by (13) possesses the sufficient descent property for any line search.

Convergence Analysis
The following assumptions are often needed to prove the global convergence of nonlinear conjugate gradient methods [14,15].In this section, we also use these assumptions in the convergence analysis of the proposed method.Assumption 3. Consider the following.Proof.The results of this lemma will be proved in the following two cases.
Remark 6.If the search direction   is defined by (13) with , then the sufficient descent property and global convergence can also be proved similar to the proof of Theorems 2 and 5.

Numerical Results
In this section, some numerical results are provided to test the performance of the proposed method, and the proposed method is compared with the existing methods [9][10][11].For the sake of simplicity, the proposed method and other comparative methods are named by NSDM, LPRP [11], SSD [10], and MPRP [9], respectively.The test problems and initial points are from [16].The test problems are listed in Table 1.In our experiment, all the codes were written in MATLAB 7.0 and run on PC with 2.00 GB RAM memory, 2.10 GHz CPU, and windows 7 operation system.In all algorithms, the step size   is computed satisfying the modified Armijo-type line search (15) with  = 0.1,  = 0.1, and  = 1, and the stopping condition is ‖  ‖ ≤ 10 −5 .We also stop these algorithms if CPU time is over 500(s).
In Table 2, P, N, NI, NF, NG, and CPU stand for th number of test problems, the dimension of the vectors, the number of iterations, the number of function evaluations, the number of gradient evaluations, and the run time of CPU in seconds, respectively.The symbol "-" means that the corresponding method fails in solving the test problems when the CPU time is more than 500 seconds, and the star * denotes that the numerical result is the best one among all the comparative methods.
In Table 2, we compare the performance of the new method by testing 28 different problems.According to the distribution of the star * , one can see that the NSDM method performs better than the LPRP, MPRP, and SSD methods with 14 test problems, worse than the MPRP method with 1 test problem and worse than the LPRP method with 6 test In order to compare the performance of these methods clearly, we adopt the performance profiles introduced by Dolan and Moré [17].The performance results are shown in Figures 1-4, respectively.In [17], Dolan and Moré introduced the notion as a means to evaluate and compare the performance of the set solvers  on a test set . Assuming   solvers and   problems exist, for each problem  and solver , they defined  , = computing time (the number of iterations or others) required to solve problem  by solver . (32) The performance ratio is given by  method.Although the LPRP method outperforms the NSDM method for 1.2 <  < 2.4 in Figure 1, 1.2 <  < 3.2 in Figure 2, 1.2 <  < 2.2 in Figure 3, and 1.1 <  < 2.8 in Figure 4, the NSDM method is superior to the LPRP method in the remaining interval.Moreover, from Figures 1-4, we can see that the NSDM method can solve 100% of the test problems, while the LPRP method can solve about 96% of the problems.Hence, the NSDM method is superior to the LPRP method.By comparing the value of   (1) in Figures 1-4, one can have a conclusion that the NSDM method is competitive to others; for example, the NSDM method is superior to other methods at least 45% in the number of iterations.In a word, one can have a conclusion that the presented method is much better  than the LPRP, MPRP, and SSD methods from the analysis of the numerical results.

Conclusions
In this paper, we have proposed a new formula (11) that can generate different search directions by taking different parameters.Based on this formula, we have proposed a new sufficient descent method for solving unconstrained optimization problems.At each iteration, the generated direction is only related to the gradient information of two successive points.
We have shown that this method is globally convergent.The numerical results indicate that the given method is superior to other methods for the test problems.In the future, we will study much better iterative methods according to (11) and perform new convergence analysis on them.

Figure 2 :
Figure 2: Performance profiles about the number of function evaluations.

Figure 3 :
Figure 3: Performance profiles about the number of gradient evaluations.

Figure 4 :
Figure 4: Performance profiles about CPU time.

Table 1 :
The test problems.
Assume that a parameter   ≥  , for all ,  is chosen, and  , =   if and only if solver s does not solve problem .The performance profile is defined by   () is the probability for solver  ∈  that a performance ratio  , is within a factor  ∈  of the best possible ratio.The performance profile   :  → [0, 1] for a solver was nondecreasing, piecewise, and continuous from the right.The value of   (1) is the probability that the solver will win over the rest of the solvers.In general, a solver with high values of   () or at the top right of the figure is preferable or represents the best solver.From Figures1-4, we can obviously see that the NSDM method performs better than the MPRP method and SSD