A Conjugate Gradient Method for Unconstrained Optimization Problems

A hybrid method combining the FR conjugate gradient method and the WYL conjugate gradient method is proposed for unconstrained optimization problems. The presented method possesses the su ﬃ cient descent property under the strong Wolfe-Powell (cid:3) SWP (cid:4) line search rule relaxing the parameter σ < 1. Under the suitable conditions, the global convergence with the SWP line search rule and the weak Wolfe-Powell (cid:3) WWP (cid:4) line search rule is established for nonconvex function. Numerical results show that this method is better than the FR method and the WYL method.


Introduction
Consider the following n variables unconstrained optimization problem: where f : R n → R is smooth and its gradient g x is avaible.The nonlinear conjugate gradient CG method for 1.1 is designed by the iterative form where x k is the kth iterative point, α k > 0 is a steplength, and d k is the search direction defined by International Journal of Mathematics and Mathematical Sciences where β k ∈ R is a scalar which determines the different conjugate gradient methods 1, 2 , and g k is the gradient of f x at the point x k .There are many well-known formulas for β k , such as the Fletcher-Reeves FR 3 , Polak-Ribière-Polyak PRP 4 , Hestenses-Stiefel HS 5 , Conjugate-Descent CD 6 , Liu-Storrey LS 7 , and Dai-Yuan DY 8 .The CG method is a powerful line search method for solving optimization problems, and it remains very popular for engineers and mathematicians who are interested in solving large-scale problems 9-11 .This method can avoid, like steepest descent method, the computation and storage of some matrices associated with the Hessian of objective functions.Then there are many new formulas that have been studied by many authors see 12-20 etc. .The following formula for β k is the famous FR method:

1.5
The numerical results show that this method is competitive to the PRP method, the global convergence of this method with the exact line search and Grippo-Lucidi line search conditions is proved.Huang et al. 25 proved that by restricting the parameter σ < 1/4, under the SWP line search rule, this method has the sufficient descent property.Then it is an interesting task to extend the bound of the parameter σ and get the sufficient descent condition.
The sufficient descent condition where c > 0 is a constant, is crucial to insure the global convergence of the nonlinear conjugate gradient method 23, 26-28 .In order to get some better results of the conjugate gradient methods, Andrei 29,30 proposed the hybrid conjugate gradient algorithms as convex combination of some other conjugate gradient algorithms.Motivated by the ideas of Andrei 29,30 and the above observations, we will give a hybrid method combining the FR method and the WYL method.The proposed method, relaxing the parameter σ < 1, under the SWP line search technique, possesses the sufficient descent condition 1.6 .The global convergence with the SWP line search and the WWP line search of our method is established for the nonconvex functions.Numerical results show that the presented method is competitive to the FR and the WYL method.
In the following section, the algorithm is stated.The properties and the global convergence of the new method are proved in Section 3. Numerical results are reported in Section 4 and one conclusion is given in Section 5.

Algorithm
Now we describe our algorithm as follows.
Algorithm 2.1 the hybrid method .
Step 1. Choose an initial point Step 2. If g k ≤ ε, then stop; otherwise go to the next step.
Step 3. Compute step size α k by some line search rules.
Step 5. Calculate the search direction where Step 6. Set k : k 1, and go to Step 3.

The Properties and the Global Convergence
In the following, we assume that g k / 0 for all k, otherwise a stationary point has been found.The following assumptions are often used to prove the convergence of the nonlinear conjugate gradient methods see 3, 8, 16, 17, 27 .

Assumption 3.1. (i) The function f x has a lower bound on the level set
In an open convex, set Ω 0 that contains Ω, f is differentiable, and its gradient g is Lipschitz continuous, namely, there exists a constants L > 0 such that 3.1

The Properties with the Strong Wolfe-Powell Line Search
The strong Wolfe-Powell SWP search rule is to find a step length α k such that where δ ∈ 0, 1/2 , σ ∈ 0, 1 .

International Journal of Mathematics and Mathematical Sciences
The following theorem shows that the hybrid algorithm with the SWP line search possesses the sufficient condition 1.6 only under the parameter σ ∈ δ, 1 .Theorem 3.2.Let the sequences {g k } and {d k } be generated by Algorithm 2.1, and let the stepsize α k be determined by the SWP line search 3.2 and 3.3 , if σ ∈ 0, 1 , 2λ 1 λ 2 ∈ 0, 1/2σ , then the sufficient descent condition 1.6 holds.
Proof.By the definition λ 1 , λ 2 and the formulae 1.4 and 1.5 , we have 3.4 3.5 Using 3.3 and the above inequality, we get By 2.1 , we have We prove the descent property of {d k } by induction.Since g T 0 d 0 − g 0 2 < 0, if g 0 / 0, now we suppose that d i , i 1, 2, . . ., k, are all descent directions, for example, d T i g i < 0. By 3.6 , we get 3.9 However, from 3.7 together with 3.9 , we deduce 3.10 Repeating this process and using the fact d T 0 g 0 − g 0 2 imply

3.19
We prove the result of this theorem by contradiction.Assume that this theorem is not true, then there exists a positive constant γ > 0 such that Squaring both sides of 2.1 , we obtain

3.22
Using 3.3 and 1.6 , we have 3.23 Repeating this process and using the fact t 0 1/ g 0 2 , we get 3.24 Now, combining 3.24 and 3.20 , we get another formula

3.30
Similar to the way of 31 , it is not difficult to get 3.28 .Secondly, using 3.30 and the Cauchy-Schwarz inequality implies that or see 31

3.31
Moreover, repeating the process of the first inequality of 3.10 and using r 0 1, we get

3.35
This contradiction shows that 3.16 is true.

The Properties with the Weak Wolfe-Powell (WWP) Line Search
The weak Wolfe-Powell line search is to find a step length α k satisfying 3.2 and where δ ∈ 0, 1/2 , σ ∈ δ, 1 .

International Journal of Mathematics and Mathematical Sciences 9
From the computation point of view, one of the well-known formulas for β k is the PRP method.The global convergence with the exact line search had been proved by Polak and Ribière 4 when the objective function is convex.Powell 33 gave a counter example to show that there exist nonconvex functions on which the PRP method does not converge globally even if the exact line search is used.He suggested that β k should not be less than zero.Considering this suggestion, under the assumption of the sufficient descent condition, Gilbert and Nocedal 27 proved that the modified PRP method β k max{0, β PRP k } is globally convergent with the WWP line search technique.For the new formula β H k , we know that it is always larger than zero.Then we can also get the global convergence of the hybrid method with the WWP line search. 37 The following Property 1 was introduced by Gilbert and Nocedal 27 , which pertains to the PRP method under the sufficient descent condition.The WYL also has this property.Now we will prove that this Property 1 pertains to the new method.

3.38
We say that the method has Property 1 if for all k, there exists constants b > 1 and λ > 0 such that |β k | ≤ b and

3.40
Then we get

3.41
International Journal of Mathematics and Mathematical Sciences By 3.1 again, we obtain

3.42
Combing the above inequality and 3.41 implies that

3.43
By 3.5 and 3.38 , we have If s k ≤ λ, using 3.1 , 3.38 , 3.43 , and the above equation, we obtain

3.45
Therefore, the conclusion of this lemma holds.function for the performance ratio.The performance profile ρ s : R → 0, 1 for a solver was a nondecreasing, piecewise constant function, continuous from the right at each breakpoint.The value of ρ s 1 was the probability that the solver would win over the rest of the solvers.
According to the above rules, we know that one solver whose performance profile plot is on top right will win over the rest of the solvers.
Figures 1-2 show that the performances of these methods are relative to the iteration number NI and the number of the function and gradient NFN , where the "FR" denotes the FR formula with WWP rule, the "WYL" denotes the WYL formula with WWP rule, and Algorithm 2.1 denotes the new method with WWP rule, respectively.
From Figures 1-2, it is easy to see that Algorithm 2.1 is the best among the three methods, and the WYL method is much better than FR methods.Notice that the global convergence of the FR method with the WWP line search has not been established yet.In other words, the given method is competitive to the other two normal methods and the hybrid formula is notable.

Conclusions
This paper gives a hybrid conjugate gradient method for solving unconstrained optimization problems.Under the SWP line search, this method possesses the sufficient descent condition only with the parameter σ < 1.The global convergence with the SWP line search and the WWP line search is established for the nonconvex functions.Numerical results show that the given method is competitive to other two conjugate gradient methods.
For further research, we should study the new method with the nonmonotone line search technique.Moreover, more numerical experiments for large practical problems such as the problems 36 should be done, and the given method should be compared with other famous formulas in the future.How to choose the parameters λ 1 and λ 2 in the algorithm is another aspect of future investigation.

. 14 thisLemma 3 . 3 . 6 InternationalTheorem 3 . 4 .
implies that 1.6 holds.The proof is complete.Suppose that Assumption 3.1 holds.Let the sequences {g k } and {d k } be generated by Algorithm 2.1, let the stepsize α k be determined by the SWP line search 3.2 and 3.3 , and let the conditions in Theorem 3.2 hold.Then the Zoutendijk condition [22] Journal of Mathematics and Mathematical Sciences By the same way, if Assumption 3.1 and the condition g T k d k < 0 for all k hold, 3.15 also holds for the exact line search, the Armijo-Goldstein line search, and the weak Wolfe-Powell line search.The proofs can be seen in 31, 32 .Now, we prove the global convergence theorem of Algorithm 2.1 with the SWP line search.Suppose that Assumption 3.1 holds.Let the sequence {g k } and {d k } be generated by Algorithm 2.1, let the stepsize α k be determined by the SWP line search 3.2 and 3.3 , let the conditions in Theorem 3.2 hold, and let the parameter 2λ 1 λ 2 ≤ 1.Then lim k → ∞ inf g k 0.

Lemma 3 . 6
see 31, Lemma 3.3.1 .Let Assumption 3.1 hold and let the sequences {g k , d k } be generated by Algorithm 2.1.The stepsize α k is determined by 3.2 and 3.36 .Suppose that 3.20 is true, and the sufficient descent condition 1.6 holds.Then we have d k / 0 and ∞ k 0

Figure 1 :
Figure 1: Performance profiles of these three methods NI .
International Journal of Mathematics and Mathematical SciencesProof.By 2.1 , 3.3 , 3.27 , and 2λ 1 λ 2 ≤ 1, we can deduce that 3.10 holds for all σ < 1 and σ * 2λ 1 λ 2 σ < 1.According to the second inequality of 3.10 , we have Theorem 3.5.Suppose that Assumption 3.1 holds.Let the sequence {g k , d k } be generated by Algorithm 2.1, let the stepsize α k be determined by the SWP line search 3.2 and 3.3 , let the parameter 2λ 1 λ 2 ≤ 1, and let 3.27 hold.Then, for all k ≥ 0, the following inequalities holds: Let Assumption 3.1 hold, let the sufficient descent condition 1.6 hold, and let the sequences {g k , d k } be generated by Algorithm 2.1.Suppose that there exists a constant M > 0 such that d k ≤ M for all k.Then this method possesses Property 1.
Lemma 3.8 see 31, Lemma 3.3.2 .Let the sequences {g k } and {d k } be generated by Algorithm 2.1 and the conditions in Lemma 3.7 hold.If β H k ≥ 0 and has Property 1, then there exists λ > 0 such that for any Δ ∈ N and any index k 0 , there is an index k > k 0 satisfying {i ∈ N : k ≤ i ≤ k Δ − 1, s i > λ}, Ndenotes the set of positive integers, |κ λ k,Δ | denotes the numbers of elements in κ λ k,Δ .