Global Convergence of a Modified LS Method

The LS method is one of the effective conjugate gradient methods in solving the unconstrained optimization problems. The paper presents a modified LS method on the basis of the famous LS method and proves the strong global convergence for the uniformly convex functions and the global convergence for general functions under the strong Wolfe line search. The numerical experiments show that the modified LS method is very effective in practice.


Introduction
The conjugate gradient method is one of the most common methods used in the optimization, which is especially effective in solving the unconstrained optimization problem: where f : R n → R is continuously differentiable nonlinear function.
The LS method introduced by Liu and Storey 1 is one of the conjugate gradient methods, and its iteration formulas are as follows: −g k β k d k−1 , for k ≥ 2; 1.3 where g k is the gradient of f at x k ; α k > 0 is a step length which is determined by some line search; d k is the search direction.

Mathematical Problems in Engineering
The convergence properties of the LS method have been studied extensively.For example, Yu et al. 2 proposed a modified LS method called the LS1 method in this paper, in which parameter β k satisfies the following formula: , 0 e l s e .

1.5
They proved the global convergence property of the LS1 method under the Wolfe line search and verified that the LS1 method was very effective in solving the large unconstrained optimization problems.Liu et al. 3 further studied the LS method on the basis of 2 and proposed the parameter β k : where ρ > 1 ξ, and ξ is sufficiently small positive number.The corresponding method is called the LS2 method in this paper.Jinkui Liu, among others, proved the global convergence properties of the LS2 method under the Wolfe line search and showed that the achievements of the LS2 method was comparable with the PRP method.
In this paper, a modified LS method is proposed on the basis of 2 , which can guarantee generate the sufficient descent direction in each step under the strong Wolfe line search.Moreover, the new method has the strong global convergence properties for uniformly convex functions and the global convergence properties for ordinary functions.

2.1
Step 3. Let Step 4. Compute d k 1 by 1.3 , and generate β k 1 by where Mathematical Problems in Engineering 3 Step 5. Set k k 1, go to Step 2.
Theorem 2.1.Let the sequences {g k } and {d k } be generated by MLS method, then Proof.The conclusion can be proved by induction.Since −g T 1 d 1 / g 1 2 1, the conclusion holds for k 1.Now, we assume that the conclusion is true for k − 1, for k ≥ 2. We need to prove that the conclusion holds for k.
Multiplying 1.3 by g T k , we have

2.5
From the above inequalities, we obtain that the conclusion holds for k.

The Global Convergence Properties
In order to prove the global convergence of the MLS method, we assume that the objective function f satisfies the following assumption.

Mathematical Problems in Engineering
ii In a neighborhood V of L, f is continuously differentiable and its gradient g is Lipschitz continuous, namely, there exists a constant M > 0 such that From Assumption H, there exists a constant r > 0, such that Firstly, we prove that the MLS method has the strong global convergence property for uniformly convex functions.
Proof.From f x is uniformly convex function and u ≥ 1, we have

3.7
From Lipschitz condition and 1.2 , we have

3.8
Then, from 1.3 , 2.2 , and 3.2 , we have 3.9 From the above inequality, we obtain that the conclusion 3.3 holds.Then, from Lemma 3.1, the conclusion 3.4 holds; and f x is uniformly convex function, so the conclusion 3.6 holds.
Secondly, we prove that the MLS method has the global convergence for ordinary functions.

3.13
Say that the MLS method has property ( * ), that is,

Mathematical Problems in Engineering
Proof.Firstly, from the strong Wolfe line search, 2.3 and 3.13 , we have

3.14
From the Assumption H i , there exists a positive constant a, such that x ≤ a, for all x ∈ L. So, from 2.3 and Lipschitz condition, we have

Numerical Results
In this section, we test the MLS method for problems from 7 , and we compare its performance to that of the LS method, LS1 method, and LS2 method under the strong Wolfe line search.The parameters δ 0.01, σ 0.1, ς 1.25, ρ 1.5, ξ 0.001, u 1.0, ν 0.35.The termination condition is g k ≤ 10 −6 or It-max >9999.It-max denotes the Maximum number of iterations.
The numerical results of our tests are reported in Table 1.The column "Problem" represents the problem's name in 7 ."Dim" denotes the dimension of the test problems.The detailed numerical results are listed in the form NI/NF/NG, where NI, NF, NG denote the number of iterations, function evaluations, and gradient evaluations, respectively."NaN" means the calculation failure.
In order to rank the average performance of all the above conjugate gradient methods, one can compute the total number of function and gradient evaluation by the following formula: where l is some integer.According to the results on automatic differentiation 8, 9 , the value of l can be set to 5.That is to say, one gradient evaluation is equivalent to five function evaluations if automatic differentiation is used.By 4.1 , we compare the MLS method with LS method, LS1 method, and LS2 method as follows: for the ith problem, compute the total numbers of function evaluations and gradient evaluations required by the MLS method, LS method, LS1 method, and LS2 method, and denote them as N total,i MLS , N total,i LS , N total,i LS1 , and N total,i LS2 , respectively.Then, we calculate the ratio: N total,i MLS .If the i 0 th problem is not run by the method, we use a constant λ max{γ i method | i ∈ S 1 } instead of γ i 0 method , where S 1 denotes the set of test problems which can be run by the method.

4.2
The geometric mean of these ratios for VLS method over all the test problems are defined by γ LS  where S denotes the set of the test problems, and |S| denotes the number of elements in S. One advantage of the above rule is that the comparison is relative and hence does not be dominated by a few problems for which the method requires a great deal of function evaluations and gradient functions.
From the above rule, it is clear that γ MLS 1.The values of γ LS , γ LS1 , and γ LS2 are computed and listed in Table 2.
From Table 2, it is clear that the MLS method is superior to the LS method and the LS1 method, and it is comparable with the LS2 method for the given test problems.So, the MLS method has certain value of research.
Lemma 3.1 see 4 .Suppose Assumption H holds. Consider any iteration of the form 1.2 -1.3 , where d k satisfies g T k d k < 0 for k ∈ N and α k satisfies the strong Wolfe line search.If Theorem 3.2.Suppose Assumption H holds. Let the sequences {g k } and {d k } be generated by MLS method.If f x is a uniformly convex function, that is, there exists t > 0, for all x, y ∈ L, subject to 3.3 see 5 .Suppose Assumption H holds. Let the sequence {x k } be generated by the iteration of the form 1.2 -1.3 , where d k satisfies g T k d k < 0 for k ∈ N and α k satisfies the strong Wolfe line search.Then, Lemma 3.4 see 6 .Suppose Assumption H holds. Consider any iteration of the form 1.2 -1.3 , where β k ≥ 0, and α k satisfies the strong Wolfe line search and 2.3 holds.If there exists a constant r > 0, such that g k ≥ r, ∀k ≥ 1 Suppose Assumption H holds. Consider any iteration of the form 1.2 -1.3 , where β k ≥ 0, and α k satisfies the strong Wolfe line search and 2.3 holds.If β k has the property ( * ), and if there exists a constant r > 0, subject tog k ≥ r ∀k ∈ N ,3.17 then there exits λ > 0, for any Δ ∈ Z and k 0 ∈ Z , and for all k ≥ k 0 , such that Suppose Assumption H holds. Consider any iteration of the form 1.2 -1.3 , where β k ≥ 0, and α k satisfies the strong Wolfe line search and 2.3 holds.If β k has the property ( * ), then Suppose Assumption H holds. Consider the method 1.2 -1.3 , where β k is computed by 2.2 , and α k satisfies the strong Wolfe line search and 2.3 holds, then denotes the number of the λ k,Δ .Lemma 3.7 see 6 .

Table 1 :
The numerical results of LS method, LS1 method, LS2 method, and MLS method.

Table 2 :
Relative efficiency of the MLS method, LS method, LS1 method and LS2 method.