Journal of Applied Mathematics

Volume 2013 (2013), Article ID 986317, 6 pages

http://dx.doi.org/10.1155/2013/986317

Research Article

A Conjugate Gradient Type Method for the Nonnegative Constraints Optimization Problems

Can Li

College of Mathematics, Honghe University, Mengzi 661199, China

Received 16 December 2012; Accepted 20 March 2013

Academic Editor: Theodore E. Simos

Copyright © 2013 Can Li. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

We are concerned with the nonnegative constraints optimization problems. It is well known that the conjugate gradient methods are efficient methods for solving large-scale unconstrained optimization problems due to their simplicity and low storage. Combining the modified Polak-Ribière-Polyak method proposed by Zhang, Zhou, and Li with the Zoutendijk feasible direction method, we proposed a conjugate gradient type method for solving the nonnegative constraints optimization problems. If the current iteration is a feasible point, the direction generated by the proposed method is always a feasible descent direction at the current iteration. Under appropriate conditions, we show that the proposed method is globally convergent. We also present some numerical results to show the efficiency of the proposed method.

1. Introduction

Due to their simplicity and their low memory requirement, the conjugate gradient methods play a very important role for solving unconstrained optimization problems, especially for the large-scale optimization problems. Over the years, many variants of the conjugate gradient method have been proposed, and some are widely used in practice. The key features of the conjugate gradient methods are that they require no matrix storage and are faster than the steepest descent method.

The linear conjugate gradient method was proposed by Hestenes and Stiefel [1] in the 1950s as an iterative method for solving linear systems

where is an symmetric positive definite matrix. Problem (1) can be stated equivalently as the following minimization problem

This equivalence allows us to interpret the linear conjugate gradient method either as an algorithm for solving linear systems or as a technique for minimizing convex quadratic functions. For any , the sequence generated by the linear conjugate gradient method converges to the solution of the linear systems (1) in at most steps.

The first nonlinear conjugate gradient method was introduced by Fletcher and Reeves [2] in the 1960s. It is one of the earliest known techniques for solving large-scale nonlinear optimization problems

where is continuously differentiable. The nonlinear conjugate gradient methods for solving (3) have the following form:

where is a steplength obtained by a line search and is a scalar which deternimes the different conjugate gradient methods. If we choose to be a strongly convex quadratic and to be the exact minimizer, the nonliear conjugate gradient method reduces to the linear conjugate gradient method. Several famous formulae for are the Hestenes-Stiefel (HS) [1], Fletcher-Reeves (FR) [2], Polak-Ribière-Polyak (PRP) [3, 4], Conjugate-Descent (CD) [5], Liu-Storey (LS) [6], and Dai-Yuan (DY) [7] formulae, which are given by

where and stands for the Euclidean norm of vectors. In this paper, we focus our attention on the Polak-Ribière-Polyak (PRP) method. The study of the PRP method has received much attention and has made good progress. The global convergence of the PRP method with exact line search has been proved in [3] under strong convexity assumption on . However, for general nonlinear function, an example given by Powell [8] shows that the PRP method may fail to be globally convergent even if the exact line search is used. Inspired by Powell’s work, Gilbert and Nocedal [9] conducted an elegant analysis and showed that the PRP method is globally convergent if is restricted to be nonnegative and is determined by a line search satisfying the sufficient descent condition in addition to the standard Wolfe conditions. Other conjugate gradient methods and their global convergence can be found in [10–15] and so forth.

Recently, Li and Wang [16] extended the modified Fletcher-Reeves (MFR) method proposed by Zhang et al. [17] for solving unconstrained optimization to the nonlinear equations

where is continuously differentiable, and proposed a descent derivative-free method for solving symmetric nonlinear equations. The direction generated by the method is descent for the residual function. Under appropriate conditions, the method is globally convergent by the use of some backtracking line search technique.

In this paper, we further study the conjugate gradient method. We focus our attention on the modified Polak-Ribière-Polyak (MPRP) method proposed by Zhang et al. [18]. The direction generated by MPRP method is given by

where ,  ,  ,  and  . The MPRP method not only reserves good properties of the PRP method but also possesses another nice property; that it is, always generates descent directions for the objective function. This property is independent of the line search used. Under suitable conditions, the MPRP method with the Armoji-type line search is also globally convergent. The purpose of this paper is to develop an MPRP type method for the nonnegative constraints optimization problems. Combining the Zoutendijk feasible direction method with MPRP method, we propose a conjugate gradient type method for solving the nonnegative constraints optimization problems. If the initial point is feasible, the method generates a feasible point sequence. We also do numerical experiments to test the proposed method and compare the performance of the method with the Zoutendijk feasible direction method. The numerical results show that the method that we propose outperforms the Zoutendijk feasible direction method.

2. Algorithm

Consider the following nonnegative constraints optimization problems:

where is continuously differentiable. Let be the current iteration. Define the index set

where is the  th component of . In fact the index set is the active set of problem (10) at .

The purpose of this paper is to develop a conjugate gradient type method for problem (10). Since the iterative sequence is a feasible point sequence, the search directions should be feasible descent directions. Let be the current iteration. By the definition of feasible direction, we have that [19] is a feasible direction of (10) at if and only if . Similar to the Zoutendijk feasible direction method, we consider the following problem:

Next, we show that, if is not a KKT point of (10), the solution of problem (12) is a feasible descent direction of at .

Lemma 1. Let and let be a solution of problem (12); then . Moreover if and only if is a KKT point of problem (10).

Proof. Since is a feasible point of problem (12), there must be . Consequently, if , there must be . This implies that the direction is a feasible descent direction of at .

We suppose that . Problem (12) is equivalent to the following problem:

Then there exist and such that the following KKT condition holds:

Multiplying the first of these expressions by , we obtain

where . By combining the assumption with the second and the third expressions of (14), we find that . Substituting it into the first expressions of (14), we obtain that

Let ,  ; then ,  . Moreover, we have

This implies that is a KKT point of problem (10).

On the other hand, we suppose that is a KKT point of problem (10). Then there exist , such that the following KKT condition holds:

From the second of these expressions, we get . Substituting it into the first of these expressions, we have and , so that . However, we had shown that , so .

By the proof of Lemma 1 we find that and are necessary conditions of the fact that is a KKT point of problem (10). We summarize these observation results as the following result.

Lemma 2. Let ; then is a KKT point of problem (10) if and only if and .

Proof. Firstly, we suppose that is a KKT point of problem (10). Similar to the proof of Lemma 1, it is easy to get that and .

Secondly, we suppose that and . Let ,  ; then the KKT condition (18) holds, so that is a KKT point of problem (10).

Based on the above discussion, we propose a conjugate gradient type method for solving problem (10) as follows. Let feasible point be current iteration. For the boundary of the feasible region , we take

where . For the interior of the feasible region , similar to the direction in the MPRP method, we define by the following formula:

where ,  ,  ,  and  .

It is easy to see from (19) and (20) that

The above relations indicate that

where .

Theorem 3. Let , be defined by (19) and (20) then

Moreover, is a KKT point of problem (10) if and only if .

Proof. Clearly, inequality (22) implies that

If is a KKT point of problem (10), similar to the proof of Lemma 1, we also get that .

If , by (22), we can get that

The equality and the definition of (19) imply that

Let ;  , then the KKT condition (18) also holds, so that is a KKT point of problem (10).

By combining (22) with Theorem 3, we conclude that defined by (19) and (20) provides a feasible descent direction of at , if is not a KKT point of problem (10).

Based on the above process, we propose an MPRP type method for solving (10) as follows.

Algorithm 4 (MPRP type method).   

Step  0. Given constants ,  ,  . Choose the initial point ; Let .

Step  1. Compute by (19) and (20). If , then stop. Otherwise, go to the next step.

Step  2. Determine satisfying and

Step  3. Let the next iteration be .

Step  4. Let and go to Step  1.

It is easy to see that the sequence generated by Algorithm 4 is a feasible point sequence. Moreover, it follows from (28) that the function value sequence is decreasing. In addition if is bounded from below, we have from (28) that

In particular we have

Next, we prove the global convergence of Algorithm 4 under the following assumptions.

Assumption A. The level set is bound.

In some neighborhood of , is continuously differentiable, and its gradient is the Lipschitz continuous; namely, there exists a constant such that

Clearly, Assumption A implies that there exists a constant such that

Lemma 5. Suppose that the conditions in Assumption A hold; and are the iterative sequence and the direction sequence generated by Algorithm 4. If there exists a constant such that

then there exists a constant such that

Proof. By combining (19), (20), and (33) with Assumption A, we deduce that

By (30), there exists a constant and an iteger such that the following inequality holds for all :

Hence, we have for any

Let

Then

Theorem 6. Suppose that the conditions in Assumption A hold. Let and be the iterative sequence and the direction sequence generated by Algorithm 4. Then

Proof. We prove the result of this theorem by contradiction. Assume that the theorem is not true; then there exists a constant such that

So by combining (41) with (23), it is easy to see that (33) holds.

 (1) If , we get from (30) that , so that . This contradicts assumption (41). (2) If , there is an infinite index set such that

It follows from Step  2 of Algorithm 4, that when is sufficiently large, does not satify ; that is

By the mean-value theorem, Lemma 1, and Assumption A, there is such that

Substituting the last inequality into (43), we get for all sufficiently large

Taking the limit on both sides of the equation, then by combining and recalling , we obtain that . This also yields a contradiction.

3. Numerical Experiments

In this section, we report some numerical experiments. We test the performance of Algorithm 4 and compare it with the Zoutendijk method.

The code was written in Matlab, and the program was run on a PC with 2.20 GHz CPU and 1.00 GB memory. The parameters in the method are specified as follows. We set ,  . We stop the iteration if or the iteration number exceeds 10000.

We first test Algorithm 4 on small and medium size problems and compared them with the Zoutendijk method in the total number of iterations and the CPU time used. The test problems are from the CUTE library [20]. The numerical results of Algorithm 4 and the Zoutendijk method are listed in Table 1. The columns have the following meanings.

Table 1: The numerical results.

Dim
Algorithm 4
Zoutendijk method

Iter
Time
Iter
Time

3
2
1973
1.5710
—
—

4
2
201
0.2290
—
—

6
2
30
0.0160
—
—

3
35
0.0160
—
—

4
39
0.0470
—
—

10
124
0.1210
—
—

50
220
0.5370
—
—

8
3
44
0.0150
40
0.2188

11
3
3
0.0000
4
0.1094

15
4
10
0.0160
20
0.1563

18
6
322
0.0690
1936
12.0938

19
11
438
0.5440
8338
72.4219

23
50
12
0.0300
4
0.5000

24
100
142
0.3750
—
—

25
100
38
0.0810
6
0.3438

26
100
8
0.0470
6
0.1250

1000
4
47.9060
4
190.1406

is the number of the test problem, Dim is the dimension of the test problem, Iter is the number of  iterations, and Time is CPU time in seconds.

We can see from Table 1 that Algorithm 4 has successfully solved 12 test problems, and the Zoutendijk method has successfully solved 8 test problems. From the number of iterations, Algorithm 4 has 12 test results better than Zoutendijk method. From the computation time, Algorithm 4 performs much better than the Zoutendijk method did. We then test Algorithm 4 and the Zoutendijk method on two problems with a larger dimension. The problem of VARDIM comes from [20], and the following problem comes from [16]. The results are listed in Tables 2 and 3.

Table 2: Test results for VARDIM with various dimensions.

Problem
Dim
Algorithm 4
Zoutendijk method

Iter
Time
Iter
Time

VARDIM
1000
46
13.4485
—
—

2000
55
49.0090
—
—

3000
65
97.1020
—
—

4000
78
164.6213
—
—

5000
90
271.0340
—
—

Table 3: Test results for Problem 1 with various dimensions.

Problem
Dim
Algorithm 4
Zoutendijk method

Iter
Time
Iter
Time

Problem 1
1000
17
0.1400
8
110.2578

2000
26
16.8604
8
263.2660

3000
39
39.6561
11
554.0310

4000
51
68.1729
30
910.1090

5000
55
110.5660
—
—

Problem 1. The nonnegative constraints optimization problem

with Engval function is defined by

We can see from Table 2 that Algorithm 4 has successfully solved the problem of VARDIM whose scale varies from 1000 dimensions to 5000 dimensions. However, the Zoutendijk method fails to solve the problem of VARDIM with larger dimension. From Table 3, although the number of iterations of Algorithm 4 is more than the Zoutendijk method, the computation time of Algorithm 4 is less than the Zoutendijk method, and this feature becomes more evident as increase of the dimension of the test problem.

In summary, the results from Tables 1–3 show that Algorithm 4 is more efficient than the Zoutendijk method and provides an efficient method for solving nonnegative constraints optimization problems.

Acknowledgment

This research is supported by the NSF (11161020) of China.

References

M. R. Hestenes and E. Stiefel, “Methods of conjugate gradients for solving linear systems,” Journal of Research of the National Bureau of Standards, vol. 49, pp. 409–436, 1952.

R. Fletcher and C. M. Reeves, “Function minimization by conjugate gradients,” The Computer Journal, vol. 7, pp. 149–154, 1964.

B. Polak and G. Ribire, “Note sur la convergence de directions conjugees,” Revue Française d'Informatique et de Recherche Opérationnelle, vol. 16, pp. 35–43, 1969.

B. T. Polyak, “The conjugate gradient method in extremal problems,” USSR Computational Mathematics and Mathematical Physics, vol. 9, no. 4, pp. 94–112, 1969.

R. Fletcher, Practical Methods of Optimization, John Wiley & Sons Ltd., Chichester, UK, 2nd edition, 1987.

Y. Liu and C. Storey, “Efficient generalized conjugate gradient algorithms. I. Theory,” Journal of Optimization Theory and Applications, vol. 69, no. 1, pp. 129–137, 1991.

Y. H. Dai and Y. Yuan, “A nonlinear conjugate gradient method with a strong global convergence property,” SIAM Journal on Optimization, vol. 10, no. 1, pp. 177–182, 1999.

M. J. D. Powell, “Convergence properties of algorithms for nonlinear optimization,” SIAM Review, vol. 28, no. 4, pp. 487–500, 1986.

J. C. Gilbert and J. Nocedal, “Global convergence properties of conjugate gradient methods for optimization,” SIAM Journal on Optimization, vol. 2, no. 1, pp. 21–42, 1992.

R. Pytlak, “On the convergence of conjugate gradient algorithms,” IMA Journal of Numerical Analysis, vol. 14, no. 3, pp. 443–460, 1994.

G. Li, C. Tang, and Z. Wei, “New conjugacy condition and related new conjugate gradient methods for unconstrained optimization,” Journal of Computational and Applied Mathematics, vol. 202, no. 2, pp. 523–539, 2007.

X. Li and X. Zhao, “A hybrid conjugate gradient method for optimization problems,” Natural Science, vol. 3, no. 1, pp. 85–90, 2011.

Y. H. Dai and Y. Yuan, “An efficient hybrid conjugate gradient method for unconstrained optimization,” Annals of Operations Research, vol. 103, pp. 33–47, 2001.

W. W. Hager and H. Zhang, “A new conjugate gradient method with guaranteed descent and an efficient line search,” SIAM Journal on Optimization, vol. 16, no. 1, pp. 170–192, 2005.

D.-H. Li, Y.-Y. Nie, J.-P. Zeng, and Q.-N. Li, “Conjugate gradient method for the linear complementarity problem with -matrix,” Mathematical and Computer Modelling, vol. 48, no. 5-6, pp. 918–928, 2008.

D.-H. Li and X.-L. Wang, “A modified Fletcher-Reeves-type derivative-free method for symmetric nonlinear equations,” Numerical Algebra, Control and Optimization, vol. 1, no. 1, pp. 71–82, 2011.

L. Zhang, W. Zhou, and D. Li, “Global convergence of a modified Fletcher-Reeves conjugate gradient method with Armijo-type line search,” Numerische Mathematik, vol. 104, no. 4, pp. 561–572, 2006.

L. Zhang, W. Zhou, and D.-H. Li, “A descent modified Polak-Ribière-Polyak conjugate gradient method and its global convergence,” IMA Journal of Numerical Analysis, vol. 26, no. 4, pp. 629–640, 2006.

D. H. Li and X. J. Tong, Numerical Optimization, Science Press, Beijing, China, 2005.

J. J. Moré, B. S. Garbow, and K. E. Hillstrom, “Testing unconstrained optimization software,” ACM Transactions on Mathematical Software, vol. 7, no. 1, pp. 17–41, 1981.