Journal of Control Science and Engineering
Volume 2015 (2015), Article ID 674617, 8 pages
http://dx.doi.org/10.1155/2015/674617

Research Article

A New BFGS Algorithm Using the Decomposition Matrix of the Correction Matrix to Obtain the Search Directions

Li Liu

Teaching Department, Jilin City Radio and TV University, Jilin 132002, China

Received 30 September 2014; Revised 3 January 2015; Accepted 3 January 2015

Academic Editor: Shengwei Mei

Copyright © 2015 Li Liu. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

We present an improved method for determining the search direction in the BFGS algorithm. Our approach uses the equal inner product decomposition method for positive-definite matrices. The decomposition of an approximated Hessian matrix expresses a correction formula that is independent from the exact line search. This decomposed matrix is used to compute the search direction in a new BFGS algorithm.

1. Introduction

Using the equal inner product decomposition algorithm [1–3] for positive-definite matrices with a vector, a novel quasi-Newton method has been proposed [4, 5]. In this algorithm, the equal inner product decomposition matrix of the correction matrix is utilized to obtain the search directions. That is, for the unrestricted optimization problem

where is a continuously differentiable function; the equal inner product decomposition correction formula for the approximate Hessian matrix with the objective function gradient vector is used to compute the search directions.

The BFGS (Broyden-Fletcher-Goldfarb-Shanno) algorithm [6–9] is a typical quasi-Newton method. If we consider to be an iteration point, and , and , , then the correction formulae for the approximate Hessian matrix and its inverse can be expressed as

In computing the search direction at the iteration point , we generally solve the equation:

but we would rather avoid using (3) to calculate

There are clearly more computations involved in (4) than in (5). We take this apparently more complex route because may appear to be non-positive-definite or even singular when we calculate (3). Equation (2) can be transformed to

Using this form, Gill and Murray [10] developed a method for solving (4) by Cholesky decomposition. First, they found the Cholesky decomposition of  :

where is a unit lower triangular matrix and is a diagonal matrix, and then they successively solved

Because (6) establishes the iterative formula from and to and , the Cholesky decomposition of need not be repeated on each iteration. The detailed calculation method is described in [10–12].

Unlike the above approach, the algorithm proposed in [4, 5] utilizes the equal inner product decomposition algorithm for a positive-definite matrix with a vector. In this improved technique, the matrix is computed. This is the equal inner product decomposition matrix of with ; that is,

where , . The search direction at the iteration point is then given by

where is the sum of the elements in the th column of .

Instead of using the correction formula (2), the method of [4, 5] directly establishes the correction formula from to , which are equal inner product decomposition matrixes of with the objective function gradient vectors. When the correction matrix is calculated, the search direction is immediately given by (10).

Thus, [4, 5] presented a novel approach to the computation of search directions. However, all the conclusions in [4, 5] relate to the exact line search condition. Therefore, this algorithm is feasible in theory but has restricted applicability.

The BFGS algorithm with the Wolfe line search has global, superlinear convergence [13–15]. Compared with other quasi-Newton methods, BFGS is the most popular and efficient algorithm. In this paper, we focus on the BFGS algorithm and set up a correction formula expressed by the decomposition matrix that is independent of the exact line search. This leads to a new BFGS algorithm that computes the search directions using the equal inner product decomposition matrix of the approximate Hessian matrix.

In Section 3, the procedures of computing search directions in the usual BFGS and in the new BFGS have been compared by a computing example.

2. Correction Formula Expressed by Decomposition Matrix

At the iteration point , we use the method of [3] to directly compute the equal inner product decomposition matrix of   with . However, this decomposition method is computationally expensive. One alternative is to set up a similar correction formula between and as that between and or between and .

We denote , , and . By Theorem 2.2 in [4], if the equal inner product decomposition matrix of with has already been computed in the exact line search condition for the BFGS correction formula (2), then the equal inner product decomposition matrix of with can be expressed as

where , ,, and . By Theorem 2.3 in [4], the equal inner product decomposition matrix of the initial correction matrix with is

where .

As defined by (11), only satisfies under the exact line search condition, that is, when . If the iteration point from to is obtained through an inexact line search, then obviously in (11) is no longer the decomposition matrix of defined by (2). The following conclusions are independent of the exact line search but are dependent on the correction matrix definition for BFGS.

2.1. Main Results

Lemma 1. Suppose . If  , , , then the matrix is nonsingular and

The proof is simple and is therefore omitted from this paper.

Lemma 2. For the BFGS correction formula (11), if   is the equal inner product decomposition matrix of   with , we denote

Then, and

Proof. Because , where is the step length, we have . By a direct calculation, we have

Therefore, holds. Since

Lemma 1 implies

Thus, we have . This completes the proof.

Lemma 3. Writing , where , if , one has .

Proof. A direct computation gives . Thus, we have

This completes the proof.

Writing

where , , we can state the following results.

Theorem 4. For the BFGS correction formula (2), if the equal inner product decomposition matrix of with has already been obtained, then the equal inner product decomposition matrix of   with is

where and .

Proof. By Lemma 2 and (21), we have

Because is the Householder matrix, . Thus, we have

Therefore, Lemma 2 gives . By Lemma 3, we get

where . Thus, defined by (21) is the equal inner product decomposition matrix of with . This completes the proof.

From the above proof, it can be found that in (21) is the equal inner product decomposition matrix of defined by (2), which is independent of the exact line search.

Corollary 5. Under the exact line search condition, (21) is equivalent to (11); that is, (11) is a special case for (21).

Proof. Under the exact line search condition, we have and .
From (20), we have

Thus, , where .
Therefore, . Under the exact line search condition, . From the above results, (11) can be directly deduced from (21). This completes the proof.

2.2. Extension

We can extend the conclusion of Theorem 4 to the following BFGS correction formula using self-scaling [16–18]:

where and . When , , so (26) can be written as

Equation (27) is well known for its practical applicability. Here, we have only transformed (27) into the form expressed by the decomposition correction matrix. When , similar methods can be used.

Applying the Sherman-Morrison formula [13] to (27), we can deduce the correction matrix for :

Writing , where and , we can state the following for the inexact line search condition.

Theorem 6. For the self-scaling BFGS correction formula (27), if the equal inner product decomposition matrix of with is already known, then the equal inner product decomposition matrix of   with is

where , , and for step length .

Proof. The proof is similar to that for Theorem 4.
Using (29), we have

Because of

(30) gives

Therefore, and we have

that is, , where . This completes the proof.

Theorem 6 admits the following corollary.

Corollary 7. Under the exact line search condition, if the equal inner product decomposition matrix of with for the self-scaling BFGS correction formula (27) is already known, then the equal inner product decomposition matrix of   with is

where , , and   for step length .

Proof. The proof is simple and is therefore omitted.

3. A New BFGS Algorithm

3.1. Algorithm

Replacing the corresponding steps of the BFGS algorithm by (12), (21), and (10) establishes a new BFGS algorithm that uses the decomposition correction matrix to determine the search direction.

The New BFGS Algorithm

Step 1. Choose a start point , , and compute . If , terminate the algorithm; otherwise, go to Step 2.

Step 2. Let . For the initial matrix , form the equal inner product decomposition with to give the decomposition matrix and .

Step 3. Compute the search direction as

where , is the sum of the elements in the th column of  .

Step 4. Perform a one-dimensional line search (exact line search or Wolfe line search), and compute the step length factor .

Step 5. Let . Compute . If , terminate the algorithm; otherwise, go to Step 6.

Step 6. Using (21), calculate the equal inner product decomposition matrix of with and the coefficient .

Step 7. Let and go to Step 3.

3.2. Example

Here, by an example, we compare the two methods of computing the search directions between the usual BFGS algorithm [10–12] and the new BFGS algorithm. Considering the optimization problem [19],

when solving the problems with BFGS algorithm, from the second iterative computation, all the following iteration procedures are similar, so the difference of the two algorithms can be clearly seen by one iterative computing procedure. Here we assume that the first iterative computing procedure is finished and at the first iterative computing, we choose the initial point , the initial matrix . Noting , the initial searching direction is

At the point , we have done the Wolfe-Powell search along the and obtained the step length which satisfies the Wolfe-Powell conditions [13]. Thus we get

Now we compare the two different algorithms of computing search directions by the second iterative computation.

We first compute the search direction with the usual BFGS algorithm.

Step 1. Compute Cholesky decomposition of  .
Obviously, the Cholesky decomposition of   is

Let , definite , and .
Compute the following in turn [13]:

Hence,

Then let . Computing the equation

we get .
Compute the following in turn [13]:

Thus we obtain

Therefore, we get the Cholesky decomposition of :

Step 2. Solve the following equations:

From , there is

Solving the equations, we get . Then from , there is

Solving the equations, we get the search direction

We then use the new BFGS algorithm to compute the search direction .

Step 1. First compute the initial correction matrix and the coefficient .
For , we get

Step 2. By , we compute .
Make use of known data, and we get

Thus there is

Therefore we have

Because

we obtain

And , , so

Thus we can obtain , in which

Therefore, the search direction is

3.3. Analysis

The above example clearly shows the difference between the two algorithms in computing the search directions. In the new algorithm, correction matrixes are directly used to compute search directions. There is no procedure of Cholesky decomposition and no procedure of solving equations. Thus it simplifies the computing procedure of the search directions.

To compute the decomposition correction matrix at point using the new BFGS algorithm, we first need to compute a Householder matrix . This increases the computational complexity. However, the search direction can be directly computed by (10) at the same time. Consequently, the two steps of the correction matrix calculation and search direction calculation are performed concurrently. In practice, the computation of  (21) can be simplified using the properties of the Householder matrix.

Let the row vectors of be denoted as ; that is,

Thus, we can write

where are the components of . Therefore, we have

where . Hence,

As a result, after and have been computed, the decomposition correction matrix can be directly obtained from (62), removing the need to compute the Householder matrix .

We can also compare the condition numbers of   and in the correction formulae (3) and (21). The condition number is directly related to the possible magnification in relative error of the matrix calculations. Thus, a smaller condition number implies better numerical stability.

Because the correction matrix is symmetric and positive-definite, if the eigenvalues , of have size order

then the condition number of in terms of the spectral norm [20] is

Because ,

and therefore

Clearly, only when is a diagonal matrix whose diagonal elements are all the same. Otherwise, . Furthermore, and , which is beneficial to the calculation stability of (21). Note that (66) holds for all , so the numerical stability of correction formula (21) should be much better than that of (3).

In addition, the iterative calculation of the decomposition correction matrix involves only nonsingular matrices. Thus, the calculation accuracy requirements should be lower than in the original BFGS algorithm, which considers symmetric positive-definite matrices.

Conflict of Interests

The author declares that there is no conflict of interests regarding the publication of this paper.

Acknowledgment

This research was financially supported by Twelfth Five-Year-Plan Science and Technology Research Project of Educational Department, Jilin Province (2014598).

References

L. Liu and Y. Liu, “Matrix decomposition from the hypothesis testing,” Journal of Beihua University (Natural Science), vol. 2, no. 6, pp. 469–471, 2001.

Y. Liu and L. Liu, “Improvement on the algorithm of equal line sum decomposition for sample correlation coefficient matrix,” Journal of Wuhan University of Technology, vol. 31, no. 12, pp. 165–168, 2009.

L. Liu, “Extension and application of a matrix decomposition algorithm,” Mathematics in Practice and Theory, vol. 41, no. 21, pp. 239–243, 2011.

L. Liu, “Quasi-Newton methods using equal inner product decomposition matrix of the correction matrix to obtain the search direction,” Mathematics in Practice and Theory, vol. 43, no. 10, pp. 214–219, 2013.

L. Liu and Y. Liu, “Correction formula of Broyden's convex class expressed with decomposition matrix,” Journal of Beihua University (Natural Science), vol. 15, no. 2, pp. 179–183, 2014.

C. G. Broyden, “The convergence of a class of double-rank minimization algorithms. II. The new algorithm,” Journal of the Institute of Mathematics and Its Applications, vol. 6, pp. 222–231, 1970.

R. Fletcher, “A new approach to variable metric algorithms,” Computer Journal, vol. 13, no. 3, pp. 317–322, 1970.

D. Goldfarb, “A family of variable-metric methods derived by variational means,” Mathematics of Computation, vol. 24, pp. 23–26, 1970.

D. F. Shanno, “Conditioning of quasi-Newton methods for function minimization,” Mathematics of Computation, vol. 24, pp. 647–656, 1970.

P. E. Gill and W. Murray, “Quasi-Newton methods for unconstrained optimization,” Journal of the Institute of Mathematics and its Applications, vol. 9, pp. 91–108, 1972.

S. L. Xi, Nonlinear Optimization Method, Higher Education Press, Beijing, China, 1992.

N. Y. Deng, M. F. Zhu, D. F. Liu, Z. H. Yang, L. Wang, and Y. Tang, Unconstrained Optimization Calculation Method, Science Press, Beijing, China, 1982.

Y. X. Yuan and W. Y. Sun, Optimization Theory and Method, Science Press, Beijing, China, 1997.

M. J. D. Powell, “Some global convergence properties of a variable metric algorithm for minimization without exact line searches,” in Nonlinear Programming, R. W. Cottle and C. E. Lemke, Eds., American Mathematical Society, Providence, RI, USA, 1976.

R. H. Byrd, J. Nocedal, and Y. X. Yuan, “Global convergence of a class of quasi-Newton methods on convex problems,” SIAM Journal on Numerical Analysis, vol. 24, no. 5, pp. 1171–1190, 1987.

S. S. Oren and D. G. Luenberger, “Self-scaling variable metric (SSVM) algorithms. I. Criteria and sufficient conditions for scaling a class of algorithms,” Management Science, vol. 20, pp. 845–862, 1974.

S. S. Oren, “Self-scaling variable metric (SSVM) algorithms. II. Implementation and experiments,” Management Science, vol. 20, pp. 863–874, 1974.

S. S. Oren, “On the selection of parameters in self scaling variable metric algorithms,” Mathematical Programming, vol. 7, pp. 351–367, 1974.

H. X. Huang and J. Y. Han, Mathematical Programming, Tsinghua University Press, Beijing, China, 2006.

Q. S. Liu, X. D. Zhang, and B. T. Wang, Numerical Calculation Method Cause, Metallurgical Industry Press, Beijing, China, 2005.