Journal of Optimization

Volume 2014 (2014), Article ID 237279, 8 pages

http://dx.doi.org/10.1155/2014/237279

Research Article

A Nonmonotone Adaptive Trust Region Method Based on Conic Model for Unconstrained Optimization

Zhaocheng Cui

Department of Mathematics and Physics, Shandong Jiaotong University, Ji’nan, Shandong Province 250023, China

Received 20 August 2013; Accepted 29 November 2013; Published 27 January 2014

Academic Editor: Adil M. Bagirov

Copyright © 2014 Zhaocheng Cui. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

We propose a nonmonotone adaptive trust region method for unconstrained optimization problems which combines a conic model and a new update rule for adjusting the trust region radius. Unlike the traditional adaptive trust region methods, the subproblem of the new method is the conic minimization subproblem. Moreover, at each iteration, we use the last and the current iterative information to define a suitable initial trust region radius. The global and superlinear convergence properties of the proposed method are established under reasonable conditions. Numerical results show that the new method is efficient and attractive for unconstrained optimization problems.

1. Introduction

In this paper, we consider the following unconstrained optimization problem:

where : is a continuously differentiable function.

Trust region method is effective for solving (1). In 1970s, Powell [1] established the convergence result of trust region method. Yuan [2], Nocedal, and Yuan [3] proposed various trust region methods for optimization problems. It calculates a trial step by solving the subproblem

where is the gradient of the objective function at , is a symmetric matrix which is either the Hessian matrix of at or an approximation to it, refers to the Euclidean norm, and is the radius of an area, called the trust region, where the model is trusted. Let be the solution of (2). The actual reduction of the objective function is defined by

and the predicted reduction of is defined by

The ratio between these two reductions is defined by

and it is normally used to test whether the trial step is accepted or the trust region radius needs to be adjusted. The next iterate is chosen by the following formula:

where is a constant. The next trust region radius is chosen as

where , are constants. In comparison with quasi-Newton methods, trust region methods converge to a point which not only is a stationary point but also satisfies second-order necessary conditions. Trust region method has strong convergence and robustness, so many authors have studied it; see [4–7]. Conn et al. [8], Nocedal and Wright [9], and Yuan and Sun [10] presented the complete introduction.

Recently, Grippo et al. [11] presented the nonmonotone line search method for unconstrained optimization problems. Due to its high efficiency of nonmonotone method, this technique has been combined with trust region methods; we can see [12–16].

Sartenaer [17] presented a strategy for determining automatically an initial trust region radius. Fan and Yuan [18] proposed a trust region method with the trust region radius converging to zero. Recently, Zhang et al. [19] gave an adaptive trust region method, and they solve the subproblem (2) with , and is a nonnegative integer. is a safely positive definite matrix based on Schnabal and Eskow modified Cholesky factorization. Therefore, instead of adjusting , one adjusts at each iteration. Based on [19] and a simple subproblem model, Sang and Sun [20] proposed another adaptive trust region method with line search. The numerical results show that the adaptive trust region algorithms [21–24] are more effective than traditional trust region methods for unconstrained optimization problems.

The adaptive trust region methods listed above are based on quadratic model; however, when the objective function has strong nonquadratic behavior, the quadratic model methods often produce a poor prediction of the minimizer of the function. In 1980, Davidon [25] proposed the conic model methods for unconstrained optimization problems. A typical conic trust region subproblem is as follows:

where is called conic model which is an approximation to . The vector is the associated vector for the collinear scaling in the th iteration, and it is normally called the horizontal vector. If , the conic model reduces to a quadratic model. Also, one can see that is quadratic along any direction satisfying . Recently, trust region methods based on conic model have been presented and studied; see [26–32]. Numerical results show the efficiency of the conic trust region methods.

In the traditional trust region methods, the next trust region radius is updated by the current ratio , which is reasonable if the matrix is exactly as the Hessian and if the trust region subproblem (2) is solved exactly. However, in practical computations, the matrix is often obtained approximately and the subproblem is solved approximately. In such a case, it may be more reasonable to adjust the next trust region according to not only but also the previous ratios , where is some nonnegative integer.

In order to measure the agreement between the model function and the objective function, we define the following ratio:

where is the weight of , such that

In this paper, we propose a new adaptive trust region method which combines the conic model and nonmonotone technique which was given in [33] and a new update rule for adjusting the trust region radius. Our method is different from other adaptive trust region methods in three points. First, the subproblem is conic model which can produce a better prediction of the minimizer of the function than quadratic model methods when the objective function has strong nonquadratic behavior. Second, at each iteration, our method generates a suitable trust region radius automatically based on the current and last iterative information. Last, we use a new update rule (8) to adjust the next trust region radius. The new trust region model is more consistent with the objective function at the current iterative point. According to the nonmonotone technique in [33], our method constructs an iterative series such that the sequence is not monotonic, which can accelerate the convergence rate of the minimization process, especially in the narrow curved valley. The global and superlinear convergence properties of the proposed method are proved under reasonable conditions. The numerical results show that the new method is more effective than other trust region methods.

The rest of this paper is organized as follows. In next section, we describe a new nonmonotone adaptive trust region method based on conic model for unconstrained optimization problems. In Sections 3 and 4, we prove the global and superlinear convergence properties of the proposed method under reasonable conditions, respectively. Some numerical results are given in Section 5. Conclusions are summarized in Section 6.

2. Algorithm Description

In this section, we describe a new nonmonotone adaptive trust region method based on conic model for unconstrained optimization problems.

We obtain the trial step by solving the following subproblem:

where , , . is updated according to (8), is the solution of the last subproblem. is generated by the procedure and is the smallest nonnegative integer such that

The new reduction of is defined by

where

is a positive integer. We define and choose

The predicted reduction of is

The new ratio is defined by

In what follows, we describe our new nonmonotone adaptive trust region method based on conic model.

Algorithm 1 (nonmonotone adaptive trust region method based on conic model). Consider the following.

Step . Given , is a symmetric positive definite matrix, , , , , , , and an integer constant ; set , .

Step . Compute . If , then stop. Otherwise, go to Step 2.

Step . Solve the subproblem (11) and let be an approximate solution of the subproblem (11).

Step . Compute , and .

Step . If , set and go to Step 2. Otherwise, go to Step 5.

Step . Choose that satisfying (9). Compute and, by (8), set and

Step . Update , and the symmetric matrix . Let and set and go to Step 1.

Remark 2. In Algorithm 1, we define a new reduction of which is different from the actual reduction in other methods. Instead of requiring to be smaller than , it is only required that is either less than or less than the weighted mean of the function values at the last iterates.

Remark 3. How to choose the horizontal vector and the approximate Hessian matrix is one of the crucial issues of a conic model method. More details for generating and can be found in [25]. If , Algorithm 1 reduces to the quadratic model trust region method. If we choose , we obtain a monotonic adaptive trust region method. Hence, Algorithm 1 is a generalization and development of quadratic model trust region method and monotonic trust region method.

Remark 4. “Step 2-Step 3-Step 4" are called the internal circulation.

3. Convergence Analysis

In this section we discuss the global convergence of Algorithm 1. Before we address some theoretical issues, we would like to give the following assumptions.

Assumption 5. The level set is bounded for any given and is twice continuously differentiable in . The sequences and are uniformly bounded; that is, there exist two positive scalars such that and hold for all .   There exists a positive constant , such that .

It is similar to Theorem  3.1 in [28], and we can prove the following lemma.

Lemma 6. Suppose that Assumption 5 holds. Then, there exists a scalar such that

holds for all , where is an inexact solution of the subproblem (11).

Lemma 7. Suppose that Assumption 5 holds. Then,

where is the solution of (11) and is sufficiently close to zero.

Proof. Since is bounded, we can have . Then, from the boundedness of and , we obtain

By which is twice continuously differentiable in , there exists such that holds for all Hessian and all .

Therefore, we have that

This completes the proof.

Lemma 8. Suppose that Assumption 5 holds. Then, Algorithm 1 is well defined; that is, Algorithm 1 can not cycle infinitely in the internal circulation.

Proof. Suppose that Algorithm 1 cycles infinitely many times between Steps 2 and 4 at the current point . We denote the solution of the conic model (11) by and the corresponding predicted reduction by with the cycling index . Then, we have

From Lemmas 6 and 7, we can have

Hence, the following

holds for sufficiently large. From , we obtain that

holds for sufficiently large, which contradicts with (23). The proof is completed.

In the next lemma, an important decrease property of the function value is established.

Lemma 9. Suppose that Assumption 5 holds and is generated by Algorithm 1. Then, there exists a constant such that the following

holds for .

Proof. We prove it by induction. For simplicity, we denote

If , by , , and Lemma 6, we have

where .

Now, we assume that it holds for , and we consider two cases.

Case 1. If , by , , and Lemma 6, then we have

Case 2. . Let , by , , and Lemma 6. Then, we have

Using , , , we have

Lemma 10. Suppose that the conditions in Lemma 9 hold. Then, .

Proof. By Lemma 9, , the conclusion holds obviously.

Now, we prove the global convergence of Algorithm 1.

Theorem 11. Suppose that Assumption 5 holds. If , Algorithm 1 either terminates in finite iterations or generates an infinite sequence which satisfies

Proof. If Algorithm 1 terminates in finite iterations, the theorem is obviously true. Assume that Algorithm 1 generates an infinite sequence in the following proof.

If (34) is not true, then there exists a positive constant , such that

From Lemma 9, , and , we can have

By Assumption 5(ii), we can obtain

Therefore, by Assumption 5(i), we have

Thus, we can obtain

By Assumption 5(ii), then . From Algorithm 1, we know that

By (39) and (40), we obtain

On the other hand, we have from Lemma 6 that

The above inequality implies that

holds for sufficiently large. Hence, there exists a positive constant such that

holds for all sufficiently large , which contradicts with (41). The proof is completed.

4. Superlinear Convergence

In general, the superlinear convergence property of Algorithm 1 requires further assumptions, we give the assumptions and the superlinear convergence property in the following theorem.

Theorem 12. Suppose that Assumption 5 holds and Algorithm 1 generates an infinite sequence which converges to , where is positive definite and is Lipschitz continuous in a neighborhood of . If the following condition holds

then the rate of convergence is superlinear; that is, .

Proof. From , we can have . By the Taylor expansion, we can obtain

where .

So,

Dividing both sides by , we get

By (45), , and Lipschitz continuous property of , we can have

By which is positive definite, and which is a twice continuously differentiable function, there exists , such that

for sufficiently large .

By (49) and (50), it follows that

and thus

which implies that the sequence converges to superlinear.

5. Numerical Results

In this section, we compare the performance of Algorithm 1, denoted by NACTR, with NAQTR if for all in Algorithm 1, MACTR if in Algorithm 1, CTR method which is a conic trust region algorithm without adaptive technique, and QTR method which is the traditional trust region method without adaptive technique. All programs are written in MATLAB with double precision.

For these methods, the trial step is computed approximately by the algorithm proposed by Lu and Ni in [34] for solving the subproblem (11). is chosen as the identity matrix. The parameters in our algorithm are chosen as , , , , , , , and . In our implementation, we prefer the following choice

where .

The stopping condition is

The iteration is also terminated if the number of evaluations exceeds 300.

We provide the results of our tests in Table 1. The numerical results are given in the form of , where denote the numbers of iterations and function evaluations, respectively. We chose the test problems from [35]. From Table 1, we can see that, in most of the problems considered, the numbers and for the conic model methods are considerably smaller than those required for the quadratic model methods, especially for those functions with strong nonquadratic behavior. This means that the conic model methods are very effective to many unconstrained optimization. Compared with the methods without adaptive technique, the adaptive trust region methods are more attractive. Hence, our new nonmonotone adaptive conic trust region method is an improvement of the existing trust region methods.

Table 1: Numerical comparisons.

Problems
QTR
CTR
NAQTR
MACTR
NACTR

Helical Valley
62/74
43/53
56/61
32/45
29/42

Gaussian
7/12
3/8
3/5
3/5
3/5

Box 3-D
41/45
33/37
36/41
30/35
30/33

Var. Dimen.
17/26
11/11
18/27
12/18
13/21

Waston
72/78
66/73
62/81
54/98
48/86

Penalty I
18/21
16/17
18/24
15/20
15/18

Penalty II
21/26
21/24
23/31
17/26
16/25

Brown-Dennis
62/66
78/84
62/75
55/67
53/58

Gulf R.D.
88/83
96/98
66/46
38/42
46/47

Trigonometric
46/52
38/39
45/48
32/39
34/36

Ex. Ros.
91/124
Failed
48/62
33/46
35/48

6. Conclusions

In this paper, we propose a new nonmonotone adaptive conic model trust region method for unconstrained optimization. The global and superlinear convergence properties of the new method are proved under reasonable conditions. Our method is attractive in the following aspects. First, the subproblem is conic model which can produce a better prediction of the minimizer of the function than quadratic model methods when the objective function has strong nonquadratic behavior. Second, at each iteration, our method generates a suitable trust region radius automatically based on the current and last iterative information. Last, we use a new update rule (8) to adjust the next trust region radius. According to the nonmonotone technique in [33], our method constructs an iterative series such that the sequence is not monotonic, which can accelerate the convergence rate of the minimization process, especially in the narrow curved valley. Numerical results show that the new method is efficient for unconstrained optimization problems.

Conflict of Interests

The author declares that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

The author is thankful to the editor and the reviewers for their constructive comments, which greatly improved this paper. This work was supported by the National Science Foundation of China (Grant no. 11226229) and Foundation of Shandong Jiaotong University.

References

M. J. D. Powell, “Convergence properties of a class of minimization algorithms,” in Nonlinear Programming, J. B. Rosen, O. L. Mangasarian, and K. Ritter, Eds., pp. 1–27, Academic press, New York, NY, USA, 1975.

Y. Yuan, “On the convergence of trust region algorithms,” Mathematica Numerica Sinica, vol. 16, pp. 333–346, 1996.

J. Nocedal and Y. Yuan, “Combining trust region and line search techniquesAdvances in Nonlinear Programming,” Y. Yuan, Ed., pp. 153–175, Kluwer Academic, Dordrecht, The Netherlands, 1996.

M. J. D. Powell, “A new algorithm for unconstrained optimization,” in Nonlinear Programming, J. B. Rosen, O. L. Mangassarian, and K. Ritter, Eds., pp. 31–65, Academic press, New York, NY, USA, 1970.

M. J. D. Powell, “On the global convergence of trust region algorithms for unconstrained optimization,” Mathematical Programming, vol. 29, no. 3, pp. 297–303, 1984.

T. Steilhaug, “The conjugate gradient method and trust region in large scale optimization,” SIAM Journal on Numerical Analysis, vol. 20, pp. 626–637, 1983.

E. M. Gertz, “A quasi-Newton trust-region method,” Mathematical Programming A, pp. 1–24, 1963.

A. R. Conn, N. I. M. Gould, and P. L. Toint, Trust-Region Methods, Society for Industrial and Applied Mathematics, Philadelphia, Pa, USA, 2000.

J. Nocedal and S. J. Wright, Numerical Optimization, Springer, New York, NY, USA, 1999.

Y. X. Yuan and W. Y. Sun, Optimization Theory and Methods, Science Press, Beijing, China, 1997.

L. Grippo, F. Lampariello, and S. Lucidi, “A nonmonotone line search technique for Newton's method,” SIAM Journal on Numerical Analysis, vol. 23, no. 4, pp. 707–716, 1986.

N. Y. Deng, Y. Xiao, and F. J. Zhou, “Nonmonotonic trust region algorithm,” Journal of Optimization Theory and Applications, vol. 76, no. 2, pp. 259–285, 1993.

W. Y. Sun, “Nonmonotone trust region method for solving optimization problems,” Applied Mathematics and Computation, vol. 156, no. 1, pp. 159–174, 2004.

P. L. Toint, “Non-monotone trust-region algorithms for nonlinear optimization subject to convex constraints,” Mathematical Programming B, vol. 77, no. 1, pp. 69–94, 1997.

J. T. Mo, C. Y. Liu, and S. C. Yan, “A nonmonotone trust region method based on nonincreasing technique of weighted average of the successive function values,” Journal of Computational and Applied Mathematics, vol. 209, no. 1, pp. 97–108, 2007.

X. Ke and J. Han, “A class of nonmonotone trust region algorithms for unconstrained optimization problems,” Science in China A, vol. 41, no. 9, pp. 927–932, 1998.

A. Sartenaer, “Automatic determination of an initial trust region in nonlinear programming,” SIAM Journal on Scientific Computing, vol. 18, no. 6, pp. 1788–1803, 1997.

J. Y. Fan and Y. X. Yuan, “A new trust region algorithm with trust region radius converging to zero,” in Proceedings of the 5th International Conference on Optimization: Techniques and Applications, Hong Kong, China, December 2001.

X. S. Zhang, J. L. Zhang, and L. Z. Liao, “An adaptive trust region method and its convergence,” Science in China A, vol. 45, no. 5, pp. 620–631, 2002.

Z. Y. Sang and Q. Y. Sun, “A self-adaptive trust region method with line search based on a simple subproblem model,” Journal of Computational and Applied Mathematics, vol. 232, no. 2, pp. 514–522, 2009.

L. Hei, “A self-adaptive trust region algorithm,” Journal of Computational Mathematics, vol. 21, pp. 229–236, 2003.

G. D. Li, “A trust region method with automatic determination of the trust region radius,” Chinese Journal of Engineering Mathematicss, vol. 23, pp. 843–848, 2006.

Z. J. Shi and J. H. Guo, “A new trust region method for unconstrained optimization,” Applied Mathematics and Computation, vol. 213, pp. 509–520, 2008.

J. L. Zhang and X. S. Zhang, “A nonmonotone adapyive trust region method and its convergence,” Computers and Mathematics with Application, vol. 45, pp. 1469–1477, 2003.

W. C. Davidon, “Conic approximation and collinear scaling for optimizers,” SIAM Journal on Numerical Analysis, vol. 17, pp. 268–281, 1980.

S. Di and W. Y. Sun, “A trust region method for conic model to solve unconstrained optimization,” Optimization Methods and Software, vol. 6, no. 4, pp. 237–263, 1996.

Q. Ni, “Optimality conditions for trust-region subproblems involving a conic model,” SIAM Journal on Optimization, vol. 15, no. 3, pp. 826–837, 2005.

S.-J. Qu, K.-C. Zhang, and J. Zhang, “A nonmonotone trust-region method of conic model for unconstrained optimization,” Journal of Computational and Applied Mathematics, vol. 220, no. 1-2, pp. 119–128, 2008.

W. Sun, “On nonquadratic model optimization methods,” Asia and Pacific Journal of Operational Research, vol. 13, pp. 43–63, 1996.

H. Gourgeon and J. Nocedal, “A Conic algorithm for optimization,” SIAM Journal of Scientific Stochastical Computation, vol. 6, pp. 253–267, 1985.

W. Y. Sun and Y. X. Yuan, “A conic trust-region method for nonlinearly constrained optimization,” Annal of Operational Research, vol. 103, pp. 175–191, 2001.

S. Sheng, “Interpolation by conic model for unconstrained optimization,” Computing, vol. 54, pp. 83–98, 1995.

Z. S. Yu and D. G. Pu, “A new nonmonotone line search technique for unconstrained optimization,” Journal of Computational and Applied Mathematics, vol. 219, no. 1, pp. 134–144, 2008.

X. P. Lu and Q. Ni, “A quasi-Newton trust region method with a new conic model for the unconstrained optimization,” Applied Mathematics and Computation, vol. 204, no. 1, pp. 373–384, 2008.

J. J. Moré, B. S. Grabow, and K. E. Hillstrom, “Testing unconstrained optimization software,” ACM Transactions on Mathematical Software, vol. 7, pp. 17–41, 1981.