Journal of Applied Mathematics

Volume 2014 (2014), Article ID 610612, 8 pages

http://dx.doi.org/10.1155/2014/610612

Research Article

A Novel Self-Adaptive Trust Region Algorithm for Unconstrained Optimization

Yunlong Lu, Wenyu Li, Mingyuan Cao, and Yueting Yang

School of Mathematics and Statistics, Beihua University, Jilin 132013, China

Received 28 August 2013; Revised 18 March 2014; Accepted 19 March 2014; Published 15 April 2014

Academic Editor: Kazutake Komori

Copyright © 2014 Yunlong Lu et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

A new self-adaptive rule of trust region radius is introduced, which is given by a piecewise function on the ratio between the actual and predicted reductions of the objective function. A self-adaptive trust region method for unconstrained optimization problems is presented. The convergence properties of the method are established under reasonable assumptions. Preliminary numerical results show that the new method is significant and robust for solving unconstrained optimization problems.

1. Introduction

Consider the following unconstrained optimization problem:

where is continuously differentiable.

The trust region methods calculate a trial step by solving the subproblem at each iteration,

where and is symmetric matric approximating the Hessian of at , and is the current trust region radius. Throughout this paper, denotes the -norm. We define the ratio,

and the numerator and the denominator are called the actual reduction and the predicted reduction, respectively. For basic trust region (BTR) method, if the sufficient reduction predicted by the model is realized by the objective function, the trial point is accepted as the next iterate and the trust region is expanded or kept the same. If the model reduction turns out to be a poor predictor of the actual behavior of the objective function, the trial point is rejected and the trust region is contracted, with the hope that the model provides a better prediction in the smaller region. More formally, basic trust region radius update rule can be usually summarized as follows:

where the constants ,,, and satisfy

The iteration is said to be successful if . If not, the iteration is unsuccessful, and the trial point is rejected. If , the iteration is said to be very successful iteration [1]. If is significantly larger than one, that is, , the iteration is called too successful iteration [2].

Sartenaer [3] developed an elaborate strategy which can automatically determine an initial trust region radius. The basic idea is to determine a maximal initial radius through many repeated trials in the steepest descent direction in order to guarantee a sufficient agreement between the model and the objective function. This strategy requires additional evaluations of the objective function. Zhang et al. [4] presented another strategy of determining the trust region radius. Their basic idea is originated from the following subproblem in an artificial neural network research [5]:

where , ,, and is a nonnegative integer. Therefore, instead of adjusting , one adjusts at each iteration. Motivated by this technique, they solved the trust region subproblem with

where , is a nonnegative integer, and is a positive definite matrix for some . However, their method needs to estimate or at each iteration, which leads to some additional cost of computation. As a result, Shi and Guo [6] proposed a simple adaptive trust region method. The new trust region model is more consistent with the objective function at the current iteration. Fu et al. [7] developed an adaptive trust region method based on the conic model by using the above adaptive strategy [4]. Sang and Sun [8] gave a new self-adaptive adjustment strategy to update the trust region radius, which makes full use of the information at the current point. Yu and Li [9] proposed a self-adaptive trust region algorithm for solving this nonsmooth equation. Some authors [1, 10] adopted different values for parameters (5) but seldom questioned the radius update formula (4). In addition, many adaptive nonmonotonic trust region methods have been proposed in [11–18]. Hei [19] proposed a self-adaptive update method, in which trust region radius is a product of a so-called -function and ; that is, , where is the -function. As the ratio is larger than one, is nondecreasing and bounded. However, as the iteration is very successful, the ratio is larger than one; it implies that the local approximation of the objective function by the model function is not good. Walmag and Delhez [2] suggested that it is not overconfident in the model at too successful iterations. They presented a self-adaptive update method, in which trust region radius is , where is the -function. If is significantly larger than one, -function is nonincreasing. However, they took but close to one to match the convergence criteria presented by Conn et al. [1].

In our opinion, the agreement between the model and the objective function is not good enough at too successful iterations. We take that the updated trust region radius is less than and is bounded lower away from zero. It implies that is not always necessary. This strategy can also match the convergence criteria presented by Conn et al. [1].

In the paper, the -function is introduced, which is a variant of -function. A new self-adaptive trust region method is proposed, in which the trust region radius is . The new method is more efficient at too successful iterations.

The rest of the paper is organized as follows. In Section 2, we define -function to introduce new update rules and a new self-adaptive trust region algorithm is presented. In Section 3, the convergence properties of proposed algorithm are investigated. In Section 4, numerical results are given. In Section 5 conclusions are summarized.

2. -Function and the New Self-Adaptive Trust Region Algorithm

To obtain the new trust region radius update rules, we define -function ,.

Definition 1. A function is called an -function if it satisfies the following:

(1) is nondecreasing in and nonincreasing in , , for ,(2),(3),(4),(5),(6), for ,(7)and ,

where the constants ,,,,,, are positive constants such that

It is easy to prove that the -function is a bounded function in . In the following, we show the differences between the usual empirical rule, the -function rule, the -function rule, and the -function rule.

The usual empirical rules ([1, 20, 21]) (Figure 1(a)) can be usually summarized as follows:

where ,,, and are predefined constants such that

(a)

(b)

(c)

Figure 1: (a) Step function. (b) -function . (c) -function .

The -function rule and the -function rule can be described as follows:

where the -function (Figure 1(b)) proposed by Hei [19] is chosen as

where , , , , and are constants, and the -function (Figure 1(c)) proposed by Walmag and Delhez [2] is chosen as

where ,,, and are constants.

The -function rule can be described as follows:

where the -function (Figure 2) is chosen as

Figure 2: -function .

where , , , , , and are constants.

The -function generalizes the -function and the -function. It contains some favorable features of the -function [19] and the -function [2].

Now describe the new self-adaptive trust region algorithm with improved update rules.

Algorithm 2. One has the following.

Step 1. Given , , ,,, and ; ; set .

Step 2. If or , stop. Otherwise solve subproblem (2) to get .

Step 3. Compute

Compute as follows:

Update the trust region radius

where is defined by (15).

Step 4. Compute and ; set ; go to step 2.

3. Convergence of Algorithm 2

In the section, we investigate the convergence properties of Algorithm 2. Since it can be considered as a variant of the basic trust region method of Conn et al. [1], we expect similar results and significant similarities in their proofs under the following assumptions.

Assumption 3. Consider the following.

(i)The sequence is uniformly bounded in norm; that is , for some constant .(ii)The function is bounded on the level set .(iii)The solution of the subproblem (2) satisfies

 where .(iv)The solution of the subproblem (2) satisfies

 for positive constant .

Lemma 4. Suppose that Assumption 3 holds. Then

where arbitrarily decreases with decreasing.

Proof. Since from Taylor theorem, we have that

it follows from the definition of in (2) that

By Algorithm 2, we are capable of showing that the iteration must be very successful but not too successful if the trust region radius is sufficiently small enough and also that the trust region radius has to increase in this case. The following lemma’s proof is a bit different from the proof of Theorem in [1].

Lemma 5. Suppose that Assumption 3 holds. If and there exists a constant such that

then

Proof . Using Assumption 3 and and assuming that there is such that , we obtain that

Combining (21) and (26), we have

By (24), we can choose sufficient small , such that

Using (27) and (28), we have ; that is, . Since , then . And so, by Algorithm 2, we have , where falls below the threshold .

The proof of Lemma 5 efficiently uses the conditions to explain the case of too successful iteration, as distinguished from the proof of Theorem in [1].

Theorem 6. Suppose that Assumption 3 holds. Let the sequence be generated by Algorithm 2. Then

Proof. Assume, for the purpose of deriving a contradiction, that, for all ,

Suppose that there is an infinite iteration subsequence such that . Using Algorithm 2 and (21), we have

where . Since is bounded below, then

which contradicts (25). Hence (30) holds.

In our strategy of the trust region radius’ adjustment, most of the iterations are indeed very successful but not too successful; the trust region constraint becomes irrelevant in the local subproblem. Therefore, superlinear convergence of trust region algorithm is preserved by the proposed self-adaptive radius update.

4. Numerical Experiments

In this section, we present preliminary numerical results to illustrate that the performance of Algorithm 2, denoted by LATR, the basic trust region method in [1], denoted by BTR, and the parameters needed in (9) are chosen that ,,, and ; the adaptive trust region method in [19], denoted by RATR, and the parameters needed in (12) are chosen that ,,, and ; and the adaptive trust region method in [2], denoted by ATR, and the parameters needed in (12) are chosen that ,,, and . In Algorithm 2, , , , , , , and . All tests are implemented by using MATLAB R2012a on a PC with CPU 2.67 GHz and 8.00 GB RAM. The first eleven problems are taken from [22]; others are from the CUTEr collection [15, 23]. The discrete Newton method is used to update the approximate Hessian matrix . To stabilize the algorithms, the approximate Hessian matrix can be chosen as follows:

where is obtained by forward difference formula at (see [20]) and is an identity matrix.

In all trust region algorithms in this paper, the trial step is computed by CG-Steihaug algorithm in [20]. The iteration is terminated by the following condition:

except for problem Waston, which will exceed 500 iterations. For the problem, the stopping criterion is

In Table 1, we give the dimension (Dim) of each test problem, the number of function evaluations (nf), and the number of gradient evaluations (ng). In many cases, algorithm LATR is superior than algorithms BTR, RATR, and ATR, especially for solving problems (1), (7), (8), (13), (19), and (28), although the numbers of gradient evaluations are a bit more than others sometimes. Further result is shown by Figure 3, which is characterized by means of performance profile proposed in [24]. Consider the following performance profile function:

where and

is the performance ratio of a solver on a problem; denotes the number of the tested problems, the number of the solvers, and the number of the function evaluations (or the CPU time, the number of gradient evaluations, number of iterations, etc.) required to solve the problem by the solver .

Table 1: Numerical comparisons of BTR, RATR, ATR, and LATR.

Problem
Dim

BTR
RATR
ATR
LATR

nf
ng
nf
ng
nf
ng
nf
ng

(1) Trigonometric
200
214
17
205
10
—
—
78
9

(2) Extended Powell singular
200
28
20
26
20
32
21
30
24

(3) Schittkowski function 302
200
239
160
363
275
117
24
83
65

(4) Linear function full rank
200
23
4
21
4
15
3
11
7

(5) Watson
200
30
14
22
10
16
10
18
11

300
39
15
115
78
73
49
67
48

400
84
39
27
12
38
25
107
78

500
95
43
29
14
68
48
39
26

(6) Nearly separable
200
41
21
27
15
23
16
24
17

300
46
25
34
18
24
16
23
16

400
52
25
39
23
33
22
26
19

500
—
—
—
—
—
—
37
26

(7) Yang tridiagonal
200
67
40
61
44
99
32
51
41

300
151
113
72
57
105
38
74
61

400
133
93
141
106
151
68
109
90

500
148
97
120
95
257
149
109
90

(8) Allgower
200
40
23
53
38
57
2
83
69

300
18
1
17
1
24
1
7
1

400
25
15
17
3
56
2
21
17

500
20
3
17
3
52
3
9
2

(9) Linear function rank 1
200
101
42
62
27
285
30
39
24

300
102
40
66
25
300
30
35
16

400
110
45
69
25
319
32
39
20

500
108
42
70
25
317
31
47
32

(10) Linear function rank 1 with zero columns and rows
200
64
3
60
2
21
2
22
2

300
67
3
64
3
21
1
21
1

400
66
1
65
1
29
5
22
1

500
78
6
67
2
24
2
26
4

(11) Discrete integral equation
200
54
19
24
4
39
14
27
18

300
31
5
70
46
42
22
30
28

400
44
12
43
32
32
16
31
29

500
49
15
37
17
45
23
31
29

(12) CRAGGLVY
200
36
9
35
9
22
10
19
9

(13) GENHUMPS
200
26
0
26
0
74
0
9
0

(14) BROYDNBD
200
32
7
20
14
47
0
17
15

(15) PENALTY
200
71
33
69
33
59
33
59
37

300
75
35
74
35
63
38
59
35

400
74
33
76
35
64
36
61
37

500
77
35
80
37
58
36
62
38

(16) PENALTY2
200
46
8
42
8
88
9
25
11

300
149
72
159
129
78
50
150
116

400
434
208
—
—
—
—
465
390

500
—
—
330
228
—
—
—
—

(17) CHEBYQAD
200
75
55
79
70
39
7
83
67

300
28
12
83
70
41
6
89
70

400
95
63
99
85
108
64
104
82

500
32
16
37
22
119
52
103
83

(18) GENROSE
200
194
161
253
248
265
172
241
205

300
308
235
381
373
394
248
357
303

400
405
301
487
479
499
323
468
399

500
—
—
—
—
—
—
—
—

(19) INTEGREQ
200
15
1
14
1
214
0
6
1

300
15
1
14
1
206
0
6
1

400
15
1
14
1
203
0
6
1

500
15
1
14
1
201
0
6
1

(20) FLETCHCR
200
113
88
151
146
157
91
158
132

300
147
122
218
213
224
132
233
195

400
183
157
292
284
295
175
309
258

500
217
190
357
351
362
211
384
321

(21) ARGLINB
200
6
5
6
5
6
5
6
5

300
7
6
7
6
7
6
7
6

400
7
6
7
6
300
5
7
6

500
7
6
7
6
424
5
7
6

(22) NONDIA
200
7
6
7
6
93
6
7
6

300
7
6
7
6
7
6
7
6

400
6
5
6
5
6
5
6
5

500
6
5
6
5
6
5
6
5

(23) EG2
200
14
9
11
9
59
23
10
8

300
14
9
12
9
60
27
6
5

400
9
7
7
6
180
30
8
6

500
12
5
17
13
111
34
14
11

(24) CURLY20
200
13
11
19
17
16
14
11
10

300
13
11
21
18
17
15
13
12

400
10
9
11
11
11
10
9
9

500
10
9
11
11
11
10
9
9

(25) CUBE
200
52
20
39
25
95
16
37
29

300
50
16
77
65
98
20
40
32

400
52
16
45
31
83
16
35
28

500
52
16
52
38
128
14
35
28

(26) EXPLIN1
200
30
2
28
2
12
3
13
4

300
29
1
28
1
33
4
13
4

400
31
2
29
2
23
4
21
16

500
32
2
30
2
15
3
13
4

(27) SINQUAD
200
68
55
18
17
119
14
19
17

300
186
175
16
15
225
20
64
60

400
25
12
21
19
224
25
48
45

500
31
15
43
41
229
40
35
33

(28) LIARWHD
200
29
1
28
1
50
2
14
4

300
32
2
29
2
18
2
12
2

400
30
1
30
1
30
2
12
1

500
30
1
30
1
33
2
11
1

— means that the algorithm reaches 500 iterations.

Figure 3: Performance profile comparing the sum of the number of function and gradient evaluations.

From Table 1, we know that and ; then performance profile is given on the sum of the number of function and gradient evaluations to solve the problem. As we can see on Figure 3, the new self-adaptive algorithm is superior to the other three algorithms.

5. Conclusion

This paper presents a new self-adaptive trust region algorithm according to the new self-adaptive radius update rule. As the iteration istoo successful, we suggest reducing the trust region radius with the new rules. The convergence properties of the method are established under reasonable assumptions. Numerical experiments show that the new algorithm for solving unconstrained optimization problems is significant and robust.

For future research, we should investigate how to select an appropriate -function to conduct numerical experiments.

Conflict of Interests

The authors declare that they have no financial nor personal relationships with other people or organizations that can inappropriately influence their work; there is no professional nor another personal interest of any nature or kind in any product, service, and/or company that could be construed as influencing the position presented in or the review of the paper.

Acknowledgments

This research is partly supported by Chinese NSF under Grant 11171003, Chinese Ministry of Education, Science and Technology Research Key Project no. 2011039, and Department of Education of Jilin Province Project no. 2009158 and no. 201215102.

References

A. R. Conn, N. I. M. Gould, and P. L. Toint, Trust-Region Methods, MPS/SIAM Series on Optimization, Society for Industrial and Applied Mathematics (SIAM); Mathematical Programming Society (MPS), Philadelphia, Pa, USA, 2000.

J. M. B. Walmag and E. J. M. Delhez, “A note on trust-region radius update,” SIAM Journal on Optimization, vol. 16, no. 2, pp. 548–562, 2005.

A. Sartenaer, “Automatic determination of an initial trust region in nonlinear programming,” SIAM Journal on Scientific Computing, vol. 18, no. 6, pp. 1788–1803, 1997.

X. Zhang, J. Zhang, and L. Liao, “An adaptive trust region method and its convergence,” Science in China A, vol. 45, no. 5, pp. 620–631, 2002.

X. S. Zhang, “Trust region method in neural network,” Acta Mathematicae Applicatae Sinica, vol. 12, pp. 1–10, 1996.

Z.-J. Shi and J.-H. Guo, “A new trust region method for unconstrained optimization,” Journal of Computational and Applied Mathematics, vol. 213, no. 2, pp. 509–520, 2008.

J. Fu, W. Sun, and R. J. B. de Sampaio, “An adaptive approach of conic trust-region method for unconstrained optimization problems,” Journal of Applied Mathematics & Computing, vol. 19, no. 1-2, pp. 165–177, 2005.

Z. Sang and Q. Sun, “A self-adaptive trust region method with line search based on a simple subproblem model,” Journal of Computational and Applied Mathematics, vol. 232, no. 2, pp. 514–522, 2009.

Z. Yu and Q. Li, “A self-adaptive trust region method for the extended linear complementarity problems,” Applications of Mathematics, vol. 54, no. 1, pp. 53–65, 2009.

N. I. M. Gould, D. Orban, A. Sartenaer, and P. L. Toint, “Sensitivity of trust-region algorithms to their parameters,” 4OR. A Quarterly Journal of Operations Research, vol. 3, no. 3, pp. 227–241, 2005.

J.-L. Zhang and X.-S. Zhang, “A nonmonotone adaptive trust region method and its convergence,” Computers & Mathematics with Applications, vol. 45, no. 10-11, pp. 1469–1477, 2003.

J. Fu and W. Sun, “Nonmonotone adaptive trust-region method for unconstrained optimization problems,” Applied Mathematics and Computation, vol. 163, no. 1, pp. 489–504, 2005.

J. Zhang, K. Zhang, and S. Qu, “A nonmonotone adaptive trust region method for unconstrained optimization based on conic model,” Applied Mathematics and Computation, vol. 217, no. 8, pp. 4265–4273, 2010.

M. Ahookhosh and K. Amini, “A nonmonotone trust region method with adaptive radius for unconstrained optimization problems,” Computers & Mathematics with Applications, vol. 60, no. 3, pp. 411–422, 2010.

N. Andrei, “An unconstrained optimization test functions collection,” Advanced Modeling and Optimization, vol. 10, no. 1, pp. 147–161, 2008.

Z. Shi and S. Wang, “Nonmonotone adaptive trust region method,” European Journal of Operational Research, vol. 208, no. 1, pp. 28–36, 2011.

Z. Sang and Q. Sun, “A new non-monotone self-adaptive trust region method for unconstrained optimization,” Journal of Applied Mathematics and Computing, vol. 35, no. 1-2, pp. 53–62, 2011.

Z. Cui and B. Wu, “A new modified nonmonotone adaptive trust region method for unconstrained optimization,” Computational Optimization and Applications, vol. 53, no. 3, pp. 795–806, 2012.

L. Hei, “A self-adaptive trust region algorithm,” Journal of Computational Mathematics, vol. 21, no. 2, pp. 229–236, 2003.

J. Nocedal and S. T. Wright, Numerical Optimization, Springer, Berlin, Germany, 2000.

W. Sun and Y.-X. Yuan, Optimization Theory and Methods, Nonlinear programming, vol. 1 of Springer Optimization and Its Applications, Springer, New York, NY, USA, 2006.

J. J. Moré, B. S. Garbow, and K. E. Hillstrom, “Testing unconstrained optimization software,” Association for Computing Machinery. Transactions on Mathematical Software, vol. 7, no. 1, pp. 17–41, 1981.

N. I. M. Gould, D. Orban, and P. L. Toint, “GALAHAD, a library of thread-safe Fortran 90 packages for large-scale nonlinear optimization,” Association for Computing Machinery. Transactions on Mathematical Software, vol. 29, no. 4, pp. 353–372, 2003.

E. D. Dolan and J. J. Moré, “Benchmarking optimization software with performance profiles,” Mathematical Programming, vol. 9, pp. 1201–1213, 2002.