Discrete Dynamics in Nature and Society
Volume 2017 (2017), Article ID 6406514, 9 pages
https://doi.org/10.1155/2017/6406514

Research Article

A New Augmented Lagrangian Method for Equality Constrained Optimization with Simple Unconstrained Subproblem

Hao Zhang1,2 and Qin Ni¹

¹College of Science, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China
²College of Engineering, Nanjing Agricultural University, Nanjing 210031, China

Correspondence should be addressed to Hao Zhang; zhanghao@njau.edu.cn

Received 3 May 2017; Accepted 16 August 2017; Published 19 October 2017

Academic Editor: Seenith Sivasundaram

Copyright © 2017 Hao Zhang and Qin Ni. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

We propose a new method for equality constrained optimization based on augmented Lagrangian method. We construct an unconstrained subproblem by adding an adaptive quadratic term to the quadratic model of augmented Lagrangian function. In each iteration, we solve this unconstrained subproblem to obtain the trial step. The main feature of this work is that the subproblem can be more easily solved. Numerical results show that this method is effective.

1. Introduction

In this paper, we consider the following equality constrained optimization:

where , , and are twice continuously differentiable.

The method presented in this paper is a variant of the augmented Lagrangian method (denoted by AL). In the late 1960s, AL method was proposed by Hestenes [1] and Powell [2]. Later Conn et al. [3, 4] presented a practical AL method and proved the global convergence under the LICQ condition. Since then, AL method attracted the attentions of many scholars and many variants were presented (see [5–11]). Up to now, there are many computer packages based on AL method, such as LANCELOT [4] and ALGENCAN [5, 6]. In the past decades, AL method was fully developed. Attracted by its well performance, there are still many scholars devoted to research AL method and its applications in recent years (see [7, 8, 11–15]).

For (1), we define the Lagrangian function

and the augmented Lagrangian function

where is called the Lagrangian multiplier and is called the penalty parameter. In this paper, refers to the Euclidean norm.

In a typical AL method, at the th step, for given multiplier and penalty parameter , an unconstrained subproblem

is solved to find the next iteration point. Then, the multiplier and penalty parameter are updated by some rules. For convenience, for given and , we define

Motivated by the regularized Newton method for unconstrained optimization (see [16–19]), we construct a new subproblem of (1). At the th iteration point , is approximated by the following quadratic model:

where and is a positive semidefinite approximation of . Let

thus we have

In [14, 15], is minimized within a trust region to find the next iteration point. Motivated by the regularized Newton method, we add a regularization term to the quadratic model and define

where is called regularized parameter. At the th step of our algorithm, we solve the following convex unconstrained quadratic subproblem:

for finding the trial step . Then, we compute the ratio between the actual reduction and predicted reduction

When is close to , we accept as the next iteration point. At the same time, we think the quadratic model is a sufficiently “good” approximation of and reduce the value of . Conversely, when is close to zero, we set and increase the value of , by which we wish to reduce the length of the next trial step. This technique is similar to the update rule of trust region radius. Actually, sufficiently large indeed reduces the length of the trial step . However, the regularized parameter is different from trust region radius. In [14, 15], the authors construct a trust region subproblem

The exact solution of (12) satisfies the first-order critical conditions if there exists some such that is positive semidefinite and

while the first-order critical condition of (10) is

Equations (13) and (15) can show the similarities and differences between regularized subproblem (10) and trust region subproblem (12). It seems that the parameter plays a role similar to the multiplier in the trust region subproblem. But, actually, the update rule of (see (26)) shows that is not the approximation of . The update of depends on the quality of last trial step and has no direct relation with system (13).

To establish the global convergence of an algorithm, some kind of constraint qualification is required. There are many well-known constraint qualifications, such as LICQ, MFCQ, CRCQ, RCR, CPLD, and RCPLD. In case there are only equality constraints, LICQ is equivalent to MFCQ in which has full rank; CRCQ is equivalent to CPLD in which any subset of maintains constant rank in a neighborhood of ; RCR is equivalent to RCPLD in which maintains constant rank in a neighborhood of . RCPLD is weaker than CRCQ, and CRCQ is weaker than LICQ. In this paper, we use RCPLD which is defined in the following.

Definition 1. One says that RCPLD holds at a feasible point of (1), if there exists a neighborhood of such that maintains constant rank for all .

The rest of this paper is organized as follows. In Section 2, we give a detailed description of the presented algorithm. The global convergence is proved in Section 3. In Section 4, we present the numerical experiments. Some conclusions are given in Section 5.

Notations. For convenience, we abbreviate to , to , to , and to . In this paper, denotes the th component of the vector .

2. Algorithm

In this section, we give a detailed description of the proposed algorithm.

As mentioned in Section 1, we solve the unconstrained subproblem (10) to obtain the trial step . Since is at least positive semidefinite and , is positive definite as Therefore, (10) is a strictly convex quadratic unconstrained optimization. solves (10) if and only if

holds. Global convergence does not depend on the exact solution of (15), although the linear system (15) is easy to be solved. For minimizer of (10) along the direction , specifically, we consider the following subproblem:

If , then the minimizer of (16) is . Therefore, at the th step, it follows that

By direct calculation, we have that

In Section 3, we always suppose that (18) holds.

In a typical AL algorithm, the update rule of depends on the improvement of constraint violation. A commonly used update rule is that if , where , one may think that the constraint violation is reduced sufficiently and thus is a good choice. Otherwise, if , one thinks that current penalty parameter can not sufficiently reduce the constraint violation and increase it in the next iteration. In [20], Yuan proposed a different update rule of for trust region algorithm. Specifically, if

is increased. In (19), is an auxiliary parameter such that tends to zero. We slightly modify (19) in our algorithm. Specifically, if

is increased.

In typical AL method, next iteration point is obtained by minimizing . In most AL methods, satisfies that , where is controlling parameter which tends to zero. As

when is sufficiently small, is a good estimate of the next multiplier . As we obtain by minimizing , the critical point of has no direct relation to . Therefore, the update rule does not suit our algorithm. We obtain by approximately solving the following least squares problem:

Most AL algorithms require that is bounded to ensure the global convergence. Hence, all components of are restricted to certain interval . This technique is also used in our algorithm.

Now, we give the detailed algorithm in the following.

Algorithm 2.  
Step  0 (initialization). Choose the parameters , , . Determine , , , , . Let Set .
Step  1 (termination test). If and , return as a KKT point. If , , and , return as an infeasible KKT point.
Step  2 (determine the trial step). Evaluate the trial step by solving

such that (18) holds. Compute the ratio between the actual reduction to the predicted reduction

where , . Set

Step  3 (update the penalty parameter). If

set

Otherwise, set

Step  4 (update the multiplier). If , set . Evaluate by

and let

If , set and .
Set and go to Step  1.

Remark 3. In practical calculation, it is not required to solve (30) exactly to find . In our implementation of Algorithm 2, we use the Matlab subroutine minres to find an approximate solution of the linear system and take it as an approximation of .

3. Global Convergence

In this section, we discuss the global convergence of Algorithm 2. We assume that Algorithm 2 can find an infinite set and give some assumptions in the following.

Assumptions 1. (A1) and are twice continuously differentiable.
(A2) and are bounded, where is positive semidefinite approximation of .
Firstly, we give a result on the upper bound of the trial step.

Lemma 4. If solves subproblem (23), then one has

Proof. Any approximate solution of (23) satisfies . Clearly,

If , then (32) holds. If , implies that or . Thus we can obtain (32).

Now, we discuss convergence properties in two cases. One is that the penalty parameter tends to and the other is that is bounded.

3.1. The Case of

Lemma 5. Suppose that (A1)-(A2) hold and ; then there exists a constant such that .

Proof. See Lemma  3.1 in Wang and Yuan [15].

In Lemma 5, if , then any accumulation point of is infeasible. Sometimes (1) is naturally infeasible; in other words, the feasible set is empty. In this case, we wish to find a minimizer of constraint violation. Specifically, we wish to solve

The solution of this problem is characterized by

In the next theorem, we show that if is not convergent to zero, at least one of the accumulation points of satisfies (35).

Theorem 6. Suppose that (A1)-(A2) hold and . If , then

Proof. We prove this result by contradiction. Suppose that there exists some such that

By the definition of in (7), we know that

As and are bounded, we can deduce the boundedness of by (A2); that is, there exists some such that

By (37), (38), and (39), we can conclude that

holds for all sufficiently large . By the boundedness of and , we can conclude that there exists , such that

holds for all sufficiently large , where is defined by (7). By (18), (40), and (41),

holds for all sufficiently large . By the update rule of and the fact that , we have that

holds for infinitely many . As holds for all sufficiently by (40), it is easy to see that (42) contradicts to (43) as and is convergent. Thus we can prove the desired result.

Lemma 7. Suppose that (A1)-(A2) hold, , and ; then

Proof. Assume that there exists such that

Then, by (18) and (41), we know that, for all sufficiently large ,

By the update rule of and ,

holds for infinitely many . We will prove that (47) contradicts to (46). Let be the index set containing all such that (47) holds. Therefore, for all ,

If there exists an infinite subset such that holds for all , then, by (48), it holds that

As , , and , (50) implies that

holds for all sufficiently large . If there exists an infinite subset such that holds for all , then by (48) we have that

for all . Equation (52) also implies (51) as and . From (28) and (29), it follows that holds for all . Therefore, by (49) we know that, for all ,

As , (53) implies that

holds for all sufficiently large . Thus from (47), we obtain (51) and (54) which contradict to (46). Thus we can complete the proof.

Theorem 8. Suppose that (A1)-(A2) hold. If and , then there exists one cluster point of such that is a KKT point of (1) or the RCPLD condition does not hold at .

Proof. Under the assumptions of this theorem, Lemma 7 implies that there exists an index set such that converge to some ,

where is defined in (7). With the help of Theorem  2 in Andreani et al. [21], (55) imply that is a KKT point or the RCPLD condition does not hold at .

3.2. The Case of Being Bounded

In this subsection, without loss of generality, we assume that for all . Thus by the update rule (29), we have that and

holds for all . As remains constant, it follows from (A1) and (A2) that and are all bounded. If we define the index set

then for .

Lemma 9. Suppose that (A1)-(A2) hold and for all . If , as , and there exists some constant such that

then is divergent.

Proof. We prove this lemma by contradiction. We will show that if is convergent, then holds for all sufficiently large which contradicts to the fact that , as .
Suppose that

and (32) imply that

holds for all sufficiently large . By the definition of ,

Equations (59)–(61) imply that is convergent. Let

It is clear that . By Taylor’s theorem, it holds that

where is a convex combination of and . According to (60), we have that

holds for all sufficiently large and thus

The convergence of and the boundedness of imply that . Therefore, for all sufficiently large . This implies that and .

Lemma 10. Suppose that (A1)-(A2) hold and for all ; then we have that

Proof. Firstly, we prove that the sum of is bounded. Define the indices set

where is defined by Steps 0 and 4 in Algorithm 2. From Step  4 of Algorithm 2, we know that if , then and . Hence we have

where is the upper bound of . From Step  4 and (67), we have

which implies

Then, we have

where is defined by (5).
Secondly, we prove

by contradiction. Suppose that there exists some such that

Equations (56) and (73) imply that

Considering the sum of on the index set (see (57)), we have by (71) that

It can be deduced by (74) and (75) that

and thus

If is a finite set, then it follows from (57) and Step  2 that and hold for all sufficiently large . Therefore, , as . If is an infinite set, the second inequality in (77) implies that , as and . From Step  2, we know that if , then . Hence, we have , as . The fact that and (74) imply that

holds for all sufficiently large . Hence it can be deduced by Lemma 9 that is divergent which contradicts the first part in (77).
Finally, we prove (66). If is a finite set, then is convergent. Thus, (72) implies (66). From now on we assume that is an infinite set. Suppose that (66) does not hold; then there exist an infinite index set () and a constant such that

By (72), there also exists an infinite index set () such that ,

Let ; then and is an infinite index set,

Therefore, by (75), we have that

With the help of (56), (80), and (83), we obtain that

A direct conclusion which can be drawn by (84) is

Thus by Lemma 4, we have that, for all sufficiently large ,

Therefore, for sufficiently large ,

Equations (84) and (87) imply that , as . Therefore (79) contradicts (81). Thus we complete the proof.

Lemma 11. Suppose that (A1)-(A2) hold and for all ; then we have

Proof. Suppose that (88) does not hold. Then there exists , such that

By (18), we have that

As is bounded above, similar to the second part in the proof of Lemma 10, we can conclude that

and thus

By (75) and (92), we have that is convergent and thus is also convergent as is bounded. However, Lemma 9, (91), (92), and the boundedness of deduce the divergence of . This contradiction completes the proof.

With the help of Lemmas 10 and 11, we can easily obtain the following result.

Theorem 12. Suppose that (A1)-(A2) hold and for all ; there exists an accumulation point of at which the KKT condition holds.

Note that, in Theorem 12, we do not suppose that RCPLD holds.

4. Numerical Experiment

In this Section, we investigate the performance of Algorithm 2. We compare Algorithm 2 with the famous Fortran package ALGENCAN. In our computer program, the parameters in Algorithm 2 are chosen as follows:

We set to be the exact Hessian of the Lagrangian at the point . The Matlab subroutine minres is used to solve (15). All algorithms are terminated when one of the following conditions holds: and ; and ; . All test problems are chosen from CUTEst collection [22].

The numerical results are listed in Table 1 where the name of problem is denoted by , the number of its variables is denoted by , the number of constraints is denoted by , the number of function evaluations is denoted by , and the number of gradient evaluations is denoted by . In Table 1, we list the results of 38 test problems. Considering the numbers of function evaluations (), Algorithm 2 is better than ALGENCAN for 30 cases (78.9%). Considering the numbers of gradient evaluations (), Algorithm 2 is better than ALGENCAN for 31 cases (81.6%).

Table 1: Results of Algorithm 2 and ALGENCAN.

Name
n
m
Algorithm 2
ALGENCAN

AIRCRFTA
8
5
5
5
20
21

ARGTRIG
200
200
69
69
15
16

BOOTH
2
2
6
6
5
6

BROWNALE
200
200
8
8
21
22

BROYDN3D
5000
5000
23
23
22
23

BYRDSPHR
3
2
42
21
80
78

BT1
2
1
17
17
67
45

BT2
3
1
40
37
74
71

BT4
3
2
17
17
34
34

BT5
3
1
12
11
28
28

BT7
5
3
5
5
97
98

BT8
5
2
19
15
76
51

BT9
4
2
18
15
106
100

BT10
2
2
15
15
95
95

BT11
5
3
15
15
75
75

BT12
5
3
10
10
20
21

CHNRSNBE
50
98
100
37
19
20

CLUSTER
2
2
11
11
26
27

CUBENE
2
2
8
6
17
18

DECONVNE
63
40
6
6
3982
1195

EIGENB
7
7
2550
2550
2451
608

GOTTFR
2
2
6
6
24
24

HATFLDF
3
3
5
5
55
45

HEART6
6
6
122
61
387
216

HEART8
8
8
29
10
87
43

HS39
4
2
65
61
106
100

HS48
5
2
46
19
6
7

HYDCAR6
29
29
267
85
212
154

INTEGREQ
502
500
6
6
14
15

MARATOS
2
1
13
11
42
43

MWRIGHT
5
3
14
14
29
29

ORTHREGB
27
6
6
6
23
24

RECIPE
3
3
6
6
39
40

RSNBRNE
2
2
6
6
16
17

SINVALNE
2
2
6
6
8
8

TRIGGER
7
6
481
290
757
270

YFITNE
3
17
11
11
805
520

ZANGWIL3
3
3
6
6
5
6

5. Conclusions

In this paper, we present a new algorithm for equality constrained optimization. We add an adaptive quadratic term to the quadratic model of the augmented Lagrangian function. In each iteration, we solve a simple unconstrained subproblem to obtain the trail step. The global convergence is established under reasonable assumptions.

From the numerical results and the theoretical analysis, we believe that the new algorithm can efficiently solve equality constrained optimization problems.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Acknowledgments

This work is supported by NSFC (11771210, 11471159, 11571169, and 61661136001) and the Natural Science Foundation of Jiangsu Province (BK20141409).

References

M. R. Hestenes, “Multiplier and gradient methods,” Journal of Optimization Theory and Applications, vol. 4, pp. 303–320, 1969.

M. J. D. Powell, “A method for nonlinear constraints in minimization problems,” in Optimization, R. Fletcher, Ed., pp. 283–298, Academic Press, New York, NY, USA, 1969.

A. R. Conn, N. I. Gould, and P. L. Toint, “A globally convergent augmented Lagrangian algorithm for optimization with general constraints and simple bounds,” SIAM Journal on Numerical Analysis, vol. 28, no. 2, pp. 545–572, 1991.

A. R. Conn, N. I. Gould, and P. L. Toint, ,LANCELOT: A Fortran Package for Large-scale Nonlinear Optimization(Release A), vol. 17 of Springer, New York, USA, 1992.

R. Andreani, E. G. Birgin, J. M. Martnez, and M. L. Schuverdt, “Augmented Lagrangian methods under the constant positive linear dependence constraint qualification,” Mathematical Programming, vol. 111, no. 1-2, Ser. B, pp. 5–32, 2008.

R. Andreani, E. G. Birgin, J. M. Martnez, and M. L. Schuverdt, “On augmented Lagrangian methods with general lower-level constraints,” SIAM Journal on Optimization, vol. 18, no. 4, pp. 1286–1309, 2007.

E. G. Birgin and J. M. Martnez, “Augmented Lagrangian method with nonmonotone penalty parameters for constrained optimization,” Computational Optimization and Applications, vol. 51, no. 3, pp. 941–965, 2012.

E. G. Birgin and J. M. Martnez, “On the application of an augmented Lagrangian algorithm to some portfolio problems,” EURO Journal on Computational Optimization, vol. 4, no. 1, pp. 79–92, 2016.

Z. Dostál, “Semi-monotonic inexact augmented Lagrangians for quadratic programming with equality constraints,” Optimization Methods & Software, vol. 20, no. 6, pp. 715–727, 2005.

Z. Dostál, A. Friedlander, and S. A. Santos, “Augmented Lagrangians with adaptive precision control for quadratic programming with simple bounds and equality constraints,” SIAM Journal on Optimization, vol. 13, no. 4, pp. 1120–1140, 2003.

F. E. Curtis, H. Jiang, and D. P. Robinson, “An adaptive augmented Lagrangian method for large-scale constrained optimization,” Mathematical Programming, vol. 152, no. 1-2, Ser. A, pp. 201–245, 2015.

F. E. Curtis, N. I. Gould, H. Jiang, and D. P. Robinson, “Adaptive augmented Lagrangian methods: algorithms and practical numerical experience,” Optimization Methods & Software, vol. 31, no. 1, pp. 157–186, 2016.

A. F. Izmailov, M. V. Solodov, and E. I. Uskov, “Global convergence of augmented Lagrangian methods applied to optimization problems with degenerate constraints, including problems with complementarity constraints,” SIAM Journal on Optimization, vol. 22, no. 4, pp. 1579–1606, 2012.

L. Niu and Y. Yuan, “A new trust-region algorithm for nonlinear constrained optimization,” Journal of Computational Mathematics, vol. 28, no. 1, pp. 72–86, 2010.

X. Wang and Y. Yuan, “An augmented Lagrangian trust region method for equality constrained optimization,” Optimization Methods & Software, vol. 30, no. 3, pp. 559–582, 2015.

C. Cartis, N. I. Gould, and L. Toint, “Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results,” Mathematical Programming, vol. 127, no. 2, Ser. A, pp. 245–295, 2011.

C. Cartis, N. I. Gould, and L. Toint, “Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity,” Mathematical Programming, vol. 130, no. 2, Ser. A, pp. 295–319, 2011.

K. Ueda and N. Yamashita, “A regularized Newton method without line search for unconstrained optimization,” Computational Optimization and Applications, vol. 59, no. 1-2, pp. 321–351, 2014.

H. Zhang and Q. Ni, “A new regularized quasi-Newton algorithm for unconstrained optimization,” Applied Mathematics and Computation, vol. 259, pp. 460–469, 2015.

Y. X. Yuan, “On the convergence of a new trust region algorithm,” Numerische Mathematik, vol. 70, no. 4, pp. 515–539, 1995.

R. Andreani, G. Haeser, M. a. Schuverdt, and P. J. Silva, “A relaxed constant positive linear dependence constraint qualification and applications,” Mathematical Programming, vol. 135, no. 1-2, Ser. A, pp. 255–273, 2012.

N. I. M. Gould, D. Orban, and P. L. Toint, “CUTEst: a Constrained and Unconstrained Testing Environment with safe threads for mathematical optimization,” Computational Optimization and Applications, vol. 60, no. 3, pp. 545–557, 2015.