A Novel Approach for Solving Semidefinite Programs

A novel linearizing alternating direction augmented Lagrangian approach is proposed for effectively solving semidefinite programs (SDP). For every iteration, by fixing the other variables, the proposed approach alternatively optimizes the dual variables and the dual slack variables; then the primal variables, that is, Lagrangemultipliers, are updated. In addition, the proposed approach renews all the variables in closed forms without solving any system of linear equations. Global convergence of the proposed approach is proved under mild conditions, and two numerical problems are given to demonstrate the effectiveness of the presented approach.


Introduction
Minimizing a linear function of a symmetric positive semidefinite matrix subject to linear equality constraints is called semidefinite programs (SDP), whose form can be given as follows: min ⟨, ⟩ s.t.A () = ,  ⪰ 0, where A : S  → R  is a linear operator, which can be expressed as A () := (⟨ 1 , ⟩ , . . ., ⟨  , ⟩)  . ( ,   ∈ S  ,  = 1, . . ., , are all matrices, and  ∈ R  is a vector, and  ⪰ 0 stands for the fact that  is a symmetric positive semidefinite matrix.Here, S  stands for the space of  ×  symmetric matrices and ⟨, ⟩ = trace() stands for the standard inner product in S  .A * () := ∑  =1     stands for the adjoint operator A * : R  → S  of A. The dual problem of (1) is min −   s.t.A * () +  = ,  ⪰ 0, where  ∈ R  and  ∈ S  .
SDP problem has been always a very active area in optimization research for many years.It has broad applications in many areas, for example, system and control theory [1], combinatorial optimization [2], nonconvex quadratic programs [3], and matrix completion problems [4].We refer to the reference book [5] for theory and applications of SDP.Interior point approaches (IPMs) have been very successful for solving SDP in polynomial time [6][7][8][9].For small and medium sized SDP problems such as  ≤ 1, 000 and  ≤ 10, 000, IPMs are generally efficient and robust.However, for large-scale SDP problems with large  and moderate , IPMs become very slow due to the need of computing and factorizing the  ×  Schur complement matrix.In order to improve this shortcoming, by using an iterative solver to compute a search direction at each iteration, [10,11] proposed inexact IPMs which manage to solve certain types of SDP problems with  up to 125,000.Based on the augmented Lagrangian approach, many variants for SDP were proposed.For example, [12] introduced the so-called boundary point approach; using an eigenvalue decomposition to maintain complementarity, [13] presented a dual augmented Lagrangian approach.More recently, Huang and Xu [14] proposed a trust region algorithm for SDP problems by performing a number of conjugate gradient iterations to solve the subproblems.Zhao et al. [15] designed a Newton-CG augmented Lagrangian approach for solving SDP problems from the perspective of approximate experimental results demonstrate that the performance of the proposed approach can be significantly better than that reported in [16].
The remaining section of this paper is described as follows.In Section 2 a novel linearizing alternating direction augmented Lagrangian approach is proposed for solving SDP problems.The convergence of the proposed approach is proved in Section 3. In Section 4, some implementation issues of the proposed approach are discussed.Two numerical examples for frequency assignment problem and binary integer quadratic programs problems are used to demonstrate the performance of the proposed approach in Section 5.
Some notations: S  + represents the set of  ×  symmetric positive semidefinite matrices. ≻ 0 represents the fact that  is positive definite.The notation ‖ ⋅ ‖ stands for the Euclidean norm and ‖ ⋅ ‖  stands for the Frobenius norm.vec() denotes a vector obtained by stacking 's columns one by one. denotes the identity matrix in proper order.

Linearizing Alternating Direction Augmented Lagrangian Approach
In this section, a linearizing alternating direction augmented Lagrangian approach is proposed for solving (1) and (3).
Without loss of generality, we assume that matrix  is a full row rank and there exists a matrix X ≻ 0 such that A( X) = .It is well known that, with the above assumption, a point (, , ) is optimal for SDP problems (1) and (3) if and only if Given a penalty parameter  > 0, the augmented Lagrangian function for the dual SDP (3) is defined as where  ∈ S  .For given  0 ,  0 ∈ S  , the alternating direction augmented Lagrangian approach for solving problems (1) and (3) generates sequences {  } ⊂ R  , {  } ⊂ S  , and {  } ⊂ S  as follows: Apparently, we can obtain  +1 by solving the first-order optimality conditions for (6), which is a system of linear equations associated with   .Since   is a  ×  matrix, it is difficult to get  +1 exactly when  is large.In order to alleviate this difficulty, we use the quadratic approximation of   (  , ,   ) in (6) around   as follows: where   > 0 and We replace step (6) by Then, we have As pointed out in [16], problem ( 7) is equivalent to min where  +1 =  − A * ( +1 ) −   .Denote the spectral decomposition of the matrix  +1 by where Σ + and Σ − are the nonnegative and negative eigenvalues of  +1 .We then obtain the fact that where  +1 − = − 2 Σ −  2 .Now we present the linearizing alternating direction augmented Lagrangian approach in Algorithm 1.
Remark 1.We can choose  +1 = ‖  ‖  to satisfy the condition of Algorithm 1.If AA * =  and   = 1 for all  ≥ 0, then Algorithm 1 is same as the approach proposed in [16].

The Convergence of the Proposed Approach
In this section, we prove the convergence of Algorithm 1 using the argument similar to the one in [34].Let    = A * (  ) +   − ; then we have the following proposition.Lemma 2. Let   = (  ,   ,   ) be generated by Algorithm 1 and let  * = ( * ,  * ,  * ) be an optimal solution of (1) and (3); then one has where Proof.From ( 8), there holds By ( 12), we know that That is, By substituting (8) into the above equality, using the fact A( * ) = , and rearranging the terms, one has Since  +1  +1 = 0, we have By substituting  =  * into (22), we get Initialize  0 ∈ R  ,  0 ⪰ 0, and  0 ⪰ 0. Choose initial step size  0 greater than the maximum eigenvalue of the matrix   .For  = 0, 1, . . .do Compute   and  +1 =   − /    .Compute  +1 and its eigenvalue decomposition, and set Choose  +1 greater than the maximum eigenvalue of the matrix   .end Algorithm 1: Linearizing alternating direction augmented Lagrangian algorithm for SDP.
By adding ( 18), (21), and ( 23) together, we obtain where the last inequality comes from ( 8) and (22).Note that   ,  +1 , and  * are semidefinite positive matrices; then It follows (25) that By (26) and the fact that we have which completes the proof.
Proof.By ( 16), we know that the sequence {  } is bounded and the sequence where  = (, , ) can be any limit point of {  }.It follows that lim Since   are greater than the maximum eigenvalue of matrix   , then the matrix   is positive definite.By the definition of   , we obtain lim From the update formula (8), we have lim By (12) and the definition of   , one has lim which together with (32) imply that lim By combining (32), (34),     = 0, and   ,   ⪰ 0 for all  ≥ 1, we know that any limit point of {  }, say  = (, , ), satisfies which means  is a solution of problems ( 1) and (3).By Lemma 2, {(  ,   ,   )} converges to a solution of problems ( 1) and (3).

Implementation Issues
The proposed algorithm is carried out by modifying the code of the alternating direction approach in [16] which is referred to as SDPAD.Before presenting the numerical results, we discuss some implementation issues of Algorithm 1 in this section.
In order to improve the computational performance of Algorithm 1, using the similar method as many alternating direction approaches [35][36][37], we replace step (8) by We can use an argument similar method to the one in [34] to prove the following theorem.

Numerical Results
In this section, we report our numerical results.We compare solutions obtained from Algorithm 1 and SDPAD on the SDP relaxations of frequency assignment problems and binary integer quadratic programs problems.All the procedures were carried out by MATLAB 2011b on a 3.10 GHz Core i5 PC with 4 GB of RAM under Windows 7.
In Tables 1 and 2, the first column gives the problem name; some notations have been also used in column headers, : the size of the matrix ; : the total number of equality and inequality constraints; "itr": the number of iterations; "cpu": the CPU time in the format of "hours, minutes, and seconds."
We did not run SDPAD on our own computer on the problem "fap36" and the results presented here were taken from Table 1 in [16].From Table 1, it can be observed that Algorithm 1 is often faster than SDPAD for achieving a duality gap of the same order.The infeasibility achieved by Algorithm 1 is satisfactory as well.

Binary Integer Quadratic Programs Problem.
In this subsection, we present numerical results of Algorithm 1 and SDPAD on binary integer quadratic (BIQ) problems [41] through SDP relaxations which have the following form: where  ∈ R (−1)×(−1) .The   , = were replaced by √2/3(  −  , ) 0 and the matrix  was scaled by its Frobenious norm.We set ℎ to 50 for updating the penalty parameter .Table 2 lists the results of Algorithm 1 and SDPAD on the BIQ problems.By comparing the results in Table 2, we can conclude that Algorithm 1 applied to BIQ problems is superior to SDPAD in terms of CPU time and number of iterations.In addition, the accuracy of the approximate optimal solutions computed by Algorithm 1 is as good as that obtained by SDPAD.
Figure 1 shows the performance profiles [42] of Algorithm 1 and SDPAD for the number of iterations, Figure 1(a), and CPU time, Figure 1(b).We observe that Algorithm 1 is better than SDPAD in terms of number of iterations and CPU time.

Conclusion
In this paper, a novel linearizing alternating direction augmented Lagrangian approach is proposed for solving semidefinite programs (SDP).The algorithm updates the dual variables without solving any system of linear equations.Moreover, all the variables are updated in closed forms.Preliminary numerical results show the efficiency of the proposed algorithm.However, there are still some unsettled issues for implementation.For example, efficient strategies to update penalty parameter  and choose step size   deserve more work for applications of the algorithm.

Figure 1 :
Figure 1: Performance profiles for SDPAD and the present method for number of iterations (a) and CPU time (b).