A Singular Value Thresholding with Diagonal-Update Algorithm for Low-Rank Matrix Completion

The singular value thresholding (SVT) algorithm plays an important role in the well-known matrix reconstruction problem, and it has many applications in computer vision and recommendation systems. In this paper, an SVT with diagonal-update (D-SVT) algorithm was put forward, which allows the algorithm to make use of simple arithmetic operation and keep the computational cost of each iteration low. The low-rank matrix would be reconstructed well. The convergence of the new algorithm was discussed in detail. Finally, the numerical experiments show the eﬀectiveness of the new algorithm for low-rank matrix completion.


Introduction
e problem of completing low-rank and sparse matrices from some of its observed entries occurs frequently in many areas of engineering and applied science such as machine learning [1,2], model reduction [3], compressed sensing [4], control [5], pattern recognition [6], signal and imaging inpainting [7][8][9][10], and computer vision [11]. From the pioneering work on low-rank approximation by Fazel [12] as well as on matrix completion by Candès and Recht [13], there has been a lot of study (see  and references therein) both from theoretical and algorithmic aspects on the problem of recovering a low-rank matrix from partial entries, also known as matrix completion. ere is a rapidly growing interest for this issue. Explicitly seeking the lowest rank matrix consistent with the known entries is mathematically expressed as min X∈R m×n rank(X), where the matrix M ∈ R m×n is the underlying matrix to be reconstructed and p Ω is the associated sampling orthogonal projection operator which acquires only the entries indexed by Ω ⊂ 1, 2, . . . , m { } × 1, 2, . . . , n { } with Ω is a random subset of indices for the known entries. e general problem (1), however, is nonconvex and is NP hard [14] due to the rank objective. ere are a few of algorithms for solving this model directly. Alternatively, Candès and Rechat [13] switched (1) to another simple convex optimization problem (2) as follows: where the nuclear norm ‖X‖ * is the sum of all singular values of the matrix X. Furthermore, it has proved that the sequence X k converges to the unique solution of the following optimization problem closely related to (2) in [15]: s.t. p Ω (X) � p Ω (M). (3) As for the solution of problems (2) and (3), there have been many computational efficient algorithms which are designed for a broad class of matrices such as mainly the accelerated proximal gradient (APG) algorithm [16], the augmented Lagrange multiplier (ALM) algorithm [17], several methods [18][19][20][21] resulted in alternating optimization based on the bilinear factorization M � XY with X ∈ R m×r , Y ∈ R r×n and rank (M) � r, and the singular value theresholding (SVT) algorithm, as well as its improvements [15,[22][23][24]. However, the computations of partial singular value decomposition (SVD) were required at each iteration in the most direct implementation of these algorithms. e computational cost of computing the SVD has complexity of O(n 3 ) when the rank r and matrix-size n are proportional, resulting in computing the SVD to be the dominant computational cost at each iteration and then limits their applicability for large n. In view of its outstanding performance and elegant mathematical properties, the SVT algorithm obtains widespread attention [25][26][27]. e variants and extended applications of the SVT algorithm have been studied later: Candès et al. [28] presented a unbiased risk estimate formula of the SVT for the noisy observations; Chatterjee [29] studied a general method for matrix denoising using SVT, which covers the stochastic block model as a special case; Donoho and Gavish [30] pointed out several ways that these matrix denoising results for singular value soft thresholding (SVST) estimation of low-rank matrices parallel results for soft thresholding of sparse vectors; Dutta et al. [31] proposed an alternative solution to the sensitivity of the classical principal component analysis (PCAs) to the outliers for solving the problem min X∈R m×n with the nonsingular weight matrix W ∈ R n×m which is user provided or automatically inferred from the data, called as WSVT problem; Klopp [32] introduced a variant of the SVT iteration; Ma and Xu [33] recovered the received signal strength (RSS) reading and achieve good localization performance based on the SVT theory; Zhang et al. [34] put forward a lower bound guaranteeing exact matrix completion via the SVT algorithm; Zhang et al. [35] have considered the low-rank tensor completion problem through a hybrid singular value thresholding scheme. is paper develops a modification of the SVT algorithm for approximately solving the nuclear norm minimization problem (2). By using a diagonal-update technique for the approximated sequence at each step, the iteration matrices generated by the new algorithm are approximated to the true solution well, which saves significant computational cost of the SVT algorithm performance. And we also establish the convergence theory in detail. Experimental results show that the new algorithm outperforms the standard SVT and its several variants algorithms, especially when problem (2) comes to large-scale issues. e rest of the paper is organized as follows. After we provide some notations in this section, we review briefly the standard SVT algorithm, the accelerated singular value thresholding (ASVT) algorithm as well as its modification, and a modified algorithm of the SVT algorithm with diagonal-update (D-SVT) is proposed in Section 2. In Section 3, the convergence theory of the new algorithm is established. en, numerical experiments are shown and compared in Section 4. Finally, we end the paper with the concluding remarks in Section 5.
Here are some notations. R m×n denotes m × n real matrices set, R m×n + represents the m × n nonnegative and real matrices' set. e nuclear norm ‖X‖ * of a matrix X is defined by ‖X‖ * � r k�1 σ k (X), σ k (X) denotes the kth largest singular value of the real matrix X ∈ R m×n of rank r, and the Frobenius norm } is the indices of the observed entries, and Ω is the complementary set of Ω. p Ω is the orthogonal projector onto Ω, satisfying,

Related Algorithms
In order to completing comparison subsequently, we briefly review and introduce some algorithms for solving the matrix completion problem (2).

e Standard Singular Value resholding (SVT) Algorithm
e singular value decomposition (SVD) of a matrix X ∈ R m×n of rank r is where U ∈ R m×r and V ∈ R n×r are two orthogonal matrices, σ 1 ≥ σ 2 ≥ · · · ≥ σ r > 0.

Remark 1.
Due to the ability of producing low-rank solutions with the soft-thresholding operator, the SVT algorithm was shown to be extremely efficient at addressing problems with low-rank optimal solutions such as recommender systems.
where Y is the Lagrangian variable.
In terms of the dual approach, f(Y) � inf X L(X, Y) is the dual function of L(X, Y), which is concave. Define which is convex. us, we can solve problem (3) by firstly minimizing the objective function h(Y), namely, Problem (10) was computed via Nesterov's method with an adaptive line search scheme. en, Algorithm 2 has been provided.
Based on the above, the accelerated singular value thresholding (ASVT) algorithm has been proposed in [22]. Furthermore, Wang et at. [23] presented the Ne-SVT by replacing the adaptive linear search with Nemirovski's technique and the M-ASVT algorithms by using the same search technique in the ASVT algorithm. e overall steps of the later can be organized as Algorithm 3.
It is reported that M-ASVT needs much fewer iterations than ASVT algorithm under the same level of accuracy and the same cost of computing.

e Singular Value resholding with Diagonal-Update
We are now in the position to introduce a modified singular value thresholding algorithm by using a diagonal-update technique, as shown in Algorithm 4.
Set D: � p Ω (X k ) for short. e difference with the standard SVT algorithm may seem at k th step, replacing the iteration matrix X k with the diagonal-update DW k of its Equation (11) is easy to compute since it is so simple just some arithmetic operation required, without extra cost. In fact, the exact solution of (11) is given by Remark 2. e sequence matrices generated by the new algorithm are approximated to the true solution well, which saves significant computational cost of the SVT algorithm performance, without actually extra complexity. It is designed as Algorithm 4 by plugging some steps into the SVT method. It should be seen that Algorithm 4 has three lines (as shown in lines 10-12) more than Algorithm 1. e new algorithm includes Algorithm 1 as special case when W � I.

Convergence Analysis
In this section, the convergence theory is discussed for the singular value thresholding which involves diagonal-update algorithm.
end if (11) end while (12) Set t k � (1 + ������� 1 + 4t 2 k− 1 )/2 (13) end for ALGORITHM 3: e modified ASVT (m-asvt) algorithm, algorithm 1 of [23] 4 Mathematical Problems in Engineering In term of the assumption of this theorem, ∀k, ∃Θ such that . We note that X k � p Ω (X k )W k from Algorithm 4, and by substituting the following inequalities we have from the equation Moreover, since lim k⟶∞ W k � I. e theorem has been proved. □ Theorem 2. Let X †,τ be the limiting point of the sequence X k generated by Algorithm 4. en, X †,τ is the solution of the optimization problem (3).
Proof. It is obtained that X k is the optimal solution of (8) for that value of Y k− 1 from Algorithm 4. Hence, for any feasible matrix X, it is yielded that us, X †,τ is the unique solution of (3).
Proof. Note that X † is the optimal solution of the optimization problem (2) if and only if From eorem 2, we have which implies that erefore, since X † is bounded. us,

Numerical Experiments
In this section, we provide the performance of our D-SVT algorithm in comparison with the SVT, ASVT, and M-ASVT algorithms mentioned in Section 2 and report the running time in seconds (denoted by "time (s)"), the numbers of iterations (denoted by "IT") it takes to reach convergence, and the relative errors of the reconstruction (denoted by" Error 1 and Error 2″) as follows: All the experiments are conducted on the same workstation with an Intel (R) Core (TM) i7-6700 CPU @3.40GHZ that has 16 GB memory and 64 bit operating system, running Windows 7, and Matlab (vision 2016a). For conciseness, the tests presented consider square matrices as is typical in the study. at is to say, suppose to simplify that the unknown matrix M ∈ R n×n is square and that one has available s sampled entries M ij : (i, j) ∈ Ω , where Ω is a random subset of cardinality s. By the way, the iteration fails if the number of iterations is up to 1000.
In our implement, we generate n × n matrices of rank r by sampling uniformly at random among all sets of cardinality s; then, p � s/n 2 denotes the observation ratio. Let ε � 2 × 10 − 4 . As discussed earlier [15], δ � 1.2 p − 1 and the step sizes are constant and the parameter τ is chosen empirically. And as presented earlier [23], the parameters L 1 � 1 and η � 0.8. As for D-SVT algorithm, we choose δ � 1.68 p − 1 , and τ is the same as the SVT algorithm. e tested matrix dimensions (denoted by "size (n)") are from 1,000 to 12,000. e experimental results are shown in Tables 1-6. Our experiments suggest that Algorithm 4 is fast and significantly outperforms the other algorithms in terms of both number of iteration steps and computing time. e     Mathematical Problems in Engineering 9 Note. e mark "-" indicates that the iteration is failing.  Note. e mark "-"indicates that the iteration is failing.  new algorithm is especially well suited for problems of very large sizes.
In order to show convergence behave of the algorithms briefly, convergence curves of several algorithms are clearly given, which are shown in Figure 1 for the different parameters. It is easy to see that our algorithm takes much less computational cost from iteration steps and computing time. at is to say, the D-SVT algorithm is more efficient than the other algorithms especially when the size of matrix M is large.

Concluding Remarks
In this paper, we focus on the problem of completing a lowrank matrix from a small subset of its entries. is model can characterize many applications arising from the areas of signal and image processing, statistical inference, and machine learning. We have proposed a modification of the SVT algorithm for solving the low-rank matrix sparse model. e key step of the algorithm is to update each iteration matrix by a weighting diagonal matrix, without the extra cost. e weighting matrix W was determined adaptively in iteration process. is algorithm is easy to implement and surprisingly effective both in terms of computational cost and storage requirement. Consequently, the matrix would be completed well.

Data Availability
We generate a matrice of rank r by sampling randomly in our implement. e readers can access the data supporting the conclusions of the study by MATLAB codes.

Conflicts of Interest
e authors declare that they have no conflicts of interests.