A Coordinate Descent Method for Total Variation Minimization

Total variation (TV) is a well-known image model with extensive applications in various images and vision tasks, for example, denoising, deblurring, superresolution, inpainting, and compressed sensing. In this paper, we systematically study the coordinate descent (CoD) method for solving general total variation (TV) minimization problems. Based on multidirectional gradients representation, the proposed CoD method provides a unified solution for both anisotropic and isotropic TV-based denoising (CoDenoise). With sequential sweeping and small random perturbations, CoDenoise is efficient in denoising and empirically converges to optimal solution. Moreover, CoDenoise also delivers new perspective on understanding recursive weighted median filtering. By incorporating with the Augmented Lagrangian Method (ALM), CoD was further extended to TV-based image deblurring (ALMCD).The results on denoising and deblurring validate the efficiency and effectiveness of the CoD-based methods.


Introduction
Total variation (TV), also known as the ROF model [1], was introduced by Rudin et al.The TV model is effective in preserving sharp and salient edges while suppressing noise and has been extensively adopted as a regularizer in various image restoration applications, for example, deblurring [2,3], superresolution [4,5], inpainting [6,7], and compressed sensing [8,9].
Recently, other image models, such as dictionary-based sparse coding [10][11][12] and nonlocal similarity [13][14][15][16][17], have been developed.Compared with these models, TV is much more efficient to be solved, making TV-based methods remain active in image and vision studies [17][18][19][20][21][22][23][24].Moreover, TV may be complementary with the other models, and thus proper combination of them can lead to better performance [25,26].Besides, extensions of TV regularizer were also studied.For color images, TV can be extended to a class of vectorial TV (VTV) [27,28], where interchannel correlation is taken into account to reduce the uneven color effects.While TV only considers first-order gradients, Total Generalized Variation (TGV) [29] was proposed to involve higher-order derivatives.For structure extraction, relative TV [30] was employed to distinguish structure from textures.Considering that the gradient distribution of each pixel is actually spatially variant, nonlocal extension of TV model [17,31] was presented to leverage the similar patches for adaptive distribution estimation.
A basic TV minimization problem is TV-based image denoising formulated as where TV(x) is the TV regularizer,  is the trade-off parameter, and x and y are the latent clear image and the noisy observation, respectively.Various methods to solve TV denoising problem had been proposed and can be roughly categorized from three directions, that is, gradient based, Markov Random Fields (MRF) based, and CoD-based methods.First, Our contribution can be summarized from two aspects: (i) We systematically study the CoD-based methods for TV minimization and develop an extremely simple unified CoD-based solution for both anisotropic and isotropic TV minimization.The resulting CoDenoise algorithm is more efficient than gradient based and MRF based methods and achieves satisfactory denoising results.
(ii) By incorporating with ALM, CoDenoise is extended to image deblurring problem.In the deblurring problems, the blurry images usually suffer from severe blur and relatively low level noises, and thus the proposed ALMCD algorithm with CoDenoise embedded for denoising subproblem is much more efficient and can concurrently provide satisfactory deblurring quality compared with several state-of-the-art methods.
This paper is organized as follows: Section 2 presents some preliminaries, including definition of TV regularizers and multidirectional gradient approximation of TV regularizers.The CoDenoise algorithm together with its convergence proof and computational complexity is proposed in Section 3. In Section 4, we embed CoDenoise to image deblurring.Section 5 demonstrates experimental results, and Section 6 ends this paper with some concluding remarks.

Preliminaries
In this section, we first present the definitions of the discrete anisotropic and isotropic TV operators.In previous studies, CoD-based solution is only available for anisotropic TV minimization problem.To address this, we then introduce the multidirectional gradient representation to establish the connection between the anisotropic and isotropic TV models, making it possible to use the unified CoD method for TV minimization.
2.1.The Discrete TV Operators.For an image x with  =  ×  pixels, the discrete gradient operators D including both horizontal gradient operator D ℎ and vertical gradient operator D V are defined as where  = 1, 2, . . .,  and  = 1, 2, . . ., .The anisotropic TV regularizer [50,60] is defined as With this definition, it is easy to obtain the anisotropic TV regularization with respect to coordinate as Thus, the CoD method can be directly used to solve the anisotropic TV minimization problem.Similarly, the isotropic TV regularizer [50,60] is defined as Apparently the isotropic TV cannot be decomposed with respect to coordinate (, ) since the quadratic interactions with horizontal and vertical gradients, making the CoD method unfeasible to solve isotropic TV minimization problem.Therefore, to extend the results of CoD to isotropic TV minimization problem, we tempt to find a connection between TV  and TV  .

Multidirectional Gradients Approximation.
The isotropic TV regularizer can be approximated by multidirectional gradients representation, and thus the anisotropic and isotropic TV models can be connected in a unified formulation [61].
For any pair of real numbers  and , the identity where   = (∑  =1 (cos   + sin   )) −1 and d ℎ = D ℎ x and d V = D V x.Equation ( 8) provides a unified formulation of anisotropic and isotropic TV models, In later context, we will use TV  to represent the TV regularizers.

The Unified Coordinate Descent Method for TV-Based Denoising
With TV  regularizer, anisotropic and isotropic TV denoising models are reformulated in the unified form which is exactly anisotropic TV-based denoising when  = 1 and infinitely approximates isotropic TV-based denoising when  increases.We thus can decompose the objective function into a sequence of one-dimensional subproblems, which can be solved efficiently via simple convex optimization.With simple sequential updating pattern, we then obtain the unified CoD denoising algorithm for both anisotropic and isotropic TV minimization.
where   = cos   and   = sin   .Vectors a and c are both of length 4 × , which are the coefficients of  , and the combinations of its 4 neighbourhoods, respectively.

CoDenoise.
Therefore, the subproblem with respect to  , (11) can be solved by The following question is how to choose coordinate updating pattern.Li and Osher adopted the checkerboard pattern [55], in which the pixels are divided into black and white blocks.The pixels in the same group are not neighbors, and then the pixels in two blocks can be alternatively updated.Another greedy strategy is also popular [62], in which the selected coordinate makes the biggest contribution to the decrease of the energy function.And by the divide and conquer strategy, the corresponding coordinate can be searched with complexity (log ) [63].
The proposed CoDenoise algorithm adopted the simple cyclic updating pattern, sequentially sweeping each pixel.If the computed solution at new selected coordinate makes a big progress than that in last iteration (evaluated by a tolerance  0 ), then it will be updated.In our implementation, we use a binary mask matrix M to indicate whether a pixel will be updated or not.If any four neighbor of pixel (, ) is updated, M(, ) is marked as 1, and the pixel (, ) will be updated in the next iteration, otherwise 0. For the nondifferentiability of TV norm, the solution generated by CoDenoise may get stuck at nonstationary points, which can be easily bypassed by adding small random perturbations.The perturbations decrease along with the increasing iteration number.
To stop the CoDenoise algorithm, we check whether the relative difference between two iterations is below tolerance The CoDenoise algorithm is summarized as Algorithm 1.

Convergence and Complexity.
We first discuss the convergence of the CoDenoise algorithm and then analyze its computational complexity.

Computational Complexity.
First, we present the analysis of computational complexity of the operator P. The operation with the heaviest computational cost is to sort vector which can be done by existing sorting algorithms, for example, max-heap sort, and thus the sorting the vectors in ( 11) can be done with computational complexity ( log ).Then, the optimal solution can be searched in () at worst.And thus, the complexity of proximal operator P at worst is ( 2 log ).
Then, CoDenoise requires calling operator P   times in each iteration, where   is the number of nonzero entries of mask matrix M, proportional to the noise level, and thus the computational complexity of CoDenoise is (   2 log ).
With the equivalence of proposed operator (20) and finding median value (28), the CoDenoise algorithm provides an interesting interpretation of the recursive weighted median operations on noisy image.By the recently great progress of studies on median filter or reweighted filter [58,59], the computational efficiency of the proposed CoDenoise algorithm should be further improved.

CoD for TV-Based Image Deblurring
TV  based image deblurring problem is formulated as where A is convolution matrix, which is an ill-posed problem.By combining variable splitting and ALM, we employ ADMM to solve this problem.First, by introducing an auxiliary variable u, the TV based image deblurring problem is reformulated as Then the augmented Lagrangian function of ( 33) is where  is a positive penalty parameter and q is related to Lagrangian vector.Then, the two variables u and x can be updated alternatively until some convergence criterion is satisfied.Given x, the u-subproblem can be efficiently solved in Fourier domain, where F and F −1 are Fourier and inverse Fourier transformation, respectively.Given u, the x-subproblem can be directly solved by the proposed CoDenoise algorithm.Finally, the parameters q and  are updated.The overall algorithm is summarized as Algorithm 2.

Experimental Results
In this section, we report the experimental results on image denoising and deblurring to validate the proposed CoDbased methods.First, as to the image denoising, CoDenoise is  compared with three Chambolle's works, that is, fixed points (CFP) algorithm on dual variables [43], first-order primal dual (CPD) algorithm [46], and graph cut-(CGC-) based algorithm [53].Then, we compare ALMCD with several stateof-the-art deblurring algorithms with the denoising subproblem embedded, including accelerated IST algorithms, that is, TwIST [49] and FISTA [50], and ALM-based algorithm, that is, SALSA [51].All the experiments ran on a 2.40 GHz Core(TM) i7-4700MQ processor.The CoDenoise algorithm is coded in C/C++, and ALMCD is coded in Matlab.We provide Matlab wrapper of CoDenoise which can be called by ALMCD.And for the parameter settings, the updating tolerances  and  0 are both set as 10 −3 .We set  = 1 for anisotropic TV minimization and  = 3 for isotropic TV minimization.Corresponding to each noise level, the trade-off parameter was chosen as 0.04, 0.09, 0.16, and 0.23, respectively, with best PSNR values.

Image
For the convexity of TV minimization problem, CFD and CPD are proved to converge to global optimal solution.From the PSNR and SSIM comparison of different algorithms, shown in Tables 2 and 3, CoDenoise can also converge to the same solutions with CFD and CPD for each noise level.As to the computational efficiency shown in Table 1, for anisotropic TV denoising CoDenoise is much faster than all the competing algorithms, especially for low level noises, and for isotropic TV denoising CoDenoise has to handle more extra entries, thus being little computational expensive than CFP and comparable to CPD. Figure 2 delivers the visual denoising effect of different algorithms, and CoDenoise can achieve satisfactory denoising results.for all the four algorithms.Since blurry images usually suffer from severe blur and low level noise, the subproblem involved should be more efficiently solved by CoDenoise.Table 4 presents the deblurring results comparison for anisotropic TV deblurring, and one can see that ALMCD is significantly faster than all the competing algorithms.Even though CoDenoise is slower than CFP for isotropic TV minimization, ALMCD is instead more efficient than the competing algorithms, shown in Table 5. Particularly, SALSA adopted the same variable splitting strategy with ALMCD, generating the same subproblems, so the efficiency superiority of ALMCD over SALSA heavily confirms that CoD-based method contributes more to efficiency improvement of ALMCD.In terms of deblurring quality, both PSNR and SSIM for anisotropic and isotropic TV models, shown in Tables 4 and 5, achieved by ALMCD are comparable to all the other competing algorithms.Figure 3 presents the visual deblurring results of Boat, from which one can see that the ALMCD can obtain visually plausible deblurring results.As a summary, CoD-based methods can provide comparable solutions compared with competing algorithms, while CoDenoise for anisotropic TV model is much more efficient than all the competing denoising algorithms, and ALMCD with CoDenoise embedded to solve denoising subproblem is much faster than state-of-the-art deblurring algorithms for both anisotropic and isotropic TV models.

Conclusion
In this paper, we propose a novel unified solution based on CoD method to solve TV minimization problems.With the unified formulation, both anisotropic and isotropic TV Mathematical Problems in Engineering
Denoising.As to the competing denoising algorithms, CFP and CPD are both only designed for isotropic TV minimization, and by modifying the projection step, CFP and CPD are easily applied to anisotropic TV minimization.Since only anisotropic TV model can be mapped to binary MRF, CGC is only feasible to anisotropic TV minimization.The denoising experiments were conducted on four 512 × 512 images, that is, Couple, Man, Hill, and Boat, shown in Figure 1, which were degraded by different Gaussian noise levels, with standard deviation (std.) as 0.05, 0.10, 0.15, and 0.20.

Figure 2 :
Figure 2: Denoising results comparison for isotropic TV model.The Gaussian noise is with std.0.15, and the two values in each bracket are PSNR and SSIM, respectively.

Figure 3 :
Figure 3: Deblurring results for anisotropic TV model.The two values in each bracket are PSNR and SSIM, respectively.

Table 1 :
CPU time (seconds) comparisons of different noise levels.

Table 2 :
PSNR comparisons of different noise levels.

Table 3 :
SSIM comparisons of different noise levels.

Table 4 :
Deblurring results comparison for anisotropic TV model.T stands for TwIST, F stands for FISTA, S stands for SALSA, and A stands for ALMCD.The proposed ALMCD algorithm is compared with TwIST, FISTA, and SALSA, where CFP is adopted to solve the involved denoising subproblem.In the experiments, the test images were degraded with Gaussian kernel with std.7 and Gaussian noise with zero mean value and std. 1 × 10 −3 .The trade-off parameter  is set as 5 × 10 −5

Table 5 :
Deblurring results comparison for isotropic TV model.T stands for TwIST, F stands for FISTA, S stands for SALSA, and A stands for ALMCD.decomposed into scalar problems that can be efficiently solved by convex optimization.With simple cyclic updating pattern and random perturbations, CoDenoise can empirically converge to the optimal solution.Also when applied in image deblurring, the CoDenoise algorithm embedded in ALMCD makes significant contributions in terms of efficiency, compared with competing deblurring algorithms.In terms of deblurring quality, ALMCD can provide comparable or superior results, validating the effectiveness of CoD-based methods.Furthermore, with the great improvements in weighted median filter or parallel implementation, the CoDenoise algorithm should be much more efficient.Also, CoD could be extended to other TV variants, for example, nonlocal TV and vectorial TV.