Adaptive Sparse Norm and Nonlocal Total Variation Methods for Image Smoothing

In computer vision and graphics, it is challenging to decompose various texture/structure patterns from input images. It is well recognized that how edges are defined and how this prior information guides smoothing are two keys in determining the quality of image smoothing. While many different approaches have been reported in the literature, sparse norm and nonlocal schemes are two promising tools. In this study, by integrating a texturemeasure as the spatially varying data-fidelity/smooth-penalty weight into the sparse norm and nonlocal total variationmodels, two newmethods are presented for feature/structure-preserving filtering.The first one is a generalized relative total variation (i.e., GRTV) method, which improves the contrast-preserving and edge stiffnessenhancing capabilities of the RTV by extending the range of the penalty function’s norm from 1 to [0, 1].The other one is a nonlocal version of generalized RTV (i.e., NLGRTV) for which the key idea is to use a modified texture-measure as spatially varying penalty weight and to replace the local candidate pixels with the nonlocal set in the smooth-penalty term. It is shown that NLGRTV substantially improves the performance of decomposition for regions with faint pixel-boundary.


Introduction
Many practical applications of computer vision and computer graphics involve estimating spatially varying image contents from noisy original images.One important requirement of such estimation is feature-preserving filtering, which has become a fundamental tool in many applications.Unfortunately, feature-preserving filtering is inherently challenging because it is usually difficult to distinguish useful features from noise.There are a number of different featurepreserving filtering methods, which can be roughly classified into two categories, that is, the spatial filter [1][2][3][4][5][6] and variational model [7][8][9][10].These unified formulations simply decompose a given image into structures and details by smoothing the image, simultaneously preserving or even enhancing image edges.The differences among them lie in the ways that edges are defined and this prior information is used to guide the smoothing process.
The anisotropic diffusion model [1] falls into the category of spatial filters.It employs a PDE-based formulation in which pixel-wise spatially varying diffusivities are estimated from image gradients.The gradient of the filtered image guides the diffusion process in order to avoid smoothing edges.Bilateral filtering (BF) [2,11] is another widely used model for removing noise from images while simultaneously performing detail flattening and edge preservation.It averages the nearby pixels by calculating weights from spatial and range domains [2] and smoothes low-contrast regions while preserving high-contrast edges.Due to its simplicity and effectiveness, BF has been successfully applied to many computational photography applications [11][12][13][14].By taking advantage of the local linear relationship between the guidance image and the filtering output at a small neighboring window, a spatially guided filter (GF) was developed in [15].In [3], the authors proposed a relatively simple technique to decompose an image into structures and oscillatory texture components by using a pair of nonlinear low-pass and highpass filters.Inspired by the fact that more pixels may be included by extending the concept of neighborhood in a nonlocal way [16], smoothing at nonlocal/semilocal regions has drawn much attention in recent years [4][5][6].The Regcovsmooth scheme developed by Karacan et al. used the secondorder statistic descriptor region covariance as a similar weight to average the pixels in a squared neighborhood [4].Ye et al. proposed a sparse norm filter (SNF) defined in square neighborhood [6].Rousselle et al. used the variant of the nonlocal means filter for adaptive rendering [5].
Variational models are based on the energy minimization formulation with a data fidelity term and a smoothingpenalty term [7][8][9][10].The data term measures the difference between the filtered signal  and the original signal , while the smoothing term measures the extent to which the filtered signal is not piece-wise smooth.That is,  = arg min  { (, ) +  ()} , where  is a nonnegative parameter controlling the weight of the smoothing term.While the design of the data fidelity term (, ) is usually relatively simple (for instance, the squared L2 distance between the filtered signal and the original signal ‖ − ‖ 2 2 is often used), the choice of the smoothing-penalty term () is critical.A representative piece of work is the total variation (TV) [7], which uses L1-norm based regularization constraints to penalize large gradient magnitudes.In its original formulation, the TV model provides fairly good separations of structures from textures.Further studies extended the standard TV formulation with different norms for both data fidelity and smoothing-induced regularization terms and demonstrated that more robust norms could improve the performance of image decomposition [8][9][10].The success of the weighted least square (WLS) optimization method proposed by Farbman et al. in [8] is attributed in part to the   norm in the iterative reweighed least square (IRLS) framework.Recently, the L0 smoothing presented by Xu et al. [9] used  0 term instead of  1 term to directly measure the gradient sparsity in the context of image smoothing and achieved promising results.Later, they proposed a special method, called relative total variation (RTV) [10], dealing with textured images.The efficiency of RTV depends heavily on the defined windowed total variation (WTV) and windowed inherent variation (WIV) involving spatial information.
In summary, most existing image decomposition models aim to extract structures from textures and noise while preserving edges and each of them has its limitations.For instance, BF, WLS, and L0 smoothing work poorly for tackling images with nonuniform texture details, while RTV and Regcovsmooth may blur the edges for images containing cartoon patterns.In this work, we propose a general relative total variation (GRTV) method and its nonlocal version (i.e., NLGRTV).Motivated from the reinterpretation of L0 smoothing from the IRLS iterative-solver viewpoint, the key idea of the proposed GRTV and NLGRTV methods is to incorporate the sparse norm and nonlocal strategy into the model, taking advantage of both spatial filters and variational models.They are very different from RTV, which is based on statistical verification, in the following two senses.First, GRTV extends the norm of WTV from 1 to [0, 1] and sets the norm of WIV inversely proportional to the norm of WTV.As a benefit, the amplitudes of noise-like structures will become smaller and the global edges will become more prominent and sharper when -values are getting smaller, which is shown in Figure 1 (seen from left to right in the top line).Second, the nonlocal GRTV (NLGRTV) uses a different smooth-penalty weight and a nonlocal sparse penalty term.By integrating the nonlocal information into the smoothing process, NLGRTV can distinguish well textures from the main structures and the information of shade is well preserved as shown in the bottom line of Figure 1.All these nice features of the proposed GRTV and NLGRTV are demonstrated later on through several applications including structure edge detection and texture enhancement, image abstraction and composition, and artifact removal.

Background and Motivation
In this section, two representative image smoothing techniques, L0 smoothing and RTV, will be briefly reviewed, with one focusing on the sparse and robust norm and the other emphasizing the spatially varying total variation measure.Subsequently their strengths and drawbacks are discussed in detail.Finally, the analysis and reinterpretation of the L0 smoothing model from the perspective of IRLS iterativesolver are derived, which leads to the motivation of our new method.

Previous Works: L0
Smoothing and RTV Model.In [9], Xu et al. proposed an image smoothing method based on L0 gradient minimization.Let ∇ denote the gradients of ; its energy function can be defined as follows: where the data term is the squared L2 distance between  and  and the smooth term is the L0 norm of ∇.The L0 norm of a vector is the number of nonzero value, which directly measures the sparsity.Compared to the   (0 <  < 1) norm regularization such as WLS filter [6,8], L0 smoothing can remove low-amplitude structures and globally preserve and enhance salient edges, even if they are the boundaries of very narrow objects.Like other edge-preserving smoothing approaches [1,2,7,8], one main drawback associated with L0 smoothing is that the local contrast-based definition of edges may fail to capture high-frequency components that are related to fine image details or textures, because differences in the brightness values or gradient magnitudes are employed as the main cues for edge indicator at an image pixel, and this information is used to guide the smoothing process.This means that L0 smoothing cannot separate completely textured regions from the main structures as they are treated as part of the structure.Recently some advanced algorithms were proposed; for example, an L0 smoothing-L1 method was presented in [17], where a L1-defility term instead of L2-norm was used to deal with heavy noise.In [18], rather than using L0-L2 iteration algorithm to solve the model, the L0 smoothing FCD (fused coordinate descent) algorithm iteratively repeated coordinate descent step and fusion step to approximately solve the model.In [10], a method extracting structure from textured image was developed by employing relative total variation (RTV).For the purpose of better texture removal, the developed RTV model does not assume the type of textures in advance; instead it introduces a novel map of windowed inherent variation (WIV).The WIV map in a region that only contains texture is generally smaller than that in a region also including structural edges.Applying WIV map as a weight of TV model in the vertical and horizontal directions yields adaptive edge-preserving and texture removal.Mathematically, the objective function can be expressed as  [10],  and  are pixel indexes, and  and  are pixel coordinates.The second term enforces the structure part to be sparse in the gradient domain.RTV is modeled as variational formulation and its texture suppression weights are spatially variant.However, Despite having good performance in edge-preserving and texture-eliminating, its discriminative power is still limited.} .

Analysis and Motivation
In the case of one dimension and for each pixel index , (4) turns to ( The importance of rewriting (4) to ( 5) is that we can analyze the iterative evolution of each pixel in detail.The weight ℎ   measures the similarity of pixels    and    .Generally, the pixel in the image can be classified in the flat/smooth region, edge boundary, and "high variance" area.For the pixel in the smooth region or edge region, there must be some similar pixels hence ∑ =−1,+1 ℎ   to be large, therefore For the pixel in "high variance" area, there may be no pixel similar to the reference pixel and thus its update value largely depends on   .Totally, as the iterative progresses, pixels at some regions change/diffuse .Since L0 smoothing targets globally preserving salient structures, even if they are small in resolution, it fails to remove the noise with large magnitude.L0 smoothing produces speckle-like artifacts.Although L0 smoothing FCD improved the noise insensitivity ability compared to that of L0 smoothing-L1 and L0 smoothing, it still retained some points near the edge, distorting the edge boundary.The result of RTV is slightly blurred.Our method performs the best to remove high-frequency noise while preserving major edges.
much faster while some others change lately.One example is illustrated in Figure 2, where L0 smoothing produces speckle-like artifacts for dealing with image corrupted by heavily noise.For better tackle with limits in (5), some points need to be fully considered for the weight term and candidate neighbor pixel sets as follows.
(i) The weight term ∑ =−1,+1 ℎ   and ℎ   may not be designed adaptively well in its definition.It accounts for not only the fidelity balance but also the measurement between the nearby pixels.Its functional can be better modified by spatially varying adaptation.In this sense, both L0 smoothing-L1 method and RTV can be seen as the improvements along this direction.
(ii) The estimate depends not only on the weights but also on the candidate pixel sets.Nonlocal property is a universal prior in the community of image processing.L0 smoothing FCD can be seen as its improvement by adding fusion step.We note that this discussion has universal guiding meaning.Recently proposed method SNF [6], Regcovsmooth [4], and adaptive NLM rendering algorithm [5] all utilized this property.

Approaches
In this work, our aim is to smooth a given image to suppress the textures to the most extent meanwhile retaining the edge sharpness.In general, from the viewpoint of iterative evolution, the change of each pixel    depends on three factors: the weight of data fidelity/smoothing-penalty to the original pixel   and how to define the candidate pixel sets and the weight of its relation to the neighboring pixels.In the following, we exploit the sparse norm and nonlocal scheme as two bridges to achieve the purpose of both properties of edge-preserving and texture-smoothing.Specifically, the proposed local Model 1 utilizes the sparse norm of the total variation and data fidelity measure to calculate the degree of the weight of data fidelity and its relation to the nearby pixels.By incorporating the nonlocal strategy into Model 1 and modifying the weight of data fidelity, the nonlocal Model 2 is derived.

Model 1:
The Local Model GRTV.One possible objective function we choose is expressed as follows: The proposed method jointly enhances edge and eliminates texture with 0 <  ≤ 1. Intuitively, on one hand, the operators    and    with 0 <  < 1 have bigger edgepreserving ability than that with  = 1.On the other hand,  2−  and  2−  with 0 <  < 1 will pose better texturesuppressing capability than that with  = 1.
In the following content, we will discuss the solver of general model.After some mathematical manipulations, the smooth term in -direction can be written as where  is used to prevent numerical instability.Equation (7b) is approximated by the Iteratively Reweighted Norm (IRN) approach proposed by Rodríguez and Wohlberg [19], which is closely related to the iteratively reweighted least squares (IRLS) method [20,21] where   is a Gaussian filter with standard deviation  and   is a small positive number to avoid division by zero.The division in ( 8) is point-wise and * is the convolution operator.Equation (9) indicates that  2− , at pixel point  incorporates neighboring gradient information in an isotropic spatial filter manner.Meanwhile,  2− , in ( 9) is only related to the pixelwise gradient.The weights (8) and ( 9) pose the neighboring and the pixel constraint for the same point .The penalty in the -directional dimension is the same as that in directional dimension.
In summary, ( 6) can be written in a matrix form as follows: where V  and V  are the vector representation of  and , respectively.  and   are the Toeplitz matrices from the discrete gradient operators with forward difference.  ,   ,   , and   are diagonal matrices whose elements are defined in ( 8) and ( 9).Similar to RTV, ( 6) is solved by a reweighted strategy; that is, update the block variables   ,   ,   , and   as a function of V  calculated from the previous iteration and then update V  by minimizing (10) with the last values of   ,   ,   , and   .Practically speaking, using   =            +            , the minimization of ( 6) is Since (1 +   ) is a symmetric positive definite Laplacian matrix, efficient solvers are available for it [10].
RTV is a special case of GRTV with  = 1.As expected, the extended range with 0 <  < 1 will substantially improve the geometry and texture separation ability.A visual illustration is displayed in Figure 3.As the parameter  decreases, the weights  2− , and  2− , of texture and edge differ more.Figures 3(b) and 3(c) correspond to the case of  = 1 and  = 0.5, respectively.It can be observed that both filters smooth small fluctuations and meanwhile preserve edges.However, as compared in Figures 3(e) and 3(d), the estimates of the textural regions with  = 0.5 are bigger than with  = 1.At the same time the edge value with  = 0.5 is more stiffness.

Model 2:
The Nonlocal Model NLGRTV.Recently, the nonlocal similarity property inherent in nature image has been successfully applied in many image processing tasks due to the work of nonlocal means (NLM), which averages over pixels value in nonlocal neighbor region with similar patch structure [16].Later this idea was successfully generalized to variational framework based on nonlocal operators [22] and the nonlocal   (0 <  ≤ 1) patch regression [23,24].In this subsection, we consider introducing the nonlocal configuration.Unlike in GRTV, in this case, the weight  2− () = ∑ ∈() ( , /( 2−  () +  2−  () + )) is viewed as a weight of spatially varying smooth-penalty term.The objective function can be expressed as where Ω  denotes a squared neighborhood set centered at pixel  and of size (2 + 1) × (2 + 1) pixels.Similar to the technique used in Section 3.2, that is, following the reweighted norm approximation, for each pixel we have Accordingly, the update of  +1  is achieved by Our alternating update algorithms GRTV and NLGRTV are sketched in Algorithm 1, where the solution variable and weight variables are updated alternately.
Figure 4 shows the advantage of extending local pixel sets to nonlocal candidate data.As can be observed, both GRTV and NLGRTV improve the visual perception of the input image than that of RTV.However, since the texture patterns in the right-bottom of the original image are very similar and the magnitude is small, the GRTV with  = 0.5 makes their low color difference almost to zero.NLGRTV with  = 0.5 alleviates this drawback.Although the pixel magnitude nearby the edge boundary is small but the pixel number is large, subsequently NLGRTV still retains these structure objects in the geometry component.

Iterative Reweighted Least Square (IRLS) Framework
and the Computation Cost.From the view of numerical optimization, RTV, the developed GRTV and NLGRTV fall into the category of IRLS framework.They are closely related to the WLS filter, which can be viewed as a single iteration of the IRLS framework.In WLS, the weight is the 1/(|(  log())  | 2− + ) (0 ≤  ≤ 0.8), which is a modification of 1/(|(  )  | 2− +).Though the convergence of the proposed GRTV and NLGRTV may not be proved strictly as in [19], our numerical experiments demonstrate good convergence behaviors.Figure 5 shows one example of the input image depicted in Figure 3(a).The intermediate structure images obtained by GRTV and NLGRTV with  = 1 are displayed, where we can find the proposed method quickly updates the saliency sharpened image in iterations.It indicates the effectiveness of the IRLS strategy adopted by our method.By the way, empirical experiments show that, for both GRTV and NLGRTV, the smaller the value of  and the bigger the value of , the more iterations the algorithm needed to converge.
The computation cost of GRTV is the same as that of RTV. that is, at each iteration, the computational cost of ( 8) is linear with respect to problem size with () by   taking advantage of the nice separation property of Gaussian kernel, where  is the total number of pixels in an image.As stated in [10], the step of calculating V  is to solve a linear system with a five-point spatially inhomogeneous sparse Laplacian matrix.Fast solvers such as the multiresolution preconditioned conjugate gradient (PCG) can reach () complexity.In all the experiments of this paper, it usually takes 3 s to process an 800 × 600 color image on an Intel Corei3CPU@2.93G with our matlab implementation.For the computation cost of NLGRTV, it differs from GRTV mainly on the updating of  +1  in (14).Admittedly, the complexity of computing ( 14) is almost the same as that in SNF [6]; that is, it can be calculated in linear time () by employing off-theshelf acceleration methods [25,26], where  is the number of bins for quantization and the pixel number .

More Analysis and Comparisons
In this section, we reveal the differences between our proposed methods and recently developed L0 smoothing, RTV, and RegionCov.We also compare our methods with some existing edge-aware image smoothing methods, including BF and WLS, and some advanced improvements of L0 smoothing.Specifically, we have hand-tuned parameters carefully for these methods to achieve the best performance.

Performance on Varying Values of 𝑝 and 𝑟.
Parameter  controls the degree of sparse norm and  controls the search window size.The effect of varying parameters is demonstrated in Figure 6.Seen from Figure 6, it can be found from the second and third rows that both GRTV and NLGRTV make the image more sharpen.Furthermore, the results in the third row are more smooth and robust than the second line due to the nonlocal pixel diffusion.By the way, although L0 smoothing method uses the penalty with norm  = 0, its smoothing result may not have visual effect due to the lack of adaptive smooth-penalty weighting term.According to the figure, it is expected that the user can obtain the specific effect by tuning parameters  and .In the following content of this paper, we set  = 2 for all the experiments.7 contains textural sward and slightly smooth clouds with small magnitude.The result of RTV in Figure 7(c) cannot simultaneously maintain the lines on the girl's skirt and smooth the sward.With  = 0.5, our GRTV largely improves this simplification ability.For this test image, the behavior of GRTV is very close to that of L0 smoothing by specifying  = 0.0004 and  = 1.As seen from Figure 7(d), the boundary of the cloud obtained by our method is more rich and clear.Besides, by tuning the parameters  and , our method yields meaningful decomposition as depicted in Figures 7(e) and 7(f).

Relationship to L0 Smoothing and Comparisons with
We can consider the relation between L0 smoothing, RTV, and GRTV from the global and local respects.L0 smoothing uses L0 gradient minimization, which can globally control how many nonzero gradients are involved to approximate prominent structure in a sparsity-control manner.It does not depend on local features but instead globally locates important edges.RTV mainly emphasizes the textureweights, which is locally defined.Our presented method GRTV falls in between and inherits the advantages of both algorithms in an approximate way.
The flower shown in Figure 8 contains smooth branches and leaves and flower bud with small magnitude.The result of L0 smoothing, RTV, and L0 smoothing L1 cannot balance   detail smoothing and edge enhancing.Since L0 smoothing FCD uses the fuse technique, that is, a group of variables rather than a single variable to be updated, it may cause overfitting.On the other hand, our method GRTV obtains a visually satisfactory result for better preserving the contrast around edges than the original RTV filter.As stated in [9], although L0 smoothing cannot efficiently tackle texture images, it still produces novel effects by applying first the local filtering BF and then the global filter L0 smoothing.Figure 9 shows the example reproduced in [9], where BF yields overblurring result and L0 smoothing maintains fluff textures.However, combining them in a consecutive way yields an image with sharpen prominent edges, as shown in Figure 9(e).In Figure 9(d) RTV gives a slightly blurred image.Finally, we can observe that our result with  = 0.5 in Figure 9(f) sharpens the large-scale salient edges more than that of RTV.

Relationship to RTV and Comparisons with Related
Stateof-the-Art Methods.Essentially, both RTV and our methods use a weight function which is very similar to the fast cartoon + texture method by Buades et al. [3].In [3], they introduced the relative reduction rate of local total variation as a weight function to measure to what extent the considered point belongs to a textured region.One main drawback of this filter is that its output result is directly computed by weighted average of the original image and a low-pass filtered image.However, RTV and our methods work in an iterative scheme that estimates the weights adaptively.
When visually evaluating the performance of standard decomposition method, one important criterion is to consider how strong the remaining parts of one component on the image of another component.For a part of the image "Barbara, " we can see in Figures 10(b) and 10(c) that Subr et al., 2009 [27] and Buades et al., 2010 [3] method cannot completely eliminate the texture from the table cover.RTV also provides similar results [10].The proposed method NLGRTV with  = 1 gives well-separated results and exhibits contrast-preserving feature, by eliminating the texture from the cartoon (see Figure 10(e)).Lowering the value of  to be 0.5, its output image changes to be more sharpening, as presented in Figure 10(f).
In the following, some examples with ample nonuniform and anisotropic texture are shown to exhibit the flexibility of our method.Note that the TV model, BF, and WLS filters were used in natural image smoothing and do not have effective terms to tackle textures.L0 smoothing also has limitation in dealing with the "structure + texture" images, since it usually cannot suppress the texture component.Figure 11(a) shows the image "Bishapur zan, " in which the main structures are surrounded by the background formed by many tiles with salient but fine boundaries.Form Figures 11(d) to 11(f), it can be observed that our method NLGRTV with varying parameters  = 1, 0.5, 0.1 can attain satisfied results with different styles.By tuning the regularization parameter, NLGRTV well preserves long coherent and prominent edges, meanwhile eliminating the outliers and irregular patterns.
Figure 12(a) shows another input image and the result from RTV method is presented in Figure 12(b).Despite making use of local signed gradients and exhibiting special properties, RTV fails to preserve some fine structures and blurs main edges compared to that of our GRTV method with  = 0.5 shown in Figure 12(c).Additionally, the decomposition of GRTV preserves the contrast between the girl and the background better.Our edge-preserving texturesmoothing filtering method NLGRTV with  = 0.5 improves

Comparison with Nonlocal-Based Methods.
Recently, the emergence of new nonlocal-based filters which have provided new insights and attained potential performances for image smoothing problem has been witnessed.Sparse norm filter (SNF), Regcovsmooth, and our NLGRTV all use the nonlocal neighborhood pixels.Specifically, the latter two approaches focus on weight adaption to achieve texturestructure separation.The Regcovsmooth method built on the patch-based nonlocal region covariance descriptor similarity framework, considering the region covariance matrices as image features to estimate the similarity between two image patches.Instead, our method NLGRTV lies on the nonlocal pixel-difference framework.An illustration is depicted in Figure 13.As can be observed, the nonlocal candidate scheme makes both Regcovsmooth and NLGRTV better preserve shading information than RTV.Besides, our method with  = 1 better preserves the edge geometries such as the regions in eye and cheek.Note that the search neighborhood size in Regcovsmooth is 21 × 21 while ours is only 5 × 5.

Applications
The incorporation of sparse norm and nonlocal strategy in variational model largely improves the ability of the existing models.By varying values of  and  as shown in Figure 6, the good separation to various types of meaningful cartoon and texture patterns under different parameter values indicates the diverse usage of the proposed method.For instance, with smaller values of  and , the method will prefer to enable applications of enhancement, image abstract/rendering, and vectorization.On the other hand, it may be better to handle images containing complex patterns and for the applications of detail enhancement, HDR compression, and haze removal.We show a few applications in image editing by using the property of effectively eliminating texture, meanwhile preserving/enhancing structure.

Structure and Texture Enhancement.
Edge extraction from images is usually the basic preprocessor to natural image editing (Bae and Durand 2007 [28]) and high-level structure inference.High quality results that are continuous, accurate, and thin are generally very difficult to produce due to high susceptibility of edge detectors to complex structures and inevitable noise.Our method is able to suppress lowamplitude details, which remarkably stabilizes the extraction process.One example is shown in Figure 14; the boundaries   are provided.As can be observed from enhancements of the small petal on the upper region of the image, the edge-aware filters WLS and L0 smoothing lead to gradient reversal artifacts due to their edge sharpening results on their corresponding base layers.GF alleviates this drawback because of its local linear model nature.Since NLGRTV employs a nonlocal scheme, it also attains visual pleasure estimate.Figure 16 shows another example.As illustrated in [10] and in Figures 16(b) and 16(c), both RTV and GRTV can nicely separate the cloth patterns from the image.Then the magnification effect can be created followed by enhancing the texture contrast and adding this layer back to the base layer.One main drawback is that the white smog is also magnified.NLGRTV alleviates the shortcoming as it can preserve the main slow-varying shape of white smog in the structure part and thus avoid being magnified.

Image Composition.
Mosaics images, paintings, and drawings sometimes cannot be directly used in image composition because the source and target textures are incompatible.Our proposed GRTV and NLGRTV methods can largely attenuate this shortcoming by first smoothing the original input with our structure-preserving filter and then performing composition between the resulting base layers with another image.As shown in Figure 17, the well-smoothed image by GRTV enables the composed image in Figure 17(e) to look more visually pleasing than Figure 17(d).Besides, by tuning parameters, the nonlocal method NLGRTV even obtains one composed image containing the shade information inherent in the original image.
Another composition example is displayed in Figure 18, in which the input image was partly analyzed in Figure 4.There the local smoothed methods RTV and GRTV cannot well preserve the subtle color information in the boy. Figure 18(b) presents the result of Regcovsmooth, from which we find that the image is blurred with the purpose of diminishing the knitted textiles.This makes the composition image in Figure 18(e) looks a little indistinct.The results in Figures 18(c) and 18(f) indicate that our method can achieve the goal of joint edge-preserving and texture-diminishing elegantly and hence yield the composed image to be more impressive.

Image Abstraction.
Nonphotorealistic abstraction with simultaneous detail simplifying and edge emphasizing usually gives a nonrealistic look of images.The conventional methods mainly consist of the edge-preserving smooth step and line extraction step, which is often accomplished by BF (Winnemöller et al. 2006 [13]) and the difference of Gaussian (DoG) filtering, respectively.Finally, the enhanced extracted lines are composed back to augment the visual distinctiveness of different regions.As discussed in Section 4.2 and shown in Figure 19, both L0 smoothing and our method can simultaneously achieve the two goals.By setting larger regularization parameter  to smooth the image, the resulting structure component can be directly applied on edge detection.

Artifact Removal.
When cartoon-like images such as the clip-art were heavily compressed, this usually causes severe compression artifacts.In [9], Xu et al. found that the general denoising approaches even with the state-of-the-art BM3D [29] cannot achieve satisfying output and their developed method L0 smoothing gives appear performance.In fact, as can be expected, this artifact that strongly correlated with edges can be viewed as a special nonuniform texture and hence can be properly coped by our texture-smoothing and     and 21, where one input image contains acute edges and the other one in contrast has smooth edges.In Figure 20 GRTV outperforms RTV mainly that it better lowers the amplitudes of noise-like structures on the animal's claws.In Figure 21, the nonlocal scheme NLGRTV corrects the locally shifted color nearby the edges.

Conclusions and Discussions
In this paper, models can effectively eliminate texture; meanwhile preserving/enhancing structure was proposed.The image decomposition results were improved by assigning with different norms for the spatially varying smooth-penalty weights and regularization terms.The proposed model can be applied to images with high noise levels and complex content and thus possesses advantages such as contrast-preserving, edge-preserving/sharpening, and data-driven scale selection.
Additionally, the proposed model allows a high level of randomness due to the nonlinear formulation.With the aid of iteratively reweighted least squares (IRLS) technique, the original nonlinear problem was transformed to the alterative iterations of weights updating and the least square solver that are much easier to solve quickly.
Experimental results demonstrate the effectiveness and robustness of the proposed method.With smaller -norm and neighbor size, the method will prefer the piece-wise image.On the other hand, nonlocal sampling strategy enables it better to handle images containing complex patterns.Particularly, our method GRTV can enhance the edge boundary by increasing steepness of transition as displayed in Figures 7,  8, and 9.This property may be in favor of the image segmentation such as the variational active contour/snake models [30,31]; therefore our developed weight can be incorporated to their weighted TV-norm for better segmentation.An important issue that needs further investigation in NLGRTV is replacing the current WIV map by other texture measurements as adaptive varying smooth-penalty weight.As stated in methods RTV and Regcovsmooth, there is no texture/feature measurement which is perfect and it may misinterpret part of structures as texture when it is statically similar in shape and scale.State-of-the-art texture measurements discussed in [3,32] can be considered to be integrated into our framework.This is an interesting direction for future work that may lead to substantial improvement.

2 MathematicalFigure 1 :
Figure 1: Examples of weighted total variation filtering results by using the sparse norm and nonlocal pixel sets.(a) From left to right are the input images and the local method results with -norm of 1, 0.5, and 0.1.(b) From left to right are the input images and the local method results with -norm of 1 and the nonlocal version of 0.1.Smaller -values give more prominent edge sharpening results while the nonlocal method preserves the shade information well.

Figure 2 :
Figure 2: Comparison with state-of-the-art methods on image created by Farbman et al.[8].Since L0 smoothing targets globally preserving salient structures, even if they are small in resolution, it fails to remove the noise with large magnitude.L0 smoothing produces speckle-like artifacts.Although L0 smoothing FCD improved the noise insensitivity ability compared to that of L0 smoothing-L1 and L0 smoothing, it still retained some points near the edge, distorting the edge boundary.The result of RTV is slightly blurred.Our method performs the best to remove high-frequency noise while preserving major edges.

Figure 4 :
Figure 4: Demonstration of introducing nonlocal strategy to achieve better shade-preserving smoothing result.

Figure 5 :
Figure 5: Structure images in different iterations obtained by the algorithm GRTV (top line) and NLGRTV (bottom line).

Figure 6 :
Figure 6: Effect of varying parameters.Smaller -values give more prominent edges sharpening results and nonlocal way in favor of preserving shade information.

Figure 10 :
Figure 10: Comparison with the extracted structures from the image, Barbara image, by different methods.Our method in (e) extracts the structure from the texture well while preserving shading information.

Figure 11 :
Figure 11: Smoothing results on the marble mosaic image.NLGRTV is processed under  = 2.

Figure 17 :
Figure 17: Image composition example.(a) Input; (b), (c) structure image by GRTV and NLGRTV; (d), (e), and (f) composed images obtained with the original and the structure images (a), (b), and (c) in top row.

Figure 19 :Figure 20 :
Figure 19: Image abstraction results.Our model effectively removes texture from structure and the extracted structure can be used to obtain a nonphotorealistic rendering of the scene.