Image Denoising Using Nonlocal Means with Shape-Adaptive Patches and New Weights

. Digital images captured from CMOS/CCD image sensors are prone to noise due to inherent electronic ﬂuctuations and low photon count. To eﬃciently reduce the noise in the image, a novel image denoising strategy is proposed, which exploits both nonlocal self-similarity and local shape adaptation. With wavelet thresholding, the residual image in method noise, derived from the initial estimate using nonlocal means (NLM), is exploited further. By incorporating the role of both the initial estimate and the residual image, spatially adaptive patch shapes are deﬁned, and new weights are calculated, which thus results in better denoising performance for NLM. Experimental results demonstrate that our proposed method signiﬁcantly outperforms original NLM and achieves competitive denoising performance compared with state-of-the-art denoising methods.


Introduction
Digital imaging devices such as digital cameras and camera phones are ubiquitous in our daily life, which use complementary metal oxide semiconductors (CMOS) or charged coupled devices (CCD) image sensors to acquire images. However, since the CMOS and CCD image sensors are subject to noise from two notable sources, i.e., electronic instruments and the photo-sensing devices [1,2], the quality of captured images is usually not satisfactory, especially when images are taken in low light condition, which leads to degraded imaging results. Hence, denoising has become a fundamental image restoration problem in image signal processor (ISP).
During the past decades, image denoising has been widely studied. However, until now, how to remove the noise efficiently while preserving significant image details has remained a challenge. Early smoothing methods, such as Gaussian filter [3], anisotropic filter [4], total variation [5], and bilateral filter [6], perform noise removal solely based on the information provided in a local neighborhood, thereby resulting in disturbing artifacts around edges. Later, transform domain-based denoising methods have been developed and extensively studied as well [7][8][9][10][11][12][13][14][15][16][17][18][19]. Wavelet transform (WT) [7] decomposes the image into multiple frequency components, where the noise is removed with thresholding [8,9] or statistical modeling [10][11][12]. By transforming back the processed wavelet coefficients into spatial domain, denoising is accomplished. Late development of WT denoising includes ridgelet [13] and curvelet [14] methods. In [15], adaptive principal components-based denoising method was proposed. Compared with WT that uses a fixed wavelet basis, it computes the locally fitted basis to decompose the image. In [16,17], the highly overcomplete dictionary was trained by using K-SVD algorithm for sparse and redundant image representation. In [18,19], discrete cosine transform (DCT) was applied to the local neighborhood, which achieves very sparse representation of the image and hence leads to effective denoising performance.
Recently, nonlocal methods, that exploit the image nonlocal self-similarity, have achieved outstanding denoising performance. In [20,21], Buades et al. first proposed the nonlocal principle-based denoising method, called nonlocal means (NLM). In this method, noise-free pixel is estimated as a weighted average of all pixels in the image, where the weights are determined based on the similarity between the patch centered at the pixel being estimated and the patches centered at other pixels. Since NLM exploits the fact that similar patches appear abundantly in the image and can contribute for denoising, it obtains high quality denoising performance. Subsequently, numerous extensions of NLM have been developed. In [22], the optimal neighborhood for each pixel was chosen during the iteration procedure to balance the accuracy of approximation and the stochastic error. In [23], the noisy image was classified into several region types, according to which the patch size was adaptively adjusted to match the local property. In [24], adaptive patch size and bandwidth were selected pixel-wise, relying on the feature metric that can provide a quantitative measure of local geometric structures. In [25], the smoothing parameter was chosen automatically based on noise estimation, and then the two-stage NLM with adaptive smoothing parameter was performed. In [26], quadtree-based NLM was proposed, which employs quadtree decomposition on each image patch to obtain more subpatches of various sizes. In addition to considering the patch sizes, some methods try to handle variable patch shapes. In [27], the adaptive binary shape for each patch was estimated by thresholding the difference between the central pixel and other patch pixels. In [28], shape-adaptive patches were constructed to match more homogeneous pixels successfully, especially in textured areas. In [29], several types of patches with various shapes were predefined and applied in NLM, respectively. en, local estimates associated with these shapes were combined using Stein's unbiased risk estimate (SURE). To improve the accuracy of similarity measure, nonlocal similarity of residual image structures in method noise was further exploited in [30,31]. Besides, rotation invariant patch comparison, that can handle rotational similarity existing in the image, was also studied in [32][33][34][35][36]. Analogously in [37], affine invariant similarity measure was applied to find more similar patches. Although NLM and its extensions have achieved significant denoising results, only exploiting the spatially nonlocal redundancy still limits their performance. erefore, some methods combine the nonlocal principle with other techniques [38,39], resulting in state-of-the-art denoising performance [40][41][42][43].
In this paper, we address these issues and propose an efficient denoising method, as shown in Figure 1. First, the original NLM is employed to obtain an initial estimate of the noisy image. However, due to inaccurate weight computation with noise interference, the initial estimate does not contain complete image details, which means the method noise still contains residual image information. For well preserving the residual image in method noise, the wavelet thresholding is used to smooth noise as much as possible.
en, the preserved residual image is combined with initial estimate to obtain a basic denoising result, based on which spatially adaptive patch shapes are defined using LPA-ICI and new weights are calculated. Finally, NLM denoising is implemented again but with the shape-adaptive patches and new weights. e remainder of this paper is structured as follows: in Section 2, original NLM is briefly reviewed. In Section 3, our proposed denoising method is described in detail. In Section 4, we present and analyse the comparative experimental results. Finally, Section 5 concludes this paper.

Nonlocal Means
Given a noise-free image u defined on a discrete grid I, the noisy observation of u at pixel i ∈ I is defined as where n(i) is the zero-mean white Gaussian noise perturbation at pixel i. Let N i denote the patch centered at pixel i, and its noisy observation is defined as In classical NLM [20], for a pixel i, the estimated value of its noise-free version, u(i), is calculated as the weighted average of all noisy observations of the pixels in the image: e weight w i,j depends on the similarity of patch N i and patch N j , which is defined as where ‖ · ‖ 2 denotes the Euclidean norm to measure the similarity between the patches, and the parameter λ acts as a degree of filtering, therefore controlling the decay of the weight. Since the classical NLM only estimates a single pixel, it is referred as the pixel-wise NLM.  Later, the patch-wise NLM does also exist [21]. Here, the whole patch is estimated as follows: As the patches are overlapped, each pixel in a patch achieves multiple estimates. erefore, for a pixel i, its different estimates are aggregated to obtain the final result: where A i � j|i ∈ N j and u(N j , i) denote the estimated value of noise-free patch N j at pixel i. anks to the multiple estimations for each pixel, the patch-wise NLM achieves better denoising performance than the pixel-wise one. In this paper, when NLM is mentioned, it refers to the patch-wise NLM.
Based on the denoising result, method noise is defined as follows: It can be inferred that, if NLM performs well, the method noise must look like a noise and should contain as little structure as possible. In Figures 2(a) and 2(b), an example image and its noisy version (σ � 20) are shown, respectively. Figure 2(c) shows the NLM denoising result, while Figure 2(d) shows the corresponding method noise. As can be seen, obvious image structure appears in the method noise, which means that some image details are removed from the denoised image. erefore, we make use of the residual image in method noise to exploit nonlocal selfsimilarity further.

Method Noise
resholding. Since the method noise contains obvious image structure, the residual image in it is estimated firstly. Here, we use the BayesShrink wavelet thresholding method [44] to suppress the noise as much as possible.
BayesShrink is an adaptive, data-driven thresholding strategy via soft-thresholding which derives the threshold in a Bayesian framework, assuming a generalized Gaussian distribution for the wavelet coefficients. is method is adaptive to each sub-band because it depends on data-driven estimates of the parameters. e threshold for a given sub-band is derived by minimizing Bayesian risk as follows: where σ 2 n is the noise variance estimated from sub-band HH 1 by a robust median estimator, given by and σ 2 w is the variance of wavelet coefficients in that subband, whose estimate is computed using where σ 2 Figure 2(e) shows the filtered method noise thresholding with wavelet. As can be seen, the residual image is preserved well while the noise is smoothed efficiently. en, we combine the initial estimate, denoted as u initial , and the filtered method noise, denoted as n f , together to obtain a basic estimate: In Figure 2(f ), we show the combined result. It can be seen that the basic estimate preserves more image details than the initial estimate, such as the regions marked by green boxes, which means that it will be more accurate to estimate the spatially adaptive patch shapes and to calculate the weights based on the basic estimate.

Spatially Adaptive Patch Shape Estimating.
e anisotropic local polynomial approximation-(LPA-) intersection of confidence intervals (ICI) technique is used to estimate the spatially adaptive shape for each patch in the image. Figure 3 shows the implementation of the LPA-ICIbased patch shape estimating. For a pixel i, eight directions are first predefined. For every specified direction θ k � ((k − 1)/4)π, k � 1, . . . , 8, a varying-scale family of narrow "linewise" directional LPA convolution kernels g h,θ k h∈H is used to obtain a corresponding set of directional varying-scale estimates u h,θ k h∈H , u h,θ k � v ⊗ g h,θ k , h ∈ H, where H ∈ R + is the set of scales. en, for each estimate, a confidence interval is built as follows: where Γ > 0 is a tuning parameter and σ is the noise standard deviation. Based on ICI rule, an adaptive scale h + (i, θ k ) ∈ H is defined for every direction θ k . Finally, the shape-adaptive patch N + i is constructed as the polygonal hull of supp g h + (i,θ k ),θ k 8 k�1 .
In Figure 4, we show some examples of the estimated shape-adaptive patches in the noise-free, noisy, initially estimated, and basically estimated images, respectively. It can be seen that, due to the influence of strong noise, the estimated patch shapes in the noisy image are incorrect. e same goes for those in the initially estimated image but because of the loss of image details during NLM denoising. By contrast, patch shapes in the basically estimated image are more accurate.

New Weight
Calculating. Based on the basic estimate, weights between the patches are calculated again. Since we estimate the spatially adaptive shape for each patch, the new weight is calculated as follows:

Shape-adaptive patch
Adaptive-scale kernel support Figure 3: Implementation of the LPA-ICI-based patch shape estimating. "Linewise" one-dimensional directional LPA kernels are used for 8 directions. e shape-adaptive patch N + i is constructed as the polygonal hull of the adaptive-scale kernel supports. Shock and Vibration 5 where N + i denotes the estimated shape-adaptive patch, N i,+ j denotes the patch using the same shape with N + i , and c is the filtering factor.

NLM with Shape-Adaptive Patches and New Weights.
With shape-adaptive patches and new weights, we implement the NLM denoising again to remove the noise in the noisy image. For a shape-adaptive patch N + i , the estimated value of its noise-free version, u(N + i ), is calculated as follows:

Shock and Vibration
Similarly, for a pixel i, its different estimates are aggregated to obtain the final result: where A + i � j|i ∈ N + j and u(N + j , i) denote the estimated value of noise-free patch N + j at pixel i.

Computational Complexity. For the image of size
, the computational complexity of the original NLM for initial estimate is Ο(n p N 2 ), where n p denotes the patch size used by NLM. In our practical implementation, a limited window of size �� � w 1 √ × �� � w 1 √ is used to restrict the search of similar patches, which reduces the complexity to Ο(n p w 1 N). Besides, by using the moving average filter together with weight symmetry, the complexity can be further brought down to Ο (w 1 N).
en, the computational complexity of method noise thresholding with BayesShrink wavelet is usually Ο(N). For the spatially adaptive patch shape estimation with LPA-ICI technique, since it is based on convolutions against one-dimensional kernels for a very limited number of directions, its computational overhead for the whole noise removal processing is negligible. In the final estimating procedure, for the search window of size �� � w 2 √ × �� � w 2 √ , the computational complexity is Ο(w 2 N).

Results and Discussion
In this section, we compare our proposed method with original NLM [21] and other two state-of-the-art denoising methods: shape-adaptive patches-based NLM (NLM-SAP) [29] and shape-adaptive PCA-based BM3D (BM3D-SAPCA) [41]. In the experiments, a set of 8 natural images commonly used in the literature of image denoising are used for the comparison, and their noisy versions are simulated by adding independent white Gaussian noise with varying noise levels. e results of our proposed method are generated using the scales H � 1, 2, 3, 4, 5, 6 { } and a search window of 21 × 21. e parameter c is selected experimentally as c � 0.4σ. e results of other three methods are obtained by using the codes available online with recommended parameters. To evaluate the quality of denoised images, the popular peak signal to noise ratio (PSNR) and the structural similarity index (SSIM) are calculated. e results of the experiments are shown in Tables 1 and  2, where the best results among the four methods are highlighted. It can be seen that our proposed method significantly outperforms NLM and NLM-SAP and achieves competitive denoising performance compared with BM3D-SAPCA. Particularly, in some cases of high noise levels, our method performs even slightly better than BM3D-SAPCA. In terms of SSIM results, our proposed method is quite close to BM3D-SAPCA, and its superiority exists in all cases with respect to NLM.
Let us then focus on the visual quality of the denoised images by the four methods. In Figures 5 and 6, we show the denoising results on two typical images with moderate and strong noise corruption, respectively. It can be seen that our proposed method is very effective in reconstructing both the smooth and the texture/edge regions. When the noise level is not very high, as shown in Figure 5, our proposed method performs better than NLM and NLM-SAP on edge preservation, and BM3D-SAPCA achieves the best visual output. When the noise level is high, as shown in Figure 6, however, details in the denoised images by NLM and NLM-SAP become blurred, and BM3D-SAPCA tends to generate many visual artifacts. By contrast, our proposed method performs much better, which preserves the image details well and generates much less artifacts.

Conclusions
In this work, we have presented an efficient image denoising method. By exploiting the residual image in the method noise, spatially adaptive patch shapes are defined, and new weights are calculated to improve the denoising performance of NLM further. Experimental results demonstrate that our proposed method is effective in noise removal and texture/edge preservation and can achieve competitive denoising performance compared with state-of-the-art denoising methods.

Data Availability
e data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest
e authors declare that they have no conflicts of interest.