NAMF: A Nonlocal Adaptive Mean Filter for Removal of Salt-and-Pepper Noise

In this paper, a novel algorithm called a Nonlocal Adaptive Mean Filter (NAMF) for removing salt-and-pepper (SAP) noise from corrupted images is presented. We employ an eﬃcient window detector with adaptive size to detect the noise. The noisy pixel is then replaced by the combination of its neighboring pixels, and ﬁnally, a SAP noise based nonlocal mean ﬁlter is used to reconstruct the intensity values of noisy pixels. Extensive experimental results demonstrate that NAMF can obtain better performance in terms of quality for restoring images at all levels of SAP noise.


Introduction
Digital images are often corrupted by noises in the process of image acquisition and transmission [1,2]. Image noise is a random change in the brightness or color information in an image, which will bring troubles for image processing and computer vision, even causing some seriously ill-posed problems [3]. Hence, image denoising has been taken as one of the most important tasks in image processing [4,5]. As a preprocessing step in image processing, image denoising can protect edges, textures, and other details in the image [6]. Generally, image noises can be divided into Gaussian noise, Poisson noise, and Salt-and-Pepper (SAP) noise in terms of discreteness and randomness. SAP noise commonly exists in natural images, and the pixels contaminated by SAP noise take the maximum or minimum value and can be represented as black or white points [7,8]. Considering the universality of SAP noise, we focus our attention on the SAP noise and develop a denoising method that provides some help for further image analysis.
To remove SAP noise, lots of computational methods have been proposed. Among them, median filter (MF) and adaptive median filter (AMF) [9] are the two most popular methods in the early stage. MF can restore image details well under low noise intensity, but it performs poorly when noise intensity is high [10]. AMF adopts the measure of the window with adaptive size, which makes it perform well in high noise intensity [11]. In addition, based on AMF, noise adaptive fuzzy switching median filter (NAFSMF) recognizes SAP noise by analyzing the histogram of noisy images and takes a fuzzy method to denoise [12]. Adaptive weighted mean filter (AWMF) uses two successive windows to detect noisy pixels and processes them with a weighted mean filter [13]. e method proposed in [14] is based on NAFSMF and AWMF, and it uses a new adaptive fuzzy switching weighted mean filter to remove SAP noise. In [15], researchers proposed a method, based on pixel density filter (BPDF), to remove SAP noise through searching the repeated numbers of the pixels, and achieves a good performance under low SAP noise intensity. Different applied median filter (DAMF) is proposed for removing SAP noise at all densities [16]. In [17], adaptive frequency median filter (AFMF) is proposed to remove the SAP noise, which employs frequency median to restore gray values of the corrupted pixels. e adaptive switching weight mean filter (ASWMF) proposed in [18] repairs noisy pixels by assigning a value of a switching weight mean to the gray value of the center pixel of an adaptive window. e processes of the above denoising methods can be divided into two stages, the detection of noisy pixels and the restoration of noisy pixels. For the latter one, the restoration of SAP noise is using the rest information (uncontaminated pixels) to repair the absent information (contaminated pixels). As Figure 1(b) shows, the limitation of existing stateof-the-art methods is that under high SAP noise level, the boundary of the restored image is jagged and the details are blurred. When SAP noise intensity is too high, the image loses too much information. e consequence is that less information is available to be used for restoring the image.
To solve the above problem, in this paper, we propose a nonlocal adaptive mean filter (NAMF) to remove SAP noise. NAMF can be divided into two stages: (1) SAP noise detection; (2) SAP noise elimination. Firstly, in the stage of SAP noise detection, we find out pixels whose gray values are equal to the global maximum or minimum gray value of the image. We then take them as noisy candidates and use a window with adaptive size to confirm them. For the possible noisy pixel, we calculate the proportion of the pixels with the same value of candidate in the window and then filter it by a threshold. If it is smaller than the threshold, the candidate is regarded as a noisy pixel. Otherwise, it is noiseless and will not be processed. Secondly, in the stage of SAP noise elimination, the noisy pixel will be replaced by the mean of its neighboring pixels. Finally, we use a SAP noise based nonlocal mean filter to further restore it. e main contributions of this paper can be concluded as follows: (i) A proportion based method is proposed to distinguish the noise pixels and texture pixels in the SAP noise detection, which can effectively avoid noiseless texture pixels being processed as noisy pixels. (ii) An improved nonlocal mean method based on the characteristics of SAP noise is developed to further restore noisy images, which can further enhance the quality of restored images.
In the experiments, we use 16 typical images and 40 test images collected from the TEST IMAGES Database [19] to evaluate our method. e experimental results demonstrate that NAMF outperforms the existing state-of-the-art methods at both high SAP noise levels and low SAP noise levels.

Related Works
In this section, we will firstly review traditional MF based methods used for SAP noise removal and then we introduce the classical nonlocal means method.

MF Based Methods for SAP Noise Elimination.
MF is the most commonly used algorithm to remove pulse interference and SAP noise. e main idea is that all pixels are replaced by neighborhood median pixels whether pixels are noiseless or not. Detailedly, let X be an original 8-bit graylevel image with a size of M × N, and x i,j be the gray value of the pixel at the location (i, j). en, for MF with a filter size of 3 × 3, the new gray value x i,j of the pixel at the location (i, j) can be calculated by the following equation: Motived by this idea, many MF methods have been proposed. However, the large distance between the noisy pixel and its neighborhood may result in a large difference between the median value pixel and the original noiseless pixel, which would cause the denoised image to be degraded [14]. In [13], the median value of neighborhoods is equipped with different weights, which are calculated by the distance from the center noisy pixel. Besides, when noise density is too high, for some noisy pixels, there are no noiseless neighborhoods for restoring. In [14], to tackle the problem, the researchers provide a method by using neighborhoods in the restored image to calculate the median value.

Nonlocal Mean (NLM) Method.
e NLM method combined the idea of the Yaroslavsky filter [20] and bilateral filter [21]. For the noisy pixel, NLM uses the intensity patch feature instead of single pixel feature to compute the denoised mean value [22].
Given a pixel x ∈ I, let N(x) denote an image block centered at x. en, NLM calculates a weighted average of all the pixels in the searching window B by the following equations: where x is the restored gray value of x, while N(x) denotes a block centered at x, and h is the smoothing parameter. u(x, y) denotes the similarity between pixels x and y which depends on the weighted Euclidean distance between two image blocks N(x) and N(y), ‖ · ‖ 2 is the Euclidean norm, and a is the standard deviation of the Gaussian kernel with zero mean and standard deviation, for details about these, please refer to [22].

The Proposed Nonlocal Adaptive Mean (NAMF) Filter
In this section, we will present the details of our proposed nonlocal adaptive mean (NAMF) filter. Same as the settings of many traditional SAP noise removal filters, NAMF can be divided into two stages: (1) SAP noise detection; (2) SAP noise elimination. Details are presented as follows.

SAP Noise Detection.
In accordance with the mathematical notation as the previous section, represent the original 8-bit gray-level image and noisy image corrupted by SAP noise, respectively. x i,j and y i,j represent the gray value of the pixel at the location (i, j) of X and Y, respectively, In a corrupted image, the value of a "salt" pixel equals the maximum gray value 255, and the value of a "pepper" pixel equals the minimum value 0. us, y i,j is defined by the following equation: 255 or 0, with probability α, where α is the density of SAP noise of Y, α is unknown in practical problems, here we just use α to represent the degree of noise. And in the process of image denoising, we use S i,j (w) to represent a (2w + 1) × (2w + 1) window centered at (i, j) with the radius w.
Considering the characteristics of SAP noise, pixel y i,j corrupted by SAP noise is 0 or 255. at is to say, noisy pixel candidate y i,j only has 2 possible values: y min � 0 and y max � 255. Following is a prior decision condition in noise detection.
where O � (o i,j ) ∈ R M×N is an indicator matrix with binary value. o(i, j) � 1 means that pixel y i,j is the noisy pixel candidate, while o(i, j) � 0 that means pixel y i,j is noiseless. For a natural image, the pixels with high or low value are also possible to be the texture of the image. For example, as shown in Figure 1(a), E and F are both candidates with a gray value of 0 (black). However, F has a larger possibility to be a noisy pixel, and E has a larger possibility to be a texture pixel. erefore, it is very necessary to further confirm these pixels with maximum or minimum value. In fact, in natural image, pixels are supposed to change smoothly, and adjacent pixels tend to have similar values. e value of pixels in an image with SAP noise may change suddenly. Motivated by these facts, we propose an adaptive SAP noise detection method by considering the neighborhood to identify noise candidates as noisy pixels or not. Its procedures can be divided into two steps: If not, then w + 1 and continue to compute, where S sum i,j (w) is the number of pixels within S i,j (w) which are not equal to y min and y max , w max is the maximum size of window. If S sum i,j (w) > 0, we take the pixel y i,j as a noisy pixel, the detection for y i,j ends. Otherwise, we go to Step 2. (ii) Step 2. If S sum i,j (w) � 0, y i,j maybe considered as a texture pixel. en, we calculate the proportion ρ � S num i,j (w)/(2w + 1) × (2w + 1), where S num i,j (w) is the number of pixels owning the same value as the candidate pixel y i,j in the window S i,j (w). After that, we set a threshold T to identify the noisy pixel. If ρ < T, the candidate pixel y i,j is regarded as a noisy pixel, else it is noiseless.
For the pixel y i,j , if it is finally detected as noisy, we mark it with the discriminant matrix L � l(i, j) ∈ R M×N , and l(i, j) � 1, else l(i, j) � 0 and will not be processed.

SAP Noise Elimination.
We restore noisy pixels in two steps. Let Z represent the initially restored image, Z represent the final output image. Before processing, we initialize Z with Z � Y.
Firstly, when a pixel y i,j is detected as a noisy pixel, we use S mean i,j (w) to restore it. e calculation of S mean i,j (w) is based on S sum i,j (w) as shown in equation (7). When S sum i,j (w) ≠ 0, S mean i,j (w) is the mean of the noiseless pixels in S i,j (w), otherwise it is the mean of three processed neighboring pixels of y i,j in Z. Unlike the four neighbors adopted in [14], the utilization of four neighboring pixels will lead to residual noisy pixels on the boundary, as shown in Figure 1(b) (even if the image boundary is expanded during the process, some noisy pixels located in boundary can still not be restored). Hence, we Figure 1: (a) "Lena" corrupted by SAP noise ratio of 10%, and the gray values of noisy candidates in E and F are both 0. (b) e result of the method proposed [14] for "Lena" corrupted by SAP noise ratio of 90%.
Secondly, considering the problem that restoration will be hard under high noise intensity, we introduce the nonlocal mean (NLM) method [22,23] to further restore noisy pixels. NLM can restore noisy pixels by using all neighbors instead of part of neighbors, which are detected as noiseless. In this way, we can use NLM to further enhance the restoration effect. Considering that the original NLM is designed for Gaussian noise or Speckle noise, which are quite different from SAP noise, we modify it based on the characteristics of SAP noise and make it applicable for our method.
In the noise detection stage, a pixel is identified and marked by the discriminant matrix L. l(i, j) � 1 means the pixel y i,j is identified as a noisy pixel, otherwise noiseless. After the restoration of the noisy pixel y i,j , we get z i,j . e modified NLM method is used to further restore z i,j . We use z i,j to represent the value of the further restored z i,j , and it can be calculated as follows: where B(z i,j , r) represents a searching window of size (2r + 1) × (2r + 1) centered at z i,j , and u(z i,j , z e,f ) represents the weight of pixel z e,f in B(z i,j , r). N(z i,j ), also called similarity window, is a square block centered at z i,j , so is N(z e,f ). As shown in equation (10), the similarity between z i,j and z e,f is measured by the Gaussian weighted Euclidean distance d(z i,j , z e,f ) between N(z i,j ) and N(z e,f ), where a is the standard deviation of the Gaussian kernel, and h is the smoothing parameter for NLM. When processing the noisy pixel z i,j , the original NLM assigns the weight based on the similarity. at is to say, the weight of pixel z i,j itself is the largest. Different from the original NLM, in our method, the noisy pixel to be processed will not participate in the process of NLM. us the weight of the pixel z i,j should be set as 0, as shown in equation (9).
In the NLM algorithm, the higher the noise intensity is, the larger the smoothing parameter hshould be. But the intensity of noise is not easy to be confirmed. Considering that SAP noise can be significantly detected, we use the intensity of SAP noise to confirm h. As shown in equation (10), L represents the total number of non-zero elements in the discriminant matrix L, that is, the more noisy pixels are detected, the larger h should be; β 0 , β 1 and β 2 are the parameters used to fit h.
Due to the high computational cost of the NLM algorithm, here we introduce a kind of fast implementation of the NLM algorithm [24] based on the computation of patch distances using sums of lines to accelerate our NAMF algorithm. e details of the proposed NAMF are shown in Algorithm 1.
We use 4 typical image quality metrics, peak signal-tonoise ratio (PSNR) [25], structural similarity (SSIM) [26], mean absolute error (MAE) [27], and image enhancement factor (IEF) [28] to evaluate the experimental results. For an image U and an image V with the same size of M × N , PSNR can be calculated by equation (11).
PSNR(U, V) � 10 * log 10 255 2 /MSE , where MSE is the mean square error of two images, and u i,j , v i,j are pixels of image U and V, respectively. SSIM for image U and V can be defined as follows: where μ u and μ v are the average intensities of image U and V, respectively. σ u and σ v are standard deviations, σ u,v is the covariance, c1 and c2 are some constants. Here c1 and c2 are set to be (0.01 * 255) 2 and (0.03 * 255) 2 as in [26], respectively. MAE can be computed as shown in the following equation: Different from traditional image quality metrics, IEF is designed for evaluating the respiring effects of the denoising method. IEF can be obtained by the following equation: where _ I, € I, I ∘ denote the original image, the noisy image, and the repaired image, respectively. In this paper, we set β 2 � 2.2186, β 2 � 2.2186, and β 0 � 4.5595 to fit h. e size of the searching window is 5 × 5. And the size of the similarity window is 41 × 41.
rough the test, we finally take the threshold T � 0.8 for our method. Other methods keep the default parameters. e experiments are performed on a personal computer with Intel Core i7 2.2 GHz processor and 16 GB RAM. Figure 3 shows the restored results of all methods for "Barbara" with SAP noise ratio of 10%. By observing the enlarged area in Figure 3, we can see that AMF, AWMF, BPDF, and ASWMF cannot restore the details of the original image very well. And the result of our proposed NAMF is nearly the same as the original image. Figure 4 shows the restored results for "Lena" with SAP noise ratio of 90%. In the enlarged area in Figure 4, it can be found that the performances of AMF, NAFSMF, BPDF, Figure 2: Y is a noisy image, Z is an image being restored, the red pixel in Z is prepared to process, the yellow pixels are pixels that have been restored, other blue pixels in Z are pixels that remained to be restored. When processing noisy pixels y i,j , neighborhoods in the window are all noisy, i.e., S sum i,j (w) � 0. So here we select three neighboring pixels z i−1,j−1 , z i−1,j and z i,j−1 in the restored image Z for restoring pixel y i,j in the original image Y.
Compute the indicator matrix O For each pixel (i, j) ∈∧ in the noisy image Y and the initially restored image Z, do (1) If o i,j �� 0, l(i, j) � 0, z i,j � y i,j , break; Otherwise, go to step 2).
Mathematical Problems in Engineering 5 AFMF, and ASWMF are very poor. And the restored images by other methods are also very blurred, while the result of our method looks more natural and smooth. e curves of average PSNR, SSIM, MAE, and IEF are shown in Figure 5. Figure 5(a) illustrates that NAMF obtains the highest PSNR under both low and high noise intensity, and the PSNR of our method is much higher than the results of other methods. Figure 5(b) shows that the SSIM curves of most methods are basically the same under low SAP noise intensity. However, with the increase of noise level, the superiority of NAMF is getting more obvious. When the noise ratio exceeds 30%, the SSIM obtained by NAMF is significantly higher than other methods. Figure 5(c) gives the MAE curves of all methods. We find that NAMF owns the lowest MAE at all SAP noise levels. e IEF curves of all methods are given in Figure 5(d). It is clear that our method can obtain the highest IEF score at all SAP noise levels. Furthermore, our NAMF can significantly outperform other methods in the low SAP noise level. Figure 6 illustrates the average running time of different methods at all noise levels. With the increase of noise intensity, the running time of all methods except AWMF and AFMF increases. Clearly, the average running time of the method proposed in [14] is the shortest. As shown in the enlarged area of Figure 6, although the rank of NAMF is in the middle, its processing speed is superior to NAFSMF and similar to DAMF under high noise intensity.
Besides, to further illustrate the function of our proposed SAP based NLM method, we make the ablation experiments to show its effectiveness. As shown in Figure 7, we compare the NAMF without the SAP based NLM and the original NAMF with the SAP based NLM. e former one uses a traditional denoising mechanism, and the latter one adopts the SAP based NLM to further repair the noisy images. Figure 7 shows that the performances of the NAMF without the SAP based NLM are significantly weakened compared with the original NAMF. (a)

Conclusion
In this paper, a method called NAMF for SAP noise denoising is proposed, which adopts a SAP noise based nonlocal mean method. NAMF can get much higher restoring quality than state-of-the-art methods at all SAP noise levels. e processing time of NAMF is comparable to most state-of-the-art methods. e experimental results show that NAMF can get much better PSNR and SSIM at all SAP noise levels. Moreover, NAMF can preserve more details even at a noise level as high as 90%.
Data Availability e data and codes in this study can be accessed at https:// github.com/zhuyuan-cug/NAMF.

Conflicts of Interest
e authors declare that they have no conflicts of interest. Mathematical Problems in Engineering 9