Mathematical Problems in Engineering
Volume 2020 (2020), Article ID 9483521, 15 pages
https://doi.org/10.1155/2020/9483521

Research Article

Blind Poissonian Image Deblurring Regularized by a Denoiser Constraint and Deep Image Prior

Yayuan Feng,1,2,3 Yu Shi,1,2,3 and Dianjun Sun1,2,3

¹School of Electrical and Information Engineering, Wuhan Institute of Technology, Wuhan 430205, China
²Hubei Key Laboratory of Optical Information and Pattern Recognition, Wuhan 430205, China
³Laboratory of Hubei Province Video Image and HD Projection Engineering Technology Research Center, Wuhan 430205, China

Correspondence should be addressed to Yu Shi; shiyu0125@163.com

Received 4 April 2020; Revised 21 June 2020; Accepted 22 July 2020; Published 24 August 2020

Academic Editor: Lotfi Senhadji

Copyright © 2020 Yayuan Feng et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

The denoising and deblurring of Poisson images are opposite inverse problems. Single image deblurring methods are sensitive to image noise. A single noise filter can effectively remove noise in advance, but it also damages blurred information. To simultaneously solve the denoising and deblurring of Poissonian images better, we learn the implicit deep image prior from a single degraded image and use the denoiser as a regularization term to constrain the latent clear image. Combined with the explicit L₀ regularization prior of the image, the denoising and deblurring model of the Poisson image is established. Then, the split Bregman iteration strategy is used to optimize the point spread function estimation and latent clear image estimation. The experimental results demonstrate that the proposed method achieves good restoration results on a series of simulated and real blurred images with Poisson noise.

1. Introduction

The process of image acquisition in electron microscope imaging, astronomical imaging, and medical imaging is inevitably affected by environmental factors, which often cause the captured image to be disturbed by Poisson noise and blur degradation. The image degradation process can be modeled as follows:

where is the observed degraded image, represents the process in which the image is corrupted by Poisson noise, represents the point spread function, and is the convolution operator. The problem of blind Poisson image deconvolution is to recover clear images and estimate the point spread function from the degraded image. The denoising and deblurring of Poisson images are opposite inverse problems. Single deblurring methods are sensitive to noise, and even a small amount of noise leads to biased point spread function estimation. Although the noise filter can effectively remove noise in advance, it also damages the blurred information and introduces more serious blur, which leads to amplified and biased point spread function estimation. To suppress Poisson noise and restore the blurred image simultaneously, we impose the denoiser as a regularization term to constrain the latent clear image, combined with an implicit deep image prior and an explicit L₀ regularization prior of the image. Then, the energy functional regularization framework for the latent clear image and point spread function is modeled as follows:

where is the data term, represents a vector with all elements equal to , is the latent clear image, which usually ensures that the clear image does not contain negative gray values, is the point spread function, is the regularization constraint term of the latent clear image, and is the regularization constraint term of a point spread function. Among them, regularization constraint terms and play a vital role in the restoration of latent clear images and the estimation of point spread functions.

1.1. Relevant Work

Poisson image denoising and blurring is an image deconvolution problem. Image noise can be divided into three categories: additive noise, multiplicative noise, and Poisson noise. The logarithm of the multiplicative noise can be processed as additive noise of Gaussian distribution. Different from the first two kinds of noise, Poisson noise generally appears in the case of very small illuminance and amplification with high power electronic circuits and obeys a Poisson distribution. Using variance stabilizing transformation [1–5] and Anscombe transformation [6–9], the transformed data can be regarded as additive noise of Gaussian distribution. However, such methods may cause some data loss during the conversion process. Thus, based on these transformations, the denoising and deblurring results need to be further improved. In recent years, many scholars have proposed algorithm models for Poissonian image deblurring. Poisson log-likelihood is inseparable in Poisson image deblurring. Many scholars [10–15] transform this problem into an equivalent constrained optimization problem through the split Bregman method and then use the augmented Lagrange method to solve this constrained optimization problem. Setzer et al. [12] developed a Poisson denoising model based on minimizing the total variation regularization term (TV) and the I-divergence [16] as a similarity term to restore blurred images. The TV regularization term is sensitive to the regularization parameter setting; thus, Yan et al. [17] proposed an improved spatially adaptive total variation regularization algorithm, which can automatically balance the regularization strength between different regions and retain more edge information in restored images, to solve this problem. Most TV regularization-based restoration models can effectively reduce the noise in flat regions, but large staircase effects are also introduced in flat regions, and fine details are not preserved in complex structural regions, which limits the practical application of the TV regularization term. To solve this problem, Fang et al. [13] developed a Poisson image deconvolution method based on framelet regularization, which aims to adaptively capture multiscale edge structures in images. To make full use of the sparse information and nonlocal information of the image, Shi et al. [18] proposed a latent image estimation method based on nonlocal total variation and framelet regularization constraints, which achieved smooth denoising and deblurring results while maintaining details and edges.

Generally, images corrupted by Poisson noise are accompanied by blur degradation. Therefore, the deblurring problem of Poisson images is not only a denoising problem but also a deblurring problem. In the process of image restoration, the prior information of degraded image plays a very important role [17–28]. In recent years, many scholars have used regularization priors to denoise or deblur images, such as dictionary learning [19], total variation [17, 18, 21, 22], nonlocal mean [26–28], and L₀ regularization prior [20]. In the field of image denoising, a nonlocal mean regularization term was proposed by Buades et al. [26, 27] in 2005 to remove image noise. Almahdi and Hardie [28] developed a new recursive nonlocal mean denoising algorithm. In 2017, Romano et al. [29] provided a new idea for removing noise; they provided an alternative, stronger, and more flexible framework for removing additive white Gaussian noise by proposing an explicit image-adaptive Laplacian-based regularization function, which can use the selected denoiser to define the regularization term, called Regularization by Denoising (RED). The advantage of RED is that it can flexibly select the denoising engine and use existing denoising algorithms to define regularization terms. When using RED to restore degraded images, the authors mainly focus on images contaminated by additive white Gaussian noise. They tested and proved that RED can achieve good restoration results as a regularization term to remove additive white Gaussian noise. Different from the image corrupted by Gaussian noise, the image corrupted by Poisson noise causes low image contrast and the image details to be covered. To make full use of the texture region information in the image and preserve the details of the image to the greatest extent while removing Poisson noise, we select the nonlocal mean denoiser as the regularization term of Poisson image denoising in the proposed method in this paper. In the field of image deblurring, Tang et al. [21] proposed a nonblind image deblurring method by local and nonlocal total variation models. Gradient priors are commonly used for image deblurring. However, in practical applications, if only the gradient prior is used, degraded images cannot be restored well. In [20], Pan et al. proposed an image deblurring method based on the L₀ regularization intensity and gradient prior. However, these deblurring methods all assume that noise is additive white Gaussian noise or impulse noise. To recover clear images from Poisson images effectively, we introduce the intensity prior of L₀ regularization and the prior of RED as explicit prior regularization terms in the proposed method.

With the success of deep networks in image processing, an increasing number of scholars have begun to use neural networks to solve the restoration problem of degraded images [22, 30–37]. Most scholars usually need the model to train a large number of data sample sets when using the deep CNN network to restore degraded images so that the model can learn the prior information of these data, thus restoring degraded images. Therefore, when using a neural network for image restoration, a large number of data sample sets need to be prepared first, and then the degraded image can be restored based on the trained model parameters. However, Ulyanov et al. [33] found in their research that the deep network structure is sufficient for capturing a large number of low-level image statistics before any learning. In other words, the only prior information when restoring degraded images is derived from the structure of the network. Based on this discovery, they proposed a method of learning implicit priors from a single degraded image by using a network, which is called DIP. The authors noted that an untrained CNN network was used to restore degraded images, while only a randomly initialized network was required. Inspired by this idea, Mataev et al. [32] proposed bringing in the concept of Regularization by Denoising (RED). They boost DIP by adding an explicit prior, which enriches the overall regularization effect. In the same year, Liu et al. [22] proposed combining implicit priors in DIP with a traditional TV regularization prior to improve the image quality of denoising or deblurring in DIP. However, these methods are all aimed at restoring degraded images contaminated by additive white Gaussian noise. It is still a challenge to denoise and deblur Poisson images using neural networks. At present, all blind Poisson image deconvolution methods have been proposed to use traditional explicit regularization methods to restore degraded images. Therefore, in this paper, we propose an algorithm for denoising and deblurring Poisson images by using neural networks. We combine an implicit regularization prior with two explicit regularization priors that are the prior of RED and the prior of L₀ regularization based on intensity and use a neural network to restore degraded images from Poisson images. Experimental results demonstrate that, compared with traditional methods, Poisson images restored by deep networks can effectively suppress Poisson noise while preserving the real image edge details and detailed texture information of degraded images.

2. Proposed methodology

2.1. RED

The regularization term plays an important role in image deconvolution. Compared with other regularization terms, RED [29] has the advantage that existing denoising algorithms can be flexibly used to define regularization terms. The regularization term by denoising, which is an explicit image-adaptive Laplacian-based regularization function, uses the selected denoiser to define the regularization term. RED, as a regularization function, is given by the following formula:

where is a degraded image, is the selected denoising engine, which is applied to degraded image , and the selection of the denoising engine is flexible. is the denoising residual. We found in the experimental research that the nonlocal mean denoising device can make full use of the texture region information in the degraded image, which can remove the Poisson noise while maintaining the details of the image features to the greatest extent. Therefore, we use the nonlocal mean denoiser as the regularization term of Poisson image denoising in the proposed method in this paper.

2.2. L₀ Regularization Term

Based on the observation of blurred images [20], the pixel intensity histogram of a blurred image is different from that of a clean image. In the experimental observation, it is found that the number of nonzero elements of pixel intensity in blurred images is large and the distribution is relatively dense. According to the pixel intensity property of blurred images, for blurred images, we have

where represents the number of nonzero values in image , and clear images and blurred images can be distinguished according to the criterion of the pixel value intensity distribution. Using this property of pixel value intensity distribution, clear images and blurred images can be distinguished in the process of image restoration, thus speeding up the convergence time of the algorithm. Therefore, the L₀ regularization term based on the pixel intensity prior is adopted as the regularization prior term in the proposed method in this paper. The result analysis of convergence of L₀ regularization is described in detail in Section 3.2.

2.3. DIP

In [33], the authors determined that the process of recovering degraded images in DIP is different from other CNN networks. It does not require a large amount of data-driven training but only requires the network to learn the implicit prior of degraded images in a randomly initialized neural network framework. The authors found that the neural network can learn the undamaged part of the image and then learn the damaged part of the image. For example, if the degraded image of white Gaussian noise is input to the network, the network will learn how to copy an image without noise first and then learn to copy the noise. Therefore, we can remove the network learning before the network learns to copy the noise so that we can obtain an image without noise. DIP defines the output of the randomly initialized neural network as

where represents the parameters of the network, z is a fixed random vector, and represents the parameterized form of the network. Then, the objective function of DIP is as follows:

where is a degraded operator and is a degraded image. The network consists of linear convolution, upsampling, and nonlinear activation functions. Based on the given degraded image and the observation model, the model parameters can be approximated to the maximum likelihood through iteration.

2.4. Proposed Model

In equation (6), to restore degraded images through a deep network, data items for restoring additive white Gaussian noise images cannot solve the problem of Poisson image deblurring well. To obtain good denoising results while preserving the image details, we introduce equation (6) into the objective function 2 for Poissonian image deblurring and use RED as the denoising regularization term in the Poisson image deblurring model. We take the L₀ norm of the latent clear image as the regularization constraint in the Poisson image deblurring model. Then, the objective function 2 can be written as follows:

where is the weight of RED and is the weight of the L₀ norm constrained regularization term.

To solve the inseparable Poisson log-likelihood problem in equation (7), we use the split Bregman method to introduce auxiliary variable and transform equation (7) into a constrained problem. In combination with equation (5), if the output of the neural network is introduced into equation (7) as a constraint condition, equation (7) becomes the following equation:

2.5. Optimal Estimation

To avoid the differentiation of the explicit denoising function, two auxiliary variables, and , are introduced by using the split Bregman method. The existence of the L₀ norm in the third term of equation (8) makes the solution somewhat difficult, the split Bregman method is used to introduce auxiliary variable , and equation (8) is changed to the following equation:

According to the split Bregman method, seven unknown variables , , , , , , and are iteratively optimized and updated. Variables , , , and are fixed, and variable can be solved by the following formula:

To solve equation (10), we use backpropagation to optimize update variable , where variables and are infinitely close to and , respectively.

Given fixed variables , , and , the point spread function can be solved by the following formula:

The update of variable is obtained by the following equation:

Given fixed variables , , and , variable can be solved by the following formula:

The update of variable , which can be solved by equation (13), is obtained by the following equation:

where is the value of in the th iteration and is the output of the neural network in the th iteration.

Given fixed variables , , and , variable can be solved by the following formula:

We use gradient descent to update :

where should be selected to ensure a decrease.

Given fixed variables , , and , auxiliary variable can be updated by the following formula:

Given fixed variables and , auxiliary variable can be updated by the following formula:

Given fixed variable , auxiliary variable can be updated by the following formula:

Algorithm 1 proposed in this paper is as follows:

Input: degraded image

Initialization: k = 0, ,  = 0,  = 0, , , , randomly

While is not converged do

    Update by equation (10);

    Update by equation (12);

    Update by equation (14);

    Update by equation (16);

    Update by equation (17);

    Update by equation (18);

    Update by equation (19);

End

Output: recovered clear image

Algorithm 1: Proposed method.

3. Experimental Results

In this section, we carry out experiments and image quality evaluation on simulated blurred images and real blurred images that are corrupted with Poisson noise and compare the experimental results with those of four methods: PID-Split [12], PIDSB-FA [13], PIDSB-NLFA [18], and DeepRED [32]. The experiment of the proposed method and the DeepRED method is implemented on a GTX 1080i GPU computer using the Python language in the Linux system. PID-Split, PIDSB-FA, and PIDSB-NLFA algorithms are tested by MATLAB 8a on a computer with an Intel(R) Core(TM) i5-7400 CPU and 8 GB RAM. In the section of parameter setting, we set the parameters of the PID-Split, PIDSB-FA, and PIDSB-NLFA methods according to the methods suggested in the original paper. For different Poisson degraded images, we set different parameters to obtain the best recovery results. The parameter settings in our proposed method are as follows: , , , , , . To evaluate the image quality, the Peak Signal-to-Noise Ratio (PSNR) and Visual Information Fidelity (VIF) are used in this section to evaluate the Poisson image recovered by each algorithm. Finally, the PSNR value of each algorithm is evaluated by using the “compare_psnr” function in the “skimage.measure” module in the Python language, and the VIF value of each algorithm is evaluated by using the “vifp_mscale” function in MATLAB. The higher the value of PSNR, the better the image recovery effect, the higher the value of VIF, and the higher the visual information fidelity of the image.

3.1. Simulated Poisson Image Denoising and Blurring

In this section, we compare the proposed method with the other four algorithms (PID-Split, PIDSB-FA, PIDSB-NLFA, and DeepRED) on five images with simulated Poisson image denoising and blurring experiments. The five images are Lena (size = 256 × 256), Cameraman (size = 256 × 256), Butterfly (size = 256 × 256), Zebra (size = 584 × 387), and House (size = 256 × 256), as shown in Figure 1.

(a)

(b)

(c)

(d)

(e)

Figure 1: Original images. (a) Lena (size = 256 × 256). (b) Cameraman (size = 256 × 256). (c) Butterfly (size = 256 × 256). (d) Zebra (size = 584 × 387). (e) House (size = 256 × 256).

In the simulated Poisson image experiment, we applied different levels of blur and noise to different images. The Gaussian blur kernel parameters we set for the Lena image of Figure 1(a) were (size = 5, σ = 1.6), (size = 15, σ = 1.6), and (size = 25, σ = 2). Three different Gaussian blur kernels were used to convolve the remaining four images, and the Gaussian blur kernels were set to (size = 5, σ = 1.6), (size = 15, σ = 2), and (size = 25, σ = 1.6). Then, the “skimage.util.random_noise” function of the image processing algorithm set in Python was used to add three different Poisson noise levels to the five images, and the noise levels “Imax” were set to 3,000, 4,000, and 5,000. Particularly, since the Butterfly image with an image size of 256 × 256 in Figure 1(c) was insensitive to noise levels of 5,000 when the Gaussian blur kernel was (size = 15, σ = 2), we applied noise levels of 2,000, 3,000, and 4,000 to this image.

In the experiment of deblurring simulated Poisson images with different noise levels and different blur kernel sizes, the image quality evaluation data recovered by the PID-Split, PIDSB-FA, PIDSB-NLFA, and DeepRED algorithms and the proposed method in this paper are shown in Tables 1 and 2. According to the image quality evaluation data in Tables 1 and 2, the PSNR value and VIF value of the proposed method in this paper were higher than those of the other four algorithms. We conducted a set of comparison experiments on the House image in Figure 1(e), and the results of the comparison experiments are shown in Figure 2. The House images have a large size of flat regions and a small amount of detailed information. In this group of comparative experiments, as shown in Figure 2(b), the House image was degraded by a Gaussian blur kernel of size 15 × 15 and standard deviation σ = 2 and polluted by Poisson noise (Imax = 3,000). As shown in Figure 2(c), although the image restored by the PID-Split algorithm suppressed noise well, it also caused staircase effects. The images recovered by the PIDSB-NLFA and PIDSB-FA algorithms are shown in Figures 2(d) and 2(e). Compared with Figure 2(c), the staircase effects were suppressed, and the noise in the flat regions was suppressed better. Figure 2(f) shows the result of restoration of the DeepRED, which retains more details but cannot remove Poisson noise well. The image restored by the proposed method in this paper is shown in Figure 2(g). Compared with the other four methods, it not only suppressed noise to the greatest extent but also had clearer details and the best visual perception.

Table 1: PSNR (dB) values of the five methods.

Image
Gaussian blur
Imax
Degraded image
PID-Split
PIDSB-FA
PIDSB-NLFA
DeepRED
Our method

Lena
Size = 5, σ = 1.6
3,000
24.84
25.06
25.22
25.57
25.80
26.91

4,000
24.87
25.07
25.25
25.62
25.82
26.94

5,000
24.90
25.11
25.29
25.68
25.85
27.11

Size = 15, σ = 1.6
3,000
24.36
25.01
25.12
25.26
25.81
26.33

4,000
24.38
25.03
25.16
25.28
25.92
26.47

5,000
24.40
25.07
25.20
25.29
26.09
26.57

Size = 25, σ = 2
3,000
22.99
23.70
24.44
24.55
24.57
24.61

4,000
23.00
23.72
24.49
24.57
24.59
24.64

5,000
23.02
23.75
24.51
24.59
24.61
24.66

Cameraman
Size = 5, σ = 1.6
3,000
23.63
25.38
26.68
26.73
27.12
27.46

4,000
23.66
25.51
26.73
26.75
27.20
28.05

5,000
23.67
25.69
26.85
26.92
27.40
28.14

Size = 15, σ = 2
3,000
21.73
23.39
24.02
24.09
24.24
24.59

4,000
21.75
23.43
24.15
24.17
24.38
24.64

5,000
21.76
23.48
24.20
24.23
24.47
24.71

Size = 25, σ = 1.6
3,000
22.83
24.39
25.29
25.37
25.51
26.29

4,000
22.84
24.49
25.32
25.40
25.74
26.30

5,000
22.88
24.66
25.35
25.42
25.92
26.50

Butterfly
Size = 5, σ = 1.6
3,000
22.88
24.50
25.23
25.52
25.56
25.95

4,000
22.90
24.63
25.26
25.55
25.62
26.01

5,000
22.91
24.78
25.27
25.58
25.66
26.12

Size = 15, σ = 2
2,000
20.57
21.96
22.99
23.29
23.47
23.98

3,000
20.60
22.04
23.03
23.47
23.59
24.05

4,000
20.63
22.22
23.07
23.57
23.69
24.13

Size = 25, σ = 1.6
3,000
21.81
23.17
23.96
24.33
24.46
24.81

4,000
21.83
23.33
23.99
24.36
24.52
24.89

5,000
21.84
23.48
24.00
24.39
24.59
24.98

Zebra
Size = 5, σ = 1.6
3,000
23.58
24.81
25.89
26.50
26.87
27.17

4,000
23.61
24.86
25.96
26.56
26.96
27.24

5,000
23.64
24.92
26.02
26.62
27.05
27.30

Size = 15, σ = 2
3,000
21.69
23.95
25.05
25.32
26.38
26.76

4,000
21.71
24.03
25.12
25.43
26.57
26.85

5,000
21.73
24.11
25.20
25.54
26.69
26.92

Size = 25, σ = 1.6
3,000
22.68
24.41
25.51
26.04
26.78
27.22

4,000
22.71
24.47
25.57
26.17
26.90
27.35

5,000
22.73
24.54
25.65
26.29
27.03
27.42

House
Size = 5, σ = 1.6
3,000
28.81
30.08
30.71
30.43
32.44
32.71

4,000
28.93
30.17
30.91
30.48
32.54
32.88

5,000
28.99
30.28
31.10
30.60
32.61
33.18

Size = 15, σ = 2
3,000
26.11
27.12
27.69
27.52
28.36
28.47

4,000
26.14
27.20
27.70
27.57
28.37
28.54

5,000
26.18
27.33
27.73
27.62
28.39
28.62

Size = 25, σ = 1.6
3,000
27.71
28.70
29.61
29.41
30.29
30.87

4,000
27.84
28.78
29.73
29.49
30.47
31.08

5,000
27.85
28.87
29.81
29.56
30.56
31.15

Table 2: VIF values of the five methods.

Image
Gaussian blur
Imax
Degraded image
PID-Split
PIDSB-FA
PIDSB-NLFA
DeepRED
Our method

Lena
Size = 5, σ = 1.6
3,000
0.4482
0.4310
0.4513
0.5147
0.5182
0.5779

4,000
0.4553
0.4327
0.4559
0.5122
0.5131
0.5925

5,000
0.4603
0.4369
0.4614
0.5147
0.5136
0.5984

Size = 15, σ = 1.6
3,000
0.4026
0.4307
0.4298
0.4938
0.5178
0.5585

4,000
0.4074
0.4338
0.4346
0.4942
0.5282
0.5683

5,000
0.4116
0.4393
0.4404
0.4946
0.5410
0.5745

Size = 25, σ = 2
3,000
0.3484
0.3946
0.4761
0.4949
0.5130
0.5290

4,000
0.3531
0.3973
0.4812
0.4962
0.5116
0.5297

5,000
0.3566
0.4003
0.4842
0.4969
0.5290
0.5373

Cameraman
Size = 5, σ = 1.6
3,000
0.3086
0.4045
0.4332
0.4336
0.4513
0.4533

4,000
0.3107
0.4133
0.4436
0.4395
0.4563
0.4675

5,000
0.3140
0.4233
0.4450
0.4458
0.4573
0.4723

Size = 15, σ = 2
3,000
0.2255
0.3132
0.3260
0.3860
0.3743
0.3981

4,000
0.2280
0.3166
0.3347
0.3867
0.3890
0.4017

5,000
0.2301
0.3195
0.3372
0.3861
0.3950
0.4105

Size = 25, σ = 1.6
3,000
0.2705
0.3730
0.3854
0.3860
0.3996
0.4295

4,000
0.2733
0.3805
0.3894
0.3876
0.4040
0.4407

5,000
0.2760
0.3910
0.3903
0.3861
0.4185
0.4447

Butterfly
Size = 5, σ = 1.6
3,000
0.3940
0.4709
0.5535
0.5597
0.5768
0.5887

4,000
0.3967
0.4815
0.5577
0.5614
0.5909
0.6028

5,000
0.3982
0.4933
0.5602
0.5635
0.6074
0.6102

Size = 15, σ = 2
2,000
0.2925
0.3818
0.4622
0.4717
0.5018
0.5124

3,000
0.2957
0.3877
0.4661
0.4849
0.5228
0.5284

4,000
0.2991
0.4002
0.4713
0.4967
0.5335
0.5390

Size = 25, σ = 1.6
3,000
0.3471
0.4119
0.5117
0.5292
0.5478
0.5613

4,000
0.3492
0.4189
0.5142
0.5325
0.5488
0.5630

5,000
0.3507
0.4263
0.5163
0.5358
0.5660
0.5800

Zebra
Size = 5, σ = 1.6
3,000
0.3913
0.4007
0.4905
0.5192
0.4913
0.5206

4,000
0.3979
0.4038
0.4957
0.5278
0.5106
0.5302

5,000
0.4020
0.4070
.04992
0.5109
0.5179
0.5355

Size = 15, σ = 2
3,000
0.2832
0.3348
0.3982
0.4094
0.4295
0.4490

4,000
0.2878
0.3387
0.4026
0.4155
0.4394
0.4590

5,000
0.2911
0.3429
0.4071
0.4219
0.4448
0.4622

Size = 25, σ = 1.6
3,000
0.3402
0.3697
0.4066
0.4712
0.4763
0.4933

4,000
0.3463
0.3731
0.4100
0.4789
0.4850
0.5016

5,000
0.3504
0.3768
0.4136
0.4857
0.4959
0.5082

House
Size = 5, σ = 1.6
3,000
0.3842
0.4028
0.4472
0.5040
0.5062
0.5156

4,000
0.3898
0.4081
0.4563
0.5049
0.5103
0.5172

5,000
0.3924
0.4127
0.4659
0.5047
0.5133
0.5320

Size = 15, σ = 2
3,000
0.2914
0.3620
0.4378
04357
0.4334
0.4559

4,000
0.2929
0.3688
0.4380
0.4373
0.4370
0.4680

5,000
0.2946
0.3753
0.4404
0.4375
0.4389
0.4749

Size = 25, σ = 1.6
3,000
0.3429
0.3743
0.4653
0.4719
0.4522
0.4946

4,000
0.3496
0.3783
0.4741
0.4724
0.4686
0.5037

5,000
0.3495
0.3823
0.4752
0.4663
0.4703
0.5091

Figure 2: Restoration of the House image. (a) Ground truth image. (b) Degraded image blurred by Gaussian kernel of size 15 × 15, σ = 2 and corrupted by Poisson noise with Imax = 3,000. (c) Restored image by the PID-Split method. (d) Restored image by the PIDSB-FA method. (e) Restored image by the PIDSB-NLFA method. (f) Restored image by the DeepRED method. (g) Restored image by the proposed method. (h) Close-up views (close-up views of (a)–(g) image regions extracted from this example, respectively).

Figure 3 shows a set of comparative results for Zebra images in Figure 1(d). The size of Zebra image is 584 × 387. Figure 3(b) is an image degraded by a Gaussian blur kernel with a size of 25 × 25 and a standard deviation of 1.6 and Poisson noise with a size of 3,000. As can be seen from Figure 3(a), the zebra has a large size of flat regions and the grassland contains considerable detailed information. As seen in Figure 3(g), the proposed method in this paper not only suppressed noise but also preserved considerable detailed information. Figure 3(c) shows that the image restored by the PID-Split algorithm was too smooth and had staircase effects. Figures 3(d)–3(f) show the images recovered by the PIDSB-NLFA, PIDSB-FA, and DeepRED algorithms, respectively. Compared with Figure 3(c), they suppressed the staircase effects while retaining some details. Compared with the other four method, the proposed method in this paper not only suppressed the staircase effects but also preserved the details of the image to the greatest extent and was clearer.

Figure 3: Restoration of the Zebra image. (a) Ground truth image. (b) Degraded image blurred by Gaussian kernel of size 25 × 25, σ = 1.6 and corrupted by Poisson noise with Imax = 3,000. (c) Restored image by the PID-Split method. (d) Restored image by the PIDSB-FA method. (e) Restored image by the PIDSB-NLFA method. (f) Restored image by the DeepRED method. (g) Restored image by the proposed method. (h) Close-up views (close-up views of (a)–(g) image regions extracted from this example, respectively).

In addition, the Butterfly image in Figure 1(c) was degraded by the Gaussian blur kernel (size = 5 × 5, σ = 1.6) and then corrupted by Poisson noise (Imax = 5,000). As shown in Figure 4, compared with the images restored by the other four algorithms, the proposed method in this paper suppressed noise and restored clear images to the greatest extent. Figure 4(g) restored by our method looks natural and preserved the most detailed information of the images.

Figure 4: Restoration of the Butterfly image. (a) Ground truth image. (b) Degraded image blurred by Gaussian kernel of size 5 × 5, σ = 1.6 and corrupted by Poisson noise with Imax = 5,000. (c) Restored image by the PID-Split method. (d) Restored image by the PIDSB-FA method. (e) Restored image by the PIDSB-NLFA method. (f) Restored image by the DeepRED method. (g) Restored image by the proposed method. (h) Close-up views (close-up views of (a)–(g) image regions extracted from this example, respectively).

3.2. Convergence Analysis

The explicit regularization prior term adopts the combination of the intensity-based L₀ regularization term and RED. The purpose is to be able to use the pixel intensity distribution of the blurred image and the clear image to distinguish the restored image pixels while effectively filtering the image pixels and removing the Poisson noise of the degraded image. To verify the effectiveness of the L₀ regularization prior, we used the method with the L₀ norm regularization term and the method without the L₀ norm to carry out simulation and comparison experiments on the House images of Figure 1(c) in Section 3.1. The experimental results are shown in Figure 5. Figure 5(a) shows the PSNR values of the image restored by the two algorithms. It can be clearly seen that the method with the L₀ regularization term has the characteristic of fast convergence. With the increase in iteration times, the restored image reached a higher PSNR. Figure 5(b) shows the enlarged graph of the abscissa range (0∼500) in Figure 5(a). Figure 5(c) shows the loss of the objective function, and Figure 5(d) shows an enlarged graph with abscissa (0∼200) in Figure 5(c). It can be seen that the method with the L₀ regularization term also achieved a smaller loss value.

(a)

(b)

(c)

(d)

Figure 5: PSNR values and loss comparison. (a) PSNR. (b) PSNE (zoomed in). (c) Loss. (d) Loss (zoomed in).

3.3. Denoising and Deblurring of Real Poisson Images

This section is an experimental comparison of denoising and deblurring of real Poisson images. Tests were carried out on four real astronomically degraded images. To evaluate the real Poisson image quality, we used the Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) to evaluate the real data. The smaller the value of BRISQUE, the better the image quality. The image quality evaluation data recovered by the PID-Split, PIDSB-FA, PIDSB-NLFA, and DeepRED algorithms and the proposed method in this paper are shown in Table 3.

Table 3: BRISQUE values of the five methods.

Image
PID-Split
PIDSB-FA
PIDSB-NLFA
DeepRED
Our method

Saturn
45.13
43.31
48.96
41.42
39.22

Moon
49.81
43.34
47.75
39.28
38.56

Docking
48.65
49.14
49.05
41.95
39.85

Phoebe
36.61
32.94
31.26
30.44
24.25

Figure 6(a) shows a satellite image, and Figure 6(b) shows an image restored by the PID-Split algorithm; the restored image is too smooth. Figures 6(c)–6(e) show the images recovered by the PIDSB-FA, PIDSB-NLFA, and DeepRED algorithms, respectively. Although the staircase effects were suppressed to a large extent, ring effects exist at the edge. The image restored by the proposed method in this paper is shown in Figure 6(f). The restored image not only restored a large amount of detailed information and texture information but also suppressed the ring effects at the edge while suppressing the step effect. The image restored by the proposed method looks more natural.

Figure 6: Restoration of real blurred image. (a) Saturn truth image. (b) Restored image by the PID-Split method. (c) Restored image by the PIDSB-FA method. (d) Restored image by the PIDSB-NLFA method. (e) Restored image by the DeepRED method. (f) Restored image by the proposed method. (g) Close-up views (close-up views of (a)–(f) image regions extracted from this example, respectively).

The second real image is a moon image, as shown in Figure 7(a). Figures 7(b)–7(e) show the images restored by the PID-Split, PIDSB-FA, PIDSB-NLFA, and DeepRED algorithms, respectively. It can be seen from these three images that the images recovered by these methods had ringing effects of the edge, and the flat areas were too smooth. Figure 7(f) shows the image restored by the proposed method in this paper. It can be seen that the image restored by the proposed method in this paper retained more detailed information while suppressing noise and ringing effects.

Figure 7: Restoration of real blurred image. (a) Moon truth image. (b) Restored image by the PID-Split method. (c) Restored image by the PIDSB-FA method. (d) Restored image by the PIDSB-NLFA method. (e) Restored image by the DeepRED method. (f) Restored image by the proposed method. (g) Close-up views (close-up views of (a)–(f) image regions extracted from this example, respectively).

Figure 8 shows a set of comparative results for the docking images of Shenzhou-9 and Tiangong-1. Restored images by RLTV, PID-Split, PIDSB-FA, DeepRED, and the proposed method are shown in Figures 8(b)–8(f), respectively. The image restored by the proposed method suppressed the ring effects at the edge and retained more detailed information. Figure 9(a) shows the Phoebe image. Comparing the restored images by the other four methods shown in Figures 9(b)–9(e), respectively, the image restored by the proposed method in this paper looks clear with fine details.

Figure 8: Restoration of real blurred image. (a) The docking truth image of Shenzhou-9 and Tiangong-1. (b) Restored image by the PID-Split method. (c) Restored image by the PIDSB-FA method. (d) Restored image by the PIDSB-NLFA method. (e) Restored image by the DeepRED method. (f) Restored image by the proposed method. (g) Close-up views (close-up views of (a)–(f) image regions extracted from this example, respectively).

Figure 9: Restoration of real blurred image. (a) Phoebe truth image. (b) Restored image by the PID-Split method. (c) Restored image by the PIDSB-FA method. (d) Restored image by the PIDSB-NLFA method. (e) Restored image by the DeepRED method. (f) Restored image by the proposed method. (g) Close-up views (close-up views of (a)–(f) image regions extracted from this example, respectively).

4. Conclusions

This paper proposed an algorithm for denoising and deblurring Poisson images by using neural networks. We combine an implicit regularization prior with two explicit regularization priors that are the prior of RED and the prior of L₀ regularization based on intensity and use a neural network to restore degraded images from Poisson images. In the proposed method, it has been proved that Poisson images restored by deep networks can effectively suppress Poisson noise. Meanwhile, the method with the L₀ regularization term has the characteristic of fast convergence.

Data Availability

The data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Acknowledgments

This work was supported by a project of the National Science Foundation of China (61701353, 61801337, and 61671337).

References

J. Boulanger, C. Kervrann, P. Bouthemy, P. Elbau, J.-B. Sibarita, and J. Salamero, “Patch-based nonlocal functional for denoising fluorescence microscopy image sequences,” IEEE Transactions on Medical Imaging, vol. 29, no. 2, pp. 442–454, 2010.

B. Zhang, M. J. Fadili, and J. L. Starck, “Multi-scale variance stabilizing transform for multi-dimensional Poisson count image denoising,” in Proceedings of the 2006 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1329–1332, Toulouse, France, May 2006.

B. Zhang, M. J. Fadili, J. L. Starck et al., “Multiscale variance-stabilizing transform for mixed-Poisson-Gaussian processes and its applications in bioimaging,” in Proceedings of the 2007 IEEE International Conference on Image Processing, pp. 233–236, San Antonio, TX, USA, October 2007.

M. Makitalo and A. Foi, “A closed-form approximation of the exact unbiased inverse of the Anscombe variance-stabilizing transformation,” IEEE Transactions on Image Processing, vol. 20, no. 9, pp. 2697-2698, 2011.

B. Zhang, J. M. Fadili, and J. L. Starck, “Wavelets, ridgelets, and curvelets for Poisson noise removal,” IEEE Transactions on Image Processing: A Publication of the IEEE Signal Processing Society, vol. 17, no. 7, pp. 1093–1108, 2008.

M. Makitalo and A. Foi, “Optimal inversion of the generalized Anscombe transformation for Poisson-Gaussian noise,” IEEE Transactions on Image Processing, vol. 22, no. 1, pp. 91–103, 2013.

F. J. Anscombe, “The transformation of Poisson, binomial and negative- binomial data,” Biometrika, vol. 35, no. 3-4, pp. 246–254, 1948.

R. Liu, Z. Lin, D. L. T. Fernando, and Z. Su, “Fixed-rank representation for unsupervised visual learning,” in Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 598–605, Providence, RI, USA, June 2012.

M. Makitalo and A. Foi, “Optimal inversion of the Anscombe transformation in low-count Poisson image denoising,” IEEE Transactions on Image Processing, vol. 20, no. 1, pp. 99–109, 2011.

M. A. T. Figueiredo and J. M. Bioucas-Dias, “Deconvolution of Poissonian images using variable splitting and augmented Lagrangian optimization,” in Proceedings of the 2009 IEEE Workshop on Statistical Signal Processing, pp. 733–736, Cardiff, UK, August 2009.

M. A. T. Figueiredo, J. M. Bioucas-Dias, and M. V. Afonso, “Fast frame-based image deconvolution using variable splitting and constrained optimization,” in Proceedings of the 2009 IEEE Workshop on Statistical Signal Processing, pp. 109–112, Cardiff, UK, August 2009.

S. Setzer, G. Steidl, and T. Teuber, “Deblurring Poissonian images by split Bregman techniques,” Journal of Visual Communication and Image Representation, vol. 21, no. 3, pp. 193–199, 2010.

H. Fang, H. Yan, Y. Liu, and Y. Chang, “Blind Poissonian images deconvolution with framelet regularization,” Optics Letters, vol. 38, no. 4, pp. 389–391, 2013.

D. Q. Chen and L. Z. Cheng, “Spatially adapted regularization parameter selection based on the local discrepancy function for Poissonian image deblurring,” Inverse Problems, vol. 28, no. 1, Article ID 015004-1, 2012.

M. A. T. Figueiredo and J. M. Bioucas-Dias, “Restoration of poissonian images using alternating direction optimization,” IEEE Transactions on Image Processing, vol. 19, no. 12, pp. 3133–3145, 2010.

L. A. Shepp and Y. Vardi, “Maximum likelihood reconstruction for emission tomography,” IEEE Transactions on Medical Imaging, vol. 1, no. 2, pp. 113–122, 2007.

L. Yan, H. Fang, and S. Zhong, “Blind image deconvolution with spatially adaptive total variation regularization,” Optics Letters, vol. 37, no. 14, pp. 2778–2780, 2012.

Y. Shi, J. Song, and X. Hua, “Poissonian image deblurring method by non-local total variation and framelet regularization constraint,” Computers & Electrical Engineering, vol. 62, pp. 319–329, 2017.

M. Elad and M. Aharon, “Image denoising via sparse and redundant representations over learned dictionaries,” IEEE Transactions on Image Processing, vol. 15, no. 12, pp. 3736–3745, 2006.

J. Pan, Z. Hu, Z. Su, and M. H. Yang, “Deblurring text images via L₀ -regularized intensity and gradient prior,” in Proceedings of the 2014 International Conference on Computer Vision and Pattern Recogintion, pp. 2901–2908, Columbus, OH, USA, June 2014.

S. Tang, W. Gong, W. Li, and W. Wang, “Non-blind image deblurring method by local and nonlocal total variation models,” Signal Processing, vol. 94, pp. 339–349, 2014.

J. Liu, Y. Sun, X. Xu, and U. S. Kamilov, “Image restoration using total variation regularized deep image prior,” in Proceedings of the 2019 International Conference on Acoustics, Speech and Signal Processing, pp. 7715–7719, Brighton, UK, May 2019.

H. Hong, X. Hua, X. Zhang, and Y. Shi, “Multi-frame real image restoration based on double loops with alternative maximum likelihood estimation,” Signal, Image and Video Processing, vol. 10, no. 8, pp. 1489–1495, 2016.

H. Hong, L. Li, I. k. Park, and T. Zhang, “Universal deblurring method for real images using transition region,” Optical Engineering, vol. 51, no. 4, 2012.

H. Hong, Restoration Method and Application for MultispectralImage in Object Detection, National Defense Industry Press, Beijing, China, 2017.

A. Buades, B. Coll, and J. M. Morel, “A non-local algorithm for image denoising,” in Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 60–65, San Diego, CA, USA, June 2005.

A. Buades, B. Coll, and J. M. Morel, “A review of image denoising algorithms, with a new one,” Multiscale Modeling & Simulation, vol. 4, no. 2, pp. 490–530, 2005.

R. Almahdi and R. C. Hardie, “Recursive non-local means filter for video denoising,” Eurasip Journal on Image and Video Processing, vol. 2017, no. 1, p. 29, 2017.

Y. Romano, M. Elad, and P. Milanfar, “The little engine that could: regularization by denoising (RED),” SIAM Journal on Imaging Sciences, vol. 10, no. 4, pp. 1804–1844, 2017.

K. Zhang, W. Zuo, S. Gu, and L. Zhang, “Learning deep CNN denoiser prior for image restoration,” in Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3929–3938, Honolulu, HI, USA, July 2017.

X. Xu, J. Pan, Y.-J. Zhang, and M.-H. Yang, “Motion blur kernel estimation via deep learning,” IEEE Transactions on Image Processing, vol. 27, no. 1, pp. 194–205, 2018.

G. Mataev, M. Elad, and P. Milanfar, “DeepRED: deep image prior powered by RED,” 2019, https://arxiv.org/abs/1903.10176.

D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” in Proceedings of the 2018 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, June 2018.

K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising,” IEEE Transactions on Image Processing, vol. 26, no. 7, pp. 3142–3155, 2017.

K. Zhang, W. Zuo, and L. Zhang, “Deep plug-and-play super-resolution for arbitrary blur kernels,” in Proceedings of the 2019 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1671–1681, Long Beach, CA, USA, June 2019.

Z. Huang, Y. Zhang, Q. Li et al., “Joint analysis and weighted synthesis sparsity priors for simultaneous denoising and destriping optical remote sensing images,” IEEE Transactions on Geoscience and Remote Sensing, pp. 1–25, 2020.

Z. Huang, Y. Zhang, Q. Li et al., “Unidirectional variation and deep CNN denoiser priors for simultaneously destriping and denoising optical remote sensing images,” International Journal of Remote Sensing, vol. 40, no. 15, pp. 5737–5748, 2019.