Blind Poissonian Image Deblurring Regularized by a Denoiser Constraint and Deep Image Prior

+e denoising and deblurring of Poisson images are opposite inverse problems. Single image deblurring methods are sensitive to image noise. A single noise filter can effectively remove noise in advance, but it also damages blurred information. To simultaneously solve the denoising and deblurring of Poissonian images better, we learn the implicit deep image prior from a single degraded image and use the denoiser as a regularization term to constrain the latent clear image. Combined with the explicit L0 regularization prior of the image, the denoising and deblurring model of the Poisson image is established.+en, the split Bregman iteration strategy is used to optimize the point spread function estimation and latent clear image estimation. +e experimental results demonstrate that the proposed method achieves good restoration results on a series of simulated and real blurred images with Poisson noise.


Introduction
e process of image acquisition in electron microscope imaging, astronomical imaging, and medical imaging is inevitably affected by environmental factors, which often cause the captured image to be disturbed by Poisson noise and blur degradation. e image degradation process can be modeled as follows: where y is the observed degraded image, P represents the process in which the image is corrupted by Poisson noise, H represents the point spread function, and * is the convolution operator. e problem of blind Poisson image deconvolution is to recover clear images and estimate the point spread function from the degraded image. e denoising and deblurring of Poisson images are opposite inverse problems. Single deblurring methods are sensitive to noise, and even a small amount of noise leads to biased point spread function estimation. Although the noise filter can effectively remove noise in advance, it also damages the blurred information and introduces more serious blur, which leads to amplified and biased point spread function estimation. To suppress Poisson noise and restore the blurred image simultaneously, we impose the denoiser as a regularization term to constrain the latent clear image, combined with an implicit deep image prior and an explicit L 0 regularization prior of the image. en, the energy functional regularization framework for the latent clear image and point spread function is modeled as follows:

E(x, H) � min
x,H 〈1, Hx − y log Hx〉 + R(x) + K(H), (2) where 〈1, Hx − y log Hx〉 is the data term, 1 represents a vector with all elements equal to 1, x is the latent clear image, which usually ensures that the clear image does not contain negative gray values, H is the point spread function, R(x) is the regularization constraint term of the latent clear image, and K(H) is the regularization constraint term of a point spread function. Among them, regularization constraint terms R(x) and K(H) play a vital role in the restoration of latent clear images and the estimation of point spread functions.

Relevant Work.
Poisson image denoising and blurring is an image deconvolution problem. Image noise can be divided into three categories: additive noise, multiplicative noise, and Poisson noise. e logarithm of the multiplicative noise can be processed as additive noise of Gaussian distribution. Different from the first two kinds of noise, Poisson noise generally appears in the case of very small illuminance and amplification with high power electronic circuits and obeys a Poisson distribution. Using variance stabilizing transformation [1][2][3][4][5] and Anscombe transformation [6][7][8][9], the transformed data can be regarded as additive noise of Gaussian distribution. However, such methods may cause some data loss during the conversion process. us, based on these transformations, the denoising and deblurring results need to be further improved. In recent years, many scholars have proposed algorithm models for Poissonian image deblurring. Poisson log-likelihood is inseparable in Poisson image deblurring. Many scholars [10][11][12][13][14][15] transform this problem into an equivalent constrained optimization problem through the split Bregman method and then use the augmented Lagrange method to solve this constrained optimization problem. Setzer et al. [12] developed a Poisson denoising model based on minimizing the total variation regularization term (TV) and the I-divergence [16] as a similarity term to restore blurred images. e TV regularization term is sensitive to the regularization parameter setting; thus, Yan et al. [17] proposed an improved spatially adaptive total variation regularization algorithm, which can automatically balance the regularization strength between different regions and retain more edge information in restored images, to solve this problem. Most TV regularization-based restoration models can effectively reduce the noise in flat regions, but large staircase effects are also introduced in flat regions, and fine details are not preserved in complex structural regions, which limits the practical application of the TV regularization term. To solve this problem, Fang et al. [13] developed a Poisson image deconvolution method based on framelet regularization, which aims to adaptively capture multiscale edge structures in images. To make full use of the sparse information and nonlocal information of the image, Shi et al. [18] proposed a latent image estimation method based on nonlocal total variation and framelet regularization constraints, which achieved smooth denoising and deblurring results while maintaining details and edges.
Generally, images corrupted by Poisson noise are accompanied by blur degradation. erefore, the deblurring problem of Poisson images is not only a denoising problem but also a deblurring problem. In the process of image restoration, the prior information of degraded image plays a very important role [17][18][19][20][21][22][23][24][25][26][27][28]. In recent years, many scholars have used regularization priors to denoise or deblur images, such as dictionary learning [19], total variation [17,18,21,22], nonlocal mean [26][27][28], and L 0 regularization prior [20]. In the field of image denoising, a nonlocal mean regularization term was proposed by Buades et al. [26,27] in 2005 to remove image noise. Almahdi and Hardie [28] developed a new recursive nonlocal mean denoising algorithm. In 2017, Romano et al. [29] provided a new idea for removing noise; they provided an alternative, stronger, and more flexible framework for removing additive white Gaussian noise by proposing an explicit image-adaptive Laplacian-based regularization function, which can use the selected denoiser to define the regularization term, called Regularization by Denoising (RED). e advantage of RED is that it can flexibly select the denoising engine and use existing denoising algorithms to define regularization terms. When using RED to restore degraded images, the authors mainly focus on images contaminated by additive white Gaussian noise. ey tested and proved that RED can achieve good restoration results as a regularization term to remove additive white Gaussian noise. Different from the image corrupted by Gaussian noise, the image corrupted by Poisson noise causes low image contrast and the image details to be covered. To make full use of the texture region information in the image and preserve the details of the image to the greatest extent while removing Poisson noise, we select the nonlocal mean denoiser as the regularization term of Poisson image denoising in the proposed method in this paper. In the field of image deblurring, Tang et al. [21] proposed a nonblind image deblurring method by local and nonlocal total variation models. Gradient priors are commonly used for image deblurring. However, in practical applications, if only the gradient prior is used, degraded images cannot be restored well. In [20], Pan et al. proposed an image deblurring method based on the L 0 regularization intensity and gradient prior. However, these deblurring methods all assume that noise is additive white Gaussian noise or impulse noise. To recover clear images from Poisson images effectively, we introduce the intensity prior of L 0 regularization and the prior of RED as explicit prior regularization terms in the proposed method.
With the success of deep networks in image processing, an increasing number of scholars have begun to use neural networks to solve the restoration problem of degraded images [22,[30][31][32][33][34][35][36][37]. Most scholars usually need the model to train a large number of data sample sets when using the deep CNN network to restore degraded images so that the model can learn the prior information of these data, thus restoring degraded images. erefore, when using a neural network for image restoration, a large number of data sample sets need to be prepared first, and then the degraded image can be restored based on the trained model parameters. However, Ulyanov et al. [33] found in their research that the deep network structure is sufficient for capturing a large number of low-level image statistics before any learning. In other words, the only prior information when restoring degraded images is derived from the structure of the network. Based on this discovery, they proposed a method of learning implicit priors from a single degraded image by using a network, which is called DIP. e authors noted that an untrained CNN network was used to restore degraded images, while only a randomly initialized network was required. Inspired by this idea, Mataev et al. [32] proposed bringing in the concept of Regularization by Denoising (RED). ey boost DIP by adding an explicit prior, which enriches the overall regularization effect. In the same year, Liu et al. [22] proposed combining implicit priors in DIP with a traditional TV regularization prior to improve the image quality of denoising or deblurring in DIP. However, these methods are all aimed at restoring degraded images contaminated by additive white Gaussian noise. It is still a challenge to denoise and deblur Poisson images using neural networks. At present, all blind Poisson image deconvolution methods have been proposed to use traditional explicit regularization methods to restore degraded images. erefore, in this paper, we propose an algorithm for denoising and deblurring Poisson images by using neural networks. We combine an implicit regularization prior with two explicit regularization priors that are the prior of RED and the prior of L 0 regularization based on intensity and use a neural network to restore degraded images from Poisson images. Experimental results demonstrate that, compared with traditional methods, Poisson images restored by deep networks can effectively suppress Poisson noise while preserving the real image edge details and detailed texture information of degraded images.

RED.
e regularization term plays an important role in image deconvolution. Compared with other regularization terms, RED [29] has the advantage that existing denoising algorithms can be flexibly used to define regularization terms. e regularization term by denoising, which is an explicit image-adaptive Laplacian-based regularization function, uses the selected denoiser to define the regularization term. RED, as a regularization function, is given by the following formula: where x is a degraded image, f(·) is the selected denoising engine, which is applied to degraded image x, and the selection of the denoising engine is flexible. x − f(x) is the denoising residual. We found in the experimental research that the nonlocal mean denoising device can make full use of the texture region information in the degraded image, which can remove the Poisson noise while maintaining the details of the image features to the greatest extent. erefore, we use the nonlocal mean denoiser as the regularization term of Poisson image denoising in the proposed method in this paper.

L 0 Regularization
Term. Based on the observation of blurred images [20], the pixel intensity histogram of a blurred image is different from that of a clean image. In the experimental observation, it is found that the number of nonzero elements of pixel intensity in blurred images is large and the distribution is relatively dense. According to the pixel intensity property of blurred images, for blurred images, we have where ‖x‖ 0 represents the number of nonzero values in image x, and clear images and blurred images can be distinguished according to the criterion of the pixel value intensity distribution. Using this property of pixel value intensity distribution, clear images and blurred images can be distinguished in the process of image restoration, thus speeding up the convergence time of the algorithm. erefore, the L 0 regularization term based on the pixel intensity prior is adopted as the regularization prior term in the proposed method in this paper. e result analysis of convergence of L 0 regularization is described in detail in Section 3.2.

DIP.
In [33], the authors determined that the process of recovering degraded images in DIP is different from other CNN networks. It does not require a large amount of datadriven training but only requires the network to learn the implicit prior of degraded images in a randomly initialized neural network framework. e authors found that the neural network can learn the undamaged part of the image and then learn the damaged part of the image. For example, if the degraded image of white Gaussian noise is input to the network, the network will learn how to copy an image without noise first and then learn to copy the noise. erefore, we can remove the network learning before the network learns to copy the noise so that we can obtain an image without noise. DIP defines the output of the randomly initialized neural network as where θ represents the parameters of the network, z is a fixed random vector, and F θ (z) represents the parameterized form of the network. en, the objective function of DIP is as follows: where H is a degraded operator and x 0 is a degraded image. e network consists of linear convolution, upsampling, and nonlinear activation functions. Based on the given degraded image and the observation model, the model parameters can be approximated to the maximum likelihood through iteration.

Proposed Model.
In equation (6), to restore degraded images through a deep network, data items for restoring additive white Gaussian noise images cannot solve the problem of Poisson image deblurring well. To obtain good denoising results while preserving the image details, we introduce equation (6)  en, the objective function 2 can be written as follows: where λ is the weight of RED and β is the weight of the L 0 norm constrained regularization term.
To solve the inseparable Poisson log-likelihood problem in equation (7), we use the split Bregman method to introduce auxiliary variable d 1 � Hx and transform equation (7) into a constrained problem. In combination with equation (5), if the output x � F θ (z) of the neural network is introduced into equation (7) as a constraint condition, equation (7) becomes the following equation:

Optimal Estimation.
To avoid the differentiation of the explicit denoising function, two auxiliary variables, u 1 and u 2 , are introduced by using the split Bregman method. e existence of the L 0 norm in the third term of equation (8) makes the solution somewhat difficult, the split Bregman method is used to introduce auxiliary variable v, and equation (8) is changed to the following equation: According to the split Bregman method, seven unknown variables d 1 , θ, x, u 1 , u 2 , v, and H are iteratively optimized and updated. Variables d 1 , u 1 , x, and H are fixed, and variable θ can be solved by the following formula: To solve equation (10), we use backpropagation to optimize update variable θ, where variables u 1 and u 2 are infinitely close to d 1 − HF θ (z) and x − F θ (z), respectively.
Given fixed variables d 1 , u 1 , and θ, the point spread function H can be solved by the following formula: e update of variable H k+1 is obtained by the following equation: Given fixed variables θ, u 1 , and H, variable d 1 can be solved by the following formula: e update of variable d k+1 1 , which can be solved by equation (13), is obtained by the following equation: where d k+1 1 is the value of d 1 in the k + 1th iteration and F θ (z) k+1 is the output of the neural network in the k + 1th iteration.
Given fixed variables θ, u 2 , and v, variable x can be solved by the following formula: We use gradient descent to update x k+1 : where c should be selected to ensure a decrease. Given fixed variables d 1 , H, and θ, auxiliary variable u 1 k+1 can be updated by the following formula: Given fixed variables x and θ, auxiliary variable u k+1 2 can be updated by the following formula: Given fixed variable x, auxiliary variable v k+1 can be updated by the following formula: Algorithm 1 proposed in this paper is as follows:

Experimental Results
In this section, we carry out experiments and image quality evaluation on simulated blurred images and real blurred images that are corrupted with Poisson noise and compare the experimental results with those of four methods: PID- Update θ k+1 by equation (10); Update H k+1 by equation (12); Update d k+1 1 by equation (14); Update x k+1 by equation (16); Update u k+1 1 by equation (17); Update u k+1 2 by equation (18)     Split [12], PIDSB-FA [13], PIDSB-NLFA [18], and DeepRED [32]. e experiment of the proposed method and the DeepRED method is implemented on a GTX 1080i GPU computer using the Python language in the Linux system. PID-Split, PIDSB-FA, and PIDSB-NLFA algorithms are tested by MATLAB 8a on a computer with an Intel(R) Core(TM) i5-7400 CPU and 8 GB RAM. In the section of parameter setting, we set the parameters of the PID-Split, PIDSB-FA, and PIDSB-NLFA methods according to the methods suggested in the original paper. For different Poisson degraded images, we set different parameters to obtain the best recovery results. e parameter settings in our proposed method are as follows: μ � 0.01, c � 0.04, λ � 0.02, α � 0.01, β � 0.002, τ � 0.04. To evaluate the image quality, the Peak Signal-to-Noise Ratio (PSNR) and Visual Information Fidelity (VIF) are used in this section to evaluate the Poisson image recovered by each algorithm. Finally, the PSNR value of each algorithm  is evaluated by using the "compare_psnr" function in the "skimage.measure" module in the Python language, and the VIF value of each algorithm is evaluated by using the "vifp_mscale" function in MATLAB. e higher the value of PSNR, the better the image recovery effect, the higher the value of VIF, and the higher the visual information fidelity of the image. In the simulated Poisson image experiment, we applied different levels of blur and noise to different images.
e Gaussian blur kernel parameters we set for the Lena image of Figure 1(a) were (size � 5, σ � 1.6), (size � 15, σ � 1.6), and (size � 25, σ � 2). ree different Gaussian blur kernels were used to convolve the remaining four images, and the Gaussian blur kernels were set to (size � 5, σ � 1.6), (size � 15, σ � 2), and (size � 25, σ � 1.6). en, the "skimage.util.random_noise" function of the image processing algorithm set in Python was used to add three different Poisson noise levels to the five images, and the noise levels "I max " were set to 3,000, 4,000, and 5,000. Particularly, since the Butterfly image with an image size of 256 × 256 in Figure 1(c) was insensitive to noise levels of 5,000 when the Gaussian blur kernel was (size � 15, σ � 2), we applied noise levels of 2,000, 3,000, and 4,000 to this image.
In the experiment of deblurring simulated Poisson images with different noise levels and different blur kernel sizes, the image quality evaluation data recovered by the PID-Split, PIDSB-FA, PIDSB-NLFA, and DeepRED algorithms and the proposed method in this paper are shown in Tables 1 and 2. According to the image quality evaluation data in Tables 1 and 2, the PSNR value and VIF value of the proposed method in this paper were higher than those of the other four algorithms. We conducted a set of comparison experiments on the House image in Figure 1(e), and the results of the comparison experiments are shown in Figure 2.
e House images have a large size of flat regions and a small amount of detailed information. In this group of comparative experiments, as shown in Figure 2(b), the House image was degraded by a Gaussian blur kernel of size 15 × 15 and standard deviation σ � 2 and polluted by Poisson noise (I max � 3,000). As shown in Figure 2(c), although the image restored by the PID-Split algorithm suppressed noise well, it also caused staircase effects. e images recovered by the PIDSB-NLFA and PIDSB-FA algorithms are shown in Figures 2(d) and 2(e). Compared with Figure 2(c), the staircase effects were suppressed, and the noise in the flat regions was suppressed better. Figure 2(f ) shows the result of restoration of the DeepRED, which retains more details but cannot remove    Poisson noise well. e image restored by the proposed method in this paper is shown in Figure 2(g). Compared with the other four methods, it not only suppressed noise to the greatest extent but also had clearer details and the best visual perception. Figure 3 shows a set of comparative results for Zebra images in Figure 1(d). e size of Zebra image is 584 × 387. Figure 3(b) is an image degraded by a Gaussian blur kernel with a size of 25 × 25 and a standard deviation of 1.6 and Poisson noise with a size of 3,000. As can be seen from Figure 3(a), the zebra has a large size of flat regions and the grassland contains considerable detailed information. As seen in Figure 3(g), the proposed method in this paper not only suppressed noise but also preserved considerable detailed information. Figure 3(c) shows that the image restored by the PID-Split algorithm was too smooth and had staircase effects. Figures 3(d)-3(f ) show the images recovered by the PIDSB-NLFA, PIDSB-FA, and DeepRED algorithms, respectively. Compared with Figure 3(c), they suppressed the staircase effects while retaining some details. Compared with the other four method, the proposed method in this paper not only suppressed the staircase effects but also preserved the details of the image to the greatest extent and was clearer.
In addition, the Butterfly image in Figure 1(c) was degraded by the Gaussian blur kernel (size � 5 × 5, σ � 1.6) and then corrupted by Poisson noise (I max � 5,000). As shown in Figure 4, compared with the images restored by the other four algorithms, the proposed method in this paper suppressed noise and restored clear images to the greatest extent. Figure 4(g) restored by our method looks natural and preserved the most detailed information of the images.

Convergence Analysis.
e explicit regularization prior term adopts the combination of the intensity-based L 0 regularization term and RED. e purpose is to be able to use the pixel intensity distribution of the blurred image and the clear image to distinguish the restored image pixels while effectively filtering the image pixels and removing the Poisson noise of the degraded image. To verify the effectiveness of the L 0 regularization prior, we used the method with the L 0 norm regularization term and the method without the L 0 norm to carry out simulation and comparison experiments on the House images of Figure 1(c) in Section 3.1.
e experimental results are shown in Figure 5. Figure 5(a) shows the PSNR values of the image restored by the two algorithms. It can be clearly seen that the method with the L 0 regularization term has the characteristic of fast convergence. With the increase in iteration times, the restored image reached a higher PSNR. Figure 5(b) shows the enlarged graph of the abscissa range (0∼500) in Figure 5(a). Figure 5(c) shows the loss of the objective function, and Figure 5(d) shows an enlarged graph with abscissa (0∼200) in Figure 5(c). It can be seen that the method with the L 0 regularization term also achieved a smaller loss value.

Denoising and Deblurring of Real Poisson Images.
is section is an experimental comparison of denoising and deblurring of real Poisson images. Tests were carried out on four real astronomically degraded images. To evaluate the real Poisson image quality, we used the Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) to evaluate the real data. e smaller the value of BRISQUE, the better the image quality. e image quality evaluation data recovered by the PID-Split, PIDSB-FA, PIDSB-NLFA, and DeepRED algorithms and the proposed method in this paper are shown in Table 3. Figure 6(a) shows a satellite image, and Figure 6(b) shows an image restored by the PID-Split algorithm; the restored image is too smooth. Figures 6(c)-6(e) show the images recovered by the PIDSB-FA, PIDSB-NLFA, and DeepRED algorithms, respectively. Although the staircase effects were suppressed to a large extent, ring effects exist at the edge. e image restored by the proposed method in this paper is shown in Figure 6(f ). e restored image not only restored a large amount of detailed information and texture information but also suppressed the ring effects at the edge while suppressing the step effect. e image restored by the proposed method looks more natural. e second real image is a moon image, as shown in Figure 7(a). Figures 7(b)-7(e) show the images restored by the PID-Split, PIDSB-FA, PIDSB-NLFA, and DeepRED algorithms, respectively. It can be seen from these three images that the images recovered by these methods had ringing effects of the edge, and the flat areas were too smooth. Figure 7(f) shows the image restored by the proposed method in this paper. It can be seen that the image restored by the proposed method in this paper retained more detailed information while suppressing noise and ringing effects. Figure 8 shows a set of comparative results for the docking images of Shenzhou-9 and Tiangong-1. Restored images by RLTV, PID-Split, PIDSB-FA, DeepRED, and the proposed method are shown in Figures 8(b)-8(f), respectively. e image restored by the proposed method suppressed the ring effects at the edge and retained more detailed information. Figure 9(a) shows the Phoebe image. Comparing the restored images by the other four methods shown in Figures 9(b)-9(e), respectively, the image restored by the proposed method in this paper looks clear with fine details.

Conclusions
is paper proposed an algorithm for denoising and deblurring Poisson images by using neural networks. We combine an implicit regularization prior with two explicit regularization priors that are the prior of RED and the prior of L 0 regularization based on intensity and use a neural network to restore degraded images from Poisson images. In the proposed method, it has been proved that Poisson images restored by deep networks can effectively suppress Poisson noise. Meanwhile, the method with the L 0 regularization term has the characteristic of fast convergence.

Data Availability
e data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest
e authors declare that there are no conflicts of interest regarding the publication of this paper.