This paper introduces an efficient deblurring image method based on a convolution-based and an iterative concept. Our method does not require specific conditions on images, so it can be widely applied for unspecific generic images. The kernel estimation is firstly performed and then will be used to estimate a latent image in each iteration. The final deblurred image is obtained from the convolution of the blurred image with the final estimated kernel. However, image deblurring is an ill-posed problem due to the nonuniqueness of solutions. Therefore, we propose a smoothing function, unlike previous approaches that applied piecewise functions on estimating a latent image. In our approach, we employ L2-regularization on intensity and gradient prior to converging to a solution of the deblurring problem. Moreover, our work is based on the quadratic splitting method. It guarantees that each subproblem has a closed-form solution. Various experiments on synthesized and real-world images confirm that our approach outperforms several existing methods, especially on the images corrupted by noises. Moreover, our method gives more reasonable and more natural deblurred images than those of other methods.
Pattani Campus Research FundSAT63030065Prince of Songkla University1. Introduction
An image deblurring is a recovering process that recovers a sharp latent image from a blurred image, which is caused by camera shake or object motion. It has widely attracted attention in image processing and computer vision fields. A number of algorithms have been proposed to address the image deblurring problem. The most common approaches to blur removal are to treat blur as a noisy convolution operated with a blur kernel. The mathematical form of the blurred image is then usually modelled as(1)y=x∗k+n,where y is a blurred image, x is a corresponding latent image, k and n denote the blur kernel and white noise, respectively, and ∗ represents the convolution operator.
A number of approaches have been proposed to address deblurring image problem, and one of popular approaches was based on the deconvolution-based concept. There are two types of approaches: nonblind and blind deconvolutions. The main difference between these two types is that the blur kernel must be known as prior in the case of nonblind deconvolution. The blur kernel estimation is an essentially important step in obtaining a high-quality sharp image. Most approaches use statistical priors on natural images and selection of salient edges for the blur kernel estimation [1–6]. Moreover, several researchers estimate a high-quality blur kernel directly from the input image with motion blur by studying the problem in the frequency domain [7–9].
In the blind deconvolution, the goal is to estimate both the corresponding latent image and the blur kernel. The blind deconvolution is an ill-posed problem since it has many pairs of solutions. To deal with the blind deblurring problem well-posed, most blind image deblurring methods tend to formulate the problem as a minimization problem. The blur kernel and the latent image are usually solved in an alternating fashion. The algorithm will converge quickly if an appropriately useful initialization is well chosen. However, various assumptions or regularizations are demanded. Pan et al.[10] presented L0-regularized intensity and gradient prior to deblurring text images. This method is suitable for text images with a clean background, but it is less useful for a cluttered background. Liu et al.[11] combined L0-regularized of both first- and second-order image gradients in order to standardize the final estimation results. Cho et al. [12] developed a method to incorporate text-specific properties for deblurring. This prior outperformed in two-tone images; however, it is still less effective in generic images.
Many researchers proposed numerous image-priors underlining optimization techniques to resolve the ill-posed problem in generic images. Pan et al. [13] continued to propose their work with the generic images via modifying a dark channel firstly introduced by He et al. [14]. It is based on the observation that, in most of the natural scene patches, at least one of the color channels possesses some pixels with very close to zero intensities. This algorithm enforces the sparsity of the dark channel of latent images for kernel estimation and generates better results compared to other approaches. However, the dark channel may not help intermediate latent image estimation if there are no dark pixels in the images, but instead, there is a large degree of bright pixels.
Yan et al. [15] proposed a novel natural image prior named Extreme Channels Prior (ECP) by taking the advantages of both Dark Channel Prior (DCP) and Bright Channel Prior (BCP) to refine natural images. This method is more robust and performs favourably against the state-of-the-art image deblurring methods on both synthesized and natural images. However, it has limitations and is less likely to support kernel estimation. The kernel estimation process is complicated, and the performance depends mainly on the intensity values of images. Several researchers turned to employ a gradient prior to improve the accuracy of kernel estimation. Shan et al. [16] adopted a sparse image prior and introduced a unified probabilistic model to fit the gradient distribution of natural images to solve the kernel estimation problems. The authors in [17, 18] proposed a hyper-Laplacian prior to fit the distribution of natural image gradients for avoiding local solutions. Fergus et al. [1] presented an algorithm using a zero-mean mixture of Gaussians to learn the image gradient prior to fitting the distribution of natural image gradients. Zhong et al. [19] proposed an adaptive generalized variation (TGV) regularized model to remove deblurring and maintaining fine image details.
All proposed algorithms [3–15] added the auxiliary terms in objective functions. Based on the observation, we found that the closed forms of auxiliary terms are in terms of the piecewise functions containing the parameters. These parameter values were constantly updated by scaling, leading to the loss of detail in the images. Moreover, most algorithms require specific conditions on generic images so they cannot be widely applied for nonspecific generic images.
In this paper, we present an efficient optimization algorithm based on the quadratic splitting method. The splitting method guarantees that each subproblem has a closed-form solution and ensures fast convergence. Additionally, we introduce a novel smoothing function for updating pixel values and a sigmoid function for scaling parameters. Our smoothing function based on L2-regularized intensity and gradient prior gives a significant advantage result and can reduce the loss of details in the images. Moreover, our approach is also suitable for colored images that do not require any specific conditions on the image.
This paper is organized as follows. Our L2-regularization method is described in Section 2. Section 3 describes our blind image deblurring method. In Section 4, we present some experimental results. Finally, conclusions are drawn in Section 5.
2. Our L2-Regularization Method
We first formulate the blind image deblurring to a minimization problem and introduce a new efficient regularization term for convergence to a solution of deblurring. As we mentioned, equation (1) is solvable. That is, Ey=x,k|y=x∗k+n is not empty. We seek an x corresponding with a kernel k whose norm is the minimum. However, we first need to confirm that such a solution always exists in the set Ey. Given a latent image x and a blur kernel k, a blurred image y can be rewritten as KX, where K is the Toeplitz matrix and X is the column latent image x. So, we have y=KX. To show that y=KX has the solution with the minimum norm, we need the following proposition.
Proposition 1.
Let W and Z be Hilbert space, G∈LW,Z and G∗∈LZ,W the adjoint operator; then the following statement holds:
RangG=Z if and only if there exists γ>0 such that ∥G∗z∥w≥γ∥z∥Z,zϵZ
RangG¯=Z if and only if G∗ is one to one
Now, the matrix K may be viewed as a linear operator K:ℝm⟶ℝn; therefore, K∈Lℝm,ℝn. Since y=KX is solvable for all y, K is surjective. Hence, by Proposition 1, there exists γ>0 such that ∥K∗z∥2≥γ∥z∥2,z∈ℝn.
Therefore, <KK∗z,z>≥γ2∥z∥22,z∈ℝn . This implies that KK∗ is one to one. Hence, detKK∗≠0. We can see that X=K∗KK∗−1y is a solution of y=Kz. Now, we shall see that this solution has a minimum norm. In fact, consider W∈ℝm such that KW=y and(2)∥W∥22=∥X+W−X∥22=∥X∥22+∥W−X∥22+2Re<X,W−X>.
Hence, ∥W∥22−∥X∥22=∥W−X∥22≥0. Therefore, ∥W∥2≥∥X∥2. This means that X has the minimum norm. Therefore, we can conclude that, for every blurred image, there always exists a corresponding latent image x with a kernel k whose norm of x is the minimum. Based on this fact, we confidently believe that seeking out such a latent image with the minimum norm is one of the alternative ways to solve the problem. Thus, we define Px=α∥x∥22+∥∇x∥22 as a regularization term in our approach.
3. Our Image Deblurring Method
Our image deblurring method is an iterative approach based on a convolution-based concept. In each iteration, the kernel estimation is performed, and then it is used to estimate a latent image. The estimated latent image is then to be as input in the next kernel estimation. The final deblurred image is obtained from a convolution of the blurred image with the final estimated kernel. A system overview of our proposed method is shown in Figure 1.
System overview of our proposed method.
To estimate the kernel and the latent image, equation (1) is formed as an optimization problem with a regularization term Px for image deblurring expressed as follows:(4)minx,k∥x∗k−y∥22+γ∥k∥22+λPx.
A solution directly from intensity values is often inaccurate [2, 4, 10], so we set the regularization term in an intensity space and a gradient space with Px=α∥x∥22+∥∇x∥22.
We use an efficient alternating minimization method [2, 5, 10, 13] to solve equation (4). These two subproblems are solved as follows:(5)minx∥x∗k−y∥22+λα∥x∥22+∥∇x∥22,(6)mink∥x∗k−y∥22+γ∥k∥22.
The details of the two subproblems are described in the following sections.
3.1. Estimating the Latent Image x with the Blur Kernel k
We introduce an efficient alternating minimization method to solve equation (5). Like [10, 13, 20], we propose the auxiliary variables g=gh,gvT corresponding to x and ∇x, respectively. The objective function then can be rewritten as(7)minx,u,g∥x∗k−y∥22+β∥x−u∥22+μ∥∇x−g∥22+λα∥u∥22+∥g∥22,where the constants β and μ are weights based on the latent image value and the constants λ and α are weight values.
To solve equation (7), the values of u and g are initialized to be zero in the first iteration. And then, the solution of x is obtained in each iteration by solving(8)minx∥x∗k−y∥22+β∥x−u∥22+μ∥∇x−g∥22.
Based on a least square minimization problem, we perform FFTs and the closed-form solution is given by(9)x=ℱ−1ℱk¯ℱy+βℱu+μℱ∇¯ℱgfk¯ℱk+β+μℱ∇¯ℱ∇,where ℱ⋅ and ℱ−1⋅ denote the Fast Fourier Transform (FFT) and inverse FFT, respectively. ℱ⋅¯ is the complex conjugate operator. ∇=∇h+∇v, where ∇h and ∇v are the horizontal and vertical differential operators, respectively.
To update the variable x, we first need to compute u and g, which can be done separately by(10)minuβ∥x−u∥22+λα∥u∥22,(11)mingμ∥∇x−g∥22+λ∥g∥22.
Note that equations (10) and (11) are pixel-wise minimization problems, and most importantly, they are smooth. Thus, the solutions of u and g are obtained by the simple technique of differentiation:(12)u=ββ+λαx,(13)g=μμ+λ∇x.
It is worthwhile to note that, in each iteration, the weighted values of β and μ are always updated. In most algorithms [3–15], the weighted values of β and μ are glued in u and g, but in the forms of piecewise function and they are always updated by scaling. This may force values for some pixels to be zero which will lead to the loss of the details in latent images. In fact, we confidently believe that each pixel should not be scaled by the same value but it should depend on itself. So, in our approach, we employ the sigmoid functions βx and μx for updating the value β and μ, respectively, where βx=λσex−a and μx=λex−b, where a and b are defined as the center of the transition area. Due to the smoothness of equations (12) and (13), not only does it have less artifact effect, but also it guarantees not losing details in intermediate latent images.
3.2. Estimating the Blur Kernel k with the Latent Image x
To estimate k, we employ the latent image x and the blurred image y in the gradient space as shown in equation (14) and then apply FFTs to solve the least square minimization problem(14)mink∥∇x∗k−∇y∥22+γ∥k∥22.
The closed form of k kernel estimation with the latent image x and the blurred image y is(15)k=ℱ−1ℱx¯ℱ∇yf∇x¯ℱ∇x+γ.
The proposed method, similar to the state-of-the-art methods, is used as a coarse-to-fine processing strategy for handling the blur kernel estimation. The algorithm is typically efficiently implemented using an image pyramid [2, 10]. The summarization of the main step for the latent image and the blur kernel estimation on one pyramid level is presented in Algorithm 1.
Algorithm 1: Latent image and blur kernel estimation algorithm.
Input: Blur image y;
initialize k with results from the coarser level;
fori←1to 5 do
x←y;
β←2λα;
repeat
solve for u using eq.12;
μ←2λ;
repeat
solve for g using eq. 13;
solve for x using eq.9;
μ←2μ;
untilμ>μmax;
β←2β;
untilβ>βmax;
solve for k using eq.15;
λ←maxλ/1.1,1e−4
end
Output: the latent image and blur kernel
3.3. Final Image Restoration
In the final step, we restore the final deblurred image using nonblind deconvolution, given the final kernel from equation (15). The final latent image can be restored from(16)x=ℱ−1ℱk¯ℱy+μℱ∇¯ℱgfk¯ℱk+μℱ∇¯ℱ∇.
We then compute a difference map between these two estimated images with different sigmoid parameters. Finally, we subtract the restored image with a difference map to suppress a ringing artifact.
4. Experimental Results
To evaluate the performance of our proposed method, we tested our approach on the real-image datasets [21], synthesized images, and specific domain images [14]. Moreover, we also evaluate the robustness of our approach to Gaussian noise. In all experiments, we set λ=σ=0.001 and μmax=1e3,βmax=23.
4.1. Evaluation on the Real-Image Datasets
First, we tested our approach on the image deblurring dataset [21], containing 48 images (4 images with 12 blur kernels). This dataset is used as a benchmark for performance measuring of image deblurring methods. The proposed method is compared with state-of-the-art methods: Cho and Lee [2], Fergus et al. [1], Hirsch et al. [22], Krishnan et al. [23], Whyte et al. [24], Xu and Jia [3], and Pan et al. [13]. To compare performance, we use peak signal-to-noise ratio (PSNR) value which is computed by comparing each restored image with 199 ground truth images. PSNR = 100, which corresponds to a perfect deblurring. The PSNR of our proposed method and those of compared methods on the dataset [21] are illustrated in Figure 2. We found that the PSNR of our approach, Xu and Jia [3], and Pan et al. [13] are not much different in image 2. It might be that on image 2 the balance of the dark and bright pixels is moderately the same. However, our approach gives a higher average PSNR than that of other approaches. Figure 3 shows sample images which are deblurred by the best three algorithms. However, our method gives clearer details and is more natural than that of other approaches.
Quantitative evaluations on Kohler et al.’s dataset [21].
Samples of restored image by the best three algorithms on [21]. (a) Blurred image [21]. (b) Xu and Jia [3]. (c) Pan et al. [13]. (d) Ours.
4.2. Evaluation on the Synthesized Images
We then evaluated our approach on the synthesized images with blur kernel. The comparison of results on the first synthesized blurred image is shown in Figure 4. It can be seen that the restored image of our approach gives a better result and is more reasonable than that of other methods. The results from [3, 13] methods redeem more blur than the original image. It might be the case that the original image contains lots of bright pixels. So the two methods could not control the regularization term, which leads to an unreasonable solution.
Comparison of results using a synthesized blurred image. (a) Blurred image. (b) Xu and Jia [3]. (c) Pan et al. [13]. (d) Ours.
Moreover, we demonstrated the performance of the proposed method on synthesized blurred images corrupted by different realizations of the white Gaussian noises, 5–30 dB. The samples of synthesized noise blurred images and deblurred images are shown in Figure 5.
Comparison of results using a synthesized blurred image corrupted by noise 30 dB and deblurred images. (a) Blurred image. (b) Xu and Jia [3]. (c) Pan et al. [13]. (d) Ours.
The PSNR values for the synthesized images corrupted by various white Gaussian noise levels are shown in Figure 6. Our method achieved the highest PSNR value. It is evident that our method shows outperformance. Our method obviously performs better than the other two methods when the test image contains lots of noise.
The PSNR values of restore images on the synthesized blurred image, corrupted by white Gaussian noises.
4.3. Evaluation on the Specific Domain Images
The next step, we evaluated the proposed method on the specific domain images such as face and text images. We compared with the advanced deblurring methods in each domain. We then evaluated our approach on face images against the method of Pan et al. [13] that used a dark channel in image deblurring. Moreover, we compared our proposed method to Xu and Jia [3]. As shown in Figures 7 and 8 , our method gives more details and is more natural than other methods.
Comparison of results using a face image. (a) Blurred image. (b) Xu and Jia [3]. (c) Pan et al. [13]. (d) Ours.
Comparison of results using a face image. (a) Blurred image. (b) Xu and Jia [3]. (c) Pan et al. [13]. (d) Ours.
Next, we evaluated the proposed method on selected real and challenging text image for qualitative evaluation as shown in Figures 9 and 10 . We compared our method with the methods of Pan et al. [10] and Yan et al. [25] that present specific methods for deblurring the text image. As shown in Figures 9 and 10, our method gives more reasonable, sharper, and less losing details than those of both methods.
Comparison of results using a text image [10]. (a) Blurred image. (b) Pan et al. [10]. (c) Yan et al. [25]. (d) Ours.
Comparison of results using a text image [10]. Evaluation on the robustness to Gaussian noise. (a) Blurred image. (b) Pan et al. [10]. (c) Yan et al. [25]. (d) Ours.
We evaluated the robustness of our approach to the blurred image with Gaussian noise. The comparison of results on a blurred image with Gaussian noise is shown in Figure 11. The other methods still contain blur in deblurring results, while ours is more reasonable and more natural. The method of [3] gives more artifacts than other methods. The one in [13] gives an unreasonable result since the noise changed the minimum intensity value of an image patch.
Comparison of results on a blurred image with Gaussian noises [14]. (a) Blurred image. (b) Xu and Jia [3]. (c) Pan et al. [13]. (d) Ours.
5. Conclusions
We propose a novel method for the deblurring image, especially for blind deconvolution. We present L2 regularization in the deblurring image formed as an optimization problem. This method does not require specific conditions on images, so it can be widely applied for unspecific generic images. It applies smooth functions on estimating a latent image in each iteration. It keeps more details in the restored image than that of existing approaches. To evaluate the performance of our approach, we performed experiments on both synthesized and real-world images. We discovered that our approach outperforms the previous approaches, especially in the image, which contains Gaussian noise. Moreover, our restored images give more details and are more natural than the restored images produced by other methods.
Data Availability
Data are available at https://sites.google.com/site/jspanhomepage/.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work was supported by the Pattani Campus Research Fund (SAT63030065) from Prince of Songkla University in Thailand.
FergusR.SinghB.HertzmannA.RoweisS. T.FreemanW. T.Removing camera shake from a single photograph200625378779410.1145/1141911.11419562-s2.0-33749249573ChoS.LeeS.Fast motion deblurring200928514515210.1145/1618452.16184912-s2.0-84964611239XuL.JiaJ.Two-phase kernel estimation for robust motion deblurringProceedings of the European Conference on Computer VisionSeptember 2010Heraklion, Crete, Greece157170LevinA.WeissY.DurandF.FreemanW. T.Efficient marginal likelihood optimization in blind deconvolutionProceeding of the IEEE Conference on Computer Vision and Pattern RecognitionJune 2011Colorado, CO, USA26572664XuL.ZhengS.JiaJ.Unnatural L0 sparse representation for natural image deblurringProceedings of the IEEE Conference on Computer Vision and Pattern RecognitionJune 2013Portland, OR, USA11071114PanJ.SuZ.Fast L0-regularized kernel estimation for robust motion deblurring2013209841844DashR.MajhiB.Motion blur parameters estimation for image restoration204125516341640WangZ.YaoZ.WangQ.Improved scheme of estimating motion blur parameters for image restoration2017651111810.1016/j.dsp.2017.02.0102-s2.0-85014988418PanL.HartleyR.LiuM.DaiY.Phase-only image based kernel estimation for single-image blind deblurringProceedings of the IEEE Conference on Computer Vision and Pattern RecognitionJune 2019Long Beach, CA, USA60276036PanJ.HuZ.SuZ.YangM.Deblurring text images via L0-regularized intensity and gradient,” priorProceedings of the IEEE Conference on Computer Vision and Pattern RecognitionSeptember 2014Columbus, OH, USA29012908LiuR. W.YinW.XiangS.PengS.L0-regularized hybrid gradient sparsity priors for robust single-image blind deblurringProceedings of the IEEE Int. Conference Acoustics, Speech Signal Process. (ICASSP)April 2018Seoul, Korea13481352ChoH.WangJ.LeeS.Text image deblurring using text-specific propertiesProceedings of the European Conference on Computer VisionOctober 2012Florence, Italy524537PanJ.SunD.PfisterH.YangM.-H.Deblurring images via dark channel prior201840102315232810.1109/tpami.2017.27538042-s2.0-85030699983HeK.SunJ.TangX.Single image haze removal using dark channel prior201133122341235310.1109/TPAMI.2010.1682-s2.0-80054892393YanY.RenW.GuoY.WangR.CaoX.Image deblurring via extreme channels priorProceedings of the IEEE Conference on Computer Vision and Pattern RecognitionJuly 2017Honolulu, HI, USA69786986ShanQ.JiaJ.AgarwalaA.High-quality motion deblurring from a single image200827311010.1145/1360612.13606722-s2.0-49249099286LevinA.WeissY.DurandF.FreemanW. T.Understanding and evaluating blind deconvolution algorithmsProceedings IEEE Conference on Computer Vision and Pattern RecognitionMarch 2009Miami, Florida19641971LouY.BertozziA. L.SoattoS.Direct sparse deblurring201139111210.1007/s10851-010-0220-82-s2.0-79952005317ZhongQ.WuC.ShuQ.LiuR. W.Spatially adaptive total generalized variation-regularized image deblurring with impulse noise20182705110.1117/1.jei.27.5.0530062-s2.0-85053525720TangJ.ShuX.QiG.-J.Tri-clustered tensor completion for social-aware image tag refinement20173981662167410.1109/tpami.2016.26088822-s2.0-85028330194K¨ohlerR.HirschM.MohlerB.Sch¨olkopfB.HarmelingS.Recording and playback of camera shake: benchmarking blind deconvolution with a real-world databaseProceedings of the European Conference on Computer VisionOctober 2012Florence, Italy2740HirschM.SchulerC. J.HarmelingS.Sch¨olkopfB.Fast removal of non-uniform camera shakeProceedingsof the International Conference On Computer VisionNovember 2011Barcelona, Spain463470KrishnanD.TayT.FergusR.Blind deconvolution using a normalized sparsity measureProceedings of the IEEE Conference on Computer Vision and Pattern RecognitionJuly 2011Colorado, CO, USA233240WhyteO.SivicJ.ZissermanA.Deblurring shaken and partially saturated imagesProceedings of the IEEE International Conference On Computer Vision Workshops (ICCV Workshops)March 1992Hyderabad, India745752YanY.RenW.GuoY.WangR.CaoX.Image deblurring via extreme channels priorProceedings of the IEEE International Conference on Computer Vision and Pattern RecognitionJuly 2017Honolulu, HI, USA40034011