Image restoration problem is ill-posed, so most image restoration algorithms exploit sparse prior in gradient domain to regularize it to yield high-quality results, reconstructing an image with piecewise smooth characteristics. While sparse gradient prior has good performance in noise removal and edge preservation, it also tends to remove midfrequency component such as texture. In this paper, we introduce the sparse prior in fractional-order gradient domain as texture-preserving strategy to restore textured images degraded by blur and/or noise. And we solve the unknown variables in the proposed model using method based on half-quadratic splitting by minimizing the nonconvex energy functional. Numerical experiments show our algorithm's robust outperformance.
1. Introduction
Mathematically, the image degradation is modeled as
(1)y=x⊗h+n,
where x is the original latent image and y is an observed image degraded by blur and/or noise, which is produced by convolving x with a blur point-spread-function (a.k.a. kernel) h and adding zero mean Gaussian noise n. Image restoration is recovering latent image from observed image.
Image restoration is ill-posed problem, so many methods introducing priors based on natural image statistics can regularize it. Total variation regularization is originally used for noise reduction [1, 2] and has also been used for image deblurring [3]. Chan and Wong [4] introduced total variational blind deconvolution method for motion blur kernel and out-of-focus kernel. Heavy-tailed natural image priors [5, 6] and hyper-Laplacian priors [7–10] were also extensively introduced. Numerous regularization approaches have been proposed too. Wang et al. [7] presented a fast total variation deconvolution algorithm to compute TV image deconvolution. Krishnan and Fergus [8] take a novel approach to the image restoration problem arising from the use of a hyper-Laplacian prior. Xu and Jia [11] developed a fast TV-l1 deconvolution method based on half-quadratic splitting.
While image reconstructed by algorithms above suppresses noise and preserve edges, it has piecewise smooth characteristic that the mid-frequency components such as textures are removed too.
In digital images, the gray values between neighboring pixels have high correlation. This highly self-similar fractal information of image fractal information is usually represented by complex textural features, and the works in [12–18] showed that fractional-order gradient is more suitable to deal with fractal-like textures. It has been proved in [12] that the fractional-order derivative satisfies the lateral inhibition principle of biologic visual system better than the integer-order derivative. The fractional-order derivative operators have been used in texture enhancement [13], image denoising [14, 15], and image inpainting [16, 17]. Jun and Zhihui [14] replaced the first-order derivative in the regularized term of ROF model with the fractional-order derivative. Bai and Feng [15] designed fractional-order anisotropic diffusion equation to remove noise. Zhang et al. [16] exploited fractional-order TV sinogram inpainting model to reduce metal artifacts for X-ray computed tomography. In [18], fractional total variation method was introduced to restore textured image. This work shows that the fractional-order derivative not only nonlinearly preserves the textural details but also eliminates the staircase effect caused by low integral-order derivative in image processing. Different from work in [18], the sparse prior in fractional-order gradient domain is considered in our work, which is more suitable for the texture of image. It is explained clearly in Figures 2 and 3.
This paper presents fractional-order regularization for the restoration of textured image degraded by blur and/or additive noise. R. Tony uses the Laplacian prior in fractional-order gradient domain for α=1 to preserve the texture. According to our analysis in the next section, hyper-Laplacian image prior in fractional-order gradient domain for0<α≤1 is more suitable to keep different texture for different texture image.
The outline of this paper is as follows. In Section 2, we analyze the reason why integral-order regularization fails to restore image texture. In Section 3, our fractional-order regularization model is proposed and based on half-quadratic splitting, we solve model using efficient alternating minimization method. Numerical experiments and comments are provided in Section 4 and the paper is concluded in Section 5.
2. Motivation
The prior p(x) favors natural image, usually based on the observation that their heavy-tailed gradient distribution is sparse. For example, Figure 1 shows textured image and a histogram of its gradient magnitudes in x-direction and y-direction, respectively. The distribution shows that the image contains primarily small or zero gradients, but a few gradients have large magnitudes. A common measure [19] is
(2)logp(x)=-∑i|Gx,i(x)|α+|Gy,i(x)|α+constant,
where Gx,i(x) and Gy,i(x) denote the horizontal and vertical derivatives at pixel i (here, the simple [-11] and [-11]T filters are used) and exponent value α∈(0,2]. α<1 leads to sparse prior and natural images usually correspond to α in the range of [0.5,0.8] [19]. α=1 and α=2 are Laplacian prior and Gaussian prior, respectively.
(a) A textured image. (b) The x-direction log distribution of gradient magnitudes. (c) The y-direction log distribution of gradient magnitudes.
Analysis of restoration on 1D signal using gradient prior. ((a) and (c)) Sharp and blurred signal; ((b) and (d)) sum of gradients -logp(x)=∑i|Gi(x)|α as a function of α.
Analysis of restoration on 1D signal: (a) sharp versus blurred signal; (b) sum of gradients -logp(x)=∑i|Gi(x)|α as a function of α; (c) sum of gradients -logp(x)=∑i|Giv(x)|α as a function of α (the value of v is 0.3).
The image restoration methods use the sparse prior term as a regularized term of variational energy functional [19], which is
(3)minxλ(x⊗h-y)+∑i=1I(|Gx,ix|α+|Gy,ix|α).
The failure of restoring texture with the sparse gradient prior depends on the fact that the value of energy does not always decrease during restoration process, so the no-blur explanation is usually favored. To understand this, consider the 1D signals in Figure 2.
For sharp edge in Figure 2(a), while Gaussian prior favors the blurry explanation, the sparse prior (α<1) favors the correct sharp explanation in Figure 2(b). The signal considered in Figure 2(c) illustrates that natural image contains a lot of medium contrast textures, which dominate the statistics more than step edges. As a result, blurring natural image reduces the overall contrast which cannot be restored by Gaussian prior or even sparse priors as in Figure 2(d).
The reason is that the gradient profile in fractal-like textures is close to Gaussian distribution and these small values are severely penalized by the sparse gradient prior.
A fractional-order gradient log distribution can be expressed as follows [18]:
(4)logp(x)=-∑i|Gx,ivx|α+|Gy,ivx|α+constant,
where Gx,ivx and Gy,ivx denote the horizontal and vertical fractional-order derivatives at pixel i and v is the fractional order v∈(0,4]. The exponent value is the same as α value in (2).
Compared with result in Figure 3(b), the sharp explanation in Figure 3(c) is favored by sparse prior even by Gaussian prior in fractional-order gradient domain.
3. The Proposed Model and Algorithm
The corresponding energy functional is as follows [18]:
(5)minxλ(x⊗h-y)+∑i=1(|Gx,ivx|α+|Gy,ivx|α),
where i is the pixel index and ⊗ is the 2-dimensional convolution operator, and a weighting term λ=3e3 controls the strength of the regularization. Gx,ivx and Gy,ivx denote the horizontal and vertical fractional-order derivatives at pixel i defined by our coauthor as Tables 1 and 2 [16].
Operator of x-direction: Gxvx.
⋮
⋮
⋮
⋮
⋮
⋮
⋮
0
0
0
⋯
0
⋮
0
0
0
Cs-1
Cs0
Cs1
⋯
Csk
⋯
Csn-2
Csn-1
Csn
0
0
0
⋯
0
⋮
0
0
0
⋮
⋮
⋮
⋮
⋮
⋮
⋮
⋮
⋮
⋮
⋮
⋮
⋮
⋮
0
0
0
⋯
0
⋮
0
0
0
Csn
Csn-1
Csn-2
⋯
Csk
⋯
Cs1
Cs0
Cs-1
0
0
0
⋯
0
⋮
0
0
0
⋮
⋮
⋮
⋮
⋮
⋮
⋮
7
Operator of y-direction: Gyvx.
⋯
0
Csn
0
⋯
⋯
0
Csn-1
0
⋯
⋯
0
Csn-2
0
⋯
⋮
⋮
⋮
⋯
0
Csn
0
⋯
⋮
⋮
⋮
⋯
0
Cs1
0
⋯
⋯
0
Cs0
0
⋯
⋯
0
Cs-1
0
⋯
⋯
0
Cs-1
0
⋯
⋯
0
Cs0
0
⋯
⋯
0
Cs1
0
⋯
⋮
⋮
⋮
⋯
0
Csk
0
⋯
⋮
⋮
⋮
⋯
0
Csn-2
0
⋯
⋯
0
Csn-1
0
⋯
⋯
0
Csn
0
⋯
The coefficients of the operator in Tables 1 and 2 are
(6)Cs-1=v4+v28Cs0=1-v22-v38Cs1=-5v4-5v316+v416⋮Csk=1Γ(-v)[Γ(k-v-1)(k+1)!·(v4+v28)+Γ(k-v)k!·(1-v24)+Γ(k-v-1)(k-1)!·(-v4+v28)],⋮Csn-2=1Γ(-v)[Γ(n-v-1)(n-1)!·(v4+v28)+Γ(n-v-2)(n-2)!·(1-v24)+Γ(n-v-3)(n-3)!·(-v4+v28)],⋮Csn-2=Γ(n-v-1)(n+1)!Γ(-v)·(1-v28)+Γ(n-v-2)(n-2)!Γ(-v)·(-v4+v28)Csn=Γ(n-v-1)(n-1)!Γ-v·(-v4+v28).
Equation (5) contains nonlinear penalties for regularization term, so we propose alternating minimization (AM) method, based on a half-quadratic splitting to solve it [18, 20]. We introduce auxiliary variables u and w=(wx,wy) at each pixel, so the energy functional in (5) can be modified as
(7)minx,wθ2(x⊗h-y)2+λ|u|+∑i=1(β2(∥Gx,ivx-wx,i∥22+∥Gy,ivx-wy,i∥22)+|wx,i|α+|wy,i|αβ2),
where the first two terms are used to ensure the similarity between the measures and the corresponding auxiliary variables. As β→∞ and θ→∞ the solution of (6) converges to that of (5). Equation (7) can be solved by AM method through fixing other variables to solve x,w, and u independently.
Given fixed values of u and w from the previous iteration, (7) is quadratic in x. So we compute x by minimizing
(8)E(x;u,w)=∥x⊗h-y-u∥2+∑i=1(βθ(∥Gx,ivx-wx,i∥22+∥Gy,ivx-wy,i∥22)).
The optimal x is
(9)(θβHTH+GxvTGxv+GyvTGyv)x=GxvTwx+GyvTwy+θβHT(y+u),
where Hx=h⊗x. According to Parseval’s theorem after the Fourier transform, (8) has the closed form solution in minimization, which enables us to find the optimal x directly:
(10)x=F-1(+θβF(H)*F(H))-1(∑F(Gxv)*F(wx)+F(Gyv)*F(wy)+θβF(H)*F(y+u)∑)×(∑F(Gxv)*F(Gxv)+F(Gyv)*F(Gyv)+θβF(H)*F(H)∑)-1),
where F(·) and F(·)-1 denote the fast Fourier transform and inverse fast Fourier transform, respectively. * is the complex conjugate operator.
Here, u and w belong to different terms. They are not coupled with each other in the functional, so their optimization is independent. Given fixed value of x, we compute u by minimizing
(11)E(u;x)=12∥u-(x⊗h-y)∥+λθ|u|.
According to shrinkage formula [21], the optimal u is
(12)u=sign(x⊗h-y)max(∥x⊗h-y∥-λθ,0).
We have the following:
(13)E˙(wx;x)=|wx|α+β2(∥Gxvx-wx∥22),E(wy;x)=|wy|α+β2(∥Gyvx-wy∥22).
For α=2 case, the subproblem about wx and wy is quadratic.
For α=1 case, the optimal solution for wx and wy can be derived by shrinkage formula too:
(14)wx˙=sign(Gxvx)*max(abs(Gxvx)-1β,0),wy=sign(Gyvx)*max(abs(Gyvx)-1β,0).
For the other α case, setting the derivative of (13) with regard to wx and wy to zero gives
(15)α˙|wx|α-1sign(wx)+β(wx-Gxvx)=0,α|wy|α-1sign(wy)+β(wy-Gyvx)=0.
Two special α cases are discussed here.
For α=1/2 case, about wx, (15) becomes
(16)12|wx|-1/2sign(wx)+β(wx-Gxvx)=0,(17)wx3-2(Gxvx)wx2+(Gxvx)2wx-sign(wx)4β2=0.
Because Gxvx is fixed and wx lies between 0 and Gxvx, we can replace sign(wx) with sign(Gxvx). Equation (17) can be rewritten as
(18)wx3-2(Gxvx)wx2+(Gxvx)2wx-sign(Gxvx)4β2=0.
So we can get the cubic polynomials about wx and wy:
(19)w˙x3-2(Gxvx)wx2+(Gxvx)2wx-sign(Gxvx)4β2=0,wy3-2(Gyvx)wy2+(Gyvx)2wy-sign(Gyvx)4β2=0.
The value of wx and wy is either 0 or the root of cubic polynomial in (19).
For α=2/3 case, we can get the quartic polynomials about wx and wy:
(20)w˙x4-3(Gxvx)wx3+3(Gxvx)2wx2-(Gxvx)3wx+827β3=0,wy4-3(Gyvx)wy3+3(Gyvx)2wy2-(Gyvx)3wy+827β3=0.
The value of wx and wy is either 0 or the root of cubic polynomial in (20).
Given the roots of cubic and quartic polynomials and zero solution, we need to determine which one corresponds to the global minima of (12), which can be confirmed by the following scheme.
Let r be the nonzero real root. If 0 value is the optimum solution of (13), for Gxvx, this implies
(21)|r|α+β2(r-Gxvx)2>β(Gxvx)22,(22)sign(r)|r|(α-1)+β2(r-Gxvx)≶0,r≶0.
We can use (15) to eliminate sign(r)|r|(α-1) from (15) and (22), yielding the condition
(23)r≶2Gxvxα-1α-2,
since sign(r)=sign(Gxvx). So wx=r if r is between 2Gxvx/3 and Gxvx in the α=1/2 case or between Gxvx/2 and Gxvx in the α=2/3 case. Otherwise, wx=0. The same scheme applies to Gyvx.
For other α cases, wx and wy can be computed by Newton method.
3.4. Algorithm
β and θ are positive values to enforce the similarity between the auxiliary variables and the respective terms. We empirically set β0=1, βMax=256 and θ0=1, θMax=λ.
The algorithm of this fractional-order regularization model is shown in Algorithm 1.
Input: observed image y, PSF h, penalty parameters λ, exponent α and fractional-order v
Input: β0, βMax, θ0, θMax
Initialize x=y, θ=θ0, β=β0
while θ<θMax, do
solve for u using (12)
while β<βMax do
Given x, solve for w according to our discussion
Given w, solve for x using (10)
β=2β
end while
θ=2θ
end while
Output: Estimated image x
4. Numerical Experiments
We consider the restoration of a blur- and noise-contaminated test image represented by 255×255 pixels. In order to compare the accuracy of FOR (fractional order regularization) and IOR (integer order regularization) more precisely, we list in Table 3 the peak signal-to-noise ratio (PSNR) and gray-scale structural similarity (SSIM) as quality metric. PSNR is most easily defined via the mean squared error (MSE). Given a noise-free M by N image I and its noisy approximation I^, MSE is defined as
(24)MSE=1MN∑m=1,n=1M,N[I(m,n)-I^(m,n)].
PSNR is defined as
(25)PSNR=110log(2552MSE)(dB),
and SSIM is defined as
(26)SSIM=(2μIμK+c1)(2σIK+c2)(μI2+μK2+c1)(σI2+σK2+c2),
where I and K are different images, μI and μK are the average of I and K, respectively, σI2 and σK2 are the variance of I and K, respectively, and σIK is the covariance of I and K. c1 and c2 are constant.
PSNR and SSIM of image restoration by IOR and FOR.
Image
α
PSNR_{IOR}
SSIM_{IOR}
v
α
PSNR_{FOR}
SSIM_{FOR}
Barbara (256 * 256)
0.8
—
0.8411
0.4
0.3
—
0.8580
Bubble (512 * 512)
0.4
2.5249
0.6949
1.8
0.4
2.7343
0.6987
The desired blur- and noise-free image is depicted in Figure 4. The image is contaminated by motion blur generated by Matlab function (fspecial(“motion”,10,20)). The resulting image is displayed in Figure 4(f). In Table 3 the second column, with header PSNR and SSIM values for images that have been corrupted by motion blur, is characterized by v=0.4 and α=0.3.
Deblurring: (a) clear image; (b) synthesized blurred image with PSF (fspecial(“motion”,10,20)); (c) image restoration by Lucy-Richardson algorithm; (d) image restoration by IOR; (e) image restoration by [18]; (f) image restoration by FOR (g) closeups of (d), (e), and (f).
The desired blur- and noise-contaminated image is depicted in Figure 5. The image is contaminated by motion blur, adding white Gaussian noise (its standard variance is 0.003). The resulting image is displayed in Figure 5(d). In Table 3 the third column, with header PSNR and SSIM values for images that have been corrupted by motion blur, is characterized by v=1.8 and α=0.4.
Deblurring and denoising: (a) clear image; (b) synthesized blurred image and adding white Gaussian noise (its standard variance is 0.003); (c) image restoration by IOR; (d) image restoration by FOR.
Figure 6 shows the result of deconvolving a real blurry image. We estimate the blur kernel using the algorithm in [9]. Again, textured regions are better reconstructed using our method in visual quality. Figure 6(b) is restored by total variation. Figure 6(c) is restored by fractional-order total variation. Figure 6(d) shows the details in Figure 6(b) and Figure 6(c).
Testing our algorithm with real-life blurry images. (a) Blurry image. (b) Restored image by using the algorithm in [9]. (c) Restored image by our algorithm. (d) Comparison of details of image window. Left: details in (a), middle: details in (b), and right: details in (c).
5. Conclusion
By introducing sparse prior in fractional-order gradient domain, we propose a fractional-order regularization method for the restoration of textured image degraded by blur and/or noise. The regularizer is constructed by using fractional-order derivatives, where the choice of the fractional-order is driven by different textured image. This makes the proposed model an efficient tool to preserve texture well. Numerical results show that the proposed model yields better SSIM and PSNR value and visual effects than using integral-order regularization method.
Our following work is to use an automatic texture detection procedure for textured image restoration. Different parameters are applied for different textures.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgment
This work was supported by the National Natural Science Foundation of China (NSFC) (no. 61302028). This document is a collaborative effort.
RudinL. I.OsherS.FatemiE.Nonlinear total variation based noise removal algorithmsChanT. F.OsherS.ShenJ.The digital TV filter and nonlinear denoisingLiY.SantosaF.A computational algorithm for minimizing total variation in image restorationChanT. F.WongC.-K.Total variation blind deconvolutionFergusR.SinghB.HertzmannA.RoweisS. T.FreemanW. T.Removingcamera shake from a single photographShanQ.JiaJ.AgarwalaA.High-quality motion deblurring from a single imageWangY.YangJ.YinW.ZhangY.A new alternating minimization algorithm for total variation image reconstructionKrishnanD.FergusR.Fast image deconvolution using hyper-laplacianpriorsKrishnanD.TayT.FergusR.Blind deconvolution using a normalized sparsity measureProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '11)June 2011Providence, RI, USA2332402-s2.0-8005291021210.1109/CVPR.2011.5995521YangJ.ZhangY.YinW.An efficient tv-l1 algorithm for deblurring multichannel images corrupted by impulsive noiseXuL.JiaJ.Two-phase kernel estimation for robust motion deblurringProceedings of the 11th European Conference on Computer Vision (ECCV '10)2010157170PuY.WangW.ZhouJ.WangY.JiaH.Fractional differential approach to detecting textural features of digital image and its fractional differential filter implementationYi-FeiP.Fractional differential analysis for texture of digital imageJunZ.ZhihuiW.A class of fractional-order multi-scale variational models and alternating projection algorithm for image denoisingBaiJ.FengX.-C.Fractional-order anisotropic diffusion for image denoisingZhangY.PuY.-F.HuJ.-R.LiuY.ZhouJ.-L.A new CT metal artifacts reduction algorithm based on fractional-order sinogram inpaintingZhangY.PuY.-F.HuJ.-R.ZhouJ.-L.A class of fractional-order variational image inpainting modelsChanR. H.LanzaA.MorigiS.SgallariF.An adaptive strategy for the restoration of textured images using fractional order regularizationLevinA.WeissY.DurandF.FreemanW. T.Understanding and evaluating blind deconvolution algorithmsProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops '09)June 2009Miami, Fla, USA196419712-s2.0-7045017434410.1109/CVPRW.2009.5206815ChanR. H.LiangH. X.ChambolleA.DeVoreR. A.LeeN.-Y.LucierB. J.Nonlinear wavelet image processing: variational problems, compression, and noise removal through wavelet shrinkage