The objective of superresolution is to reconstruct a high-resolution image by using the information of a set of low-resolution images. Recently, the variational Bayesian superresolution approach has been widely used. However, these methods cannot preserve edges well while removing noises. For this reason, we propose a new image prior model and establish a Bayesian superresolution reconstruction algorithm. In the proposed prior model, the degree of interaction between pixels is adjusted adaptively by an adaptive norm, which is derived based on the local image features. Moreover, in this paper, a monotonically decreasing function is used to calculate and update the single parameter, which is used to control the severity of penalizing image gradients in the proposed prior model. Thus, the proposed prior model is adaptive to the local image features thoroughly. With the proposed prior model, the edge details are preserved and noises are reduced simultaneously. A variational Bayesian inference is employed in this paper, and the formulas for calculating all the variables including the HR image, motion parameters, and hyperparameters are derived. These variables are refined progressively in an iterative manner. Experimental results show that the proposed SR approach is very efficient when compared to existing approaches.
1. Introduction
Superresolution (SR) technique [1–5] is an important branch in image fusion technology which targets the reconstruction of a high-resolution (HR) image from a set of degraded low-resolution (LR) images. The LR images are usually affected by warping, blurring, downsampling, and noising. Due to advancement of the technology, superresolution has been widely used in a broad range of applications such as computer vision, medical imaging, public safety, and military reconnaissance.
SR is a typically ill-posed inverse problem that is hard to solve without the introduction of some prior image information [6–8]. Thus, a number of regularization-based SR approaches have been proposed [9–11] by incorporating the prior knowledge of the unknown high-resolution image in the regularization strategy [12]. The authors in [9] proposed a regularization-based SR approach based on the Tikhonov regularizer using L2 norm. It can remove noises effectively; however, it also blurs the image edges. Farsiu et al. [10] proposed the bilateral total variation (BTV) regularizer, which penalized gradient magnitudes measured by L1 norm, with the intention of preserving edges in the reconstruction. However, the BTV regularizer often produces artifacts in the smoothed regions due to the use of L1 norm [11]. In [11], the authors improved BTV with the norm ρx,a=aa2+x2-a2, where x is the gradient value and a is a positive scale parameter, and proposed the bilateral edge-preserving (BEP) regularizer. This new norm can adaptively use L1 and L2 norms, and the transition from L1 to L2 can be controlled by modifying the positive scale parameter a. Unfortunately, the BEP regularizer employed a fixed scale parameter for the whole image and ignored the local image features. With this drawback, it cannot well control the level of punishment to local gradients.
It is well known that accurate registration which refers to the estimation of motion information is a very important factor in the success of the SR reconstruction method [13]. The above SR methods only registered the observations once before reconstructing the HR image. This is not a robust method. The error caused by an inaccurate registration would produce poor effects in the reconstruction process [13]. Other than the motion parameters, the hyperparameters related to the image prior model and the noise model could intimately affect the image reconstructing quality [12]. Recently, the variational Bayesian method has been used for SR [14–16]. This method approximates the joint posterior distribution of the unknowns, including the HR image, the motion parameters, and the hyperparameters, using a tractable distribution by minimizing the Kullback-Leibler (KL) divergence and by estimating the unknowns simultaneously in each iteration.
The Bayesian approach uses a prior model to estimate a priori knowledge of the unknown HR image. For edge-preservation, the variational Bayesian approaches based on TV prior model [14] and l1 prior model [16] have been proposed. In these two prior models, the gradient magnitudes are measured by the L1 norm, and consequently both of them can preserve edges. Compared with the TV prior, the l1 prior contains two model parameters, which constrain certain preferred edge directions [15, 16]. However, the L1 norm may not always distinguish the real image features from the effects of errors caused by inaccurate registration and additive Gaussian noise. Therefore, some artifacts will be produced in the smoothed image regions. Villena et al. made progress by combining the TV prior model and simultaneous autoregressive (SAR) (i.e., based on L2 norm) prior model [15]. A similar approach is also applied to the l1 prior model. The problem is that it is difficult to choose a proper parameter to balance the priors in the combination.
In this paper, a variational Bayesian estimate of the HR image, the motion parameters, and the hyperparameters are proposed and derived and then given a set of LR images. In order to overcome the difficulty of aforementioned SR methods, an adaptive image prior model is proposed, in which the degree of interaction between pixels is adjusted adaptively by using the norm ρ(x,a) in order to preserve edges and remove noises. It is important to note that we propose a method for automatically estimating the scale parameters (i.e., a) to the proposed prior model. Information needed to determine these scale parameters is updated iteratively based on the available estimated HR image. Experiment results show the effectiveness of the proposed SR method.
The rest of this paper is organized as follows: the observation model and a Bayesian framework for image SR are described in Section 2. The adaptive image prior model is sketched in Section 3. The proposed SR approach is presented in Section 4. Experimental results are demonstrated in Section 5. Finally, the conclusion is given in Section 6. In addition, Appendix is presented at the end of the paper showing the solving process in detail.
2. The Problem Formulation
We formulate the problem to be solved in this section. The formulation is divided into the observation model and hierarchical Bayesian framework which are discussed below.
2.1. The Observation Model
Formulating an observation model that relates the original HR image to the LR observations is the first step to fully analyze the SR reconstruction problem.
Let u denote the original HR image of size P1N1×P2N2, and each observed LR image is of size N1×N2. P1 and P2 denote the downsampling factors in the horizontal and vertical directions, respectively. The HR image and LR images are written in lexicographical notations as the vectors of sizes P1N1P2N2×1 and N1N2×1, respectively. In this work, the image acquisition process is modeled by geometric transformation, blurring, downsampling, and adding with white Gaussian noise. Thus, the following popular matrix notation [15] is adopted to describe the acquisition process:(1)vk=AHkCsku+εk1≤k≤L,where vk is one of a set (L) of LR images, A (the dimension of A is N1N2×P1N1P2N2) is the downsampling matrix, Hk (the dimension of Hk is P1N1P2N2×P1N1P2N2) is the blur matrix, C(sk) (the dimension of C(sk) is P1N1P2N2×P1N1P2N2) is the warp matrix, and εk (the dimension of εk is N1N2×1) is the additive white Gaussian noise.
The motion that occurs during the image acquisition is represented by warp matrix C(sk). In this paper, we assume that the motion includes global rotation and translation; that is, sk=θk,xk,ykT, where θk, xk, and yk are the rotation angle and the displacement in horizontal direction and vertical direction of the kth HR image, respectively, relative to the reference frame. In this study, we assume that the blur matrices Hk are determined by sensors. The sensor PSF is usually modeled as a spatial averaging operator, and the characteristics of this blur are usually assumed to be known. Under these assumptions, the matrices Hk and C(sk) have a block circulant with circulant block structure. The downsampling matrix A is determined by the downsampling factors P1 and P2, and it generates LR images from the warped and blurred HR image.
For convenience, we define B(sk)=AHkC(sk) and v={vk}, where · is a set.
2.2. The Hierarchical Bayesian Framework
We use a hierarchical Bayesian framework [17] to model the acquisition process, the HR image u, the motion parameters {sk}, and the hyperparameters α and {βk}. Parameters α and {βk} are the model parameters of our proposed prior model and the noise model, respectively. Thus, we can obtain the following joint posterior distribution of u, {sk}, {βk}, and α given v by using the Bayes rule:(2)pu,sk,βk,α∣v=pv∣u,sk,βkpu∣αpskpβkpαpv,where pv∣u,sk,βk represents the conditional distribution of the LR images v and pu∣α, p({sk}), p({βk}), and p(α) are prior distributions for the unknown image u, the motion parameters {sk}, and the hyperparameters {βk} and α, respectively.
Suppose that the noise εk in the LR image vk (k=1,2,…,L) is the white Gaussian noise with a zero-mean distribution εk~N(0,βk-1), and we can write(3)pεk∣βk∝βkN1N2/2exp-βk2εk22,where ·2 denotes the L2 norm of a given vector ·.
Due to εk=vk-B(sk)u, we can obtain the conditional distribution of the LR image vk,(4)pvk∣u,sk,βk∝βkN1N2/2exp-βk2vk-Bsku22.Moreover, if we assume the statistical independence of noises among the LR images, we can obtain the conditional distribution of the LR images v(5)pv∣u,sk,βk=∏k=1Lpvk∣u,sk,βk.
By using the available registration algorithm, the initial values of motion parameters can be obtained. However, the obtained initial values are often inaccurate. In this paper, we refine the values of motion parameters by modeling them as stochastic variables following Gaussian distributions (sk~N(sk0,δk)), which is similar to the model used in [15].
The prior distributions for the hyperparameters are defined to be Gamma distributions (t~Γ(at,bt), t∈{α,{βk}}). Gamma distributions for the hyperparameters were selected because they are conjugate for the variance of the Gaussian distribution; therefore, the posteriors will also have Gamma distributions in the Bayesian formulation [18].
Our proposed adaptive image prior model, denoted by pu∣α, will be presented in the next section.
3. The Proposed Adaptive Image Prior Model
In this section, we propose an adaptive image prior model and then present a method to calculate the scale parameter a used in the proposed prior model.
3.1. The Adaptive Image Prior Model
The adaptive norm ρx,a=aa2+x2-a2 is used to measure the horizontal and vertical gradients in which x is the gradient value and a is a positive scale parameter. These parameters are to determine the severity of penalizing image gradients. A new adaptive image prior model is proposed and defined as follows:(6)pu∣α∝α1α2N/4exp-∑i=1Nα1ρ∇ixu,ai,1+α2ρ∇iyu,ai,2,where N=P1N1×P2N2 is the number of pixels in the HR image. Symbols ∇ixu and ∇iyu represent the horizontal and vertical gradient components, respectively, for the pixel i, α={α1,α2} is the hyperparameter of this prior controlling the degree of regularization, and ai,1 and ai,2 are the scale parameters of the adaptive norms measuring the horizontal and vertical gradient component, respectively, of pixel i.
If we take the logarithm, (6) becomes(7)logpu∣α∝-∑i=1Nα1ρ∇ixu,ai,1+α2ρ∇iyu,ai,2,where logpu∣α is proportional to a linear combination of ρ((∇ixu),ai,1) and ρ((∇iyu),ai,2), i=1,…,N. Equation (7) is strictly a convex function because ρ(x,a) is strictly convex [11].
In addition, logpu∣α is adaptive, because the following approximations of ρ(x,a) can be obtained with the assumption that parameter a is fixed:(8)ρx,a≈12x2when x⟶0,(9)ρx,a≈ax-a2when x⟶∞.Thus, ρ((∇ixu),ai,1) or ρ((∇iyu),ai,2), i=1,…,N, in (7) can use either L1 norm (i.e., (9)) or L2 norm (i.e., (8)) adaptively based on the evaluation of the local image features.
Thus, the expression F=α1ρ(∇ixu,ai,1)+α2ρ(∇iyu,ai,2) can be approximated as(10)F≈α12∇ixu2+α2ai,2∇iyu-ai,22,when ∇ixu⟶0,∇iyu⟶∞;(11)F≈α1ai,1∇ixu-ai,12+α22∇iyu2,when ∇ixu⟶∞,∇iyu⟶0;(12)F≈α1ai,1∇ixu-ai,12+α2ai,2∇iyu-ai,22,when ∇ixu⟶∞,∇iyu⟶∞;(13)F≈α12∇ixu2+α22∇iyu2,when ∇ixu⟶0,∇iyu⟶0.
For example, if there exists an edge along the horizontal direction in an LR image, |∇iyu| of the pixel at the edge is relatively large while |∇ixu| is small. Thus, ai,2(ai,2)2+|∇iyu|2-(ai,2)2 approximates to ai,2|∇iyu|-(ai,2)2 and ai,1(ai,1)2+|∇ixu|2-(ai,1)2 approximates to 1/2|∇ixu|2. Therefore, F approximates to (10). In the SR process based on (10), when the LR pixel splits in both horizontal and vertical directions into a block of HR pixels, the large gradient magnitude in the vertical direction will be preserved due to the L1 norm, and the small gradient magnitude in the horizontal direction will be smoothed due to the L2 norm. Thus whenever there exists an edge along horizontal or vertical direction, aa2+|·|2-a2 can act as the L1 norm in the direction perpendicular to the edge, preserving large gradients, and, at the same time, aa2+|·|2-a2 can act as the L2 norm in the direction along the edge, obtaining relatively ideal smoothing effects. In (12), when |∇ixu|→∞ and |∇iyu|→∞, the L1 norm acts on both horizontal and vertical directions, to preserve the edges. In (13), when |∇ixu|→0 and |∇iyu|→0, the noise will be effectively removed.
3.2. Adaptive Calculation of Parameter a
It is known that the gradients produced by edges should be preserved, while the gradients produced by errors, which suffer from inaccurate registration and additive Gaussian noise, need to be smoothed. Thus, the severity of penalizing image gradients should be determined according to the local image features. In ρ(x,a), the scale parameter, a, can control the severity of penalizing local gradients. For the gradients produced by errors, L2 norm is a good choice. In this case, the scale parameter should be set as a large value. When the value of the scale parameter increases, ρ(x,a) accepts a larger range of errors and handles them like L2 norm [11]. Otherwise, it should be set as a small value. Therefore, the property of scale parameter should be constrained with the following conditions: (1) its value is determined according to the local image features, (2) the value is inversely proportional to gradient magnitudes, and (3) the value is larger than zero.
We use the well-known monotonically decreasing function [19] to calculate ai,1, and ai,2 in (6),(14)ai,1=11+∇ixu2,ai,2=11+∇iyu2,1≤i≤N,where ∇ixu and ∇iyu represent the magnitudes of horizontal and vertical gradient components, respectively, for the pixel i.
For convenience, we use the symbol R to represent ai,1, and ai,2; that is, R={ai,1,ai,2,1≤i≤N}.
4. Variational Bayesian Superresolution
The variational Bayesian inference [14] is utilized to estimate the unknowns. This variational Bayesian technique approximates the true posterior distribution p(u,{sk},α,{βk}∣v) analytically by a tractable distribution q(u,{sk},α,{βk}) that minimizes the KL divergence, which can measure the difference between the two distributions p(u,{sk},α,{βk}∣v) and q(u,{sk},α,{βk}). That is to say, by minimizing this KL divergence, we can obtain the optimal approximation distribution. Thus, we can obtain the following expression:(15)qΩ=argminqΩCKLqΩ∥pΩ∣v=∫qΩlogqΩpΩ∣vdΩ=∫qΩlogqΩpΩ,vdΩ+const=EqΩ,v+const, where Ω={u,{sk},{βk},α} and p(Ω,v)=pv∣u,sk,βkpu∣αpαpβkpsk.
Thus, (11) can be rewritten as(16)qΩ=argminqΩEqΩ,v.Because (16) cannot be evaluated due to the form of (6), the inequality e≤e+g/2g with e≥0, g>0 is employed in (6). This inequality was used in [14, 15]. Then, we can obtain(17)pu∣α≥α1α2N/4exp-∑i=1Nα1ai,1∇ixu2+ai,12+gix2gix-ai,12+α2ai,2∇iyu2+ai,22+giy2giy-ai,22=Zα,u,g, where g=gix,giy∣i=1,…,N, the auxiliary variables gix and giy are related to the unknown HR image, and they need to be estimated (see (22)). Thus, we can obtain the upper bound of E(q(Ω),v)(18)EqΩ,v≤∫qΩlogqΩKΩ,v,g=E^qΩ,v,g,where K(Ω,v,g)=pv∣u,sk,βkZ(α,u,g)p(α)p({βk})p({sk}).
As shown in [20], the minimization of E(q(Ω),v) can be replaced by the minimization of its upper bound E^(q(Ω),v,g). Thus, we can obtain the distributions q(u), q({sk}), q({βk}), and q(α) and g by minimizing E^(q(Ω),v,g); that is, (19)qΩ=argminqΩE^qΩ,v.
The details of equations derivation are given in Appendix at the end of this paper. We obtain for the posterior distribution q(u) the multivariate Gaussian with mean value(20)u=Σu∑kβkBskTvkand covariance (21)Σu=∑kβkBskTBsk+∑k∑i∑jξkijOkiskTOkjsk+α1∇xTWgx∇x+α2∇yTWgy∇y-1,where · denotes the expected value of ·, T denotes the transposed operator, ξkij is a 3 × 3 region in the covariance matrix Λk in (24), Okrsk=AHkNk, r=1,2,3, where N1, N2, and N3 denote partial derivatives of C(sk) for θk, xk, and yk, respectively, and Wg, ∀g∈R+N is a diagonal matrix, in which the element on the diagonal is Wgii=1/gi.
Then, the following expressions are obtained for:(22)gix=∇ixu2+a2qu,giy=∇iyu2+a2qu,where ·q∗ denotes the expected value of · using q∗.
We also obtain for the posterior distribution q({sk}) the multivariate Gaussian with mean value(23)sk=Λkδk-1sk0+βkΓksk+Ψksk+Qk-Φkand covariance(24)Λk=δk-1+βkΓk+Ψk-1.In (23) and (24), Φki=traceBskTOkjskΣu, Ψkij=traceOkiskTOkjskΣu, Γkij = ΥkiTΥkj with Υk=Ok1sku,Ok2sku,Ok3sku, and Qki=vk-BskTOkisku, for i,j=1,2,3.
Finally, we obtain the mean values of the hyperparameter distributions, which are used to estimate hyperparameters, (25)βk=N1N2/2+aβk1/2vk-Bsku22qu+bβk,(26)α1=N+aα12∑iagix-a2+bα1,(27)α2=N+aα22∑iagiy-a2+bα2.
At each iteration step, after obtaining the HR image, the scale parameters R are updated using (10). We now conclude our algorithm (Algorithm 1).
Algorithm 1: Our iterative SR algorithm.
Input:
The observations vk; The motion parameters sk0; The hyper-parameters βk0,
α0; The scale parameters R0; The parameters g0.
Output:
The HR image u;
(1) Use the Bicubic interpolation of the first LR image v1 as initial value u0 for the HR image.
(2) n=1;
(3) repeat
(4) Compute un by using (20) and (21);
(5) Compute gn by using (22);
(6) Compute skn by using (23) and (24);
(7) Compute βkn by using (25);
(8) Compute αn by using (26) and (27);
(9) Compute Rn by using (14);
(10) n=n+1;
(11) until convergence criterion is met
(12) Set u=un.
5. Experimental Results
We will give our experimental results in detail in this section. The experimental environment and parameter settings are presented in Section 5.1. The evaluation standards and simulated experimental results obtained by four popularly used SR approaches and our proposed SR approach are illustrated in Section 5.2. Some of the results on real dataset are given in Section 5.3.
5.1. General Description
We compare the performance of our proposed approach with the bicubic approach, the BEP model [11] based approach, the l1 model [16] based approach, the TV model [14] based approach, and the SAR model [15] based approach in terms of Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM) values [21], and thorough visual inspection. The images presented in Figure 1 were chosen for the simulation test. We used MATLAB R2009a on a 3.0 GHz Pentium Dual core computer with 4.0 GB RAM.
The 128 × 128 test images: (a) “zebra” and (b) “car.”
We need to derive a set of LR images from original images in the experiments. These LR images should have been generated with subpixel motion, rotation, blurring, downsampling, and noise addition. In the experiments, we let θk∈ (−3°, 3°), xk∈ (−1, 1), yk∈ (−1, 1), k = 1, 2, 3, 4, 5, and the transformation of LR images be variant from each other. 3 × 3 average and Gaussian blurring operators with deviation equal to 1 were used in the simulation. The downsampling factors were P1 = 2 and P2 = 2. Finally, additive zero-mean white Gaussian noises were added to the LR images with Signal-to-Noise Ratio (SNR) levels of 1 dB, 5 dB, 10 dB, and 30 dB, respectively. Thus, in each level, five LR images were obtained from each of the original images. The initial parameters used in the experiments were set as follows: our proposed method: gix0=∇ixu02+ai,102, giy0=∇iyu02+ai,202, ai,10=1/1+|∇ixu0|2, ai,20=1/1+|∇iyu0|2, α10=N/4/∑i(ai,10gix0-(ai,10)2), and α20=N/4/∑i(ai,20giy0-(ai,20)2); the l1 method: gix0=∇ixu02, giy0=∇iyu02, α10=N/4/∑igix0, and α20=N/4/∑igiy0; the TV method: gi0=∇ixu02+∇iyu02, α0=N/4/∑igi0; the SAR method: α0=N/2/ħu022 with Laplacian operator ħ; the BEP method: gix0=ui0-sxlsymui02+ai,102, where a is the scale parameter, sxl and sym shift ui0 by l and m pixels in the horizontal and vertical directions, respectively, and these parameters are selected to obtain the best reconstructions; consider α0=N/4/∑igi0. For all the methods, a bicubic interpolation of v1 is used as initial value u0, βk0=N/vk-B(sk)u022, the initial motion parameters were estimated using the method proposed in [22], δk=0, and at=bt=0, t∈α1,α2,βk. In our work, the convergence of the algorithms was set with the same criterion un+1-un22/un22≤10-5.
5.2. Simulation Experiments
We used LR images derived from artificial HR images to test our proposed approach. The PSNR and SSIM are used to evaluate the reconstruction quality of different approaches. The first set of experiments was with the high level of noise (i.e., SNR = 1 dB and SNR = 5 dB). For the second set of experiments, white Gaussian noises with SNR = 10 dB and SNR = 30 dB were used and the motion parameters were estimated in each iteration.
In the first set of experiments, we will show the improvement of our proposed approach in heavy noise (i.e., SNR = 1 dB and SNR = 5 dB). Tables 1 and 2 show the PSNR values of all the estimated HR images in heavy noise with average blur and Gaussian blur. Based on the PSNR values, our proposed method shows a good performance. Moreover, Tables 1 and 2 give the corresponding SSIM values that also demonstrate the efficiency of our proposed method.
Comparisons of PSNR and SSIM with average blur and white Gaussian noise with SNR = 1 dB and SNR = 5 dB.
SNR
Bicubic
SAR
l1
TV
BEP
Ours
Zebra
1 dB
PSNR
18.51
21.15
21.62
20.24
21.72
22.27
SSIM
0.5645
0.6894
0.6780
0.6410
0.6953
0.7086
5 dB
PSNR
18.63
22.39
22.63
22.29
22.70
24.06
SSIM
0.5645
0.7858
0.7889
0.7840
0.7905
0.8020
Car
1 dB
PSNR
26.20
30.33
28.27
28.48
29.93
30.73
SSIM
0.7369
0.8365
0.7601
0.7697
0.8025
0.8563
5 dB
PSNR
26.20
32.27
30.35
32.57
31.57
33.03
SSIM
0.7370
0.8938
0.8640
0.8957
0.8475
0.9121
Comparisons of PSNR and SSIM with Gaussian blur and white Gaussian noise with SNR = 1 dB and SNR = 5 dB.
SNR
Bicubic
SAR
l1
TV
BEP
Ours
Zebra
1 dB
PSNR
18.97
21.44
21.00
21.06
21.46
22.04
SSIM
0.6072
0.7053
0.6733
0.6742
0.7108
0.7255
5 dB
PSNR
19.08
22.90
23.12
23.95
22.74
24.41
SSIM
0.6072
0.8012
0.7977
0.8173
0.7197
0.8236
Car
1 dB
PSNR
26.56
30.27
29.40
29.43
30.41
31.45
SSIM
0.7511
0.8417
0.7771
0.7956
0.8573
0.8660
5 dB
PSNR
26.77
32.86
33.06
33.61
32.16
34.64
SSIM
0.7512
0.8930
0.8869
0.9021
0.6576
0.9288
For the second set of experiments, the motion parameters were estimated. In order to further illustrate the effectiveness of our approach, we will present the reconstruction results for “zebra” and “car” images in noise with SNR = 10 dB and SNR = 30 dB, respectively. Tables 3 and 4 show the comparisons of PSNR values of bicubic approach, BEP approach, l1 approach, SAR approach, TV approach, and our approach. Tables 3 and 4 also give the SSIM results of these approaches with average blur and Gaussian blur, respectively. From Tables 3 and 4, we can see the improvement of our proposed approach over other tested methods. Among all the approaches tested, the proposed method achieves the highest PSNR and SSIM values on both test images.
Comparisons of PSNR and SSIM with average blur and white Gaussian noise with SNR = 10 dB and SNR = 30 dB.
SNR
Bicubic
SAR
l1
TV
BEP
Ours
Zebra
10 dB
PSNR
19.22
23.73
26.41
25.62
25.81
26.90
SSIM
0.6745
0.8247
0.8420
0.8360
0.8102
0.8766
30 dB
PSNR
19.23
37.09
42.88
41.17
43.34
43.87
SSIM
0.6768
0.9207
0.9252
0.8744
0.9261
0.9413
Car
10 dB
PSNR
26.45
34.51
35.95
35.96
34.98
36.32
SSIM
0.7690
0.8898
0.8857
0.8866
0.8816
0.9197
30 dB
PSNR
26.45
38.73
40.29
39.81
44.78
48.74
SSIM
0.7710
0.9036
0.9160
0.9169
0.9288
0.9461
Comparisons of PSNR and SSIM with Gaussian blur and white Gaussian noise with SNR = 10 dB and SNR = 30 dB.
SNR
Bicubic
SAR
l1
TV
BEP
Ours
Zebra
10 dB
PSNR
19.39
24.20
26.74
25.33
25.80
27.72
SSIM
0.6868
0.8364
0.8537
0.8510
0.8756
0.9167
30 dB
PSNR
19.39
37.49
40.34
39.01
40.97
42.32
SSIM
0.6874
0.9128
0.9208
0.9194
0.9265
0.9491
Car
10 dB
PSNR
26.77
34.98
36.09
36.01
36.26
36.55
SSIM
0.7793
0.8969
0.8956
0.8951
0.8325
0.9246
30 dB
PSNR
26.78
44.11
46.14
46.17
47.58
49.01
SSIM
0.7812
0.9238
0.9246
0.9242
0.8624
0.9468
For visual quality comparison, the error images, that is, the difference between the estimated HR image and the original image, are shown in Figures 2 and 3. Our proposed approach can preserve edge details well. Figures 2(g)–2(l) and 3(g)–3(l) show the corresponding error images to the reconstructed images obtained with different approaches. Brighter pixels represent a large error. From these error images, the difference between different SR approaches is clearly observed. In general, our proposed approach can obtain the highest PSNR and SSIM values, with the best visual quality.
Results obtained by applying different approaches to LR zebra images corrupted with Gaussian blur and white Gaussian noise with SNR = 10 dB and their corresponding error images. Brighter pixels represent a large error in error image.
Bicubic
SAR
l1
TV
BEP
Ours
Error image of bicubic interpolation
Error image of SAR
Error image of l1
Error image of TV
Error image of BEP
Error image of ours
Results obtained by applying different approaches to LR car images corrupted with average blur and white Gaussian noise with SNR = 30 dB and their corresponding error images. Brighter pixels represent a large error in error image.
Bicubic
SAR
l1
TV
BEP
Ours
Error image of bicubic interpolation
Error image of SAR
Error image of l1
Error image of TV
Error image of BEP
Error image of ours
5.3. Experiments on Real Data
The performance of our approach is tested with real dataset. The datasets are those popular used video sequences, downloaded from Milanfar’s website https://users.soe.ucsc.edu/~milanfar/software/sr-datasets.html. The first ten LR images were used to reconstruct the HR image.
Figures 4 and 5 present the obtained HR images for two of the image sequences downloaded. In Figures 4 and 5, the HR images obtained by the bicubic and SAR approaches are still blurred. In addition, there exist artifacts in the HR images obtained by the BEP, l1, and TV approaches. Our approach is superior to the approaches compared.
The reconstructed images for “Adyoron” sequence obtained by different approaches. A subimage is enlarged and displayed at the right bottom corner in the image.
Bicubic
SAR
l1
TV
BEP
Ours
The reconstructed images for “EIA” sequence obtained by different approaches. A subimage is enlarged and displayed at the right bottom corner in the image.
Bicubic
SAR
l1
TV
BEP
Ours
6. Conclusions
The errors caused by inaccurate registration and noises in the traditional regularization-based SR methods often produce unsatisfactory results in the reconstruction. Thus, the variational Bayesian method, which can simultaneously estimate the HR image, the motion parameters, and the hyperparameters, has been used to improve the reconstruction quality. However, the existing variational Bayesian approaches cannot adapt to local image features. Therefore, in this paper, a Bayesian SR approach is proposed by designing a new adaptive image prior model, based on an adaptive norm.
Our adaptive image prior model can be adjusted automatically based on the evaluation of the local image features; therefore, this new model not only preserves edge details but also avoids artifacts in the smoothed regions. We also propose a method for automatically estimating the scale parameters for the proposed adaptive image prior model. Information needed to determine these scale parameters is updated in each iteration step based on the available estimated HR image, and they are calculated by using a monotonically decreasing function. In our approach, the acquisition process, the HR image, the motion parameters, and the hyperparameters are modeled in a stochastic sense by using a hierarchical Bayesian framework. And all unknowns are estimated by employing the variational Bayesian inference.
The experimental results show that the HR images obtained with our SR approach are better than those previously tested. In the proposed adaptive image prior model, we only calculate the gradients of neighboring pixels at the top and left of the center pixel for fast computation. We will investigate further for the prior on the selection of 4-neighbourhood or 8-neighbourhood in order to improve the proposed algorithm in our future work.
Appendix
In this section we show how the calculations of all the unknowns are carried out.
Because all the unknowns are independent, the following expressions are obtained from (19):(A.1)qu=argminquE^quaskqαqβk,v,g,(A.2)g=argmingE^quqskqαqβk,v,g,(A.3)qsk=argminqskE^quqskqαqβk,v,g,(A.4)qβk=argminqβkE^quqskqαqβk,v,g,(A.5)qα=argminqαE^quqskqαqβk,v,g,α=α1,α2.
From (A.1), we can obtain(A.6)qu∝exp-α1∑ia∇ixu2+a2+gix2gi1-a2+α2∑ia∇iyu2+a2+giy2g2-a2-∑kβkvk-Bsku22qsk,where ·q∗ denotes the expected value using q∗ and · is used for simplicity in the rest of this paper.
In order to calculate vk-Bsku22qsk, C(sk) is first expanded using its first-order Taylor value sk of q(sk) in (A.12), resulting in(A.7)Csk≈Csk+N1sk,N2sk,N3sksk-sk.Then, B(sk) is approximated by (A.8)Bsk≈Bsk+Ok1sk,Ok2sk,Ok3sksk-sk,where Okrsk=AHkNk, r=1,2,3.
Then the following approximation of vk-Bsku22qsk can be obtained:(A.9)vk-Bsku22qsk≈vk-Bsku22+∑i=13∑j=13βkξkijuTOkiskTOkjsku.Substituting (A.9) into (A.6), the covariance and mean values of q(u) can be calculated as(A.10)Σu=∑kβkBskTBsk+∑k∑i∑jξkijOkiskTOkjsk+α1∇xTWgx∇x+α2∇yTWgy∇y-1,u=Σu∑kβkBskTvk,where ξkij is a 3 × 3 region in the covariance matrix Λk of q(sk). And Wg,∀g∈R+N is a diagonal matrix, in which the element on the diagonal is Wgii=1/gi.
Then, we can get the following expressions from (A.2):(A.11)gix=∇ixu2+a2qu,giy=∇iyu2+a2qu.
From (A.3), we can obtain(A.12)qsk∝exp-12βkvk-Bsku22qu+sk-sk0Tδk-1sk-sk0,where T denotes the transposed operator.
The approximation of vk-Bsku22qu can be obtained by using (A.8) again, (A.13)vk-Bsku22qu≈vk-Bsku-Υksk-sk22qu+traceBskTBskΣu+2ΦkTsk-sk+sk-skTΨksk-sk,where Υk=[Ok1sku,Ok2sku,Ok3sku], Φki=traceBskTOkjskΣu, and Ψkij=traceOkiskTOkjskΣu, for i,j=1,2,3.
Thus, the covariance and mean values of q(sk) can be calculated as (A.14)Λk=δk-1+βkΓk+Ψk-1,sk=Λkδk-1sk0+βkΓksk+Ψksk+Qk-Φk,where Γkij=ΥkiTΥkj, for i,j=1,2,3, and Qki=(vk-BskTOkisku), for i,j=1,2,3.
From (A.5), we can obtain(A.15)qβk∝βkN1N2/2-1+aβkexp-βkbβk+vk-Bsku22qu2,qα1∝α1N/4-1+aα1exp-α1bα1+∑iagix-a2,qα2∝α2N/4-1+aα2exp-α2bα2+∑iagiy-a2.
The mean values of the distributions in (A.15) are given by(A.16)βk=N1N2/2+aβk1/2vk-Bsku22qu+bβk,α1=N+aα12∑iagix-a2+bα1,α2=N+aα22∑iagiy-a2+bα2.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
This research is supported by NSFC Grant no. 61370179, the Independent Innovation Research Funds of HUST no. 2013YLQX001, and the Fundamental Research Funds for the Central Universities HUST no. 2015YGYL012.
IzadpanahiS.DemirelH.Motion based video super resolution using edge directed interpolation and complex wavelet transform20139372076208610.1016/j.sigpro.2013.01.0062-s2.0-84875908705LuJ.WuB.Single-image super-resolution with joint-optimization of TV regularization and sparse representation2014125112497250410.1016/j.ijleo.2013.10.0932-s2.0-84901233793NasrollahiK.MoeslundT. B.Super-resolution: a comprehensive survey20142561423146810.1007/s00138-014-0623-42-s2.0-84904508167SuL.ZhouS.YuanY.High spatial resolution image restoration from subpixel-shifted hyperspectral images20159110.1117/1.jrs.9.095093Christensen-JeffriesK.BrowningR. J.TangM.-X.DunsbyC.EckersleyR. J.In vivo acoustic super-resolution and super-resolved velocity mapping using microbubbles201534243344010.1109/tmi.2014.23596502-s2.0-84922389753ShimadaK.KonishiK.UrumaK.TakahashiT.FurukawaT.Mixed lp/l1 norm minimization approach to intra-frame super-resolution2014E97D102814281710.1587/transinf.2014edl80862-s2.0-84907485596HuangC.LiangY. C.DingX. Q.FangC.Generalized joint kernel regression and adaptive dictionary learning for single-image super-resolution201410314215410.1016/j.sigpro.2013.11.0422-s2.0-84901237580TraonmilinY.LadjalS.AlmansaA.Robust multi-image processing with optimal sparse regularization201551341342910.1007/s10851-014-0532-1MR3323658ZBL064686742-s2.0-84906455045HeH.KondiL. P.An image super-resolution algorithm for different error levels per frame200615359260310.1109/tip.2005.8605992-s2.0-32944471247FarsiuS.RobinsonM. D.EladM.MilanfarP.Fast and robust multiframe super resolution200413101327134410.1109/TIP.2004.8346692-s2.0-4544374765ZengX.YangL.A robust multiframe super-resolution algorithm based on half-quadratic estimation with modified BTV regularization20132319810910.1016/j.dsp.2012.06.013MR30002312-s2.0-84869497905TianJ.MaK.-K.A survey on super-resolution imaging20115332934210.1007/s11760-010-0204-62-s2.0-80051947769ParkS. C.ParkM. K.KangM. G.Super-resolution image reconstruction: a technical overview2003203213610.1109/msp.2003.12032072-s2.0-0037648729BabacanS. D.MolinaR.KatsaggelosA. K.Variational Bayesian super resolution201120498499910.1109/TIP.2010.2080278MR28088242-s2.0-79952944722VillenaS.VegaM.BabacanS. D.MolinaR.KatsaggelosA. K.Bayesian combination of sparse and non-sparse priors in image super resolution201323253054110.1016/j.dsp.2012.10.002MR30097552-s2.0-84871456427VillenaS.VegaM.MolinaR.KatsaggelosA. K.Bayesian super-resolution image reconstruction using an ℓ1 priorProceedings of the 6th International Symposium on Image and Signal Processing and AnalysisSeptember 2009Salzburg, AustriaIEEE1521572-s2.0-70450245251GopalS.YangY. M.Hierarchical Bayesian inference and recursive regularization for large-scale classification20159312310.1145/2629585BabacanS. D.MolinaR.KatsaggelosA.Parameter estimation in TV image restoration using variational distribution approximation200817332633910.1109/tip.2007.916051MR24929772-s2.0-40849120186ChenY.TagareH. D.ThiruvenkadamS.HuangF.WilsonD.GopinathK. S.BriggsR. W.GeiserE. A.Using prior shapes in geometric active contours in a variational framework200250331532810.1023/a:10208784089852-s2.0-0036892488BabacanS. D.MolinaR.KatsaggelosA. K.Variational Bayesian blind deconvolution using a total variation prior2009181122610.1109/tip.2008.2007354MR27224072-s2.0-58049218859WangZ.BovikA. C.SheikhH. R.SimoncelliE. P.Image quality assessment: from error visibility to structural similarity200413460061210.1109/tip.2003.8198612-s2.0-1942436689LucasB.KanadeT.An iterative image registration technique with an application to stereo visionProceedings of the Imaging Understanding Workshop1981121130121130