Video Superresolution Reconstruction Using Iterative Back Projection with Critical-Point Filters Based Image Matching

To improve the spatial resolution of reconstructed images/videos, this paper proposes a Superresolution (SR) reconstruction algorithm based on iterative back projection. In the proposed algorithm, image matching using critical-point filters (CPF) is employed to improve the accuracy of image registration. First, a sliding window is used to segment the video sequence. CPF based image matching is then performed between frames in the window to obtain pixel-level motion fields. Finally, high-resolution (HR) frames are reconstructed based on the motion fields using iterative back projection (IBP) algorithm. The CPF based registration algorithm can adapt to various types of motions in real video scenes. Experimental results demonstrate that, compared to optical flow based imagematchingwith IBP algorithm, subjective quality improvement and an average PSNR score of 0.53 dB improvement are obtained by the proposed algorithm, when applied to video sequence.


Introduction
Since high-resolution (HR) images/videos are important in many applications, such as astronomy, military monitor, medical diagnosis, and remote sensing, superresolution (SR) reconstruction has a great significance in practice [1].The concept of superresolution (SR) reconstruction refers to reconstructing a high-resolution (HR) image from one or more low-resolution (LR) images.The purpose of superresolution (SR) reconstruction is using digital image processing algorithm to enhance the spatial resolution by transcending the limiting factors of optical imaging system [2,3].The innate character of superresolution (SR) reconstruction is using complementary content of multiple images to extend high frequency component.
Most superresolution reconstruction methods contain four steps: registration, map, interpolation, noise, and blur removal.Registration refers to estimating motion vectors between two different video frames or images.Then the motion vectors are used to map the pixels of the input lowresolution frames to a common high-resolution reference frame.Interpolation is used to obtain the pixel value of the superresolution grid, by utilizing the mapped pixels.Finally, noise and blur removal is applied to eliminate the optical sensor blur [4].
A variety of iterative superresolution reconstruction algorithms have been proposed.These algorithms can be divided into two types: frequency domain method and spatial domain method.The frequency domain approach was first proposed by Tsai and Huang [5].They formulated a series of equations which relate high-resolution frames to low-resolution frames, making use of the shift property of Fourier transform.But their imaging model does not consider motion blur and additive noise and is only restricted to global translational motion.Interpolation based method, iterative back projection (IBP) method [1,6,7], projection onto convex set (POCS) method [3,8], and maximum a posteriori (MAP) method [9] are four main spatial domain algorithms for superresolution reconstruction.Interpolation based method is the simplest spatial domain algorithm, which uses multiple registered images to generate the HR image, based on some interpolation approach, such as nearest-neighbor interpolation, bilinear interpolation, and cubic spline interpolation [10].The other three spatial domain SR algorithms (IBP, POCS, and MAP) are all based on iterative reconstruction and have better results than interpolation based method.Another class of 2 Advances in Multimedia superresolution reconstruction algorithm is example-based superresolution algorithm.Freeman et al. [11] presented the idea of example-based superresolution reconstruction algorithm.A state-of-the-art algorithm via structure analysis of patches was proposed by Kim et al. in [12].
The key to the spatial domain algorithms is accurate image registration.Traditional registration approaches estimate the global translational or rotational motion.But if more than one type of motion coexists in the scene such as natural video, global motion estimation does not work.In video superresolution, image registration algorithms such as block based matching [13] and optical flow based matching [14,15] are often used.But both of these two algorithms have some drawbacks and are not appropriate in the superresolution reconstruction.Motion vectors cannot be accurately obtained because of block based matching criterion, and the performance of optical flow based matching may be seriously degraded due to changes of brightness.
In this paper, image matching using multiresolutional critical-point filters (CPF-IM) [16][17][18] is proposed to be applied in the process of superresolution reconstruction.CPF-IM is suitable for representing both the global and local motions.Moreover, the influence of brightness change is small in CPF-IM.In our experiments, pixel-level motion fields are first obtained by CPF-IM.Then the iterative back projection (IBP) algorithm is employed to reconstruct the high-resolution image based on the motion fields.Experimental result shows that the IBP algorithm with CPF-IM has better performance than bilinear interpolation algorithm, bicubic interpolation algorithm, and optical flow based image matching with IBP algorithm.
The rest of this paper is organized as follows.Section 2 briefly introduces the CPF based image matching algorithm.Section 3 describes the proposed superresolution algorithm using iterative back projection (IBP) algorithm with CPF-IM.Section 4 discusses the experimental results.Finally, Section 5 concludes this paper.

Image Matching Using Multiresolutional Critical-Point Filters
Multiresolutional critical-point filters (CPF) [16][17][18] provide a means of matching two images in pixel-level accurately.Suppose there are two images, a source and a destination.A set of multiresolution subimages are constructed for both images.Then, mappings from the source to the destination subimages are performed at each level from the coarsest resolution to the finest resolution.The mapping is computed pixel by pixel constrained by the inherited and the bijectivity conditions.The mapping with the minimum energy will be selected as final correspondence.Supposing the width and the height of the original image size are  and , respectively,  denotes the hierarchy level of the finest resolution.A multiresolution hierarchy of size /2 (−) × /2 (−) images can be computed.There are four subimages to be calculated, which are formed by extraction of the minimum, the maximum, and the saddle points, respectively, at each level of the hierarchy.Let  (,) (,) denote the pixel at (, ) in the subimage, where  is the level of hierarchy and  is the type of subimage.The pixels of subimages in the hierarchy are recursively calculated from the pixels of its higher level subimages in the hierarchy as follows: where  (,0) (,) =  (,1) (,) =  (,2) (,) =  (,3) (,) =  (,) , which are the pixels of the original image.
Once the multiresolution hierarchy is constructed, a top down method is utilized to map pixels from the source image to the destination image.The number of candidate mappings at each level is constrained by the mapping at its upper level.
A pixel  at level  of the source image is searching for its corresponding pixel  in the destination image.Suppose the 4 nearest pixels of the pixel  are , , , and .Their parents (, , , ) are mapped to   ,   ,   ,   at level  − 1.For each of the parents, one child pixel is selected.The four children pixels define an inherited quadrilateral         , inside which we search the pixel  with a minimum mapping energy.
Let  (,) (, ) be the pixel to map in source image at location (, ) and  (,) (, ) the pixel to test in destination image at location (, ) =  (,) (, ).The mapping energy  (,) (, ) consists of  (,) (, ) and  (,) (, ), defined as where  ≥ 0 is a real number. (,) (, ) denotes the intensity differences between the source image pixel and its corresponding pixel in the destination image.Consider where the function (⋅) denotes the intensity of image pixel.And  (,) (, ) is the cost related to the locations of pixel.Consider where  ≥ 0 is a real number.Consider ( 0 is determined by the differences between (, ) and (, ) to prevent a pixel being mapped to a pixel too far away. 1 is determined by the distance between the displacement of  (,) (, ) and the displacement of its neighbors.This energy is used to smooth the mapping.
In the mapping procedure, the energy of the candidate pixels satisfying the above conditions will be computed and compared.Then the pixel with the minimum energy will be determined as the final corresponding pixel.
The advantage of integrating CPF based image matching into video image superresolution is that pixel-level motion fields can be accurately obtained.Shifts usually vary across small regions between video frames; therefore, block based image matching is not accurate enough.Moreover, optical flow based image matching is seriously affected by brightness change.In summary, CPF based image matching can overcome the defects of traditional image matching algorithms and is applicable to global motion and local motion model.In the process of image matching, one of the low-resolution images is selected as the reference image.CPF based image matching is performed between the reference image and the other low-resolution images to obtain motion fields.

Integrate CPF Based Image Matching into Video Image Superresolution
This section introduces how to integrate CPF-IM into video image superresolution.A new framework of iterative back projection algorithm is proposed to reconstruct the highresolution video image.

Video Superresolution Reconstruction Model.
The acquisition of low-resolution images is shown in Figure 1.The relation between the th observation image   and the original high-resolution image  can be expressed as where   represents the affine operation matrix for   ,   denotes the blur operation caused by point spread function (PSF), and   is the downsampling matrix.The point spread function (PSF) and the downsampling operator used to be the same in the process of image acquisition in an imaging system.Thus, the blur operator   and the downsampling operator   can be assumed to be constants in a sequence of images acquisition.So the imaging model ( 6) can be transformed to The purpose of superresolution is to obtain the optimal estimation of the high-resolution image  from a series of low-resolution images   .This is the inverse process of imaging as shown in (7).
The key to superresolution reconstruction algorithms is accurate image registration, which is a difficult problem, since motion between video frames is complex and accurate motion field is difficult to obtain.Most traditional superresolution reconstruction algorithms are proposed for image superresolution rather than video superresolution, as motion field between images is much easier to estimate.The algorithm proposed in this paper is suitable for video image superresolution, and the experimental results will verify its effectiveness.
For a video sequence, the motion vectors between two adjacent frames are relatively small.So we select the successive frames as the low-resolution images.A high-resolution frame can be reconstructed from the adjacent frames through (7).A video superresolution reconstruction model based on sliding window [19] is depicted in Figure 2.
As shown in Figure 2, in video superresolution reconstruction, the sliding window is moved forward to produce successive high-resolution frames in the output sequence.In order to ensure the accuracy of registration, the middle frame of the sliding window is selected as the reference frame.Image matching is performed between the reference frame and other frames in the sliding window to obtain the motion fields.Then the high-resolution frames are reconstructed based on these motion fields.If we assume that the total length of video is  and  is the length of sliding window, (+−1) high-resolution frames can be reconstructed from the low-resolution video sequence.

Original Iterative Back Projection Algorithm.
Iterative back projection [1,6,7] starts with an initial estimation of high-resolution image.The initial high-resolution image is generated by interpolating an input low-resolution image.Then, a series of simulated low-resolution images are generated from the reconstructed superresolution image by the image acquisition model.If the simulated low-resolution images are consistent with the corresponding input lowresolution images, the reconstructed superresolution image is identical with the original high-resolution image.Otherwise, the differences between the simulated low-resolution images and the input low-resolution images will be projected to the reconstructed superresolution image.This process is repeated iteratively until the differences between the simulated lowresolution images and the input low-resolution images are small enough.Iterative back projection algorithm can be formulated as where  represents the iteration number,  +1  is the reconstructed superresolution image in the ( + 1)th iteration, and    is the reconstructed superresolution image in the th iteration. is a constant value controlling the convergence speed, which is set as 0.1 for every iterative process.The length of sliding window is defined as , which also represents the number of input low-resolution images. BP  is the th back projection operation, which is the same projection operator.  iL denotes the simulated low-resolution image defined by the image acquisition model, and  iL denotes the input lowresolution image.  represents the affine operation matrix for   iL ,  denotes the blur operation caused by point spread function (PSF), and  is the downsampling matrix.
The flow of the original iterative back projection algorithm is shown in Figure 3, and deblur block denotes the inverse matrix of .

Proposed Iterative Back Projection Algorithm Combined with CPF Based Image Matching.
In traditional image superresolution, global motion estimation is usually adopted as image matching algorithm, and global motion parameters are obtained between low-resolution images.In this case, the motion parameters between high-resolution images can be directly obtained by transforming the motion parameters between low-resolution images.And thus the affine transformation is easily implemented at high resolution.In this paper, pixel-level motion fields are obtained by CPF based image matching.And the affine transformation should be performed pixel by pixel, which is difficult to implement.Moreover, the affine transformation between the input lowresolution images is already included in the process of CPF matching.Therefore, the affine transformation in the image acquisition model can be replaced by that in CPF matching, and a new framework of iterative back projection algorithm combined with CPF-IM (IBP-CPF-IM) is proposed.
The proposed IBP-CPF-IM algorithm starts with taking an input low-resolution image as the reference image.Then, CPF-IM is performed between the reference image and the other nonreference input low-resolution images to obtain pixel-level motion fields.And a series of simulated reference images  iL are produced from these nonreference low-resolution images using the motion fields.Each input low-resolution image corresponds to a simulated reference image.Next, an initial reconstructed superresolution image  0  is generated by interpolating the reference low-resolution image.Generally, better initial estimation directly leads to better quality of the final reconstructed image.In addition, high quality of the initial estimation image can speed up the convergence of the iteration algorithm.Finally, simulated low-resolution image   iL is generated through downsampling the reconstructed superresolution image.If the simulated low-resolution image   iL is consistent with the simulated reference images  iL , the reconstructed superresolution image is identical with the original high-resolution image.Otherwise, the differences between the simulated low-resolution image and the simulated reference images will be projected to the reconstructed superresolution image.This process is repeated iteratively until the differences between the simulated lowresolution image and the simulated reference images are small enough.The proposed IBP-CPF-IM algorithm can be formulated as where  iL represents the simulated reference image,   iL is the simulated low-resolution image through downsampling the superresolution image, and CPF(⋅) represents the operation which transforms input low-resolution image to simulated reference image.
In the original IBP algorithm, affine transformation is implemented between high-resolution images using global motion parameters.However, the affine transformation in the proposed algorithm is replaced by CPF matching between low-resolution images.And the convergence of the proposed algorithm is going to speed up.
The relative error used to control when to stop the iterative process is defined as follows: where ‖ ⋅ ‖ represents  2 norm.Iteration stops when error is smaller than a threshold (set as 10 −4 in our experiments) or the iteration number reaches the maximum value which is set as 100.
The processes of IBP-CPF-IM algorithm for video superresolution are presented as follows.
Step 1. Move the sliding window and take five video frames as the input low-resolution frames.In general, when the distance between the middle frame and the matching frame becomes larger, their differences also become larger.In this case, the accuracy of CPF matching will decrease, and subtle quality improvement of the generated frame can be obtained.Nevertheless, the computational complexity will increase greatly.To get a tradeoff between computational complexity and image quality, the length of sliding window  is set as five empirically.
Step 2. The middle frame of the sliding window is taken as the reference frame.CPF based image matching is conducted between the reference frame and the other frames in the sliding window.Four motion fields are obtained through this process.And four simulated reference frames are generated using the motion fields.
Step 3. The reference frame is interpolated to obtain the initial reconstructed superresolution frame.
Step 4. The simulated low-resolution frame is obtained through downsampling the reconstructed superresolution frame.
Step 5.If the simulated low-resolution frame is not the same as the simulated reference frame, differences between them are projected to the reconstructed superresolution frame and we return to Step 4. If the differences between the simulated low-resolution frame and the simulated reference frame are small enough, the reconstructed superresolution frame is the desired frame.
The flow of proposed iterative back projection algorithm is shown in Figure 4, and deblur block denotes the inverse matrix of .

Experiments on Static Image.
To evaluate the performance of the proposed algorithm, standard images are tested.The size of the input high-resolution image is 256 × 256.In order to evaluate the quality of the reconstructed image objectively, PSNR is calculated between the original high-resolution image and the reconstructed superresolution image.Table 1 gives the performance of the proposed algorithm and the compared algorithms.It is apparent that the proposed algorithm achieves an improvement of objective quality.

Experiments on Video.
To verify the effect of the proposed algorithm on video image superresolution reconstruction, a series of standard video sequences are tested.The tested videos are in format of YUV.Proposed algorithm is applied to each component of video.In order to calculate PSNR performance of reconstructed video, low-resolution videos are obtained through downsampling high-resolution videos.And the PSNR performance of the experiment results is calculated based on  component.For each sequence, only  the first 100 frames are used and the size of the video image is 176 × 144.In this experiment, the scheme of sliding window (Figure 2) is adopted.The pixel-level motion fields between the reference frame and the other frames are estimated by CPF-IM.Taking the scheme of sliding window into consideration, the length of the output reconstructed frames is 96.The proposed algorithm is compared with bilinear interpolated algorithm, bicubic interpolated algorithm, and IBP with optical flow based image matching algorithm (optical flow based image matching algorithms used in the experiment are proposed in [14] and [15], resp.).Thus the motion mode which frequency domain registration algorithm requests is only global translation or rotation.iterative back projection algorithm based on frequency domain registration is not suitable for video superresolution.For bilinear and bicubic interpolation, the output reconstructed frame is acquired by interpolating the reference frame.For each frame in the video, a stable result is obtained before iterative number reaches the maximum value.The PSNR performances of these algorithms are listed in Table 2. Compared to bicubic interpolation, IBP-CPF-IM achieves a linear average PSNR gain of 1.27 dB.These demonstrate that the amount of reconstruction information which we get from more than one frame is much more than that from only one frame, and the objective quality of the reconstructed frame by superresolution algorithm is generally higher than the reconstructed frame by interpolation.When compared with the IBP-OL2004 and IBP-OL2010 algorithms, the PSNR gain of IBP-CPF-IM is about 0.52 dB and 0.60 dB.The reason is that optical flow is sensitive to brightness change, which frequently occurs on video sequence.And the motion vectors cannot be accurately obtained by using optical flow based image matching.Figures 7 and 8 show examples of the experimental results about hall and container sequences.The original images are

Conclusion
The performance of superresolution reconstruction algorithm depends largely on the accuracy of image registration.Traditional registration approaches for superresolution reconstruction algorithm usually demand that the motion between images is global translation or rotation.For video frames, this does not hold.Moreover, block based matching and optical flow based matching algorithms have their own limitations.To improve the accuracy of image registration, this paper introduces multiresolutional critical-point filters based image matching algorithm (CPF-IM) to image/video Advances in Multimedia superresolution.The elastic motion model in CPF-IM can adapt to various nontranslational motions, such as zoom and rotation.The combination of IBP and CPF-IM, which is called IBP-CPF-IM, is applied to image and video superresolution.Experimental results show that the proposed approach achieves better performance in objective and subjective quality for both image and video frames, when compared with bilinear and bicubic interpolation algorithms and IBP-FR and IBP-OL algorithm.CPF-IM can also be integrated into other superresolution reconstruction algorithms, such as projections onto convex sets (POCS) algorithm and maximum a posteriori (MAP) algorithm.

Figure 3 :
Figure 3: Flow of the original IBP algorithm.

Figure 4 :
Figure 4: Flow of the proposed IBP with CPF-IM.

Table 1 :
PSNR of experimental image shown in Figures3 and 4and the other experimental images.