Sparse representation has recently attracted enormous interests in the field of image super-resolution. The sparsity-based methods usually train a pair of global dictionaries. However, only a pair of global dictionaries cannot best sparsely represent different kinds of image patches, as it neglects two most important image features: edge and direction. In this paper, we propose to train two novel pairs of Direction and Edge dictionaries for super-resolution. For single-image super-resolution, the training image patches are, respectively, divided into two clusters by two new templates representing direction and edge features. For each cluster, a pair of Direction and Edge dictionaries is learned. Sparse coding is combined with the Direction and Edge dictionaries to realize super-resolution. The above single-image super-resolution can restore the faithful high-frequency details, and the POCS is convenient for incorporating any kind of constraints or priors. Therefore, we combine the two methods to realize multiframe super-resolution. Extensive experiments on image super-resolution are carried out to validate the generality, effectiveness, and robustness of the proposed method. Experimental results demonstrate that our method can recover better edge structure and details.
In video surveillance, medical imaging, satellite observation, and other scenes, due to the imaging equipment, the hardware storage, natural environment, and other limited factors, we usually get low-resolution (LR) images [
As a hot research direction in the field of image processing, the problem of SR has been studied for more than three decades, and many SR approaches have been proposed. According to the number of input LR images, SR approaches can be broadly classified into two categories: single-image SR and multiframe SR [
Inspired by the work of [
The content of this paper is arranged as follows: Section
After downsampling
The quality of reconstructed image depends largely on the expression ability of overcomplete dictionary. In Yang et al. [
Based on the same sparse representation model (
Both Yang et al. [
It has been shown in [
As everyone knows, edge represents the large-scale structure of image and has the characteristics of smoothness, so human visual system is more sensitive to edge. Besides, image content is highly directional. In short, edge and direction are the most important features of an image. In order to better capture the intrinsic direction and edge characteristic of image, we design Direction and Edge dictionaries for different clusters of patches, instead of a global dictionary for all the patches.
Based on the consideration of the significant difference between edge pixels and neighborhood pixels and the strong direction performance of the image, we design a new pair of Direction and Edge templates, as Figure
Direction and Edge templates (from left to right: A template and B template).
Direction and Edge templates are used to guide the clustering of image patches and further to obtain Direction and Edge dictionaries. Firstly, each patch is clustered, and the training image patches are classified into two clusters, in which the criterion for clustering is Euclidean distance. Thus the Euclidean distances between the image patch and two templates are obtained and the smaller value determines which cluster the patch belongs to. Finally, two clusters are trained, respectively, to obtain two pairs of HR and LR dictionaries, which are referred to as the Direction and Edge dictionaries.
There are some advantages of Direction and Edge dictionaries: (i) the dictionaries are expected to better represent the intrinsic direction and edge characteristics of the natural images; (ii) the reconstructed HR image via the above dictionaries inherits the large-scale information of natural images and has more high-frequency information, which are the most important parts for SR; (iii) they reduce computational complexity due to the fact that structural dictionaries can be smaller than a global dictionary.
In order to improve the algorithm efficiency, our templates are at the size of 6 by 6. Compared with Farhadifard et al. [
The single-image SR based on Direction and Edge dictionaries includes three steps: tectonic training sets, Direction and Edge dictionary training, and image reconstruction, as shown in Figures
Two classes of tectonic HR and LR training sets.
Direction and Edge dictionary training.
Process of single-image SR.
(a) Take 91 natural images as HR image library, and the LR image library is comprised of LR images achieved from downsampling of HR images. To reach the HR image dimension, LR images are scaled up to the size of HR images via bicubic interpolation and are termed medium-resolution (MR) images.
(b) Take patches with five-pixel overlap from HR images
(c) Take the same size patch from MR image in the same position as HR image, and then use the first- and second-order gradients of the patches as the feature vector. Develop the first class LR (LR1) training set and the second class LR (LR2) training set by combining the corresponding class feature vectors.
(d) Extract image patch from HR-MR image to be columns feature vector, so as to develop the first class HR training set (HR1) and the second class HR training set (HR2) by collecting corresponding class feature vectors.
For first class, train LR1 training set by K-SVD algorithm to get first class LR dictionary
(a) Acquire MR image by interpolation amplification of the input LR image. Take patch with five-pixel overlap from MR image and classify the patches into two clusters by same method as above. Then get feature vectors by extracting the first- and second-order gradients of patches. Finally calculate the sparse coefficient
(b) Calculate high-frequency information of each patch from known
The results of single-image SR are showed in Section
The method of POCS is widely used for multiframe SR and easily available to introduce prior knowledge. However, it usually shows jagged edges in the reconstructed results when the upscaling factor is larger. Our method based on Direction and Edge dictionaries can recover more high-frequency information and preserve smooth edges. Therefore, we combine the POCS method with our single-image SR method to realize multiframe SR. It includes three steps: multiframe registration, POCS reconstruction, and single-image SR based on Direction and Edge dictionaries, like Figure
Process of multiframe SR.
In the stage of multiframe registration image, firstly extract feature points of input multiple images by SURF algorithm [
(a) Obtain LR sequence images via geometric distortion and downsampling of HR image. Then select the first frame as reference frame and other frames for the floating frame. Use SURF algorithm to extract feature points and RANSAC algorithm to remove the false matching points.
(b) The registration images are calculated on the basis of the affine transformation model with matching points.
Use POCS method to reconstruct the registration images by an upscaling factor
The result of POCS is magnified by our method by a factor of
In this section, we demonstrate the numerous experiments to verify the performance of our method. All the experiments are executed with MATLAB 8.3.0.
The experimental setting in this paper refers to Yang et al. [
We compare the proposed single-image SR based on Direction and Edge dictionaries with the bicubic interpolation method and several state-of-the-art SR methods, including Yang et al. [
SR results on image Plant (the upscaling factor 2).
SR results on image Parrot (the upscaling factor 2).
SR results on image Comic (the upscaling factor 2).
PSNR (dB) results by different methods (the upscaling factor 2).
Images | Bicubic | Yang | Zedye | NCSR | ANR | CSC | Our |
---|---|---|---|---|---|---|---|
Butterfly | 27.43 | 30.16 | 29.93 | 29.39 | 29.67 | 31.95 | 29.69 |
Child | 31.93 | 33.36 | 33.26 | 30.84 | 33.20 | 33.68 | 33.19 |
Hat | 31.73 | 33.50 | 33.28 | 31.00 | 33.24 | 34.74 | 33.30 |
Lena | 32.70 | 34.48 | 34.19 | 31.31 | 34.28 | 35.63 | 34.27 |
Parrot | 31.25 | 33.45 | 32.96 | 30.80 | 33.18 | 34.56 | 33.21 |
Plant | 34.30 | 36.56 | 36.37 | 32.53 | 36.28 | 38.75 | 36.24 |
Parthenon | 28.07 | 29.09 | 28.98 | 28.03 | 28.87 | 29.49 | 28.91 |
Bike | 25.64 | 27.68 | 27.39 | 26.87 | 27.52 | 28.93 | 27.57 |
Comic | 26.01 | 27.71 | 27.42 | 26.96 | 27.52 | 28.40 | 27.55 |
Flower | 30.36 | 32.28 | 31.97 | 30.33 | 31.96 | 33.13 | 32.05 |
Foreman | 32.76 | 34.08 | 35.92 | 31.47 | 35.83 | 36.62 | 34.07 |
Girl | 34.74 | 35.53 | 35.45 | 31.92 | 35.55 | 35.68 | 35.48 |
Pepper | 33.15 | 34.08 | 36.31 | 31.64 | 36.01 | 36.90 | 34.04 |
Raccoon | 30.95 | 32.38 | 32.04 | 29.97 | 32.33 | 32.96 | 32.43 |
Woman | 32.14 | 34.37 | 34.20 | 31.44 | 34.13 | 35.31 | 34.04 |
Zebra | 30.63 | 33.20 | 32.92 | 30.91 | 32.70 | 33.69 | 32.83 |
|
|||||||
Average | 30.862 | 32.619 | 32.662 | 30.338 | 32.642 | 33.776 | 32.429 |
SSIM results by different methods (the upscaling factor 2).
Images | Bicubic | Yang | Zedye | NCSR | ANR | CSC | Our |
---|---|---|---|---|---|---|---|
Butterfly | 0.9086 | 0.9400 | 0.9406 | 0.8402 | 0.9353 | 0.9591 | 0.9336 |
Child | 0.8922 | 0.9200 | 0.9166 | 0.7719 | 0.9190 | 0.9232 | 0.9187 |
Hat | 0.8898 | 0.9145 | 0.9151 | 0.7213 | 0.9162 | 0.9301 | 0.9104 |
Lena | 0.8990 | 0.9249 | 0.9215 | 0.7504 | 0.9240 | 0.9372 | 0.9219 |
Parrot | 0.9270 | 0.9450 | 0.9439 | 0.7590 | 0.9453 | 0.9538 | 0.9425 |
Plant | 0.9310 | 0.9521 | 0.9530 | 0.7873 | 0.9535 | 0.9675 | 0.9493 |
Parthenon | 0.7932 | 0.8357 | 0.8290 | 0.7181 | 0.8287 | 0.8466 | 0.8305 |
Bike | 0.8433 | 0.8987 | 0.8922 | 0.8238 | 0.8953 | 0.9217 | 0.8940 |
Comic | 0.8411 | 0.8973 | 0.8896 | 0.8328 | 0.8929 | 0.9143 | 0.8919 |
Flower | 0.8896 | 0.9202 | 0.9171 | 0.7955 | 0.9192 | 0.9309 | 0.9167 |
Foreman | 0.9450 | 0.9570 | 0.9590 | 0.7822 | 0.9584 | 0.9637 | 0.9551 |
Girl | 0.8450 | 0.8717 | 0.8664 | 0.7337 | 0.8707 | 0.8744 | 0.8706 |
Pepper | 0.9917 | 0.9954 | 0.9960 | 0.8923 | 0.9967 | 0.9970 | 0.9953 |
Raccoon | 0.8419 | 0.8929 | 0.8817 | 0.7871 | 0.8904 | 0.8969 | 0.8921 |
Woman | 0.9428 | 0.9592 | 0.9592 | 0.7982 | 0.9593 | 0.9663 | 0.9565 |
Zebra | 0.9860 | 0.9971 | 0.9959 | 0.9504 | 0.9970 | 0.9976 | 0.9969 |
|
|||||||
Average | 0.8980 | 0.9264 | 0.9236 | 0.7965 | 0.9251 | 0.9363 | 0.9235 |
The experiments aim to obtain a HR image (512 × 512) from 10 frames LR image (128 × 128) by an upscaling factor of 4 (
In this part, we perform SR experiments on multiframe images and the upscaling factor is 4. However, most of the state-of-the-art SR methods are for single-image SR and the upscaling factor is 2 or 3. So we compare our method with the bicubic interpolation method and POCS. As to bicubic interpolation method, we directly magnify the second frame image with a factor 4.
In order to verify the good robustness of our method for different kinds of images, Table
PSNR (dB) and SSIM results of multiframe SR (the upscaling factor 4).
Images | PSNR | SSIM | ||||
---|---|---|---|---|---|---|
Bicubic | POCS | Our | Bicubic | POCS | Our | |
Lena | 23.5296 | 23.1444 |
|
0.5720 | 0.5483 |
|
Monarch | 19.4358 | 18.5742 |
|
0.6751 | 0.6427 |
|
Pepper | 23.6443 | 23.5289 |
|
0.6761 | 0.6631 |
|
Child | 23.9282 | 23.0742 |
|
0.7041 | 0.6622 |
|
|
||||||
Average | 22.6345 | 22.0804 |
|
0.6568 | 0.6291 |
|
Figures
Results of Lena (the upscaling factor 4). Smaller: input images. Larger: from left to right and top to bottom: bicubic, POCS, our method, and original image.
Results of the Monarch (the upscaling factor 4). Smaller: input images. Larger: from left to right and top to bottom: bicubic, POCS, our method, and original image.
Results of the Pepper (the upscaling factor 4). Smaller: input images. Larger: from left to right and top to bottom: bicubic, POCS, our method, and original image.
In this paper, we present a novel approach for image super-resolution based on sparse representation in terms of Direction and Edge dictionaries. The key idea is to classify image patches based on their direction and edge features and selectively code each patch using more appropriate dictionary. According to the Euclidean distances between image patch and two new templates, image patches are divided into two clusters and then are trained to obtain two pairs of Direction and Edge dictionaries. Single-image experimental results indicate the usefulness of the proposed Direction and Edge dictionaries. Furthermore, we combine the POCS with our single-image SR method to realize multiframe SR, especially when upscaling factor is larger, while the experiments show that it has the same satisfactory results. In short, our proposed method achieves not only competitive PSNR and SSIM values, but also more pleasant visual quality of image edge structures and texture.
The authors declare that there are no conflicts of interest regarding the publication of this paper.