Mathematical Problems in Engineering
Volume 2021 (2021), Article ID 9944385, 11 pages
https://doi.org/10.1155/2021/9944385

Research Article

Single-Image Super-Resolution Using Panchromatic Gradient Prior and Variational Model

Yingying Xu, Jianhua Li, Haifeng Song, and Lei Du

School of Electronics and Information Engineering, Taizhou University, Taizhou 318000, China

Correspondence should be addressed to Yingying Xu; yyxu@tzc.edu.cn

Received 6 March 2021; Revised 12 April 2021; Accepted 25 April 2021; Published 3 May 2021

Academic Editor: Taseer Muhammad

Copyright © 2021 Yingying Xu et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

Single-image super-resolution (SISR) is a resolution enhancement technique and is known as an ill-posed problem. Motivated by the idea of pan-sharping, we propose a novel variational model for SISR. The structure tensor of the input low-resolution image is exploited to obtain the gradient of an imaginary panchromatic image. Then, by constraining the gradient consistency, the image edges and details can be better recovered during the procedure of restoration of high-resolution images. Besides, we resort to the nonlocal sparse and low-rank regularization of image patches to further improve the super-resolution performance. The proposed variational model is efficiently solved by ADMM-based algorithm. We do extensive experiments in natural images and remote sensing images with different magnifying factors and compare our method with three classical super-resolution methods. The subjective visual impression and quantitative evaluation indexes both show that our method can obtain higher-quality results.

1. Introduction

Image super-resolution (SR) is one of the most fundamental problems in the field of image processing, which aims to reconstruct clear and accurate high-resolution images from degraded low-resolution images. In other words, SR technology can recover a high-resolution (HR) image from one or more low-resolution (LR) input images [1]. In recent years, image SR has attracted great attention from academia and industry. The quality of the reconstructed SR image also greatly affects the accuracy of other computer vision tasks (such as image classification, image segmentation, and target detection). At present, SR technology has made great progress; it has been widely used in high-definition digital television, remote sensing monitoring, medical image reconstruction, image restoration, military reconnaissance, and other fields [2]. However, as an ill-posed problem, the existing image SR technology cannot always obtain satisfactory reconstruction results. Therefore, the study of SR is still a challenging but significant research topic.

According to the number of input LR images, the SR can be classified into multi-image super-resolution (MISR) [3–5] and single-image super-resolution (SISR) [6–8]. MISR uses the complementary information provided by multiple images of the same scene to enhance resolution. However, in a practical application, it is much difficult to obtain image sequences of the same scene and achieve precise subpixel registration. Meanwhile, SISR is suitable for more scenarios, which enhances the resolution of one input image based on some prior information [9, 10]. To date, SISR methods can be broadly classified into interpolation-based methods, reconstruction-based methods, and learning-based methods [11].

Interpolation methods like nearest-neighbor interpolation, bilinear interpolation, and bicubic Interpolation [12, 13] estimate the intensity at a point using the information of adjacent pixels. While these methods are easier calculated and very speedy, they tend to generate excessive smoothing and jagged artifacts. Reconstruction-based SISR methods [14–16] generally utilize image priors to restrict the possible solution space with an advantage of recovering sharp details. However, these methods are usually time-consuming, and their performance suffers a rapid degradation when the amplification factor increases.

Learning-based SISR methods include example-based methods, neighbor-embedding-based methods, sparse-representation-based methods, and deep-learning-based methods [17, 18]. The basic idea of these approaches is to obtain the correspondence between the LR image and the HR image by a priori knowledge training and then reconstruct an HR image from the input LR image using the correspondence that has been learned. Freeman et al. [19] presented an example-based method with a Markov network. Chang et al. [20] proposed a neighbor-embedding-based method that needs less training set. Yang et al. [21] proposed a sparse-representation-based algorithm to obtain more favorable results and further improve their work by jointly training coupled dictionaries for LR and HR image patch pairs [22]. Very recently, deep-learning-based SISR methods, using convolutional network [23, 24], residual network [25, 26], generative adversarial network [27], attention network [28, 29], and so on, have become popular and demonstrated good performance with the rapid development of deep learning technology.

Pan-sharpening [30] is a multispectral image fusion and super-resolution technique; the panchromatic (PAN) image has higher spatial resolution than the corresponding multispectral image but only has a single band. Pan-sharpening fuses PAN image and multispectral image together to enhance both spatial and spectral resolutions of the data. Motivated by the idea of pan-sharpening, we present a novel variational approach for single-image super-resolution. In this paper, we assume that there is a PAN image of the input LR image. By utilizing the structure tensor [31] of input image, we can construct the gradient of the PAN image; then we build a variational model combined with low-rank and sparse representation of similar patches to effectively fuse the constructed PAN information with LR image and obtain HR image. The proposed approach addresses the super-resolution problem of both single nature image and the multispectral data without panchromatic image.

The remainder of this paper is organized as follows. In Section 2, we present the proposed SISR model in detail. The numerical algorithm for solving our model is given in Section 3. The experimental results and analysis are discussed in Section 4. Finally, conclusion is presented in Section 5. In addition, the descriptions of the acronyms used in this paper are listed in Table 1.

Table 1: The descriptions of the acronyms used in this paper.

Acronym
Description

SR
Super-resolution

SISR
Single-image super-resolution

MISR
Multi-image super-resolution

HR
High resolution

LR
Low resolution

PAN
Panchromatic

ADMM
Alternating direction method of multipliers

PSNR
Peak signal-to-noise ratio

RMSE
Root mean square error

SSIM
Structure similarity index

SC
Sparse-coding-based super-resolution method

SCBP
Iterative-back-projection-based SC method

2. Variational Model

Drawing on the idea of pan-sharpening, we construct a variational fusion model to realize single-image super-resolution. As panchromatic image contains the edges and details needed for resolution enhancement, we first construct the information of PAN image which does not actually exist. According to [31], the key information of the assumed PAN image can be exploited from the structure tensor of the input image. Then the gradient of PAN image can be fused into the LR image to enhance the HR details. The similar image patch pairs extracted from LR and HR images are constrained by low-rank and sparse regularization to improve the SR performance.

2.1. Image Degradation Model

We consider that the observed low-resolution image is a downsampled version of the high-resolution image , and the degradation model [32] can be written as

where represents a downsampling operator and is the additive Gaussian white noise.

2.2. Construct the Gradient of PAN Image

We construct the gradient of the PAN image corresponding to input LR image using the knowledge of structure tensor [31]. Firstly, we apply quadratic interpolation to the given LR image to obtain the enlarged image in desired scale. Then, matrix known as the structure tensor [31] can be expressed as

where denotes the number of spectral bands; denotes the weight and it can be determined by  = .

As a symmetric matrix, can be decomposed as

where is an orthogonal matrix and the columns of which are the eigenvectors of ; is a diagonal matrix and the diagonal elements of which are the eigenvalues of . Equation (3) can also be written as

where is the maximum eigenvalue and it gives the maximum rate of change of ; is the minimum eigenvalue and it gives the minimum rate of change. The corresponding eigenvectors and give the directions of change.

The PAN image can theoretically capture the basic geometry of the image . Consider image with  =  and  = 0, whose structure tensor should approximately be equal to . Thus, we have the following equation:

In order to solve , equation (5) can be rewritten as follows:

To specify the sign of the eigenvectors, the target gradient of the constructed PAN image is thus obtained as

2.3. Variational Model Based on Gradient Consistency

In this subsection, we build a variational model to fuse the low-resolution image and the gradient of its corresponding PAN image ; then the model is optimized together with the nonlocal sparse and low-rank regularization to obtain the high-resolution image .

2.3.1. Consistency of Gradient

Bands of an ideal HR image often have similar structural information with the PAN image ; namely, bands of HR image closely approximate the PAN image in gradient information. Therefore, we can use constraint about the consistency of gradient to establish a relationship model. We use to denote the -th band of image and then set up the gradient consistency regularization term as follows:

where denotes the weight of the -th band in , and we generally set in the absence of special requirements.

2.3.2. Sparse and Low-Rank Decomposition

Motivated by the effective work of optical flow estimation with nonlocal sparse and low-rank regularization in [33], we adopt a joint sparse and low-rank decomposition term formulated as follows:

where denotes the operator that extracts the -th patch of image and groups its nonlocal similar patches together. Then the grouped similar patches are decomposed to the low-rank component and the sparse component , and denotes the Gaussian white noise.

2.3.3. Total Energy

Combining the equality constraint in (1), (8), and (9), the proposed variational SISR model can be formulated as the following optimization objective:

where is the number of image patches that are extracted from the -th band of image , is the number of spectral bands, are parameters, denotes the rank of low-rank component, and norm denotes the sparse regularization. It is well known that the rank-solution and norm are discrete combinatorial optimization problems; they are both NP-hard. Therefore, in order to make the minimization tractable, we resort to their convex relaxed modification, and model (10) can be reformulated as the following convex optimization problem:

where denotes the nuclear norm, which is the convex envelope (tightest convex surrogate) of rank, and norm is the nearest convex norm to norm.

3. Numerical Algorithm

In this section, the numerical procedure of the proposed model will be implemented. We adopt the alternating direction method of multipliers (ADMM) [34] to solve the minimization problem (11). The flexibility of the ADMM lies in the fact that it splits the initial optimization problem into several subproblems.

3.1. Solution of the Sparse Matrix

For each band, we fix and the low-rank matrix ; the optimization problem of sparse matrix is as follows:

We can obtain the solution of equation (12) by soft threshold algorithm as

where .

3.2. Solution of the Low-Rank Matrix

For each band, we fix and the sparse matrix ; the optimization problem of low-rank matrix is as follows:

We can obtain the solution of equation (14) by singular value threshold algorithm as

where , are singular values of matrix , and are left and right singular vectors of , and .

3.3. Solution of the SR Image

In order to separately solve the fidelity term and two regularization terms in (11), we introduce two intermediate variables and which are close to . Thus, we need to iteratively solve the three following subproblems to obtain the solution of SR image.

(1)We use the intermediate variable to replace in . The optimization problem for is as follows:

 where denotes the inner product, denotes the Lagrange multiplier matrix, and is a positive penalty parameter. We set the derivative of (16) with respect to to 0:

 which yields

 The Lagrange multiplier is updated as follows:

(2)We introduce an intermediate variable to replace in the gradient consistency term. The optimization problem for is as follows:

 where denotes the Lagrange multiplier matrix. We set the derivative of (20) with respect to to 0:

 Using the fast Fourier transform (FFT), we can obtain the closed-form solution of from (21) directly:

 where and denote the FFT and inverse FFT, respectively. The Lagrange multiplier is updated as follows:

(3)After and are substituted into equation (11), the optimization subproblem of becomes

 We set the derivative of (24) with respect to to 0, which yields

Overall, taking all above analyses into account, we can summarize the complete numerical procedure for the proposed method. The detailed descriptions are shown in Algorithm 1.

(i)
Input:

    The LR image ;

    Magnification factor.

Output:

    The SR image .

Initialization:

    Compute an initial SR image via bicubic

    interpolation;

    Calculate the target gradient via equation (7).

For:

    For:

        1. For each extracted image patch , search

            for a set of similar patches across the whole

            image by patch matching.

        2. Decompose the grouped patches into low-rank

            and sparse components.

            (a) Calculate the sparse matrix via (13);

            (b) Calculate the low-rank matrix via (15).

        3. Reconstruct the SR image .

            (a) Update the image via (25);

            (b) Update the intermediate variable and via (18) and (22);

            (c) Update the Lagrange multiplier and via (19) and (23);

            (d)  =  ;

    End;

End;

    =

Algorithm 1: Algorithm 1A variational approach for single-image SR.

4. Experimental Results and Analysis

In this section, we first evaluate the sensitivity of the proposed algorithm with respect to the main parameters, that is, , , , and . Then, to demonstrate the SR effectiveness on both natural images and remote sensing images with different magnifying factors (2 and 4), we compare our method with the Bicubic interpolation method, sparse-coding-based method (SC) [22], and its back-projection enhanced version (SCBP) [35]. We evaluate the outcome of various methods by using quantitative indexes: peak signal-to-noise ratio (PSNR), root mean square error (RMSE), and structure similarity index (SSIM) [36]; they are calculated on RGB channels for nature image and on each band for multispectral image. The higher PSNR and SSIM values and lower RMSE value represent the better SR performance.

4.1. Adjustment of Parameters

The parameter is the weight to the consistency of gradient, controls the low-rank and sparse regularization of the similar patches, and and control the contribution of low-rank and sparse regularization terms. We adjust the parameter values in proper ranges and demonstrate the average quantitative metrics as functions of the parameters on the experimental dataset; then the optimal parameter combination is found with extensive numerical tests. Figure 1 shows the average PSNR, RMSE, and SSIM results when the parameter varies from 0 to 0.5 with fixed  = 1.6,  = 0.2, and . From Figure 1, we can see that when , all quantitative metrics get the worst values, which illustrate the effectiveness of the gradient consistency constraint. Meanwhile, Figure 1 shows that our method achieves the best performance when . Figure 2 shows the average PSNR, RMSE, and SSIM results under various combinations of and with fixed and . The analysis of and shows that the trend of the overall performance gets better and then worse. When the maximal PSNR is selected, we have  = 1.8 and  = 0.3; when the maximal SSIM is selected, we have  = 1.4 and  = 0.1. In order to synthesize the performance of our algorithm in these two indexes, we normalize PSNR and SSIM by introducing E = 0.8PSNR + 0.2SSIM, and E gets its maximum when  = 1.6 and  = 0.2. Besides, we find that the performance of the proposed method is insensitive to the parameter , and we simply set .

(a)

(b)

(c)

Figure 1: SR results under different . (a) PSNR; (b) RMSE; (c) SSIM.

(a)

(b)

(c)

Figure 2: SR results under different and . (a) PSNR; (b) RMSE; (c) SSIM.

4.2. SR Results and Analysis

4.2.1. Natural Images Super-Resolution

To evaluate the performance of the proposed super-resolution algorithm, we use bicubic interpolation and SC and SCBP algorithms for comparison. Experimental results of the images “Flower,” “Leaf,” and “Lena” with magnification factor of 2 are shown in Table 2 and Figure 3, while the results with magnification factor of 4 are shown in Table 3 and Figure 4.

Table 2: SR results of different methods with magnification factor of 2.

Images
Bicubic
SC
SCBP
Ours

Flower
PSNR
32.7356
32.7032
34.8588
36.5869

RMSE
5.8852
5.9072
4.6089
3.7774

SSIM
0.9163
0.8998
0.9295
0.9351

Leaf
PSNR
23.7807
23.2820
25.8536
28.6582

RMSE
16.5009
17.4760
12.9975
9.4108

SSIM
0.8650
0.8386
0.8997
0.9323

Lena
PSNR
32.0533
31.7019
34.0844
36.1762

RMSE
6.3661
6.6289
5.0387
3.9603

SSIM
0.9143
0.8939
0.9289
0.9319

Figure 3: SR results of images “Flower,” “Leaf,” and “Lena” with magnification factor of 2. Each odd line from left to right: original images; results by bicubic interpolation; results by SC method; results by SCBP method; results by our method. Each even line from left to right: the magnification of the small red rectangle in the corresponding each example.

Table 3: SR results of different methods with magnification factor of 4.

Images
Bicubic
SC
SCBP
Ours

Flower
PSNR
26.1123
28.6254
28.4408
28.8676

RMSE
12.6160
9.4465
9.6494
9.1867

SSIM
0.7416
0.7922
0.7856
0.7969

Leaf
PSNR
17.2636
19.3226
19.0129
19.3259

RMSE
34.8430
27.5682
28.5689
27.5578

SSIM
0.5081
0.5919
0.5905
0.6295

Lena
PSNR
25.4408
27.9062
27.4987
28.0027

RMSE
13.6302
10.2619
10.7548
10.1486

SSIM
0.7381
0.7833
0.7761
0.7996

Figure 4: SR results of images “Flower,” “Leaf,” and “Lena” with magnification factor of 4. Each line from left to right: original images; results by bicubic interpolation; results by SC method; results by SCBP method; results by our method.

As we can see from the visual comparison, our method could obtain desired results; it effectively reduces zigzag and distortion of the recovered SR images so that it better preserves the image details compared to other competitors. From the close-ups of the regions in red boxes, we can see clearly that the SR images obtained by bicubic interpolation and SC methods are more blurred than ours, and SCBP method could restore clear edges but generate some obvious artifacts in SR images. In summary, our algorithm is superior to the contrast methods in terms of reconstructing clearer SR images with sharper edges and without artifacts. The above observations can be quantitatively confirmed by the Tables 2 and 3 which record the PSNR, RMSE, and SSIM values of the SR results of the three images with magnification factors of 2 and 4, respectively. The proposed method obtains the best metrics values compared with the other three methods.

4.2.2. Multispectral Images Super-Resolution

The adopted multispectral images contain four bands (red, blue, green, and infrared bands), and our algorithm is executed at each band. Same as the nature image experiments, we compared the performance of our method with bicubic interpolation, SC, and SCBP. Experimental results for the multispectral datasets “Field,” “Tree,” and “Beach” with magnification factor of 2 are shown in Figure 5 and Table 4, while the results for magnification factor of 4 are shown in Table 5.

Figure 5: SR results of images “Field,” “Tree,” and “Beach” with magnification factor of 2. Each odd line from left to right: original images; results by bicubic interpolation; results by SC method; results by SCBP method; results by our method. Each even line from left to right: the magnification of the small red rectangle in the corresponding each example.

Table 4: SR results of different methods with magnification factor of 2.

Images
Bicubic
SC
SCBP
Ours

Field
PSNR
24.4504
24.5058
22.7932
26.8614

RMSE
15.2764
15.1792
18.4876
11.5736

SSIM
0.7493
0.7475
0.6872
0.8255

Tree
PSNR
23.5264
24.0390
22.1848
25.9000

RMSE
16.9911
16.0166
19.8289
12.9283

SSIM
0.7237
0.7248
0.6664
0.8139

Beach
PSNR
22.9976
22.9150
20.9018
24.7080

RMSE
18.0577
18.2300
22.9854
14.8300

SSIM
0.7427
0.7401
0.6821
0.8166

Table 5: SR results of different methods with magnification factor of 4.

Images
Bicubic
SC
SCBP
Ours

Field
PSNR
19.5392
19.5944
17.8861
21.0251

RMSE
26.8894
26.7189
32.8934
22.6614

SSIM
0.4536
0.4537
0.3825
0.5437

Tree
PSNR
19.0398
19.5148
17.6925
20.5211

RMSE
28.4806
26.9560
32.9877
24.0150

SSIM
0.4018
0.4138
0.3510
0.5144

Beach
PSNR
18.3940
18.2420
16.4463
19.5784

RMSE
30.6790
31.2204
38.3908
26.7681

SSIM
0.4049
0.3946
0.3396
0.5010

The observation of visual comparison is close to that of natural image experiment. Bicubic interpolation and SC algorithms tend to reconstruct overly smooth images that lack detailed information and have fuzzy image quality, while the SR images obtained by SCBP method are overenhanced and have obvious artifacts. Comparatively speaking, our algorithm can better restore the detailed information of images and make a better texture effect. By comparing the numerical results in Tables 4 and 5, our algorithm also gets the best results for all indicators. It is worth noting that, because of the lack of training set, the results of SCBP algorithm are worse than bicubic interpolation. In conclusion, our proposed algorithm is superior to other algorithms compared in the subjective visual impression and objective indexes.

5. Conclusion

This paper presented an efficient variational method for single-image super-resolution. Motivated by image fusion and pan-sharpening, we exploit the gradient of imaginary PAN image from the input image via structure tensor and then construct a gradient consistency constraint which is supposed to fuse HR edges and details into the optimization target image. Meanwhile, we adopt the regularization of sparse and low-rank decomposition for similar image patches to further improve the SR accuracy. The variables of our variational model are iteratively optimized by ADMM-based algorithm. Extensive experiment results demonstrate the edge recovery and artifacts suppression capabilities of our method.

Data Availability

The datasets used to support the findings of this study are public available and included within the article.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

Yingying Xu was supported by the School of Electronics and Information Engineering, Taizhou University, Zhejiang, China. This research was founded by Zhejiang Provincial Natural Science Foundation of China under Grant no. LQ21F020001, Research Project of Education Department of Zhejiang Province under Grant no. Y201840245, and Agricultural Science and Technology Project of Taizhou under Grant no. 20ny13.

References

D. Glasner, S. Bagon, and M. Irani, “Super-resolution from a single image,” in Proceedings of the IEEE 12th International Conference on Computer Vision, pp. 349–356, Kyoto, Japan, October 2009.

S. Yang, M. Wang, Y. Sun, F. Sun, and L. Jiao, “Compressive sampling based single-image super-resolution reconstruction by dual-sparsity and non-local similarity regularizer,” Pattern Recognition Letters, vol. 33, no. 9, pp. 1049–1059, 2012.

Y. Wang, L. Wang, J. Yang, W. An, J. Yu, and Y. Guo, “Spatial-angular interaction for light field image super-resolution,” in Computer Vision–ECCV 2020, Springer, Berlin, Germany, 2020.

X. Ying, L. Wang, Y. Wang, W. Sheng, W. An, and Y. Guo, “Deformable 3d convolution for video super-resolution,” IEEE Signal Processing Letters, vol. 27, pp. 1500–1504, 2020.

Y. Tian, Y. Zhang, Y. Fu, and C. Xu, “Tdan: temporally-deformable alignment network for video super-resolution,” in Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3357–3366, Seattle, WA, USA, June 2020.

H. Su, N. Jiang, Y. Wu, and J. Zhou, “Single image super-resolution based on space structure learning,” Pattern Recognition Letters, vol. 34, no. 16, pp. 2094–2101, 2013.

H. Liu, Z. Fu, J. Han, L. Shao, S. Hou, and Y. Chu, “Single image super-resolution using multi-scale deep encoder-decoder with phase congruency edge map guidance,” Information Sciences, vol. 473, pp. 44–58, 2019.

S. A. Hussein, T. Tirer, and R. Giryes, “Correction filter for single image super-resolution: robustifying off-the-shelf deep super-resolvers,” in Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1425–1434, Seattle, WA, USA, June 2020.

C. Ren, X. He, Y. Pu, and T. Q. Nguyen, “Enhanced non-local total variation model and multi-directional feature prediction prior for single image super resolution,” IEEE Transactions on Image Processing, vol. 28, no. 8, pp. 3778–3793, 2019.

C. Ma, Y. Rao, Y. Cheng, C. Chen, J. Lu, and J. Zhou, “Structure-preserving super resolution with gradient guidance,” in Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7766–7775, Seattle, WA, USA, June 2020.

W. Yang, X. Zhang, Y. Tian, W. Wang, J.-H. Xue, and Q. Liao, “Deep learning for single image super-resolution: a brief review,” IEEE Transactions on Multimedia, vol. 21, no. 12, 2019.

R. Keys, “Cubic convolution interpolation for digital image processing,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 29, no. 6, pp. 1153–1160, 1981.

H. S. Hou and H. Andrews, “Cubic splines for image interpolation and digital filtering,” IEEE Transactions on Acoustics, Speech and Signal Processing, vol. 26, no. 6, pp. 508–517, 1978.

M. Irani and S. Peleg, “Improving resolution by image registration,” CVGIP: Graphical Models and Image Processing, vol. 53, no. 3, pp. 231–239, 1991.

A. J. Patti, M. I. Sezan, and A. Murat Tekalp, “Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time,” IEEE Transactions on Image Processing, vol. 6, no. 8, pp. 1064–1076, 1997.

R. R. Schultz and R. L. Stevenson, “Extraction of high-resolution frames from video sequences,” IEEE Transactions on Image Processing, vol. 5, no. 6, pp. 996–1011, 1996.

C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 295–307, 2016.

S. Gao and X. Zhuang, “Multi-scale deep neural networks for real image super-resolution,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA, June 2019.

W. T. Freeman, E. Pasztor, and O. Carmichael, “Learning low-level vision,” International Journal of Computer Vision, vol. 40, no. 1, pp. 25–47, 2000.

H. Chang, D. Yeung, and Y. Xiong, “Super-resolution through neighbor embedding,” in Proceedings of the 2004 IEEE Computer Society Conference on computer Vision and pattern recognition 2004, CVPR 2004, pp. 275–282, Washington, DC, USA, July 2004.

J. Yang, J. Wright, T. Huang, and Y. Ma, “Image super-resolution as sparse representation of raw image patches,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2008, CVPR 2008, pp. 1–8, Anchorage, AK, USA, June 2008.

J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Transactions on Image Processing, vol. 19, no. 11, pp. 2861–2873, 2010.

W. Shi, J. Caballero, F. Huszár et al., “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883, Las Vegas, NV, USA, June 2016.

J. Kim, J. Kwon Lee, and K. Mu Lee, “Accurate image super-resolution using very deep convolutional networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654, Las Vegas, NV, USA, June 2016.

B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144, Honolulu, HI, USA, July 2017.

Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2472–2481, Salt Lake City, UT, USA, June 2018.

C. Ledig, L. Theis, F. Huszár et al., “Photo-realistic single image super-resolution using a generative adversarial network,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681–4690, Honolulu, HI, USA, July 2017.

Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in Computer Vision–ECCV 2018, pp. 286–301, Springer, Berlin, Germany, 2018.

T. Dai, J. Cai, Y. Zhang, S. Xia, and L. Zhang, “Second-order attention network for single image super-resolution,” in Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11 057–111 066, Long Beach, CA, USA, June 2019.

F. Fang, F. Li, C. Shen, and G. Zhang, “A variational approach for pan-sharpening,” IEEE Transactions on Image Processing, vol. 22, no. 7, pp. 2822–2834, 2013.

G. Piella, “Image fusion for enhanced visualization: a variational approach,” International Journal of Computer Vision, vol. 83, no. 1, pp. 1–11, 2009.

K. I. Kim and Y. Kwon, “Single-image super-resolution using sparse regression and natural image prior,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 6, pp. 1127–1133, 2010.

W. Dong, G. Shi, X. Hu, and Y. Ma, “Nonlocal sparse and low-rank regularization for optical flow estimation,” IEEE Transactions on Image Processing, vol. 23, no. 10, pp. 4527–4538, 2014.

E. Esser, “Primal dual algorithms for convex models and applications to image restoration, registration and nonlocal inpainting,” University of California Los Angeles, 2010, Ph.D. thesis.

C. Liu, F. Fang, Y. Xu, and C. Shen, “Single image super-resolution based on nonlocal sparse and low-rank regularization,” Springer, Berlin, Germany, 2016.

X. Gao, K. Zhang, D. Tao, and X. Li, “Joint learning for single-image super-resolution via a coupled constraint,” IEEE Transactions on Image Processing, vol. 21, no. 2, pp. 469–480, 2011.