Mathematical Problems in Engineering

Volume 2014 (2014), Article ID 230348, 8 pages

http://dx.doi.org/10.1155/2014/230348

Research Article

Local and Nonlocal Regularization to Image Interpolation

Yi Zhan,1,2 Sheng Jie Li,³ and Meng Li⁴

¹College of Computer Science, Chongqing University, Chongqing 400030, China

²College of Mathematics and Statistics, Chongqing Technology and Business University, Chongqing 400067, China

³College of Mathematics and Science, Chongqing University, Chongqing 400044, China

⁴Department of Mathematics & KLDAIP, Chongqing University of Arts and Sciences, Yongchuan, Chongqing 402160, China

Received 13 November 2013; Revised 15 January 2014; Accepted 6 February 2014; Published 12 March 2014

Academic Editor: Yang Tang

Copyright © 2014 Yi Zhan et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

This paper presents an image interpolation model with local and nonlocal regularization. A nonlocal bounded variation (BV) regularizer is formulated by an exponential function including gradient. It acts as the Perona-Malik equation. Thus our nonlocal BV regularizer possesses the properties of the anisotropic diffusion equation and nonlocal functional. The local total variation (TV) regularizer dissipates image energy along the orthogonal direction to the gradient to avoid blurring image edges. The derived model efficiently reconstructs the real image, leading to a natural interpolation which reduces blurring and staircase artifacts. We present experimental results that prove the potential and efficacy of the method.

1. Introduction

Image interpolation has been an active research topic in both academia and industry due to its huge number of applications. It aims at producing a high-resolution (HR) image from its low-resolution (LR) counterpart . The relationship between HR and LR images can be described as

where the matrix combines both filtering and downsampling processes. Equation (1) is an ill-posed inverse problem since the null space of is nonzero as stated in [1]. This ill-posed inverse problem is generally approached in a regularization-based framework, which was formulated as an energy functional,

where is a regularization parameter that controls the tradeoff between the data fidelity term and the regularization term . The data fidelity function generally is formulated in the classical least-squares sense as .

Many regularizers have been developed to specify (2). Generally speaking, variational regularization approaches to image processing can be divided into two categories: the local regularization and the nonlocal regularization. In the local regularization approaches, the -Laplacian functional and variations of it have been extensively studied in the literature. It possesses the following form:

TV-based regularization, , as first proposed by Rudin et al. [2] does an excellent job at preserving edges in image denoising, image interpolation [1], image restoration [3–7], and image inpainting [8]. Choosing results in isotropic diffusion which solves the staircasing problem. Together with other constraints, it formulates the regularization for image segmentation [9]. Different values of result in anisotropic diffusion between TV-based and isotropic smoothing.

These local regularization methods which penalize derivatives involve only the values and derivatives of at the same point [10]. Moreover, the classical gradients which are used to extract the image feature direction are local operators [11]. The information contained in the gradient is limited to a point and its immediate neighbors, while the edge curve is not a local event. As pointed out in the literatures, for example, [12, 13], this sometimes causes “blocky" solutions, staircasing effects, and “false edges," which can mislead a human or computer into identifying erroneous features not present in the true image.

In order to better respect edges in images, the nonlocal regularization schemes have been proposed. Kindermann et al.  [10] were the first to interpret nonlocal means and neighborhood filters as regularization based on nonlocal functionals which possesses the following general form:

with an appropriate positive weight function , a differentiable filter function , and a parameter . Choosing , the nonlocal -Laplacian functional is obtained as follows:

which shares many properties of the classical -Laplacian functional (3). The nonlocal -Laplacian regularization of (5) has been used recently in the study of saliency detection [14] and image denoising and segmentation [15, 16].

Another type of nonlocal regularization is based on the graph gradient operator. The graph gradient operator is defined by [17–19]

Thus, this type of nonlocal regularization is defined by

Taking in the functional (7), Gilboa and Osher [17] focus on the nonlocal TV regularization

and use it to detect and remove irregularities from textures and image inpainting. Elmoataz et al. [18] present nonlocal discrete -Laplacian regularization on image and manifold by the choice in (7) which leads to the corresponding nonlocal regularization

The above nonlocal functionals do not take into account the gradient of the image . As pointed out in [10], if the problem is ill-posed a functional involving gradients will do better.

In order to make use of the advantages of local and nonlocal functional, local and nonlocal regularization are presented in this paper. It is a linear combination of local TV and nonlocal BV. The nonlocal BV function is an exponential function including gradient, which shares the properties of the classical Perona-Malik equation. This makes our nonlocal BV functional possess the property of anisotropic diffusion, which preserves image edges better. The rest of the paper is organized as follows. In Section 2, the motivation of our nonlocal BV regularization was introduced. The local and nonlocal regularization appear in Section 3. In Section 4, we demonstrate the experimental results to verify the effectiveness of our method, and the last section is for conclusion.

2. Background

Here we briefly review some related nonlocal functional presented in [10]. Considering the functional (4), the authors chose , which gives the functional

The corresponding Euler-Lagrange descent flow of (10) is

As stated in [10], the above functionals might suffice for denoising purposes, but not for the deblurring or interpolation case which is ill-posed in the space , since adding a Prokhorov term will in general not regularize the problem in a reasonable space. Hence, a functional involving gradients is beneficial for ill-posed problem. So for image deblurring or image interpolation it is a good choice to use nonlocal functionals derived from the space of BV. The more general functional of the this type is

In [10], the following nonlocal regularization term was selected () to image denoising:

where .

For , the corresponding Euler-Lagrange descent flow of (13) is

Note that does not include image information. Equation (14) indicates that (13) bears some resemblance to the TV regularization of which the corresponding Euler-Lagrange descent flow is . This kind of dissipation process diffuses along the image contours. It will preserve the location and the intensity transitions of the contours to avoid blurring image edges. In order to enhance image edges it is important to consider the diffusion along the direction orthogonal to the image contours, similar to the Perona-Malik anisotropic diffusion equation [20],

The general selection of is or , where is a constant. It is difficult to obtain the Euler-Lagrange equation as above from a functional, but we can derive its nonlocal form similar to (15) from a nonlocal functional motivated by (10) and (11). In the next section we consider this kind of nonlocal functional.

3. Local and Nonlocal Regularization to Image Interpolation

In this section we formulate a nonlocal BV functional by incorporating (10) and (12), which has the form

The directional derivative of is given by

We can split the integral into a term involving and . In the latter a change of variables gives the same integral as that with ; hence, we end up with

Integrated by parts, the Fréchet derivative of as a functional from to is given by

The corresponding Euler-Lagrange descent flow is

As can be seen, (20) is similar to the Perona-Malik equation (15) with . At the points lying in flat area or gentle slope, is small and (20) acts as a Laplacian evolution equation, which is used in image denoising and so forth, while, for the points on image edges, is large and (20) diffuses image energy backward like the Perona-Malik equation. This process enhances image edges, which represents our desirable requirement. To eliminate oscillatory image contour, the TV regularizer will suffice.

Based on the above, our proposed energy functional for regularized image interpolation is given by

The gradient flow associated with the functional is

where

The kernel is assumed to be nonnegative, bounded continuous radial function, with and .

We develop a fully discrete numerical method to approximate problem (22). We recall first the notations in the finite differences scheme used in our paper. Let and be the space and time steps, respectively, and let be the grid points. Let be an approximation of the function , with . Equation (22) can be discretized as follows:

In all numerical experiments, we choose the following kernel function:

the constant is selected such that and .

4. Experimental Results

In this section, we tested the proposed interpolation method (24) on a variety of images. The nine natural images of them are shown in Figure 1. The image sizes are (Barbara), (race), (fingerprint), (eye), (ball), and (hat, parrot, flower, and boat). The images were downsampled without low-pass filtering and upsampled two times, such that the peak signal-to-noise ratio (PSNR) and the mean structural similarity (MSSIM) index [21] were measured, as shown in Tables 1 and 2. The algorithms for comparison are the NEDI [22], BSAI [23], adaptive gradient magnitude self-interpolation (AGMS) (using the default parameters) [24], the LMMSE estimation [25], and nonlocal BV [10]. The MATLAB codes of the NEDI, LMMSE, and AGMS and the C codes of the BSAI were obtained from the authors' corresponding websites, respectively. The MATLAB code of AGMS is available at http://www.datatang.com/data/44058. Some of the results are shown in Figures 2–6. The choice of the parameters in (24) is based on subjective quality of the results assessed informally by our personal preference as human viewers in terms of edge sharpness, contour crispness, no ringing in smooth regions, and no ringing near edges. We use the parameters , , , and time step for the proposed interpolation method. There is no visible improvement on subjective or objective quality of the results when the parameters are not badly changed. The iteration is terminated, when , normally within a few decade iterations.

Table 1: Comparison of different interpolation algorithms using PSNR values.

Image
NEDI
LMMSE
BSAI
AGMS
NLBV
Proposed

Hat
28.6606
28.6328
28.8784
30.0459
30.9239
31.3346

Parrot
27.6830
27.6966
28.0056
28.1065
31.3969
31.6225

Boat
25.6305
25.6840
25.8219
27.3422
28.4654
28.5086

Race
23.6013
23.6792
23.8557
24.3376
26.0261
26.1027

Eye
29.7578
29.8821
29.9564
31.0001
32.7364
32.6488

Ball
31.9757
32.1733
32.2449
34.6244
36.3935
36.5681

Flower
26.7856
26.7789
26.9631
28.6568
30.2598
30.5425

Barbara
23.8678
24.3268
24.3028
25.1123
25.8749
25.5891

Fingerprint
23.7430
24.2690
24.3663
25.5444
26.3480
26.8346

Table 2: Comparison of different interpolation algorithms using MSSIM values.

Image
NEDI
LMMSE
BSAI
AGMS
NLBV
Proposed

Hat
0.8575
0.8548
0.8590
0.8807
0.8949
0.9037

Parrot
0.9018
0.9037
0.9066
0.9182
0.9378
0.9433

Boat
0.7786
0.7823
0.7862
0.8309
0.8653
0.8634

Race
0.7286
0.7367
0.7445
0.7732
0.8412
0.8344

Eye
0.8834
0.8879
0.8883
0.9226
0.9443
0.9430

Ball
0.9093
0.9147
0.9149
0.9468
0.9595
0.9630

Flower
0.8311
0.8335
0.8401
0.8741
0.9073
0.9131

Barbara
0.9046
0.9011
0.9030
0.9391
0.9811
0.9650

Fingerprint
0.8911
0.8942
0.8973
0.9364
0.9420
0.9480

Figure 1: Eight test images: (left to right and top to bottom) Barbara, race, hat, parrot, ball, eye, fingerprint, flower, and boat.

Figure 2: Portions of eye image. From left to right and top to bottom, they are the interpolated images using NEDI, LMMSE, BSAI, AGMS, NLBV, and the proposed method.

Figure 3: Portions of parrot image. From left to right and top to bottom, they are the interpolated images using NEDI, LMMSE, BSAI, AGMS, NLBV, and the proposed method.

Figure 4: Portions of race image. From left to right and top to bottom, they are the original HR image and interpolated images using NEDI, LMMSE, BSAI, AGMS, NLBV, and the proposed method.

Figure 5: Portions of flower image. From left to right and top to bottom, they are the original HR image and interpolated images using NEDI, LMMSE, BSAI, AGMS, NLBV, and the proposed method.

Figure 6: Portions of fingerprint image. From left to right and top to bottom, they are the original HR image and interpolated images using NEDI, LMMSE, BSAI, AGMS, NLBV, and the proposed method.

Figure 2 shows a portion of the eye image. It is obvious that the NEDI suffers from zigzagging around the fence in the pupil. It is observable that the fence in the eye is blurring in LMMSE and BSAI interpolation and is deformed and distorted in AGMS interpolation. The artificial effects are clear in NEDI and NLBV. The proposed method produces crisp and smooth image edges, which contributes to 2.6 dB improvement in PSNR in the eye image. In Figure 3, the interpolation artifacts around black plumage strip appear in LMMSE and NLBV result. The deformation on black plumage strip appears in AGMS result. The shape of black plumage strip is natural and clear in our result, since local TV regularizer smoothes image edges and avoids staircase effects while nonlocal BV regularizer acts as anisotropic diffusion to enhance image edges in our method. But the blurring is visible in NEDI and BSAI. The same effects arise in Figures 4 and 5. In Figure 4, the teeth in tire are clear in our result. Although the teeth in tire are the clearest in NLBV, it is a false appearance produced by zigzagging effect. In Figure 5, the NEDI and our method obtain clear pistils, but our result possesses natural petal. Figure 6 shows that our method works well for texture (fingerprint) image.

We use two measures, the classic PSNR and MSSIM, to characterize the difference between the reference images and the outputs of a method. If and are two images, the PSNR is defined as , where . The detailed definition of MSSIM refers to [21]. MSSIM index takes values in and increases as the quality increases. It is calculated by the code available at http://www.cns.nyu.edu/~lcv/ssim/, using the default parameters. From the two tables, the proposed method is about 2 dB PSNR and 0.05 MSSIM higher than the other algorithms on average.

5. Conclusion

In this paper, a local and nonlocal image interpolation model based on local TV and nonlocal BV regularization is proposed. It combines the advantages of TV regularizer and nonlocal BV regularizer, that is, allowing discontinuities and preserving 1D image structures and the diffusion of the grey values of images along image feature direction. The direction of anisotropic diffusion is indicated by the information of image feature in a larger neighborhood. This results in minimal smoothing in the directions across the image features preserving sharp edges and maximal smoothing in the directions along the image features reducing zigzagging artifacts and oscillatory. We have shown improvement over local and nonlocal regularization on a subjective scale and in many cases with an improvement in PSNR and MSSIM.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

This work was supported by the NSF of China (nos. 61202349, 11101453, and 11226268), the Natural Science Foundation Project of CQ CSTC (no. cstc2013jcyjA40058), Scientific and Technological Research Program of Chongqing Municipal Education Commission (nos. KJ120709, KJ131209), and the Research Foundation of Chongqing University of Arts and Sciences (no. R2012SC20).

References

H. A. Aly and E. Dubois, “Image up-sampling using total-variation regularization with a new observation model,” IEEE Transactions on Image Processing, vol. 14, no. 10, pp. 1647–1659, 2005.

L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D, vol. 60, no. 1–4, pp. 259–268, 1992.

P. L. Combettes and J.-C. Pesquet, “Image restoration subject to a total variation constraint,” IEEE Transactions on Image Processing, vol. 13, no. 9, pp. 1213–1222, 2004.

Y. Wang, W. Yin, and Y. Zhang, “A fast algorithm for image deblurring with total variation regularization,” Rice University CAAM Technical Report TR07-10, 2007.

D. Goldfarb and W. Yin, “Second-order cone programming methods for total variation-based image restoration,” SIAM Journal on Scientific Computing, vol. 27, no. 2, pp. 622–645, 2006.

A. Chambolle and P.-L. Lions, “Image recovery via total variation minimization and related problems,” Numerische Mathematik, vol. 76, no. 2, pp. 167–188, 1997.

Y. Wang, J. Yang, W. Yin, and Y. Zhang, “A new alternating minimization algorithm for total variation image reconstruction,” SIAM Journal on Imaging Sciences, vol. 1, no. 3, pp. 248–272, 2008.

T. F. Chan and J. Shen, “Mathematical models for local nontexture inpaintings,” SIAM Journal on Applied Mathematics, vol. 62, no. 3, pp. 1019–1043, 2002.

A. Tsai, A. Yezzi Jr., and A. S. Willsky, “Curve evolution implementation of the Mumford-Shah functional for image segmentation, denoising, interpolation, and magnification,” IEEE Transactions on Image Processing, vol. 10, no. 8, pp. 1169–1186, 2001.

S. Kindermann, S. Osher, and P. W. Jones, “Deblurring and denoising of images by nonlocal functionals,” Multiscale Modeling and Simulation, vol. 4, no. 4, pp. 1091–1115, 2005.

R. A. Carmona and S. Zhong, “Adaptive smoothing respecting feature directions,” IEEE Transactions on Image Processing, vol. 7, no. 3, pp. 353–358, 1998.

W. Ring, “Structural properties of solutions to total variation regularization problems,” Mathematical Modelling and Numerical Analysis, vol. 34, no. 4, pp. 799–810, 2000.

D. C. Dobson and F. Santosa, “Recovery of blocky images from noisy and blurred data,” SIAM Journal on Applied Mathematics, vol. 56, no. 4, pp. 1181–1198, 1996.

M. Li, Y. Zhan, and L. Zhang, “Nonlocal variational model for saliency detection,” Mathematical Problems in Engineering, vol. 2013, Article ID 518747, 7 pages, 2013.

G. Gilboa and S. Osher, “Nonlocal linear image regularization and supervised segmentation,” Multiscale Modeling and Simulation, vol. 6, no. 2, pp. 595–630, 2007.

Q. Xin, X. Zhang, and L. Xu, “Image segmentation model based on the nonlocal p-Laplacian equation,” Journal of Computational Information Systems, vol. 9, no. 1, pp. 345–351, 2013.

G. Gilboa and S. Osher, “Nonlocal operators with applications to image processing,” Multiscale Modeling and Simulation, vol. 7, no. 3, pp. 1005–1028, 2008.

A. Elmoataz, O. Lezoray, and S. Bougleux, “Nonlocal discrete p-Laplacian driven image and manifold processing,” Comptes Rendus, vol. 336, no. 5, pp. 428–433, 2008.

G. Peyré, S. Bougleux, and L. Cohen, “Non-local regularization of inverse problems,” in Proceedings of the European Conference on Computer Vision (ECCV '08), 2008.

P. Perona and J. Malik, “Scale-space and edge detection using anisotropic diffusion,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, no. 7, pp. 629–639, 1990.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004.

X. Li and M. T. Orchard, “New edge-directed interpolation,” IEEE Transactions on Image Processing, vol. 10, no. 10, pp. 1521–1527, 2001.

K.-W. Hung and W.-C. Siu, “Fast image interpolation using the bilateral filter,” IET Image Processing, vol. 6, no. 7, pp. 877–890, 2012.

L. Wang, S. Xiang, G. Meng, H. Y. Wu, and C. Pan, “Edge-directed single image super-resolution via adaptive gradient magnitude self-interpolation,” IEEE Transaction on Circuits and Systems for Video Technology, vol. 23, no. 8, pp. 1289–1299, 2013.

L. Zhang and X. Wu, “An edge-guided image interpolation algorithm via directional filtering and data fusion,” IEEE Transactions on Image Processing, vol. 15, no. 8, pp. 2226–2238, 2006.