A Bidirectional Flow Joint Sobolev Gradient for Image Interpolation

An energy functional with bidirectional flow is presented to sharpen image by reducing its edge width, which performs a forward diffusion in brighter lateral on edge ramp and backward diffusion that proceeds in darker lateral. We first consider the diffusion equations as L 2 gradient flows on integral functionals and then modify the inner product from L 2 to a Sobolev inner product. The experimental results demonstrate that our model efficiently reconstructs the real image, leading to a natural interpolation with reduced blurring, staircase artifacts and preserving better the texture features of image.


Introduction
Digital image interpolation is an inverse process of imaging process which samples higher resolution scene to lower resolution lattices.It is an important technology in digital image processing, artificial intelligence, pattern recognition, and modern industries.Producing visually pleasing resulting image is the key task in image interpolation.Many methods do their service for these problems, usually known as edge directed, level-set based, or isophote oriented, varying considerably.
The methods based on partial differential equations (PDEs) or energy variational usually have better effects in deblurring.Morse and Schwartzwald [1,2] reduced the zigzagging artifacts by restricting curvature of the interpolated isophotes (equi-intensity contours).Minimum curvature is required on isophotes of the interpolated images.Malgouyres and Guichard [3] considered the preservation of onedimensional structure to lead to the resulting images without blurring effects.Aly and Dubois [4] proposed a model-based TV regularization image up-sampling methods.Image acquisition process is modelled after a lowpass filtering followed by sampling.However, TV minimization is based on the assumption that the desirable image is almost piecewise constant, which yields a result with oversmoothed homogeneous regions.Belahmidi and Guichard [5] have improved the TVbased interpolation by developing a nonlinear anisotropic PDE, hereafter referred to as BG interpolation method.Similar to [6], the nonlinear diffusion coefficient is locally adjusted according to image features such as edges, textures, and moments, which performs forward diffusion near homogeneous regions while backward diffusion near edges with strength and orientation adapted to image structures.To sharpen edges further, bidirectional diffusion (flow) was proposed [7][8][9].In [7], a bidirectional flow is presented to sharpen image by reducing its edge width, which performs a fuzzy backward (inverse) diffusion along the gradient direction to the isophote line (edge), while it does a certain forward diffusion along the tangent direction on the contrary.A fractional-order bidirectional flow was also present in [9].
Generally, the solution of the energy functional is obtained by a steepest descent iteration with the discretized  2 gradient.Richardson [10] demonstrated substantial benefits of Sobolev gradient method to a variety of PDEs that arise in image processing in computational efficiency.Sundaramoorthi et al. [11] presented Sobolev active contours with a Sobolev gradient in a space of smooth curves.Kazemi and Danaila [12] used Sobolev gradient method in conjunction with the steady-state solution of the Navier-Stokes equation in order to fill in missing pieces in a digital image.Yuan and He [13] proposed variational level set methods for image segmentation based on both  2 and Sobolev gradients.
In this paper we focus on a new method for image interpolation based on variational regularization and Sobolev gradient.The variational regularization is a bidirectional flow in which forward diffusion and backward diffusion occur on both sides of image edge instead of along the gradient direction to the isophote line (edge) and the tangent direction as in [7].We consider the diffusion equations as  2 gradient flows on integral functionals and then modify the inner product from  2 to a Sobolev inner product.The rest of the paper is organized as follows.In Section 2, we review the method based on variational method, PDEs method, and bidirectional diffusion method.The proposed variational regularization and Sobolev gradient for image interpolation are presented in Section 3. In Section 4 we demonstrate the experimental results on natural images which suggest that this method outperforms previous work, and Section 5 is for conclusion.

Background
The imaging processing of a digital lower resolution image  0 from a high-resolution image  can be formulated as where  is a sparse matrix that combines both filtering and downsampling processes.The image interpolation is to solve unknown image  from the ill-posed inverse problem (1).It is an efficient approach to reduce it to a minimization problem in a regularization-based framework as follows [4]: where  is a regularization parameter that controls the tradeoff between   and   .The data fidelity function   is generally formulated in the classical least-squares sense as The variational regularization has been widely used in many areas of image processing [4,7,14].The main advantage of the variational regularization is that it allows incorporating various prior knowledge, such as shape and intensity distribution, smoothing image edges while preserving important features of image.All of these formulations are different forms of a unified functional form of the variational regularizer given by Using Euler's equation, the minimizer of problem ( 3) is the steady-state solution of the nonlinear PDE given by A direct calculation verifies The two second-order directional derivatives are calculated as follows: A simple and effective function  is () = , which corresponds to TV regularization.This derived the image interpolation based on the TV regularization under the level set method [4] An analogue regularization of ( 5) was proposed by Belahmidi and Guichard based on the classical heat diffusion model [5], formulated as follows: where the function () is typically defined as with  > 0 a constant to be tuned for a particular application.The role of the diffusion coefficient (|∇|) is to control the smoothing adaptively.
In order to sharpen image by reducing its edge width, a fuzzy bidirectional flow framework based on generalized fuzzy set has been presented for image denoising by [7] as follows: where , , ,  1 ,  2 are positive constant numbers, th() is a hyperbolic tangent function, and   is a Gaussian function with standard deviation .
In all these models,  2 / 2 is the second-order directional derivative in the direction of the gradient |∇| and |∇| div(∇/|∇|) =  2 / 2 in the direction orthogonal to the gradient |∇|.From the viewpoint of geometry, the evolution process in the artificial time  given by these models is seen as energy dissipation process in two orthogonal directions  and  [4].The diffusion process of (, ) along with  will preserve the location and the intensity transitions of the contours, while smoothing along them maintaining their crispness.This diffusion term is used to maintain edges with smooth isophotes in [4,5].But this process cannot decrease the width of edges caused by the upsampling process   , and it will introduce some blur.In (8), the forward diffusion process of the grey values along  conserves the edges by the stop function ().Unfortunately, the ability of the forward diffusion term in ( 8) is limited since it is a unilateral diffusion.The feature-dependent fuzzy bidirectional flow (10) has better effects in decreasing the width of edges since it is a bilateral diffusion across the image counters.However, the second-order directional derivative in the direction of gradient locate inaccurately the center of the "slop" of image edges, which reduces the ability of decreasing the width of edges.In the next section, we chose an appropriate function  and improved its Sobolev gradient to decrease the width of edges.

The Variational Regularization and Sobolev Gradient
If we consider the gray-level value at a pixel to be analogous to the amount of particles, each having one unit of "mass, " stacked at the pixel, then in order to decrease the width of edges, we would like to move particles from the lower part of a "slope" upwards and from the higher part downwards.This process enhances the ratio of gray level on both sides of image edges.It can be achieved by enlarging the small and middle gradients in interpolation process.Since the arc tangent function rapidly increases near zero while very slowly on larger arguments, we take the arc tangent function for  in (3); that is, () = arctan().Equation ( 3) is rewritten as Thus, our energy functional for image interpolation is defined as The problem now is to minimize or at least find critical points of the energy functional () for which the Euler-Lagrange equation is a nonlinear PDE.There are three approaches to minimize (): Newton's method, evolving the parabolic PDE   = −∇() to steady state, and steepest descent iteration [15].As pointed in [15], these methods all have some drawbacks.With the Sobolev gradient, however, steepest descent is often very effective [16].Next we derive the Sobolev gradient of () similar to [15,17].Define a differential operator  :  1,2 (Ω) →  2 (Ω) 3 by and let From ( 12) the Fréchet derivative of  is For  ∈  1,2 (Ω),   () is a bounded linear functional on  1,2 (Ω), and hence, by the Riesz representation theorem, there exists ∇  () ∈  1,2 (Ω), termed the Sobolev gradient, such that for all ℎ ∈  1,2 (Ω), where  * denotes the adjoint of .
For  ∈  2,2 (Ω),   () is also bounded on  2 (Ω) and is represented by an  2 gradient for all ℎ ∈  1,2 (Ω).The two gradients are thus related by where Since  2,2 (Ω) is a dense subspace of  1,2 (Ω), (18) can be extended to  ∈  1,2 (Ω) by continuity.In order to obtain an expression for ∇(), we integrate by parts in (15): with ∇ = 0 on the boundary of Ω. Hence We improve the variant coefficients of the divergence and Laplace terms in (21) to obtain terse formula and better interpolation effects as follows:    where the parameter ,  controls the tradeoff between the diffusion along the direction orthogonal to the gradient and the bidirectional diffusion.Now we illustrate the action of the regularization terms in (22).As mentioned at the outset of this section, in order to decrease the width of edges, we would like to move particles from the lower part of a "slope" upwards and from the higher part downwards.It can be achieved by an energy dissipation process in two opposite directions, forward diffusion in brighter lateral on edge ramp and backward in darker lateral.In brighter lateral on edge ramp, Δ > 0 and the second term in ( 22) is negative, and the dissipation process given by ( 22) diffuses  backward.While in darker lateral, Δ < 0 and the second term in ( 22) is positive, the dissipation process given by ( 22) diffuses  forward.

Numerical Algorithm and Experimental Results
In this section, we develop a fully discrete numerical method to approximate problem (22).We recall first the notations in the finite differences scheme used in our paper.Let ℎ and Δ be the space and time steps, respectively, and let ( 1 ;  2 ) = (ℎ; ℎ) be the grid points.Let   (; ) be an approximation of the function (Δ;  1 ;  2 ), with  ≥ 0. Then for each  > 0 (given   ), (1) discretize by finite differences and compute   := −∇() from ( 22), (2) introduce the notation  (will correspond to  = ( +1 −   )/Δ), (3) solve ( − Δ) =   with (/ ⃗ )| Ω in  using the semi-implicit scheme as in [16]: start with  0 = 0 (or the previous ), iterate for  = 1, 2, . . .until reaching the steady-state solution : We evaluate the performance of the proposed interpolation method in comparison with some representative work in the literatures: new edge-directed interpolation (NEDI) [18], the BG [5], the fuzzy bidirectional flow method (FBF) [7], and the local self-similarity algorithm (LSS) [19].Experiments results are shown in Figures 1-4.The choice of the parameters is based on subjective quality of the results assessed informally by our personal preference as human viewers in terms of edge sharpness, contour crispness, no ringing in smooth regions, and no ringing near edges.The first experiment directly interpolates a koala image and a Barbara image (portion) by a factor of 1.5 × 1.5 and 5 × 5, respectively, shown in Figures 1 and 2. In the second experiment, the images were first lowpass filtered and subsampled by a factor of 2 × 2 or 3 × 3; then the subsampled image was interpolated to the original image size, shown in Figures 3, 4, and 5.
From these figures, we can see that the LSS method sharpens the edges of image but smoothes unduly the image features, which produces the cartoon effects in result images.For example, the blocks or lumps of irregular shape appear in the fur of koala in face and forelimb (Figure 1), the strip (Figure 2), in face of old man and on the edge of hat (Figure 3), and in the petal (the detail texture disappeared in Figures 4 and 5).It is remarkable especially for larger expanded factor as shown in Figures 2 and 3, which results from using several small filters for larger factor.For example, to achieve a magnification of 5 it uses 5 : 4, 4 : 3, 4 : 3, 3 : 2, and 3 : 2. However, the effects appear even if only one filter is adopted, as shown in Figure 1.The FBF sharpens the image edges but cannot obtain smooth edges, unduly smooth image homogeneous areas, as shown in Figures 1 and 2. The BG produces better results than the LSS and FBF; the blur visual effect is still remarkable.Our approach not only sharpens the edges obtaining crisp and clear image edges not only, but also preserves the image detail feature resulting in the best visual effects.It results from the evolution process in the artificial time  given by ( 22).The diffusion process of (, ) along the orthogonal direction to the gradient smooth image contours along them while the bidirectional flow decreases the width of edges maintaining their crispness.For example, the fur of koala, the fuzz on the hat of the old man, the stamen, and the stria in petals are seen clearly.
We use the classic PSNR to characterize the difference between the reference image and the output of a method.We use several test images including parrot, old man, and two flower images.To show the true power of the interpolation algorithms, we first downsampled each image by a factor of 2 × 2 or 3 × 3 and then interpolated the result back to  1. From the table, the proposed method yields improved PSNR in all the experiments.This improvement may be attributed to the fact that our approach works better than other methods.

Conclusion
In this paper, a new image interpolation model based on energy variational is proposed.It diffuses forward and backward simultaneously on both sides of the image's edge.A Sobolev gradient method is adopted to minimize the functional.We have shown improvement over our method on a subjective scale and in many cases with an improvement in PSNR.

Table 1 :
Comparison of different interpolation algorithms using PSNR values.
its original size.The PSNR is shown in Table