Mathematical Problems in Engineering
Volume 2016 (2016), Article ID 6104196, 5 pages
http://dx.doi.org/10.1155/2016/6104196

Research Article

A Precise-Mask-Based Method for Enhanced Image Inpainting

Wanxu Zhang,¹ Yi Ru,¹ Hongqi Meng,¹ Min Liu,² Xiaolei Ma,³ Lin Wang,¹ and Bo Jiang¹

¹School of Information Science and Technology, Northwest University, Xi’an, Shaanxi 710127, China
²Key Laboratory of Space Active Opto-Electronics Technology, Shanghai Institute of Technical Physics of the Chinese Academy of Science, Shanghai 200083, China
³Department of Physics, Emory University, Atlanta, GA 30322, USA

Received 5 January 2016; Accepted 9 February 2016

Academic Editor: Maria Gandarias

Copyright © 2016 Wanxu Zhang et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

Mask of damage region is the pretreatment step of the image inpainting, which plays a key role in the ultimate effect. However, state-of-the-art methods have attached significance to the inpainting model, and the mask of damage region is usually selected manually or by the conventional threshold-based method. Since manual method is time-consuming and the threshold-based method does not have the same precision for different images, we herein report a new method for automatically constructing the precise mask by the joint filtering of guided filtering and smoothing. It can accurately locate the boundary of damaged region in order to effectively segment the damage region and then greatly improves the ultimate effect of image inpainting. The experimental results show that the proposed method is superior to state-of-the-art methods in the step of constructing inpainting mask, especially for the damaged region with inconspicuous boundary.

1. Introduction

Image inpainting as a technique of restoring missing or damaged areas of an image in an undetectable form for observers has drawn considerable attention over recent years due to the demand for revisiting damaged works of paintings or photography to make them more legible [1]. Remote sensing images are generally degraded by motion blur, the random noise, and the cloud that becomes a popular topic in remote sensing field [2]. Image inpainting can obtain the high quality image from degraded image and it can inpaint the random noise region. The noise-like white or off-white region can be located and further inpainted using the CDD or other inpainting models in this paper. The fundamental process of image inpainting is to construct a mask to locate the boundary of damaged region followed by subsequent inpainting process. State-of-the-art methods have attached significance to the inpainting model, and the mask of damage region is usually selected manually or by the conventional threshold-based method. Since manual method is time-consuming and the threshold-based method does not have the same precision for various images, therefore, it is of vital significance to automatically construct a precise mask for the damaged region of an image in order to fully restore the missing message, which is the aim of this paper.

To date, the numerous approaches [3–8] have been proposed for image inpainting since the seminal work of BSCB [3, 4]: for example, the classical Total Variation (TV) [7] and Curvature-Driven Diffusion (CDD) [8] models, which are based on solving Partial Differential Equation (PDE) to preserve the directions of the isophotes in the missing region. In practice, it is assumed that the basement layer is white and off-white photo letter or wall, so it is with the damaged region. Then a certain range of white area is taken as the mask based on the threshold which applies to the threshold of the pixels in the image to be inpainted.

However, the boundary between damaged and undamaged regions is usually inconspicuous, which may result in two main drawbacks using the threshold method: some pixels belonging to damaged region may be lost when the threshold is high, which leads to the occurrence of bleached inpainting region; some bright pixels in undamaged region are probably treated as the region which needs to be inpainted when the threshold is low. Therefore, the validity of conventional threshold method is strongly restricted; that is, it works on condition that the boundary of damaged region is conspicuous. In this work, we introduce a new method to construct a more precise mask based on guided filtering and smoothing in order to preciously locate the boundary of damaged region. Experimental results demonstrate that our method shows superiority over the conventional method in image inpainting especially for the images with inconspicuous boundary of damaged region.

2. Related Work

Two main steps comprise the conventional method for image inpainting.

Step 1. The construction of mask image: In detail, setting two values—255 and 0—to represent absolutely white and black, respectively. Supposing that the inpainting region approaches white, when each RGB value of pixels is higher than the limit value, the pixels are labeled as damaged pixels, and then the values of corresponding pixels in mask are assigned to 255. Otherwise, the values are assigned to 0. The principle of constructing mask with threshold method is shown in Figure 1, where and denote the damaged image and the mask, respectively.

Figure 1: Process of building mask by using the threshold method.

Step 2. After the construction of mask image, the damaged region is inpainted iteratively by CDD model with the mask above. CDD inpainting is like the process of heat diffusion with the diffusion intensity depending on both the gradient and curvature. The diffusion intensity is high when the curvature is large and decreases along with the isophote furtherly smoothing. Hence, the CDD model can eliminate the large curvature and stabilize the small curvature.

3. The Proposed Method

As mentioned above, it is difficult to build mask when the boundary of damaged region is inconspicuous. Inspired by dark channel prior [9], a new method is proposed to deal with this issue. The definition of dark channel is described as

where is a color channel of and is a local patch centered at . is the corresponding dark channel image. The size of is in this paper. As the natural outdoor images are usually colorful and full of shadows, the dark channel of these images is really dark. The bright parts in these images have very high intensity in each RGB channel, so that the brightest point in the dark channel image is usually close to white.

Since the damaged region (like noise or cloud region) in an image is generally white, it is assumed that the damaged region approaches white and its boundary is inconspicuous. Resultantly, the brightest pixel is considered as the most obviously damaged pixel. In addition, the colors of the damage region and its undamaged adjacent region are obviously different, which means that the damage region and undamaged region can be identified by our eyes. The most obvious damage pixel is labeled as reference point . The pixels whose RGB channel is approximately balanced may be damaged pixels. Therefore, the pixels are selected as damaged pixels according to their balanced degree of RGB and approaching degree to reference point .

The method mentioned above is sufficient for building mask when the boundary of damaged region is clear. However, it fails to work well when the boundary is inconspicuous. Motivated by this challenge, we here propose a new method to build precise mask for the boundary of damaged region based on guided filtering [10] and smoothing [11, 12]. In brief, guided filtering is employed to enhance image details, and then smoothing is used to sharpen major edges and eliminate low-amplitude structures. These two processes act as the first two steps of our proposed method as illustrated in Figure 2 and the theoretical details are provided as follows.

Figure 2: Sketch map of the proposed method.

In [10], it is assumed that the output image is linearly related to the guide image in a local place in guide filter; then the output image in window with radius centered at pixel can be expressed as the linear coefficient are constant in . This linear model ensures that and have the same edges due to .

To determine the linear coefficient , the filtering input is constricted, and the difference between output and input is considered to be noise, expressed as

and then the linear coefficient can be solved by minimizing the cost function:

where is a regularization parameter. The guided filter behaves as an edge-preserving smoothing operator when and are identical. Under this circumstance, the filtered information is the difference between and , which is expressed as . This information represents the detailed information of and is used for the following enhancement process.

The enhancement via guided filtering is written as

where is the result of the enhancement through guided filtering and is the enhancement factor. As seen from (4), increases with the increase of , meaning that the detailed information and therefore the boundary of damaged region will be enhanced as is high; otherwise, the enhancement will be weak.

smoothing is used to suppress low-amplitude details and sharpen the salient boundary. smoothing is realized by

where is the input image and the gradient of pixel in the output image is expressed as . Here and are the gradients at the directions of and , respectively. is defined as a counter to count the number of the pixels whose gradient is nonzero; that is, . The aim of minimization is to sharpen the major boundary, so that the boundary of damaged region can be preserved in and then found in the gradient of .

Figure 2 shows the sketch of our method and the steps are listed below.

Step 1. The damaged image is enhanced by guided filtering, and the result is represented by .

Step 2. is smoothed by smoothing, and the result is denoted by .

Step 3. The interim mask is built by RGB balance and approaching degree of from .

Step 4. The mask is built by and the gradient map of .

Step 5. is represented by .

Firstly, the original damaged image is enhanced by (4), and the enhancement of is represented as . The smoothed result of by smoothing is denoted as .

The brightest pixel in the dark channel of is regarded as the reference point mentioned above. Regarding the construction of mask the pixels are treated as damaged pixels when the following two conditions are satisfied: the pixels’ RGB approaching degree does not exceed the given coefficient ; the differences between RGB average of and the pixels’ RGB average do not exceed coefficient . Otherwise, the pixels will be treated as undamaged pixels. As stated in Section 2, in our method the pixels of damaged region are all set to 255 and the rest are set to 0, and we thus construct a new grayscale image denoted as mask . Generally, the white region in is mainly the inner region of damaged region; the other pixels especially the ones at the boundary in the damaged region are probably not included in . Therefore, acts as an interim mask and will be utilized to construct more precise masks in the following steps.

The next step is to locate the boundary of damaged region more precisely using guided filtering enhancement and smoothing. Then a more precise mask instead of is needed, which is constructed in the following way. We employ a small patch through which contains inner damaged region to match the gradient map of and then denote the included boundary of damaged region as mask . Likewise, the boundary is set to 255 and the rest is assigned to 0 in . Based on this, we can obtain a more precise mask defined as . Note that the size of the patch here should not be large, or the boundary will be too thick.

4. Experimental Results

In this section, we further test the performance of the proposed method using Matlab2011b in a personal computer with an Intel Core i7 processor. We collect the tested images from http://imagedatabase.cs.washington.edu/. In this paper, the damaged regions like white text and curves in original images are manually made and superimposed on undamaged images with Photoshop.

We compared the proposed method with the traditional methods, and the results are shown in Figure 3. Figures 3(a) and 3(b) are the undamaged and damaged image, respectively. Figures 3(c) and 3(d) illustrate the inpainting regions (the orange regions) found by our method and threshold method, respectively. As can be seen from Figure 3(d), some undamaged regions (e.g., mountains indicated by the red arrow) are wrongly selected to be inpainted by using the threshold method. In comparison, fewer undamaged regions are found with the proposed method as indicated by Figure 3(d).

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 3: Comparison of inpainted results. (a) Original undamaged image. (b) Damaged image. (c) and (d) are the inpainting regions found by our method and threshold method, respectively. (e) and (f) are the inpainting results using our method and threshold method, respectively. (g) and (h) are the zoom-in patch of (e) and (f), respectively.

Figures 3(e) and 3(f) show the inpainting results using the proposed and threshold methods with both basing on CDD model, respectively. The zoom-in views of the red rectangle patches in Figures 3(e) and 3(f) are shown in Figures 3(g) and 3(h), respectively. Obviously, some damaged regions are not inpainted using the threshold method as indicated by the red arrows in Figure 3(g). In contrast, our method is superior to the threshold method especially for the images with inconspicuous boundary of damaged region.

5. Conclusion

In this paper, we report a new method for building a precise mask of damaged region for image inpainting. With the precise mask and the combinations of the guided filtering enhancement and smoothing, our method exhibits satisfactory performance in image inpainting and shows superiority over conventional methods for images with inconspicuous boundary of damaged region. It is also interesting to apply our method to the inpainting of images colorful damaged region; meanwhile, a precise cloud region can be precisely detected and acquired as a byproduct of image inpainting. We leave this problem for future research.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant 61503300, Natural Science Foundation of Shaanxi Province of China under Grant 2014JQ8327, China Postdoctoral Science Foundation under Grant 2014M560801, Scientific Research Program Funded by Shaanxi Provincial Education Department under Grant 14JK1750, Science Foundation of Northwest University under Grant 13NW40, and Foundation of Key Laboratory of Space Active Opto-Electronics Technology of Chinese Academy of Sciences under Grant AOE-2016-A02.

References

C. Guillemot and O. Le Meur, “Image inpainting: overview and recent advances,” IEEE Signal Processing Magazine, vol. 31, no. 1, pp. 127–144, 2014.

H.-Y. Ding and Z.-F. Bian, “Remote sensing image restoration based on TV regularization and local constraints,” Acta Photonica Sinica, vol. 38, no. 6, pp. 1577–1580, 2009.

A. Bugeau, M. Bertalmío, V. Caselles, and G. Sapiro, “A comprehensive framework for image inpainting,” IEEE Transactions on Image Processing, vol. 19, no. 10, pp. 2634–2645, 2010.

A. Telea, “An image inpainting technique based on the fast marching method,” Journal of Graphics Tools, vol. 9, no. 1, pp. 23–34, 2004.

J.-F. Cai, R. H. Chan, and Z. Shen, “A framelet-based image inpainting algorithm,” Applied and Computational Harmonic Analysis, vol. 24, no. 2, pp. 131–149, 2008.

C. Ballester, M. Bertalmio, V. Caselles, G. Sapiro, and J. Verdera, “Filling-in by joint interpolation of vector fields and gray levels,” IEEE Transactions on Image Processing, vol. 10, no. 8, pp. 1200–1211, 2001.

T. Chan and J. Shen, “Local inpainting models and tv inpainting,” SIAM Journal on Applied Mathematics, vol. 62, no. 3, pp. 1019–1043, 2011.

T. F. Chan and J. Shen, “Nontexture inpainting by curvature-driven diffusions,” Journal of Visual Communication and Image Representation, vol. 12, no. 4, pp. 436–449, 2001.

K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 12, pp. 2341–2353, 2011.

K. He, J. Sun, and X. Tang, “Guided image filtering,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 6, pp. 1397–1409, 2013.

L. Xu, C. Lu, Y. Xu, and J. Jia, “Image smoothing via L ₀ gradient minimization,” ACM Transactions on Graphics, vol. 30, no. 6, article 174, 2011.

F. Kou, W. Chen, Z. Li, and C. Wen, “Content adaptive image detail enhancement,” IEEE Signal Processing Letters, vol. 22, no. 2, pp. 211–215, 2014.