A Criminisi-DnCNN Model-Based Image Inpainting Method

Existing image inpainting methods achieve unideal results in dealing with centralized inpainting areas. For this reason, in this study, a Criminisi-DnCNNmodel-based image inpaintingmethod is proposed. Inspired by the manual inpainting technology, the pointwise mutual information (PMI) algorithm was adopted to obtain the marginal structural map of the images to be repaired. en, the Criminisi algorithm was used to restore the marginal structure to obtain the complete marginal structure image guided by the supercial linear structure. Finally, the problem of texture inpainting was converted into the counterpart of image denoising through the separation of variables by using the denoising convolutional neural network image denoiser (DnCNN). Compared with the existing inpainting methods, this model has improved the clarity of the marginal structure and reduced the blurring of the area to be repaired.


Introduction
Image inpainting technology, a research direction in image reconstruction, is the hotspot in image processing and computer vision elds. Essentially, image inpainting aims to use the known information to ll in the missing area in the image to be repaired and restore the area with incomplete information so as to ensure the natural overall inpainting e ect and meet the visual needs of people. When the inpainting area is centralized, the lack of surrounding prior information will lead to unsatisfactory result. So, image inpainting requires further research.
Currently, image inpainting methods are divided into three main types [1][2][3][4][5][6]: the variational partial di erentialbased method, texture synthesis-based method, and deep learning-based method. In the rst type, the heat ow partial di erential equation in physics is used to di use the information around to the inside of the broken area in the image to repair images. e pivotal part of the second method lies in treating the intact areas of the image as samples. en, the areas to be repaired are dealt with to achieve consistent inpainting. As for the third approach, the outstanding learning capability of the network is given full play to extract information and perform content inference layer by layer from pixel-level raw data to abstract high-level semantic concepts. When the areas to be repaired are more centralized, the deep learning-based method represents better inpainting than the other two methods mentioned above. A model of multiscale neural patch synthesis based on joint optimization of image content and texture constraints was proposed [7]. On the one hand, the low-resolution images are restored using a context encoder, which will be optimized under the guidance of the idea of neural patch matching in style transfer. Finally, detailed texture information in existing regions of the image will be transferred and generated. On the other hand, in this model, a multiscale inpainting model is used to repair high-resolution images. e upsampling image of the texture optimization at the next scale is input as the upper scale. In this manner, the high-frequency details are continuously optimized. is model has the advantage of inpainting from coarse to ne, which makes the repaired images have a good visual e ect, but its disadvantage is time consuming. Besides, a rapid and exible nonblind denoising network is proposed [8] to accomplish the image inpainting, which transforms the inpainting problem into a subproblem of image denoising. In this denoising model, an adjustable noise level map is introduced as the network input and denoising is conducted on the downsampling subimage space. is method features weighing the degree of denoising the noise level map and detail retention. However, the method of obtaining the subimage of this model is very simple, and the method of restoring the whole image is simpler, and the effect is poor. Furthermore, an end-to-end network model for texture optimization and inpainting is proposed [9]. In this model, coarse restoration results are generated, and depth features are obtained using a content initialization network. en, substitution is performed using the most similar neural patch in known regions. Finally, the restored image is obtained after decoding. is model features high-frequency texture detail migration. It is worth noting that this approach is a meta-algorithm, so the final result can be improved by a variety of network architectures and training techniques. e artist-net network [10] is proposed to realize the image inpainting, characterized by displaying structure inference and perception, including an image structure inpainting model and a texture filling model based on structure priors. In this method, the image structure is used to represent the scene as a whole, and then texture inpainting is conducted under the guidance of the important information of the restored image structure. In this way, an intact, restored image can be obtained. However, the content and expression of this model are not good, so it needs to be further improved to improve the repair effect.
In response to the unsatisfactory edge repair like blurred or even incorrect inpainting when the inpainting area is concentrated, manual inpainting generally comes first for the edge, and then the texture information is restored. Influenced by the manual inpainting technology and the previous literature [1,8,10], a Criminisi-DnCNN modelbased method of image inpainting is proposed in this study in terms of the centralized areas to be repaired, which can avoid blurred or even wrong inpainting to achieve better results. e algorithm in this study has made the following contributions: (1) e PMI algorithm is for the marginal extraction of nonlocal features at the pixel level. is method has the advantages of simple and handy operation, as well as great adaptability. e PMI algorithm was used to extract the marginal structure of the image to be repaired, providing useful information for the following inpainting of marginal structure. (2) e Criminisi and DnCNN algorithms were combined to realize image inpainting. Among them, the Criminisi was designed to repair the marginal structure to obtain the complete map of this part. By contrast, the DnCNN was used for texture inpainting by converting the problem of inpainting into an image denoising counterpart.

Criminisi
Algorithm. e crucial part in the Criminisi [11] lies in filling the information-impaired area by matching with the existing pixel blocks in the image with the pixel on the damaged edges as the center so as to repair the image. It contained the following three steps: (1) Calculation of Priority. e priority of the Criminisi algorithm is determined by the confidence that represents the proportion of information in the original image in the block to be repaired and data items that show the strength of the linear structure. (2) Search and Filling in the Best Matching Template.
After determining the pixels getting priority repair, the search and filling of the global optimal match module were performed in the intact region on the SSD principle. (3) Updating of the Confidence. Once the best matching template was determined and filled to the corresponding position, the edges of the area to be repaired would be updated accordingly.
e above three steps were repeated continuously before the area to be repaired was filled to realize the inpainting.

Image Inpainting Model.
Typically, image inpainting is an ill-posed inverse problem, which is performed based on the existing information contained in the image. In other words, the inpainting problem can be changed from ill posed to posed by adding a suitable regularization item for constraint solving, i.e., recovering a complete image from an image with missing information as shown in the following equation: where x represents the potential complete image; y is the known observation image to be repaired; H represents the binary degenerate operator; λ represents the regularization parameter used to balance the first two terms. e commonly used prior image models include the following types [12][13][14][15][16]: the local smoothness model, nonlocal self-similarity model, low-rank representation model, sparse representation model, and deep learning model. Takeda [12] focuses on kernel regression methods, extending the classical kernel regression to the adaptive kernel regression to recover corrupted noiseless high-frequency information, achieving great results. Gilboa [13] has proposed to adopt the nonlocal operators to define new functions for image processing. Compared to the classical algorithms based on the partial differential equation, this approach can be considered an extension of spectral graph theory and diffusion geometry framework to functional analysis and partial differential equations, better handling repetitive textures and structures. Gu et al. [14] proposed the weighted nuclear norm minimization problem. Specifically, it is adaptively weighting values on different singular values, and this method has achieved notable success in the field of image reconstruction. Zhang et al. [15] presented a highquality image restoration algorithm based on the structural group sparse representation model, which can ensure an efficient sparse representation of images in the structural group domain. is algorithm effectively portrays the local sparsity and nonlocal self-similarity of the image uniformly to accomplish the image inpainting task. Dong et al. [16] proposed a new inpainting loss function that adds a supervised term to solve actual problems. Specifically, based on a trained generative model, an inpainting loss function is used to search for the nearest encoding of the corrupted image in the low-dimensional space. en, the code infers the missing content by generating a model. is method has advantages over other existing methods.
In recent years, deep learning models have made great breakthroughs, thanks to the power of computers, larger data set, and more advanced networking techniques. Deep learning models only need to be trained end to end to achieve better performance instead of bearing a high computational cost.

Criminisi-DnCNN Image Inpainting
Manual restoration generally targets the marginal structure first, followed by the inpainting of texture information. An intact marginal structure can provide reliable prior knowledge for further inpainting. e Criminisi-DnCNN method proposed in this study also adopts the above idea to accomplish the concentrated image inpainting task in inpainting areas.

Marginal Structure Extraction.
Marginal structure extraction has always been a research hotspot in image processing and pattern recognition. Figure 1 shows the contrast of marginal structure extraction using different methods. Figure 1(a) represents an original image, Figure 1(b) represents the Sobel method, Figure 1(c) represents the Roberts method, Figure 1(d) represents the Prewitt method, and Figure 1(e) represents the PMI method. e essence of the traditional marginal structure extraction method [17][18][19][20] is linear filtering to obtain local features, which is prone to noise and marginal information redundancy, limiting the application of marginal structure in image processing, as shown in Figures 1(b)-1(d). erefore, research on marginal structure detection and extraction methods has focused on the increase of neighborhood, scale, direction, and types of features. As mentioned above, the PMI algorithm proposed in literature [21]is for the marginal extraction of nonlocal features at the pixel level. In this method, the affinity is obtained using pointwise mutual information to determine whether the pixels are on the same target object, thus obtaining the marginal structure. PMI method has the advantages of simple and handy operation, as well as great adaptability, as shown in Figure 1(e). erefore, this study applies the PMI method to the marginal structure extraction of images to be repaired.

Marginal Structure Inpainting.
e Criminisi method can make full use of the peripheral information of images to be repaired and accomplish the inpainting task with linear structure, avoiding blurring and visual disconnections as much as possible. us, this method is suitable for situations where the area to be restored is concentrated. erefore, for the marginal structure inpainting of images, this study adopts the Criminisi image inpainting method to accomplish the marginal inpainting task, as shown in Figure 2. Figure 2(a) represents a missing information image, Figure 2(b) represents a marginal structure image, and Figures 2(c)-2(f) represent the marginal structure inpainting effect of Kernel regression method [11], weight nuclear norm minimization method [13], group-based sparse method [14], and Criminisi method. e Kernel regression method shows blurring. e weight nuclear norm minimization method and group-based sparse method show broken marginal inpainting and could not satisfy the connectivity. e inpainting effect of Criminisi can satisfy the subjective visual requirements and does not show significant blurring and breakage. e reason is that the priority in the Criminisi algorithm method consists of confidence and data item. Pixel blocks with more information about the intact region of confidence should be given priority to repair. Besides, the data item indicates that the superficial linear structure has high strength, which should be given priority to repair.

Texture Inpainting Method.
e structure of images is first repaired using the Criminisi method, followed by the inpainting of the image texture. By deforming equation (1) using the separation of variables technique and neglecting the iteration times, equations (2)-(4) can be obtained as follows: e subproblem of x represented by equation (2) is converted into a strictly convex quadratic minimization problem. erefore, the following equation can be obtained: By deforming equation (4), the following equation can be obtained: In equation (6), if r � x − b, then r can be viewed as an observation of some noise of x [8,11,22,23]. b represents residual, and u represents denoised image. erefore, the texture inpainting subproblem of the image can be attributed to image denoising. Currently, the image denoising problem of images can be solved by the K-SVD denoising method, BM3D denoising method, and CNN network denoising method. e image denoising network of the CNN method has the advantages of high efficiency and strong prior modeling ability compared with other denoising methods. Zhang et al. [22] developed a set of welltrained CNN denoiser for image denoising, image super-Mathematical Problems in Engineering 3 resolution, and image deblurring. Li et al. [23] proposed the structure-texture decomposition model and two fusion rules are carefully designed to fuse them. is model made full use of the details and energy in the low-frequency component. e fused image can be reconstructed by the perfused highfrequency, low-frequency structure, and low-frequency texture. In literature [24], an effective, fast, and robust medical image fusion method is proposed. A two-layer decomposition scheme is introduced by the joint bilateral filter, the energy layer contains rich intensity information, and the structure layer captures ample details. en, a novel local gradient energy operator based on the structure tensor and neighbor energy is proposed to fuse the structure layer and the l1-max rule is introduced to fuse the energy layer. In literature [25], temporally redundant information was used fully by recursive convolutional neural network for singleand multi-image image denoising and applied it to superresolution of images. In literature [26], combining BM3D and convolutional neural network, the "extraction" and "aggregation" layers were used to model the block matching phase in BM3D to form BM3D-net and perform the image denoising task. In literature [27], DnCNN image denoising models were proposed. is model can combine residual learning and batch normalization, which has the advantages of fast training and strong denoising ability. erefore, this study adopts the DnCNN model image denoising prior to  image denoising. e network structure of DnCNN is shown in Figure 3.
As shown in the figure, the DnCNN image denoising network includes three types of layers: the first type of network layer is to form 64 feature maps and improve the nonlinear relationship between the layers; the second type of network layer is to facilitate the separation of potentially complete images from noisy ones; the third type of network layer is to reconstruct the output result layer. is network introduces the residual learning strategy into the denoising method and uses residual units to predict residual images. e mean square error of the residual image and the predicted residual image are used as the loss function training parameters Θ: In equation (7), (y i , x i ) N i�1 denotes N image pairs. is network is trained using the dataset [28] with a training image block size of 40 × 40, a training image block number of 128 × 1600, and a training noise level of 15, 25, and 50.

Simulation Experiment
4.1. Image Inpainting. In this study, the Lenovo R720 computer is used as the hardware simulation platform, and the software simulation platform is MATLAB. In order to Mathematical Problems in Engineering compare the subjective inpainting effect, the Criminisi algorithm and DnCNN are used in this study, as shown in Figure 4. In order to evaluate the objective quality of the inpainting effect of the algorithm in this study, the traditional PSNR and SSIM are used, as listed in Table 1.  Table 1. It can be seen that the Criminisi method is prone to produce inpainting error results when faced with a more concentrated inpainting area, and the information of foreground and background is not better distinguished. e DnCNN method is prone to fuzzy inpainting. e algorithm in this study can not only repair the marginal structure but also recover the texture information better based on both methods.
In Table 1, the PSNR and SSIM of the three algorithms are provided. Overall, the algorithm proposed in this study has comparative advantages, with an average improvement of 0.0521 dB in PSNR and 0.0003 in SSIM.

Size of Best Matching Template.
is section shows the impact of the size of the best matching template on the inpainting.
As shown in Figure 5, the size of the best matching template for achieving a good repair effect is different for different images. erefore, we choose different best matching template sizes for different images, subject to PSNR.

Iteration Number.
is section shows the influence of iterations on the inpainting effect. About 10%, 20%, and 30% of the above original images were randomly lost and then repaired to obtain the average PSNR.
As shown in Figure 6, all PSNR curves are monotonically increasing. When the number of iterations is 30, the curve is flat and stable. erefore, the iteration number of the simulation experiment is set to 30.

Conclusion
In this study, an image inpainting method based on Criminisi-DnCNN is proposed. Guided by the idea of manual inpainting, for the case where the concentrated areas are to be repaired, using this method means that the addition of the marginal structure of images is performed first, followed by the inpainting of the texture. e recovery of marginal structure using the Criminisi method can be  complemented by fully using the surrounding marginal structure information. By variable separation technique, texture inpainting is transformed into an image denoising problem, which is solved by using a superior performance DnCNN image denoising prior method. e experimental results show that the algorithm can effectively perform the inpainting task [29].

Conflicts of Interest
e authors declare that they have no conflicts of interest.

Authors' Contributions
Methodology was proposed by Z. L.; supervision was done by Z. L.; validation was performed by Z. L.; visualization was done by Y. P. Z.; original draft was written by Z. L.; review and editing were done by Y. P. W. All authors have read and agreed to the published version of the manuscript.