Medical Image Fusion Based on Rolling Guidance Filter and Spiking Cortical Model

Medical image fusion plays an important role in diagnosis and treatment of diseases such as image-guided radiotherapy and surgery. Although numerous medical image fusion methods have been proposed, most of these approaches are sensitive to the noise and usually lead to fusion image distortion, and image information loss. Furthermore, they lack universality when dealing with different kinds of medical images. In this paper, we propose a new medical image fusion to overcome the aforementioned issues of the existing methods. It is achieved by combining with rolling guidance filter (RGF) and spiking cortical model (SCM). Firstly, saliency of medical images can be captured by RGF. Secondly, a self-adaptive threshold of SCM is gained by utilizing the mean and variance of the source images. Finally, fused image can be gotten by SCM motivated by RGF coefficients. Experimental results show that the proposed method is superior to other current popular ones in both subjectively visual performance and objective criteria.


Introduction
Multimodal medical image fusion is a hot research topic and drives a lot of attention for increasing demands for diagnosis and treatment of diseases. There are various modalities of medical images today such as computed tomography (CT), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), and functional MRI (fMRI) [1]. Different modality medical images can reflect different information of human organs such as CT can only provide dense structures like bones and implants with less distortion, while MR can provide normal and pathological soft tissues information. It is really helpful to doctor by combining complimentary features of different imaging modalities into one fused image. For example, MRI/CT imaging can be combined for diagnosis and treatment planning [2,3].
This paper focuses on the pixel level medical image fusion technology. Up to now, a lot of medical image fusion algorithms have been proposed. Examples include principal component analysis fusion algorithm (PCA) [4], guided filtering fusion algorithm (GFF) [5], medical image fusion algorithm based on wavelet in [6], fusion algorithm based on Contourlet transform (CT) in [7], fusion algorithms based on nonsubsampled Contourlet transform (NSCT) in [8], fusion algorithm based on Ripplet in [9], and fusion algorithm based on Shearlet and PCNN in [10], and so on. Although these methods produce high-quality images, they also will lead to loss of information and pixel distortion due to nonlinear operations of fusion rules and blocky artifacts [11]. To address these problems, Wang et al. proposed a new medical fusion method based on SCM in [11], which can get much better fusion effects; but, in their method, the parameters of SCM are fixed to some constants which will obviously not be widely applicable to all kinds of medical image fusion. Although the gray values of images can be used as the input of SCM like Wang's method, they are more sensitive to environment than the edge information [12].
In our paper, these disadvantages are overcome by using RGF and adaptive threshold in SCM. RGF is an edge aware filtering, and it can remove the small texture of images without blurring the image edge [13]. Therefore, in this paper, RGF is used to extract the saliency (edge information); and 2 Computational and Mathematical Methods in Medicine Input image Step 1 Small structure removal Guidance Guidance Input Input Step 2 Edge recovery Step 2 Edge recovery then, the coefficients of RGF are normalized and taken as the stimuli of the SCM. In order to be widely applied to all kinds of medical image fusion, adaptive threshold of SCM is proposed. This paper is organized as follows. In Section 2, we give a brief review of RGF and SCM. In Section 3, we give the steps of the new image fusion algorithm. In Section 4, we demonstrate the experimental results of the proposed method and the comparisons with other typical fusion methods; and, in the last section, we explore some conclusions.

Rolling Guidance Filter and
Spiking Cortical Model 2.1. Rolling Guidance Filter. Zhang et al. [13] proposed a new framework called RGF to filter images based on a rolling guidance with the complete control of detail smoothing under a scale measure. Compared to other edge preserving filters, RGF is implemented iteratively, which has a fast convergence property. It is simple and fast and also easy to understand. RGF can preserve large-scale structures automatically, where small structure removal and edge recovery are two main steps in RGF; see Figure 1 [13]. Firstly, Gaussian filter is used to remove the small structure. denotes the input image and denotes the output image. denotes the standard deviation of Gaussian filter. p and q are the indexes of pixel coordinates in the image. The filter is as follows: where p = ∑ q∈ (p) exp(−‖p − q‖ 2 /2 2 ) is for normalization and (p) denotes the set of pixels in the windows of Gaussian filter whose center is at p. Secondly, a joint bilateral filter is used to recover the edge iteratively. Initially, 1 is set as the output of the Gaussian filtering. +1 is the output of the th iteration of joint bilateral filtering with the input and . Consider is for normalization. denotes the same input image in (2). controls the range weights. Finally, two main steps in RGF can be combined into one by starting rolling guidance simply from a constant-value image. In (2), if we set all values in to a constant , that is, ∀p, = , it updates to +1 (p) = (1/ p ) ∑ q∈ (p) exp(−‖p − q‖ 2 /2 2 ) (q); the new form is exactly the same as (2).
From Figure 1, we can see that the small structure in medical images is removed by RGF. RGF can remove smallscale structures while preserving other content and is parallel in terms of importance to previous edge-preserving filters. It enlists the power of distinguishing between structures in terms of scales without knowing the exact form (or model) of texture, details, or noise.

Spiking Cortical
Model. The SCM [12] is derived from Eckhorn's model and it conforms to the physiological characteristic of human visual neural system. In fact, Wang's method [11] provides an effective means for fusion of the different kinds of medical images. In the spiking cortical model, each neuron consists of three parts: feeding and linking field, modulating product, and pulse generator; see Figure 2.
In the following expressions, the indexes and refer to the pixel location in the image, and refer to the locations of its neighboring pixels, and denotes the current iteration times. The receiving and linking field and modulating product are given by Computational

Modulating product
Modulation field Pulse generator where , ( ) is the internal activity and is the attenuation coefficient of , ( ). , is the external stimulus. is the synaptic linking weight and ( − 1) is the previous output pulse.
The pulse generator determines the firing events in the model in (4).
depends on the internal activity and threshold. Consider The dynamic threshold of the neuron is defined as where denotes the attenuation coefficient and ℎ denotes the threshold magnitude coefficient. Normally, the size of internal activity matrix , (0) is the same as the external stimulus matrix, and , (0) is always initialized to zero matrices; and the image matrix can be input as external stimulus of SCM; that is, , = , . However, the external stimulus of SCM in this paper is replaced by RGF coefficients of image. In our paper, we find that the expectation and variance of the sources images can be used to calculate threshold ℎ which can reach better fusion results. The adaptive threshold ℎ is defined as where ( = 1, . . . , ) denotes sources images needed to fuse and mean() denotes expectation function; std() denotes variance function; and the fired times can be computed as follows: where ( ) denotes the total number of the fired times of neurons after the current iteration.

Image Fusion Based on RGF and SCM
Without loss of generality, we suppose that and are two medical images with different sensor to fuse, and is the fused image. Firstly, the RGF coefficients of and can be represented as follows. Note that all input images must be registered and also have the same size and identical resolution. Consider where RGF() denotes the RGF function.
Secondly, the normalized RGF coefficients are taken as the stimulus of the two SCMs to obtain where SCM() denotes the SCM with adaptive threshold ℎ by (3)- (7). and denote the total fired times motivated by RGF coefficients RGF and RGF , respectively.
Finally, the fused image can be refined as follows: In conclusion, the framework of the proposed fusion algorithm is shown in Figure 3

The Comparison of Other Fusion Methods.
In order to evaluate the performance of the proposed fusion method, we introduce some objective criteria such as mutual information (MI) [8], / metric [14], / metric [14], and / metric [14]. MI measures the amount of information transferred to the fused image from the source images. information which is transferred from the source images to the fused image. In general, the higher MI and / values indicate the better fused result. / is introduced to evaluate the information lost during the fusion process. The lost information is available in the source images but not in the fused image.
/ represents fusion artifacts that were introduced into the fused image. It is clear that the smaller / and / the better the fused image. It is worth noting that the complimentary / , / , and / indicate that the sum of all these should result in unity [14]. Furthermore, the fused algorithms are evaluated by using the Matlab codes on Intel Core2 2.6 GHz machines with a 4 GB RAM.
To evaluate the performance of the proposed fusion method, the experiments have been performed on four pairs of multimodal medical images as shown in show that the method based on SCM can achieve much better performances. Comparing the fused image of SCM, our method not only preserves the texture information of source images but also suppresses useless image information such as block effect and artifacts, which should be attributed to the adaptive threshold in SCM and the saliency of medical images which is captured by RGF.
From the objective criteria shown in Table 1, one can find that our algorithm has the best objective criteria. The highest MI and / mean that most useful information and edge information are converted into the fused result by our algorithm. The least / means that fewest information of  source images is lost by our method. The least / means that least fusion artifacts are introduced into the fused image by our method. Therefore, our method can be regarded as a kind of good medical image fusion algorithm. Figures 7 and 8 show the fused images of group c and d by eight fused methods. The fusion results shown in Figures 7(a)-7(f) and 8(a)-8(f) indicate that our method both has a higher contrast in all the fused methods and preserves the texture information of source images, suppressing useless image information such as block effect and artifacts.
From the objective criteria shown in Table 2, we can find that our algorithm always has the best objective criteria. Therefore, our method can be regarded as a robust medical image fusion algorithm.

The Robust to Noise.
In order to validate the robustness of the algorithm, Gaussian noise with different noise variance from 5 to 50 is added to group a. The peak signal to noise ratio (PSNR) [15] is used to evaluate the performance of different fused methods. As the perfect fused image does not exist, the average of PSNR between fused image and source images is computed as measurement. It is defined as follows: where ( = 1, . . . , ) denotes source images needed to fuse. denotes the fused image. Figure 9(a) shows the fused image by SCM and Figure 9(b) shows the fused image by RGF-SCM. Obviously, our method has better visual performance than SCM. Figure 10 shows PSNR of fused images by RGF-SCM and SCM. Obviously, the PSNR of fused image by RGF-SCM is higher than that by SCM when the source images have heavy noises. When the noise variance increases, the difference Computational and Mathematical Methods in Medicine    between PSNR of fused images by RGF-SCM and SCM increases too. It means that the performance of RGF-SCM becomes more efficient than SCM when the noise variance grows. Therefore, our method can be regarded as a robust medical image fusion algorithm.

Conclusions
A new fused method based on RGF and improved SCM is proposed to improve the medical fusion effect. The new fused method can enhance robustness to noise and extend SCM to fuse other kinds of medical images. Experimental results demonstrate that the proposed method is better than state-of-the-art medical image fusion methods in both visual appearance and objective criteria. In this paper, we just only cover the fusion of 2D images; however, 3D data sets are becoming increasingly important in medical procedure. It would be interesting to know whether and how an application to 3D data sets could be achieved. In the future research, we will extend current work to 3D data sets.