^{1}

In the nonsubsampled contourlet transform (NSCT) domain, a novel image fusion algorithm based on the visual attention model and pulse coupled neural networks (PCNNs) is proposed. For the fusion of high-pass subbands in NSCT domain, a saliency-motivated PCNN model is proposed. The main idea is that high-pass subband coefficients are combined with their visual saliency maps as input to motivate PCNN. Coefficients with large firing times are employed as the fused high-pass subband coefficients. Low-pass subband coefficients are merged to develop a weighted fusion rule based on firing times of PCNN. The fused image contains abundant detailed contents from source images and preserves effectively the saliency structure while enhancing the image contrast. The algorithm can preserve the completeness and the sharpness of object regions. The fused image is more natural and can satisfy the requirement of human visual system (HVS). Experiments demonstrate that the proposed algorithm yields better performance.

Due to a tremendous growth in the application of image sensors, image fusion technique has huge potential for growth and has been used successfully in many fields, such as remote sensing, medical imaging, defense surveillance, and computer vision [

According to the level, image fusion approaches can be generally classified into three types: pixel level, feature level, or decision level [

The combination strategy of the decomposed coefficients is another key step in the MST-based fusion approaches. Fusion strategies can mainly be divided into three categories: pixel-based, window based, and area based [

The existing image fusion approaches do not take fully into account the characteristics of HVS, which the HVS tends to focus on the most relevant saliency regions in a scene. According to the visual perception mechanism, the fused image should improve the quality of object areas in a scene. The goal of the proposed algorithm is to preserve the completeness, saliency, and sharpness of object areas and satisfy the requirements of HVS. Consequently, based on NSCT and saliency-motivated PCNN, the paper proposes a novel image fusion algorithm. The visual saliency model and PCNN are two very important tools in image processing. The former is inspired by the behavior and the neuronal architecture of the early primate visual system; the latter is a visual cortex-inspired neural network and characterized by the global coupling and pulse synchronization of neurons. The saliency map produced by the visual saliency model as input to motivate PCNN is used as the fusion rule which can preserve the saliency objects from source images leading to more abundant content contained in a fused image.

The rest of the paper is organized as follows. Section

In this section, we briefly review the theory and properties of NSCT, which will be used in the rest of this paper (see [

NSCT is a kind of overcomplete transform and is a shift-invariant version of contourlet transform. NSCT has some excellent properties in the process of image decomposition, including shift invariance, multiscale, and multidirection. NSCT is used as the MST tool to provide a better representation of the contours and overcome pseudo-Gibbs phenomena. The main components of the NSCT are a nonsubsampled pyramid filter bank (NSPFB) structure for multiscale decomposition and a nonsubsampled directional filter bank (NSDFB) structure for directional decomposition. The NSCT is displayed in Figure

Nonsubsampled contourlet transform. (a) NSFB structure that implements the NSCT and (b) idealized frequency partitioning obtained with the proposed structure.

The multiscale property of the NSCT is achieved by using two-channel nonsubsampled 2-D filter banks (NSFBs), called as NSPFB. The filters for next level are obtained by upsampling the filters of the previous level, by which the multiscale property is obtained without the need for additional filter design. We assume that the NSPFB decomposition is with

Figure

In the section, the proposed image fusion algorithm based on NSCT and saliency-motivated PCNN is presented in detail. The main idea is that the visual saliency map is first built on high-pass subband coefficients of the NSCT using the visual attention model (phase spectrum of Fourier transform (PFT) model presented in Section

Architecture of the proposed algorithm.

The algorithm first decomposes source images into the low-pass subband and high-pass directional subband coefficients by the NSCT. The coarsest subband contains the main energy from source images and denotes the abundant structural information. Therefore, an adaptive weighted average fusion rule based on the firing times of PCNN is developed to merge the low-pass subband. High-pass directional subbands contain the abundant detail contents of source images, so we create a maximum selection fusion principle based on saliency-motivated PCNN for selecting the fused coefficients. The final fused image is reconstructed by applying the inverse NSCT on the merged coefficients.

The decomposition of source images employs NSCT presented in Section

The following the proposed saliency-motivated PCNN model is discussed. Eckhorn develops a novel biological neural network, called PCNN which is based on the experimental observations of synchronous pulse bursts in cat and monkey visual cortices [

Connection model of PCNN neuron.

In this paper, let

In (

The saliency maps

Phase spectrum of Fourier transform (PFT) proposed in [

PFT model is a simple and efficient saliency detection method. An example of the PFT saliency detection is shown in Figure

The results of saliency detection from two complementary input images. (a) and (c) Multifocus source images (b) and (d) saliency maps from PFT.

Source image

Saliency map

Source image

Saliency map

Consequently, in this paper, the saliency value of high-pass subbands

Then,

The high-pass subbands of NSCT decomposition contain abundant detailed information and indicate the saliency components of images, for example, lines, edges, contours, and so forth. In order to preserve the saliency components in the process of image fusion, we propose the fusion rule based on saliency-motivated PCNN for the high-pass subbands. According to the visual attention mechanism, different regions in an image have varying importance for HVS, so the saliency detection is performed on source images to yield saliency maps which indicate the significance level of every pixel in source images. Based on the characteristics, the PFT model is performed on the high-pass subbands to produce the saliency maps, which indicate the importance level of coefficients. And then, the obtained saliency maps are combined with the corresponding high-pass subband coefficients as the input to motivate PCNN. Coefficients with large firing times are selected as the fused coefficients. In addition, the low-pass subband of NSCT decomposition in the coarsest scale contains the main energy of source images and denotes abundant structural information. The fusion rule of the low-pass subband employs a weighted fusion rule based on firing times of PCNN.

The activity maps of high-pass subbands as the criteria of selecting coefficients are presented by the firing map of saliency-motivated PCNN. The activity level indicates the magnitude of coefficients. The coefficients of greater energy carry more important information, so the coefficients of greater activity level are selected as the fused coefficients. Now, according to (

The fused coefficients of low-pass subbands denoted by

Finally, apply the inverse NSCT to the fused coefficients

directional subband coefficients and the low-pass subband coefficients.

In this section, the proposed image fusion algorithm based on NSCT and saliency-motivated PCNN (named as NSCT-SPCNN) is tested on several sets of images. The goal of the tests is to validate if the proposed algorithm can be used in the real applications and varying surroundings. For comparison, besides the fusion scheme proposed in this paper, another three fusion algorithms, the Laplacian pyramid transform based (LPT), discrete wavelet transform based (DWT), and NSCT-simple based, are used to fuse the same images. All of these use averaging and absolute maximum selection schemes for merging low- and high-pass subband coefficients, respectively. The decomposition level of all of the transforms is three. Extensive experiments with multifocus image fusion and different sensor image fusion have been performed. Here, three groups of different images were tested to evaluate the performance of the proposed algorithm: a set of multifocus images, a set of multimodal medical images, and a set of artificial out-of-focus images. It is assumed that source images have been registered. The fused results were evaluated using subjective visual inspection and objective assessment tools.

The first experiment uses two multifocus source images and four fused images produced by LPT, DWT, NSCT-simple, and NSCT-SPCNN methods, shown in Figure

“Clock” source images (256 level, size of

Figures

Magnified regions from the fused images in Figures

(a) Difference image between Figures

Figure

Medical source images (256 level, size of

Spatial source images (256 level, size of

In previous discussion, the fusion results of different algorithms have been analyzed by visual aspect. However, the performance of fusion algorithms needs to be further evaluated using objective metric tools. A successful fusion technique has to satisfy many conditions, such as preserving important features of source images, enhancing contrast, and avoiding artifacts. Mutual information (MI) [

Figure

Quality metrics for the different fusion methods.

MI metric

Finally, the computational performance of the proposed NSCT-SPCNN algorithm is tested on three sets of images (Figures

The paper proposes a novel image fusion algorithm based on NSCT and saliency-motivated PCNN. In fusion for high-pass subbands, a saliency-motivated PCNN model is proposed. The key idea is that depending on the human visual attention model, the visual saliency map is first built on high-pass subband coefficients of NSCT, and then the algorithm combines the visual saliency map with the coefficients of NSCT as input to motivate PCNN. Coefficients with large firing times are employed as the fused high-pass subband coefficients. Low-pass subband coefficients are merged to develop a weighted fusion rule based on firing times of PCNN. The algorithm can preserve the completeness and the sharpness of object regions. The fused image is more natural and can satisfy the requirement of HVS. Experiments illustrate that the proposed fusion algorithm improves greatly the quality of the fused images.

This work was supported by the National Basic Research Program of China (973 Program nos. 2012CB821200 2012CB821206), the National Natural Science Foundation of China (nos. 91024001, 61070142), and the Beijing Natural Science Foundation (no. 4111002).