Fusion of IR and Visual Images Based on Gaussian and Laplacian Decomposition Using Histogram Distributions and Edge Selection

We propose a novel fusion method of IR (infrared) and visual images to combine distinct information from two sources. To decompose an image into its low and high frequency components, we use Gaussian and Laplacian decomposition.The strong high frequency information in the two sources can be easily fused by selecting the large magnitude of Laplacian images.The distinct low frequency information, however, is not as easily determined. As such, we use histogram distributions of the two sources.Therefore, experimental results show that the fused images can contain the dominant characteristics of both sources.

Forward looking infrared (FLIR) cameras can sense IR radiation (i.e., thermal radiation).Therefore, IR images can contain useful information that is not apparent in visual images.Alternatively, detailed information within the visual band is not included in IR images.Therefore, the fusion of IR and visual images can provide the advantages of both types of images.Also, the fusion of IR and visual images can apply various research fields, such as night vision [9,10], face recognition [11], human detection [12], and detecting concealed object [13].
Visual and IR images can be easily fused by averaging; however, in this scheme, the advantages of the two sources are eliminated and the details are annihilated in the worst case.
To preserve the dominant advantages in the fused images, two methods can be utilized.Various methods using multiscale decomposition, such as Laplacian pyramid [14,15], discrete wavelet transform (DWT) [16,17], and contrast pyramid [18] can be used.One stage of Laplacian pyramid decomposition is shown in Figure 1.Additionally, some methods using region-segmentation [19][20][21] have also been proposed.
The multiscale decomposition-based methods are performed with low computation; however, selecting the correct and distinct values of the low frequency information is not easily determined.So, many stages of pyramid and DWT are used to select significant values, and distinct values are only detected from Laplacian images or high frequency bands [22].Although distinct values can be selected from both low and high frequency images, most of all methods used only strong intensity and predetermined weights.
On the other hand, the latter methods can simply select the distinct values of the low frequency information; accurate segmentation is not guaranteed, and segmentation methods have a higher computation complexity.In addition, seam boundary regions should be blended by two image sources to prevent discontinuities.
In this paper we are aimed at developing a fusion method that has the advantages of the methods mentioned above.To do this, we use simple Gaussian and Laplacian decomposition and utilize histogram distributions and edge selection to determine the distinct values of the low and high frequency information, respectively.Because we use histogram distributions, significant low frequency information can be selected, such as locally hot or cold regions of IR images and locally bright or dark regions of visual images.In addition, only one decomposition is used and fast averaging filters are used, so processing speed is fast enough for real-time applications.

Fusion of IR and Visual Images
The proposed fusion method is similar to methods that use the Laplacian pyramid or DWT.We use Gaussian smoothing, as opposed to Gaussian or wavelet scaling, so our method is identical to methods that perform single scaling.A Gaussian image  is the filtered image of an original image  by the Gaussian convolution, and a Laplacian image ∇ 2  is calculated by  −  ( =  + ∇ 2 ) as shown in Figure 2.
To obtain a fused image  =  + ∇ 2 , the distinct information of ∇ 2  is related to the magnitude; a larger |∇ Finally, we determine ∇ 2  using   or   as follows: where (2a) can be used for the fusion image having smooth boundaries of objects, and (2b) can be used for the fusion image having distinct boundaries of objects.
The distinct intensity values of most images have a low population.In particular, intensities with a low population in the IR images show the highest or the lowest temperature as shown in Figure 4. Therefore, we use the histogram distribution to select .
where His  () and His  () denote the histogram distribution functions for  and , respectively.However,  ℎ has extreme discontinuities, so we substitute  ℎ for .In all processes for Gaussian filter, we use a fast mean filter using spatial buffers to reduce computation complexities.

Experimental Result
The proposed fusion method was tested with three image sequences of TNO image fusion dataset [23]: "UN Camp," "Dune," and "Trees" (360 × 270, grayscale).These test images have small intensity rages, so we tested modified images by linear histogram normalization instead of original images.In this experiment, we used two parameter sets as shown in Table 1, where the first three parameters are the mask sizes.The optimal parameters are experimentally determined to make visually nature fusion images not having artificial discontinuities.
Examples of our results using the optimal parameters are shown in Figures 5, 6  More results for the three image sequences using optimal parameters are shown in Figure 8; the red fused regions are more influenced by the visual images, while orange fused regions are more influenced by the IR images.To objectively evaluate the proposed method, we consider two evaluation metrics: entropy () and the Xydeas and Petrovic index ( /  ) [24].The performance comparison with the averaging method is shown in Table 2. To compare our results with those reported in [12], we compared the improvement in the ratio of the averaging method as shown in Table 3, because the existing methods used a different image enhancement methods which were not appeared in literature.
Although our method using the optimal parameters does not yield the best objective performance, these metrics do not completely agree with human subjective evaluation as shown in Figure 9.The performance of Figure 9    methods may be overestimated to the human subjective evaluation.
Visual comparison of two results using two parameter sets is shown in Figure 10.As shown in Table 3,  /  and  of the results using the large parameters are higher than the results using the optimal parameters; however the results using the optimal parameters show better visuality.
Using single thread processing with a 3.60 GHz CPU, the average computation time of all of the sequences is 5.237 msec.This processing time is inconceivable for  segmentation-based methods.Therefore, the proposed method can be used for real-time fusion applications.

Conclusion
In this paper, we have proposed a novel fusion method for IR and visual images based on Gaussian and Laplacian decomposition using histogram distributions and edge selection.This method can easily determine the distinct values of Gaussian and Laplacian images.The distinct values of

Figure 1 :
Figure 1: One stage of Laplacian pyramid decomposition.

Figure 3 :
Figure 3: The proposed fusion method of IR and visual images.

Figure 4 :Figure 5 :
Figure 4: Intensities with a low population: (a) IR image and its histogram, (b) the binary image (, ) >  1 of the IR image (a), and (c) the binary image (, ) <  2 of the IR image (a).

Figure 8 :
Figure 8: Examples of the proposed fusion method (from top to bottom: "UN Camp," "Trees," and "Dune"): (a) visual images, (b) IR images, and (c) fused images using the proposed method.

Figure 10 :
Figure 10: Comparison of the proposed method with averaging;(a-c) results of the proposed method (optimal parameters), results of the proposed method (large parameters), and results of the averaging method.
observed in   and  ℎ , while they are blurred in   and .It seems that the distinct information of  and  is well fused in .In ∇ 2  images, strong high frequency components are well selected.In  images, the blurred distinct values are well revealed regardless of ∇ 2  images.Particularly, isolated dark regions having low frequency are distinctly shown in  images.

Table 2 :
Comparison with the averaging method.

Table 3 :
Comparison with the existing methods (the improved ratio of the averaging method; the performance ranking is shown in parentheses).