Multifocus Image Fusion Using Biogeography-Based Optimization

For multifocus image fusion in spatial domain, sharper blocks from different source images are selected to fuse a new image. Block size significantly affects the fusion results and a fixed block size is not applicable in various multifocus images. In this paper, a novel multifocus image fusion algorithm using biogeography-based optimization is proposed to obtain the optimal block size. The sharper blocks of each source image are first selected by sum modified Laplacian and morphological filter to contain an initial fused image. Then, the proposed algorithm uses the migration and mutation operation of biogeography-based optimization to search the optimal block size according to the fitness function in respect of spatial frequency. The chaotic search is adopted during iteration to improve optimization precision. The final fused image is constructed based on the optimal block size. Experimental results demonstrate that the proposed algorithm has good quantitative and visual evaluations.


Introduction
Optical lenses with long focal lengths often suffer from the problem of limited depth of field.It is impossible to get an image that contains all relevant objects in focus.The objects only on the focus plane are sharpness and other objects in front of or behind the focus plane are blurred [1].Multifocus fusion method which synthesizes multiple images of the same view point under different focal settings can be used to extend depth of field and obtain an all-in focus image.The fused image has more useful information of the view point and is more suitable for many applications than any individual images.Multifocus image fusion technology has played important roles in many fields such as target recognition, remote sensing, medical diagnosis, and military application [2].
Many multifocus fusion algorithms have been proposed in recent years.Basically, these fusion algorithms can be categorized into two groups: spatial domain fusion and transform domain fusion [3].For spatial domain fusion, a new image is fused by directly selecting different regions from source images.Firstly, source images are divided into nonoverlapping blocks.Then, the sharpness values of blocks are calculated based on different sharpness measure methods.Finally, the sharper blocks from different source images are selected to fuse a new image.The common algorithms in spatial domain include average, variance, energy of image gradient (EOG), sum modified Laplacian (SML), and spatial frequency (SF) [4].Recently, many new algorithms in spatial domain have been proposed to improve efficiency, such as artificial neural network (ANN) [5], pulse coupled neural network (PCNN) [6], independent component analysis (ICA) [7], robust principal component analysis (RPCA) [8], and neighbor distance [9].For transform domain fusion, a new image is generated with certain frequency transforms.Firstly, source images are converted into a transform domain to obtain the corresponding transform coefficients.Then, the transform coefficients are integrated together based on different fusion rules.Finally, the fused image is constructed by applying the inverse transform.The common algorithms in transform domain are based on pyramid transform, such as Laplacian pyramid (LAP), gradient pyramid (GRP), ratio of 2 Mathematical Problems in Engineering low-pass pyramid (RAP), and morphological pyramid (ROP) [10].Some fusion algorithms based on wavelet transform are proposed which are generally superior to the fusion algorithms based on pyramid transform, such as discrete wavelet transform (DWT) [11], stationary wavelet transform (SWT) [12], and dual-tree complex wavelet transform (DTCWT) [13].Recently, many new multiscale multiresolution transform algorithms have been widely proposed in image fusion, such as curvelet transform [14], contourlet transform [15], nonsubsampled contourlet transform (NSCT) [16], and nonsubsampled shearlet transform (NSST) [17].
The transform domain algorithms have no block artifact, but they usually are complicated and time-consuming to implement.And the multiscale multiresolution transform algorithms are generally shift-variant and sensitive to noise.The spatial domain algorithms are simple to implement and have low complexity [18].However, block artifacts inevitably exist because the block size is fixed in these algorithms.If the block size is too large, there are both in-focus part and outof-focus part in the same block.The block's sharpness may be incorrect and the out-of-focus part may be selected as the part of the fused image when considering the integrity of the segmented part.If the block size is too small, the fused result is sensitive to noise and the computational complexity is too high.Obviously, a fixed block size may not be applicable in various multifocus images.
To solve the fixed block size problem, some image fusion algorithms based on optimization are proposed, such as genetic algorithm (GA) [19,20], particle swarm optimization (PSO) [21], and differential evolution (DE) [22].These fusion algorithms utilize the global optimization characteristics to obtain the optimal block size, which effectively suppress the block artifact and increase the performance of the fused images.However, the efficiency of these fusion algorithms depends largely on the performance of the optimization algorithms.GA, PSO, and DE are not so satisfactory because of low convergence rate and low optimization accuracy.
A novel multifocus image fusion algorithm using biogeography-based optimization is proposed in this paper.Biogeography-based optimization (BBO) [23] is a new swarm intelligence optimization, which is proposed by Dr. Simon in 2008.BBO obtains the global optimum through its migration mechanisms and mutation operation.BBO has fast convergence rate and high search precision compared with GA, PSO, and DE [23].These advantages make BBO solve more effectively complex optimization problem.It has been applied to image and video processing, such as image classification [24], image matching [25], image segmentation [26], image enhancement [27], and motion estimation for video coding [28].In this paper, the proposed algorithm utilizes the biogeography-based optimization technique to find the best block size.Moreover, chaotic search is embedded to improve optimization accuracy.Experiments on various multifocus images demonstrate that the proposed algorithm has good performance in terms of quantitative and visual evaluations.
The rest of this paper is organized as follows.In Section 2, the biogeography-based optimization is briefly reviewed.Section 3 describes the proposed image fusion algorithm in detail.The experiment results and discussions are presented in Section 4. The conclusions are given in Section 5.

Biogeography-Based Optimization
Dr. Simon proposed biogeography-based optimization (BBO) in 2008 [23].It is based on the mathematics of biogeography, which describes how species migrate from one island to another, how new species arise, and how species become extinct.BBO, which is similar to GA, PSO, and DE, is a stochastic algorithm to solve optimization problem.It has two important operations, one is migration and the other one is mutation.Species among neighboring islands share their information through migration.Individual species improve their diversity through mutation.Global optimum can be effectively obtained with migration and mutation.BBO has good performance because of its fast convergence rate and high search precision [23].

Migration.
In BBO, each solution of optimization problem is regarded as an island.The feature of each solution is regarded as a suitability index variable (SIV) and the fitness value of each solution is regarded as its habitat suitability index (HSI).The higher the HIS of an island, the better the performance of the optimization problem.High HSI and low HSI use the emigration and immigration rates of each solution to probabilistically share information between islands, as in Figure 1.
The immigration rate  and the emigration rate  are the functions of the number of species in the island where  is the maximum emigration rate,  is the maximum immigration rate,  is the number of species of the th individual, and  max is the maximum number of species.For simplicity, here, assume  =  = 1; that is,   +   = .The basic step of migration is as follows.Firstly, calculate the HSI values of all islands and sort them in descending order.Second, select one island that is needed to immigrate based on immigration rate and choose its adjacent islands based on emigration rate.Then, randomly select SIV value from the adjacent islands to replace the SIV value of that island.Recalculate the HSI values of all islands.The island with highest HSI value is the optimal solution.

2.2.
Mutation.An island's HSI may change suddenly due to apparently random events such as disease and natural catastrophes.In BBO, this phenomenon is called mutation.The mutation rates   is determined as follows: where  max is a mutation parameter and  max = arg max  ,  = 1, . . ., ,   is probability of species.The relationship between   and migration rate is shown as follows: This mutation approach makes low HSI solutions likely to mutate, which gives them a chance of improving.It also makes high HSI solutions likely to mutate, which gives them a chance of improving more than they already had.Thus, the BBO mutation strategy tends to increase diversity of islands.

Elitism Strategy.
BBO incorporate elitism strategy in order to save the features of the habitat that has the best solution in the iterative process as with GA, PSO, and DE.This prevents the best solutions from being corrupted by immigration or being ruined by mutation.So even if the habitat with highest HSI is destroyed, BBO has saved it and can revert back to it if needed.The proposed algorithm in this paper retains two higher islands as elites.These elites are kept from this generation to the next generation.

Multifocus Image Fusion Using Biogeography-Based Optimization
In this paper, a novel multifocus image fusion algorithm using biogeography-based optimization is proposed.Considering the characteristic of image fusion, the island, SIV, and HSI of BBO is regarded as block size, width and height of block size, and image quality assessment, respectively.So, the dimension of optimization problem is only two, the width and height of block size.Here are the key steps of the proposed algorithm.Firstly, it selects the random block size as initial islands for utilizing the global random characteristics of BBO.Then, it calculates the HSI value of each block size.During iteration process, the width and height of the block size are updated through the migration and mutation operations.The iteration stops if the termination condition is met.Finally, the block size with the highest HSI is the optimal block size.The details of the multifocus image fusion using BBO are as follows.
3.1.HSI Function Selection.Spatial frequency [4] can measure the overall activity level in images and reflect the ability to express small details in images.The larger the value of spatial frequency in fused image, the better the performance of fusion.In the image fusion algorithms based on GA, PSO, and DE, spatial frequency is chosen as the fitness function [20][21][22].It is a good fitness function to assess the fused image quality.In this paper, we also choose the spatial frequency as the HSI function of BBO HSI (, ) = SF (, ) .
Spatial frequency SF(, ) of  ×  size image is defined as follows: where are the row and column gradients, respectively.(, ) is the pixel of the image.

Sharpness Measurement.
For multifocus image, the sharpness areas of different source images are selected to construct a fused image.How to define the sharpness areas is very important.There are many typical sharpness measurements, such as EOG, SF, and SML [18].They measure the variation of pixels of blocks in source images to determine the sharpness.The blocks with greater values are considered as in-focus blocks.Experimental results of Huang and Jing [18] show that SML method provides better performance than EOG and SF.It differs from the usual Laplacian operator in that the absolute values of the partial second derivatives are summed instead of their actual values.The SML is defined as follows: where (, ) is the pixel of the image. and  is the window size.((, )) is the modified Laplacian given as follows.

Mathematical Problems in Engineering
Here, step is a variable that is used to set the distance between the central pixel and the pixels are used to compute the second order derivative.In this paper, a value of step = 1 is found to produce the best results

𝐿 (𝐼 (𝑥
Source images  and  are divided into nonoverlapped blocks with the size of  × .Refer to the th block of source images  and  images by   and   , respectively.The focus region sml   (, ) according to SML is defined as follows: However, there are always some small holes in focus region in practice.Here, the morphological filter is used to smooth blocks and remove the holes and gulfs defects, which is shown as follows: The structure element  is 5 × 5 "diamond" matrix.((, )) and ((, )) is the opening and closing operation.The focus region    (, ) according to morphological operation is defined as follows: So, the final focus region    (, ) is defined as The focus region    (, ) is defined same as    (, ).The blocks of fusion image   are combined with   and   in focused regions (13)

Chaotic Search.
To further enhance the optimization accuracy of BBO, the modified chaotic search algorithm is adopted during each late period of iteration.Chaotic search algorithm makes full use of the ergodicity, randomness, and regularity of chaotic motion.It obtains the optimal solution by traversing every state in a small area without repetition [29].Therefore, further chaotic search can generate several neighborhood islands based on the island with the highest HSI value to improve the performance of the proposed algorithm.
The basic process is that the chaotic variables are firstly generated based on the current optimal solution.Then, the fitness value of each variable is calculated and the optimal solution of these chaotic variables is compared with the previous global optimal solution.Better solution is chosen as the current global optimal solution.This paper adopts typical logistic mapping, as shown in (14), where  = 1, 2, . . ., , and  is the maximum of iteration, which is set as 5.  is the control parameter.The system of function in (14) with  = 4,  1 ∈ (0, 1), and  1 ̸ = {0.25,0.5, 0.75} is a chaotic system The chaotic search is as follows.
Step 1.The current optimal solution  (the highest HSI point of current th iteration) is mapped to domain [0, 1] of logistic formula (15).The search area is confined in Step 2. The chaotic variable   is generated by logistic function in (14) and then returned by inverse mapping in term of the following equations: Each fitness value of each solution of   = ( 1 ,  2 , . . .,   ) is calculated and the fitness values of the optimal solution of them with  are compared.If the former is better than , it replaces  as global optimal solution; otherwise, return

Fusion Algorithm Description.
The multifocus image fusion algorithm using biogeography-based optimization is summarized as follows.
Step 1. Define some initial parameters, such as the maximum number of species  and maximum iteration number  max .
Step 2. Randomly generate initial islands, Island = {Island 1 , Island 2 , . . ., Island  }.Each island contains two parameters: Island  = {  ,   }.   and   are width and height of th block size, respectively.Divide source images into nonoverlapping blocks according these initial islands.
Step 4. BBO and chaotic searches are executed to update the state of islands as follows.
(1) Calculate the immigration rate , the emigration rate , and mutation rates   of each island.
(2) Migration operation is as follows: the parameters {  ,   } of Island  will be changed to generate new parameters based on immigration rate and emigration rate.
(3) Mutation operation is as follows: if the mutation rate is not zero, mutate the island with the highest HSI value.
Update the global optimal solution.
Step 5. Repeat Steps 3 and 4 until maximum iteration number is reached.Finally, the optimal block size is obtained based on the highest HSI to construct the optimal fused image.

Experiments
The proposed algorithm is called CSBBO.In order to verify the performance of CSBBO algorithm, such classical algorithms include the pulse coupled neural network (PCNN) [6] in spatial domain, nonsubsampled contourlet transform (NSCT) [16], and nonsubsampled shearlet transform (NSST) [17] in frequency domain, genetic algorithm (GA) [20], particle swarm optimization (PSO) [21], and differential evolution (DE) [22] based on intelligence optimization.Two kinds of experiment results, with a reference image and without a reference image, are shown in this paper.Different objective evaluation is used in these two kinds of experiments.A reference image is regarded as a ground truth image and various algorithms are assessed using the difference or structure similarity between the fused image and reference image.Without reference image, various algorithms are assessed using the information which transfers from the source images to the fused image.In the implementation, it is assumed that two multifocus source images are registered before the image fusion process.All multifocus source images and reference images are downloaded from http://www.imagefusion.org/.
With the reference image, root mean squared error (RMSE), peak signal-to-noise ratio (PSNR), and structural similarity metric (SSIM) [30] are used as quantitative assessment metrics to compare various fusion algorithms.RMSE and PSNR are commonly used to evaluate the difference between the fused image and the reference image.SSIM commonly evaluates the structure similarity between the fused image and reference image.The higher PSNR and SSIM values and the less RMSE value indicate better fusion results.They are defined as where (, ) and (, ) are the pixel values of the reference image and the fused image, respectively.The image size is  × .  ,   ,  2  ,  2  ,   is mean, variance, and covariance, respectively. 1 ,  2 ,  3 are constants closing to zero.
It is impossible to guarantee that an optimal solution could be reached after a number of iterations because of the stochastic behavior of GA, PSO, DE, and BBO [20][21][22].Moreover, the optimal solution almost could not be improved in late of iterations because of the premature behavior of GA, PSO, DE, and BBO.Too much iteration will result in very high computational complexity.The experimental result in [22] has shown that the performances of image fusion based on GA and DE cannot be significantly improved after 20 iterations.There is the same trend of the image fusion based on PSO, BBO, and CSBBO in our experiments.Therefore, the maximum iteration number of all intelligence optimization algorithms is set to 20 in order to reduce complexity.
In order to analyze the parameters of CSBBO, population number, and maximum mutation parameter, the experiment of "Pepper" image fusion is given in Table 1.Three images of "Pepper" are shown as in Figure 2. Figure 2(a) is the reference image in focus everywhere.Figures 2(b) and 2(c) focus on center and surround, respectively.All images have 512 × 512 pixels with 256 level gray scales.Considering the randomness of four intelligence optimization algorithms, experiments are repeated 30 times and the average results are saved.From Table 1, we can see that the difference between RMSE values obtained for different population number (PN) is not very high.Then, PN is set to 10 taking into account the efficiency and complexity.And the most suitable mutation parameter  max that gives the best RMSE value is 0.2 when PN is 10.The fused results of various algorithms are shown in Figure 3.The value inside the parentheses represents the optimal block size.Each algorithm can obtain a good fused image, which is almost the same as the reference image.For clearer comparison, the difference results between the fused image and the reference image are shown in Figure 4. Ideally, the difference should be zero.So, less residual features in the difference results means better performance of the fusion algorithm.It can be observed that the difference images obtained by PCNN, NSCT, and NSST have more residual features than GA, PSO, DE, BBO, and CSBBO.This indicates that these fusion algorithms based on block optimization of intelligent techniques can improve the performance of image fusion.Moreover, we can see that the difference image obtained by CSBBO has the least residue among them.
Table 2 shows the quantitative evaluation results of various image fusion algorithms with the reference image.As we can see, GA, PSO, DE, BBO, and CSBBO based on intelligent optimization are able to obtain higher PSNR, SSIM values and less RMSE value.Their performances are superior to PCNN, NSCT, and NSST.Among them, the performance of BBO is slightly better than GA, PSO, and DE.And CSBBO has the highest PSNR and SSIM values and the least RMSE value, which can obtain more useful features from the source images and has better performance than other fusion algorithms.5, respectively.For these four image pairs, the fused results of various algorithms are presented in Figures 6, 8, 10, and 12, respectively.To make better comparisons, the difference results of various algorithms are presented in Figures 7, 9, 11, and 13, respectively.
From Figures 6 and 8 for gray image Plane and Clock, we can see that each algorithm can get a clear fused image except for PCNN.The fused image obtained by PCNN has no good visual effect because there are some blurred blocks in focus area, such as the front plane and the clock surface.Moreover, the fused image obtained by PCNN does not maintain continuous edge features, such as the book edge and the clock edge in Figure 8.The fused images obtained by NSCT, NSST, GA, PSO, DE, BBO, and CSBBO have almost no difference for visual effects.
However, we can distinguish the fusion performance of these algorithms from the difference results from Figures 7  and 9.It can be observed that the difference images obtained by NSCT and NSST have more residual features than GA, PSO, DE, BBO, and CSBBO.There is large residual information of front plane obtained by NSCT and NSST in Figures 7(b) and 7(c).The numbers of clock surface are obviously remained in Figures 9(b) and 9(c).From Figures 7(d)∼7(g) and Figures 9(d)∼9(g), it can be found that the difference results obtained by GA, PSO, DE, BBO, and CSBBO are better than other algorithms.Among them, it is obvious that CSBBO has the least residual features.The difference images obtained by GA, PSO, DE, and BBO have more residual information especially in the edge parts of the focus area, such as Figures 9(d)∼9(f), and they also have more block artifacts in Figures 7(d)∼7(f).Figures 6(g), 8(g), 7(g), and 9(g) show that CSBBO can detect the focus region accurately and transfer more useful features from source images to the fused image.
From Figures 10 and 12 for color image Flower and Book, we can see the performance of each algorithm in vision.The fused image obtained by PCNN has distortion brightness and blurred edges, such as the switch and bricks in Figure 10.The fused images obtained by NSCT and NSST preserve a good brightness and edge features, but they still have some obvious blurred area, such as the words at the top right corner in Figure 12.The fused images obtained by GA, PSO, DE, BBO, and CSBBO improve the visual effects because these algorithms depress efficiently the blurred vision, such as the  For objective assessment without the reference image, natural multifocus images should be evaluated by nonreference fusion metrics.In this paper, the spatial frequency (SF) [4], feature mutual information (FMI) [31], the edge information based metric    [32], and similarity based on block metric   (, , ) [33] are used to evaluate the fusion performance of various fusion algorithms.SF evaluates the ability to express small details in the fused images, defined as formula (5).FMI calculates the amount of information conducted from the source images to the fused image, defined as FMI where  is the analysis window and  is the family of all windows.(,  | ) ( = , ) is the image quality index.sim(, ,  | ) is the similarity in spatial domain between the input image and the fused image.For all the nonreference fusion metrics, the larger the value is, the better the fusion performance is.Table 3 shows the objective assessments of the fused images obtained by various fusion algorithms without the reference image.From this table, it can be observed that CSBBO outperforms other algorithms in terms of SF, FMI, and    for all images.For   (, , ), PCNN performs a little bit better than CSBBO for Plane image, but there are obvious defects in the fused image obtained by PCNN in Figure 6(a).Based on the analysis above, it can be concluded that CSBBO has good performance in terms of quantitative and visual evaluations.

Conclusion
For spatial domain fusion, the fixed block size of source images will result in block artifacts in multifocus image fusion.To solve this problem, a novel multifocus image fusion algorithm using biogeography-based optimization is proposed in this paper.An optimized block size of each source image is obtained by BBO and chaotic search.The final fused image consisted of the optimal block size, not a fixed block size.The experiment results with or without a reference image show that CSBBO outperforms the traditional PCNN, NSCT, and NSST algorithms.Furthermore, among image

Table 1 :
Average RMSE results obtained by CSBBO for Pepper images fusion.

Table 2 :
Evaluation of various image fusion algorithms with the reference image.