Improved guided image fusion for magnetic resonance and computed tomography imaging is proposed.
Existing guided filtering scheme uses Gaussian filter and two-level weight maps due to which
the scheme has limited performance for images having noise. Different modifications in filter (based
on linear minimum mean square error estimator) and weight maps (with different levels) are proposed
to overcome these limitations. Simulation results based on visual and quantitative analysis show the
significance of proposed scheme.
1. Introduction
Medical images from different modalities reflect different levels of information (tissues, bones, etc.). A single modality cannot provide comprehensive and accurate information [1, 2]. For instance, structural images obtained from magnetic resonance (MR) imaging, computed tomography (CT), and ultrasonography, and so forth, provide high-resolution and anatomical information [1, 3]. On the other hand, functional images obtained from position emission tomography (PET), single-photon emission computed tomography (SPECT), and functional MR imaging, and so forth, provide low-spatial resolution and functional information [3, 4]. More precisely, CT imaging provides better information on denser tissue with less distortion. MR images have more distortion but can provide information on soft tissue [5, 6]. For blood flow and flood activity analysis, PET is used which provide low space resolution. Therefore, combining anatomical and functional medical images through image fusion to extract much more useful information is desirable [5, 6]. Fusion of CT/MR images combines anatomical and physiological characteristics of human body. Similarly fusion of PET/CT is helpful for tumor activity analysis [7].
Image fusion is performed on pixels, features, and decision levels [8–10]. Pixel-level methods fuse at each pixel and hence reserve most of the information [11]. Feature-level methods extract features from source images (such as edges or regions) and combine them into a single concatenated feature vector [12, 13]. Decision-level fusion [11, 14] comprises sensor information fusion, after the image has been processed by each sensor and some useful information has been extracted out of it.
Pixel-level methods include addition, subtraction, division, multiplication, minimum, maximum, median, and rank as well as more complicated operators like Markov random field and expectation-maximization algorithm [15]. Besides these, pixel level also includes statistical methods (principal component analysis (PCA), linear discriminant analysis, independent component analysis, canonical correlation analysis, and nonnegative matrix factorization). Multiscale transforms like pyramids and wavelets are also types of pixel-level fusion [11, 14]. Feature-level methods include feature based PCA [12, 13], segment fusion [13], edge fusion [13], and contour fusion [16]. They are usually robust to noise and misregistration. Weighted decision methods (voting techniques) [17], classical inference [17], Bayesian inference [17], and Dempster-Shafer method [17] are examples of decision-level fusion methods. These methods are application dependent; hence, they cannot be used generally [18].
Multiscale decomposition based medical image fusion decompose the input images into different levels. These include pyramid decomposition (Laplacian [19], morphological [20], and gradient [21]); discrete wavelet transform [22]; stationary wavelet transform [23]; redundant wavelet transform [24]; and dual-tree complex wavelet transform [25]. These schemes produce blocking effects because the decomposition process is not accompanied by any spatial orientation selectivity.
To overcome the limitations, multiscale geometric analysis methods were introduced for medical image fusion. Curvelet transform based fusion of CT and MR images [26] does not provide a proper multiresolution representation of the geometry (as curvelet transform is not built directly in the discrete domain) [27]. Contourlet transform based fusion improves the contrast, but shift-invariance is lost due to subsampling [27, 28]. Nonsubsampled contourlet transform with a variable weight for fusion of MR and SPECT images has large computational time and complexity [27, 29].
Recently, guided filter fusion (GFF) [30] is used to preserve edges and avoid blurring effects in the fused image. Guided filter is an edge-preserving filter and its computational time is also independent of filter size. However, the method provides limited performance for noisy images due to the use of Gaussian filter and two-level weight maps. An improved guided image fusion for MR and CT imaging is proposed to overcome these limitations. Simulation results based on visual and quantitative analysis show the significance of proposed scheme.
2. Preliminaries
In this section, we briefly discuss the methodology of GFF [30]. The main steps of the GFF method are filtering (to obtain the two-scale representation), weight maps construction, and fusion of base and detail layers (using guided filtering and weighted average method).
Let F be the fused image obtained by combining input images A and B of same sizes (M×N). The base (I11 and I12) and detail (I21 and I22) layers of source images are
(1)[I11I12I21I22]=[A*fB*fA-A*fB-B*f],
where f is the average filter. The base and detail layers contain large- and small-scale variations, respectively. The saliency images are obtained by convolving A and B and with a Laplacian filter h followed by a Gaussian filter g; that is,
(2)[S1S2]=[|A*h|*g|B*h|*g].
The weight maps P1 and P2 are
(3)[P1P2]=[ξ(S1,S2)ξ(S2,S1)],
where ξ(S1,S2) is a function with value 1 for S1(m,n)≥S2(m,n) and value 0 for S1(m,n)<S2(m,n) (similarly for ξ(S2,S1)). S1(m,n) and S2(m,n) are the saliency values for (m,n) pixel in A and B, respectively.
Guided image filtering is performed to obtain the refined weights W11, W12, W21, and W22 as
(4)[W11W12W21W22]=[Gr1,ϵ1(P1,A)Gr2,ϵ2(P1,A)Gr1,ϵ1(P2,B)Gr2,ϵ2(P2,B)],
where r1, ϵ1, r2, and ϵ2 are the parameters of the guided filter W11 and W21 and W12 and W22 are the base layer and the detail layer weight maps.
The fused image F is obtained by weighted averaging of the corresponding layers; that is,
(5)F=∑i1=12∑i2=12Wi1i2Ii2i1.
The major limitations of GFF [30] scheme are summarized as follows.
The Gaussian filter from (2) is not suitable for Rician noise removal. Thus, the algorithm has limited performance for noisy images. Hence filter of (2) needs to be modified to incorporate noise effects.
The weight maps P1 and P2 from (3) can be improved by defining more levels. The main issue with binary assignment (0 and 1) is that when the saliency values are approximately equal, the effect of one value is totally discarded, which results in degraded fused image.
3. Proposed Methodology
The proposed scheme follows the methodology of GFF [30] with necessary modifications to incorporate the above listed limitations. This section first discusses the modification proposed due to noise artifacts and then the improved weight maps are presented.
3.1. Improved Saliency Maps
The acquired medical images are usually of low quality (due to artifacts), which degrade the performance (both in terms of human visualization and quantitative analysis).
Beside other artifacts, MR images often contain Rician Noise (RN) which causes random fluctuations in the data and reduces image contrast [31]. RN is generated when real and imaginary parts of MR data are corrupted with zero-mean, equal variance uncorrelated Gaussian noise [32]. RN is a nonzero mean noise. Note that the noise distribution tends to Rayleigh distribution in low intensity regions and to a Gaussian distribution in regions of high intensity of the magnitude image [31, 32].
Let A˙=A+NR be image obtained using MR imaging containing Rician noise NR. The CT image B has higher spatial resolution and negligible noise level [33, 34].
The source images are first decomposed into base I˙11 and I12 and detail I˙21 and I22 layers following (1):
(6)[I˙11I12I˙21I21]=[A˙*fB*fA˙-A˙*fB-B*f].I˙11 and I˙21 have an added noise term compared to (1). Linear minimum mean square error estimator (LMMSE) is used instead of Gaussian filter for minimizing RN, consequently improving fused image quality.
The saliency maps S˙1 and S˙2 are thus computed by applying the LMMSE based filter q and following (2):
(7)[S˙1S˙2]=[|A*h|*q+|NR*h|*q|B*h|*q].
The main purpose of q is to make the extra term NR*h as in S˙1 small as possible while enhancing the image details.
3.2. Improved Weight Maps
The saliency maps are linked with detail information in the image. The main issue with 0 and 1 weight assignments arises in GFF [30] when different images have approximately equal saliency values. In such cases, one value is totally discarded. For noisy MR images, the saliency value may be higher at a pixel due to noise; in that case it will assign value 1 (which is not desirable). An appropriate solution is to define a range of values for weight maps construction.
Let ΔAB=S1(m,n)-S2(m,n), and S1(m,n)≥S2(m,n),
(8)P˙1(m,n)={1ifΔAB≥0.30.8ifΔAB≥0.250.7ifΔAB≥0.20.6ifΔAB≥0.150otherwise.
Let ΔBA=S2(m,n)-S1(m,n), and S2(m,n)≥S1(m,n),
(9)P˙2(m,n)={1ifΔBA≥0.30.8ifΔBA≥0.250.7ifΔBA≥0.20.6ifΔBA≥0.150otherwise.
These values are selected empirically and may be further adjusted to improve results. Figures 1(a) and 1(b) show CT and noisy MR images, respectively. Figures 1(c)–1(f) show the results of applying different weights. The information in the upper portion of the fused image increases as more levels are added to the weight maps.
Weight maps comparison: (a) CT image, (b) noisy MR image, (c) fused image with 3 weight maps, (d) fused image with 4 weight maps, (e) fused image with 5 weight maps, and (f) fused image with 6 weight maps.
The weight maps are passed through guided filter to obtain W˙11, W˙21, W˙12, and W˙22. Finally the fused image F˙ is
(10)F˙=∑i1=12∑i2=12W˙i1i2I˙i2i1.
LMMSE based filter reduces the Rician noise and the more levels of weight maps ensure that more information is transferred to the fused image. The incorporation of the LMMSE based filter and a range of weight map values makes the proposed method suitable for noisy images.
4. Results and Analysis
The proposed method is tested on several pairs of source (MR and CT) images. For quantitative evaluation, different measures including mutual information (MI) [35] measure ζMI, structural similarity (SSIM) [36] measure ζSSIM, Xydeas and Petrović’s [37] measure ζXP, Zhao et al.’s [38] measure ζZ, Piella and Heijmans’s [39] measures ζPH1 and ζPH2, and visual information fidelity fusion (VIFF) [40] metric ζVIFF are considered.
4.1. MI Measure
MI is a statistical measure which provides the degree of dependencies in different images. Large value of MI implies better quality and vice versa [11, 33, 35]:
(11)ζMI=2[1(HA+HF)∑a,fPAF(a,f)logPAF(a,f)PA(a)PF(f)+1(HB+HF)∑b,fPBF(b,f)logPBF(b,f)PB(b)PF(f)],
where HA, HB, and HF are the entropies of A, B, and F images, respectively. PAF is the jointly normalized histogram of A and F, PBF is the jointly normalized histogram of B and F, and PA, PB, and PF are the normalized histograms of A, B, and F, respectively.
4.2. SSIM [36] Measure
SSIM [36] measure is defined as(12)ζSSIM(A,B,F)={λwζSSIM(Aw,Fw)+(1-λw)ζSSIM(Bw,Fw)ifζSSIM(Aw,Bw∣w)≥0.75max(ζSSIM(Aw,Fw),ζSSIM(Bw,Fw))ifζSSIM(Aw,Bw∣w)<0.75,
where w is a sliding window and λ(w) is
(13)λw=σAwσAw+σBw,
where σAw and σBw are the variance of images A and B, respectively.
4.3. Xydeas and Petrović’s [37] Measure
Xydeas and Petrović [37] proposed a metric to evaluate the amount of edge information, transferred from input images to fused image. It is calculated as(14)ζXP=∑m=1N∑n=1M(QAF(m,n)τA(m,n)+QBF(m,n)τB(m,n))×(∑m=1N∑n=1M(τA(m,n)+τB(m,n)))-1,
where QAF and QBF are the product of edge strength and orientation preservation values at location (m,n), respectively. The weights τA(m,n) and τB(m,n) reflect the importance of QAF(m,n) and QBF(m,n), respectively.
4.4. Zhao et al.’s [38] Metric
Zhao et al. [38] used the phase congruency (provides an absolute measure of image feature) to define an evaluation metric. The larger value of the metric describes a better fusion result. The metric ζZ is defined as the geometric product of phase congruency, maximum and minimum moments, respectively.
4.5. Piella and Heijmans’s [39] Metric
Piella and Heijmans’s [39] metrics ζP1 and ζP2 are defined as
(15)ζP1=1|W|∑w∈W[λ(w)Qo(A,F∣w)(1-λ(w))Qo(B,F∣w)ζP1=1|W|∑w∈W5+λ(w)Qo(A,F∣w)(1-λ(w))Qo(B,F∣w)]ζP2=∑w∈Wc(w)[λ(w)Qo(A,F∣w)(1-λ(w))Qo(B,F∣w)ζP2=∑w∈Wc(w)5+λ(w)Qo(A,F∣w)(1-λ(w))Qo(B,F∣w)],
where Qo(A,F∣w) and Qo(B,F∣w) are the local quality indexes calculated in a sliding window w and λ(w) is defined as in (13). Consider
(16)Qo(A,F∣w)=4σAFA-F-(A-2+F-2)(σA2+σF2),
where A- is the mean of A and σA2 and σAF are the variance of A and the covariance of A, B, respectively. Consider
(17)cw=max[σAw,σBw]∑w′∈W[σAw′,σAw′],
where σAw and σBw are the variance of images A and B within the window w, respectively.
4.6. VIFF [40] Metric
VIFF [40] is a multiresolution image fusion metric used to assess fusion performance objectively. It has four stages. (1) Source and fused images are filtered and divided into blocks. (2) Visual information is evaluated with and without distortion information in each block. (3) The VIFF of each subband is calculated and the overall quality measure is determined by weighing (of VIFF at different subbands).
Figure 2 shows a pair of CT and MR images. It can be seen that the CT image (Figure 2(a)) provides clear bones information but no soft tissues information, while in contrast to CT image the MR image in Figure 2(b) provides soft tissues information. The fused image must contain both the information of bones and soft tissues. The fused image obtained using proposed scheme in Figure 2(d) shows better results as compared to fused image obtained by GFF [30] in Figure 2(c).
Figure 3 shows the images of a patient suffering from cerebral toxoplasmosis [41]. A more comprehensive information consisting of both the CT and MR images is the requirement in clinical diagnosis. The improvement in fused image using proposed scheme can be observed in Figure 3(d) compared to image obtained by GFF [30] in Figure 3(c).
Figure 4 shows a pair of CT and MR images of a woman suffering from hypertensive encephalopathy [41]. The improvement in fused image using proposed scheme can be observed in Figure 4(d) compared to image obtained by GFF [30] in Figure 4(c).
Figure 5 shows a pair of images containing acute stroke disease [41]. The improvement in quality of fused image obtained using proposed scheme can be observed in Figure 5(d) compared to Figure 5(c) (image obtained by GFF [30]).
Table 1 shows that proposed scheme provides better quantitative results in terms of ζMI, ζSSIM, ζXP, ζZ, ζP1, ζP2, and ζVIFF as compared to GFF [30] scheme.
Quantitative analysis of GFF [30] and proposed schemes.
Quantitative measures
Example 1
Example 2
Example 3
Example 4
GFF [30]
Proposed
GFF [30]
Proposed
GFF [30]
Proposed
GFF [30]
Proposed
ζMI
0.2958
0.2965
0.4803
0.5198
0.4164
0.4759
0.4994
0.5526
ζSSIM
0.3288
0.3540
0.3474
0.3519
0.3130
0.3139
0.2920
0.2940
ζXP
0.4034
0.5055
0.4638
0.4678
0.4473
0.4901
0.4498
0.4653
ζZ
0.1600
0.1617
0.3489
0.3091
0.2061
0.2193
0.3002
0.2855
ζP1
0.4139
0.4864
0.2730
0.3431
0.2643
0.3247
0.2729
0.3339
ζP2
0.4539
0.7469
0.5188
0.6387
0.6098
0.7453
0.5268
0.6717
ζVIFF
0.2561
0.3985
0.1553
0.2968
0.1852
0.3009
0.1842
0.3487
5. Conclusions
An improved guided image fusion for MR and CT imaging is proposed. Different modifications in filter (LMMSE based) and weights maps (with different levels) are proposed to overcome the limitations of GFF scheme. Simulation results based on visual and quantitative analysis show the significance of proposed scheme.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
YangY.ParkD. S.HuangS.RaoN.Medical image fusion via an effective wavelet-based approach20102010132-s2.0-7795350971810.1155/2010/579341579341MaesF.VandermeulenD.SuetensP.Medical image registration using mutual information20039110169917212-s2.0-244247711410.1109/JPROC.2003.817864DasS.ChowdhuryM.KunduM. K.Medical image fusion based on ripplet transform type-I2011303553702-s2.0-8005233881010.2528/PIERB11040601DaneshvarS.GhassemianH.MRI and PET image fusion by combining IHS and retina-inspired models20101121141232-s2.0-7444908895210.1016/j.inffus.2009.05.003BarraV.BoireJ.-Y.A general framework for the fusion of anatomical and functional medical images20011334104242-s2.0-003572113810.1006/nimg.2000.0707PoloA.CattaniF.VavassoriA.OriggiD.VillaG.MarsigliaH.BellomiM.TosiG.De CobelliO.OrecchiaR.MR and CT image fusion for postimplant analysis in permanent prostate seed implants2004605157215792-s2.0-1004425855710.1016/j.ijrobp.2004.08.033GrosuA. L.WeberW. A.FranzM.StärkS.PiertM.ThammR.GumprechtH.SchwaigerM.MollsM.NiederC.Reirradiation of recurrent high-grade gliomas using amino acid PET (SPECT)/CT/MRI image fusion to determine gross tumor volume for stereotactic fractionated radiotherapy20056325115192-s2.0-2464449691710.1016/j.ijrobp.2005.01.056PiellaG.A general framework for multiresolution image fusion: from pixels to regions2003442592802-s2.0-024227098710.1016/S1566-2535(03)00046-0MitianoudisN.StathakiT.Pixel-based and region-based image fusion schemes using ICA bases2007821311422-s2.0-3384664200610.1016/j.inffus.2005.09.001JameelA.GhafoorA.RiazM. M.Entropy dependent compressive sensing based image fusionIntelligent Signal Processing and Communication SystemsNovember 2013YangB.LiS.Pixel-level image fusion with simultaneous orthogonal matching pursuit201213110192-s2.0-8005398723110.1016/j.inffus.2010.04.001RanchinT.WaldL.Fusion of high spatial and spectral resolution images: the ARSIS concept and its implementation200066149612-s2.0-0033982849Al-WassaiF.KalyankarN.Al-ZakyA.Multisensor images fusion fased on feature-level201215124138DingM.WeiL.WangaB.Research on fusion method for infrared and visible images via compressive sensing2013575667MitchellH. B.2010SpringerSharmaV.DavisW.Feature-level fusion for object segmentation using mutual information2008295319HallD. L.LlinasJ.An introduction to multisensor data fusion19978516232-s2.0-003073595910.1109/5.554205Ardeshir GoshtasbyA.NikolovS.Image fusion: advances in the state of the art2007821141182-s2.0-3384659442810.1016/j.inffus.2006.04.001BurtP. J.AdelsonE. H.The laplacian pyramid as a compact image code19833145325402-s2.0-0020737631ToetA.A morphological pyramidal image decomposition1989942552612-s2.0-0000434315PetrovićV. S.XydeasC. S.Gradient-based multiresolution image fusion20041322282372-s2.0-194242182210.1109/TIP.2004.823821ChitadeA. Z.KatiyarS. K.Multiresolution and multispectral data fusion using discrete wavelet transform with IRS images: cartosat-1, IRS LISS III and LISS IV20124011211282-s2.0-8485663988010.1007/s12524-011-0140-0LiS.KwokJ. T.WangY.Using the discrete wavelet frame transform to merge Landsat TM and SPOT panchromatic images20023117232-s2.0-003649555010.1016/S1566-2535(01)00037-9SinghR.VatsaM.NooreA.Multimodal medical image fusion using redundant discrete wavelet transformProceedings of the 7th International Conference on Advances in Pattern Recognition (ICAPR '09)February 20092322352-s2.0-6364913658010.1109/ICAPR.2009.97ChenG.GaoY.Multisource image fusion based on double density dualtree complex wavelet transform9Proceedings of the International Conference on Fuzzy Systems and Knowledge Discovery201218641868AliaF.El-DokanyaI.SaadaA.Abd El-SamieaF.A curvelet transform approach for the fusion of MR and CT images2010574273286WangL.LiB.TianL.-F.Multi-modal medical image fusion using the inter-scale and intra-scale dependencies between image shift-invariant shearlet coefficients20122-s2.0-8485899271110.1016/j.inffus.2012.03.002YangL.GuoB. L.NiW.Multimodality medical image fusion based on multiscale geometric analysis of contourlet transform2008721–32032112-s2.0-5594912659510.1016/j.neucom.2008.02.025LiT.WangY.Biological image fusion using a NSCT based variable-weight method201112285922-s2.0-7865057579910.1016/j.inffus.2010.03.007LiS.KangX.HuJ.Image fusion with guided filtering2013227NowakR. D.Wavelet-based Rician noise removal for magnetic resonance imaging1999810140814192-s2.0-003332213210.1109/83.791966AnandC. S.SahambiJ. S.Wavelet domain non-linear filtering for MRI denoising20102868428612-s2.0-7795380800410.1016/j.mri.2010.03.013NakamotoY.OsmanM.CohadeC.MarshallL. T.LinksJ. M.KohlmyerS.WahlR. L.PET/CT: comparison of quantitative tracer uptake between germanium and CT transmission attenuation-corrected images2002439113711432-s2.0-0036732033BowsherJ. E.YuanH.HedlundL. W.TurkingtonT. G.AkabaniG.BadeaA.KuryloW. C.Ted WheelerC.CoferG. P.DewhirstM. W.Allan JohnsonG.Utilizing MRI information to estimate F18-FDG distributions in rat flank tumors4Proceedings of the IEEE Nuclear Science Symposium Conference RecordOctober 2004248824922-s2.0-23844469210HossnyM.NahavandiS.CreightonD.Comments on 'Information measure for performance of image fusion'20084418106610672-s2.0-5084912141910.1049/el:20081754YangC.ZhangJ.-Q.WangX.-R.LiuX.A novel similarity based quality metric for image fusion2008921561602-s2.0-3974911988310.1016/j.inffus.2006.09.001XydeasC. S.PetrovićV.Objective image fusion performance measure20003643083092-s2.0-003390818410.1049/el:20000267ZhaoJ.LaganièreR.LiuZ.Performance assessment of combinative pixel-level image fusion based on an absolute feature measurement200736143314472-s2.0-63849287542PiellaG.HeijmansH.A new quality metric for image fusionProceedings of the International Conference on Image Processing (ICIP '03)September 20031731762-s2.0-0344271801HanY.CaiY.CaoY.XuX.A new image fusion performance metric based on visual information fidelity2013142127135Harvard Image Databasehttps://www.med.harvard.edu/