Multifocus image fusion is a process that integrates partially focused image sequence into a fused image which is focused everywhere, with multiple methods proposed in the past decades. The Dual Tree Complex Wavelet Transform (DTCWT) is one of the most precise ones eliminating two main defects caused by the Discrete Wavelet Transform (DWT). Q-shift DTCWT was proposed afterwards to simplify the construction of filters in DTCWT, producing better fusion effects. A different image fusion strategy based on Q-shift DTCWT is presented in this work. According to the strategy, firstly, each image is decomposed into low and high frequency coefficients, which are, respectively, fused by using different rules, and then various fusion rules are innovatively combined in Q-shift DTCWT, such as the Neighborhood Variant Maximum Selectivity (NVMS) and the Sum Modified Laplacian (SML). Finally, the fused coefficients could be well extracted from the source images and reconstructed to produce one fully focused image. This strategy is verified visually and quantitatively with several existing fusion methods based on a plenty of experiments and yields good results both on standard images and on microscopic images. Hence, we can draw the conclusion that the rule of NVMS is better than others after Q-shift DTCWT.
National Key Technology Support Program of China2015BAF10B01Science and Technology Commission of Shanghai Municipality15111104002151111063021. Introduction
Since the optical lens has limited depth of focus (DOF), it is difficult to obtain an image of an object with every part in the field focused. Due to the insensitivity of the human eyes to unsharpness or blurriness within the DOF of the lens, only the objects within the DOF appear to be in focus while objects out of the DOF are blurred [1]. The solution is to use multifocus image fusion to extract in-focus information from each partially focused image into one fully focused image, which is better for visual perception and further calculation. This technology has been extended to various fields, such as microscopic imaging, visual inspection, 3D shape recovery, and measurement [2]. Thus, it attracted the attention of lots of researchers, who presented a variety of fusion algorithms.
Existing image fusion methods can be categorized into fusion methods on pixel, feature, and decision level [3]. In this paper, only pixel level methods are discussed due to their high accuracy and less information loss, which is better for microscopic imaging. Fusion methods on pixel level are subdivided into spatial domain and transform domain fusion algorithms [2, 4]. Transform domain algorithms, namely, the multiresolution algorithms, are more robust since the human visual system deals with information in a multiresolution way, which is in line with the processing principle of transform domain algorithms [5]. So, multiresolution algorithms can bring about higher precision, while spatial domain methods are commonly used in cases when rapidity is required. Transform domain fusion methods have two major research hotspots: pyramids algorithm [2, 6, 7] and DWT algorithm [2]. Presently, the DWT based fusion algorithm is a research hotspot with better fusion effect [5]. The DTCWT proposed by Selesnick et al. [8] is a relative improvement on DWT, with several important additional properties: approximate shift invariance, good directional selectivity, and limited data redundancy. In this Q-shift solution [8], the construction process of filters is largely simplified. Image fusion with guided filtering was presented by Li et al. [9]. Yin et al. presented a method of image fusion and superresolution using sparse representation [10]. Yang and Li reported an image fusion method on pixel level with simultaneous orthogonal matching pursuit [11].
In this work, we have proposed an effective fusion rule for multifocus image fusion after using Q-shift DTCWT and performed this method over partially focused image sequences blurred by Gaussian operators and microscopic image sequences. The proposed method is compared with DWT and different pyramid algorithms. Some common objective metrics [12, 13] are used to analyze the performance of these algorithms and different fusion rules.
The following renders the framework of this paper. Section 2 deals with the preliminary details of Q-shift DTCWT. Section 3 describes the proposed approach of image fusion. Section 4 is about the fusion rules applied in the proposed approach. In Sections 5 and 6, the experimental results, performance comparisons, and time consumption are given. Finally, conclusions are drawn in Section 7.
2. Q-Shift DTCWT Related Theory
Aiming to solve the drawbacks caused by the DWT method, Kingsbury proposed Complex Wavelet Transform (CWT) method when conducting image fusion experiment. In the working process of CWT method, the plural form is needed, and perfect reconstruction filter banks should be built based on the plural form. It is relatively easy to construct such filter banks on the first decomposition layer, but it is much harder on more upper layers. In 1999, Kingsbury proposed a new form of DTCWT, which can both keep the advantages of CWT and achieve the function of perfect reconstruction [8]. DTCWT is a novel wavelet transform method using a pair of binary-structured trees to realize wavelet transforms on real and imaginary components of the signal in parallel. The principle of DTCWT decomposition process on a two-dimensional signal on level j is shown in Figure 1, while the reconstruction process is not presented as it is just an inverse transform of the decomposition one.
Decomposition diagram of the Chinese character “wood” on level j using Q-shift DTCWT.
DTCWT algorithm gains a series of advantages due to its unique structure, which are approximate shift invariance, good directional selectivity, and limited data redundancy. Due to its shift invariance, images fused by DTCWT are smooth and continuous while images fused by DWT contain irregular edges. Another major superiority of DTCWT is good directional selectivity since DTCWT produces six subbands at each scale for both imaginary and real parts in (±15°, ±45°, ±75°), while DWT only presents limited directions in (0°, 45°, 90°), which improves the transform precision and keeps more detailed information. However, in DTCWT, the process of designing filters is a little complicated due to its requirement for satisfying simultaneously the biorthogonal conditions and the phase condition [8]. Q-shift DTCWT is a solution to improve this problem as its quarter shift filters produce complex wavelets that are exactly linear phase, which largely simplifies the filter construction process.
The transform uses two parallel treelike filter series (Tree A and Tree B in Figure 1, generating the real and imaginary parts, resp.) that realize complex wavelet filtering of the input signal. In order to achieve shift invariance, filters are designed to own different delays, ensuring that every filter can sample the values discarded by other filters because of downsampling. In this way, aliasing phenomenon is minimized. The trees are implemented on rows and columns of the signal, respectively, and generate two low frequency coefficients L(j+1,1) and L(j+1,2), containing rough information, and six high frequency coefficients H(j+1,m)(m=1,…,6), containing detailed information in different directions. In Figure 1, character “wood” is decomposed into eight subimages; two low frequency coefficients appear to be similar to the original one while the other six are the extracted detailed information in different directions. The two low frequency images are used as the input signal of the next level decomposition.
3. The Proposed Fusion Methods
The general image fusion scheme using DTCWT for both grayscale and color images is shown in Figure 2. Since DTCWT can only be calculated on monochrome images, color images should be decomposed into subimages. As the commonly used RGB color model creates color distortion due to the high correlation among the R, G, and B components, we transform it into YIQ color model. Since Y (luminance) component contains much more information than I and Q (chrominance) components [14], we perform the DTCWT fusion only on Y components, while I and Q components are fused based on a mapping table corresponding to Y fusion. Finally, the images in YIQ color model are transformed in RGB color model for display. In case of dealing with grayscale images, the above steps are omitted and DTCWT fusion is performed directly on the source images.
Schematic flowchart of the proposed fusion method.
Detailed steps of DTCWT fusion process are as follows:
Perform DTCWT decomposition process on the input images a to k, which are focused in different parts. Each image is decomposed into two low frequency coefficients (La1, La2) to (Lk1, Lk2) and some high frequency coefficients (Ha1, Ha2,…, Han) to (Hk1, Hk2,…, Hkn). The high frequency coefficients represent detailed information of the input image in different directions, and their number depends on the total decomposition level.
Fusion rules are performed on low frequency coefficients and high frequency coefficients separately. More details are presented in the next section.
The selected high frequency and low frequency coefficients are then adjusted with consistency check. In multifocus images, blurred and sharp areas are generally regionally connected. For example, if the surrounding pixels are in focus, then the single pixel inside must be in focus too. Consistency check [15] is aimed at picking and changing these single pixels to be consistent with surrounding ones.
Finally, DTCWT reconstruction process is implemented on the final low frequency coefficients L1 and L2 as well as high frequency coefficients H1 to Hn to construct the fused image.
4. Fusion Rules
Fusion rule is the most important factor of the proposed fusion methods as it is the core of distinguishing focused region from defocused region. In the proposed method, low frequency coefficients and high frequency coefficients represent rough information and detailed information, respectively, requiring different fusion rules [16]. Several fusion rules are described in this section including pixel based fusion rules such as the Weighted Average (WA) method [17], the Synthesis image Module Value Maximum Selectivity (SMVMS), and regional based fusion rules like the Neighborhood Variant Maximum Selectivity (NVMS) [18], the Neighborhood Gradient Maximum Selectivity (NGMS) [19], and the SML [20]. Details of these fusion rules are discussed below.
4.1. The Weighted Average (WA) Method
WA is normally used in low frequency coefficients as differences in low frequency information are comparatively small. The final coefficient F(x,y) is defined as follows:(1)Fx,y=ω1p1x,y+ω2p2x,y+⋯+ωnpnx,y,where p1x,y to p2x,y are pixel values of source low frequency coefficients in the x,y location and ω1 to ωn are weights for p1x,y to p2x,y, generally ω1 to n=1/n.
4.2. The Neighborhood Variant Maximum Selectivity (NVMS)
NVMS is defined as follows: calculate the standard variance STDnx,y of the pixel pnx,y in the nth picture in the x,y location: (2)STDnx,y=∑i=-N-1/2N-1/2∑i=-M-1/2M-1/2Pnx+i,y+i-Pn¯x,y2N×M,where N,M are the window sizes, generally N=M=3 or N=M=5. pn¯x,y is the average value which is given by(3)pn¯x,y=∑i=-N-1/2N-1/2∑j=-M-1/2M-1/2pnx+i,y+jM×N.
The final coefficient Fx,y is equal to the kth pixel pkx,y and k is defined by picking the biggest STDkx,y from all computed values. Hence,(4)STDkx,y=maxSTD1x,y,STD2x,y,…,STDnx,y.
4.3. The Neighborhood Gradient Maximum Selectivity (NGMS)
NGMS is defined by Gnx,y which is formed as follows:(5)Gnx,y=∑i=-11∑j=-11Hi,jpnx+i,y+jH=-1-1-1-18-1-1-1-1.
Fx,y is equal to the kth pixel pkx,y and k is decided by the biggest Gkx,y. Hence,(6)Gkx,y=maxG1x,y,G2x,y,…,Gnx,y.
4.4. The Sum Modified Laplacian (SML) Method
The Sum Modified Laplacian SMLnx,y is given by(7)MLnx,y=2pnx,y-pnx-s,y-pnx+s,y+2pnx,y-pnx,y-s-pnx,y+sSMLnx,y=∑i=-N-1/2N-1/2∑j=-N-1/2N-1/2MLnx+i,y+j,where s denotes the step length and F(x,y) is the kth pixel pkx,y and k is defined by the biggest SMLnx,y. Hence,(8)SMLkx,y=maxSML1x,y,SML2x,y,…,SMLnx,y.
4.5. Synthesis Image Module Value Maximum Selectivity (SMVMS)
SMVMS is used in high frequency coefficients as high frequency information is generally bigger in focused area. In order to make the time consumption short, a synthesis image pnix,y (i is the decomposition level) is formed by combination of coefficients in six directions, and the fused coefficient Fix,y is equal to the maximum pnix,y which is calculated as(9)pnix,y=∑i=16pni,jx,yFix,y=maxp1ix,y,p2ix,y,…,pnix,y.
5. Experimental Results and Analysis
The proposed algorithm has been operated on Matlab 2014a with an Intel i5 4590 processor with 4 GB RAM. The traditional fusion methods based on DWT [16] and pyramids algorithms such as Gradient Pyramid (GRAD), Morphology Pyramid (MOP), Ratio Pyramid (RAT), and Laplacian Pyramid (LAP) [2, 13] are presented as compared to the proposed one. The DWT method, pyramids methods, and the proposed Q-shift DTCWT methods are all decomposed on 5 levels. In the aspect of fusion rules, traditional methods all take the rule of low frequency WA and high frequency SMVMS. For the proposed one, five different fusion rules described in the last section are used, and different matches are shown in Table 1. DTCWT1 is for comparing the performances among Q-shift DTCWT and the traditional methods. The remaining four are improved methods with better performance. Low frequency information has high correlation with the adjacent ones, so it is natural to reach an inference that more regional based fusion rules could improve its performance.
Results of IQA indicators on fused images.
Matches
Low frequency fusion rules
High frequency fusion rules
DTCWT1
WA
SMVMS
DTCWT2
NVMS
SMVMS
DTCWT3
NGMS
SMVMS
DTCWT4
SML
SMVMS
DTCWT5
NVMS
NVMS
In DTCWT2 to DTCWT4, three regional based fusion rules are applied on low frequency coefficients to testify this inference and, in DTCWT5, the regional based NVMS is used both on low frequency and on high frequency coefficients for potential improvement.
5.1. Experiments on Standard Images
In the first experiment, the performance of the proposed fusion method is demonstrated by fusing 20 pairs of blurred images which are generated by filtering the source images shown in Figure 3 with Gaussian filter. Due to space reasons, only six pairs are displayed but the others are similar to the presented ones. In each of these pairs, complementary regions of the source images are blurred. The source images are standard grayscale or color images and are taken as the ground truth images, serving as the templates for comparison.
Standard grayscale and color images for testing. (a) Peppers. (b) Couple. (c) Airplane. (d) Balloon. (e) Lena. (f) Flowers.
We have found that it is difficult to evaluate the quality of fused images visually, especially when the differences are too small to be observed. Hence, objective methods are used for more scientific and accurate evaluation. The first objective evaluation method is pixel value subtraction. The difference image is obtained from subtracting ground truth image from the fused image. The subtraction results of “Peppers” are shown in Figure 4 consisting of the ground truth image (Figure 4(a)), the partially blurred images (Figure 4(b) blurred in the middle Pepper and Figure 4(c) blurred in surroundings), subtraction results of the blurred images (Figures 4(d) and 4(e)), and subtraction result of Gradient Pyramid (Figure 4(f)), Morphology Pyramid (Figure 4(g)), Ratio Pyramid (Figure 4(h)), Laplacian Pyramid (Figure 4(i)), DWT (Figure 4(j)), and DTCWT1 to DTCWT5 (Figures 4(k)–4(o)).
Subtraction results of “Peppers.” (a) The ground truth image. (b) Source image blurred in the middle Pepper. (c) Source image blurred in the surroundings. (d) Subtraction image of (b). (e) Subtraction image of (c). (f) Subtraction image with Gradient Pyramid. (g) Subtraction image with Morphology Pyramid. (h) Subtraction image with Ratio Pyramid. (i) Subtraction image with Laplacian Pyramid. (j) Subtraction image with DWT. (k) Subtraction image with DTCWT1. (l) Subtraction image with DTCWT2. (m) Subtraction image with DTCWT3. (n) Subtraction image with DTCWT4. (o) Subtraction image with DTCWT5.
From analyzing the color of these subtraction images according to the color bar, we can judge the deviation between each fused image and the ground truth image. Obviously, subtraction results of GRAD and MOD bear relatively poor effect and those of RAT and LAP are better but perform extremely poor in edges. DWT performs better than the pyramids methods, and, by comparing DWT and DTCWT1, it can be concluded that DTCWT is better than DWT under the same fusion rules. But, with regional based fusion rules in low frequency coefficients, as in DTCWT2 to DTCWT4, the fused results are much better. Finally, in DTCWT5, where both low frequency and high frequency use regional based NVMS fusion rule, the deviation mainly focuses on the edge of the middle Pepper.
Another objective evaluation method is called the quantitative metrics [21]. Quantitative metrics are the objective evaluation indicators which can overcome the influence of human’s inaccurate vision judgment and make the indicators mathematically evaluate the effectiveness of image fusion. Three quantitative metrics are applied here, including mutual information (MI), peak signal-to-noise ratio (PSNR), and root mean square error (RMSE), which are described as follows.
5.1.1. Mutual Information (MI)
Let hA,hB, and hF denote the normalized histogram of the source image A, source image B, and fused image, respectively. hAF and hBF denote the joint histogram between the fused and source images. P and Q denote the row size and column size of the image. Mutual information between source images A, B and resulting fused images is as follows:(10)MIA=∑i=1P∑j=1QhAFijlog2hAFijhAijhFijMIB=∑i=1P∑j=1QhBFijlog2hBFijhBijhFij.
The total mutual information is defined as(11)MI=MIA+MIB.
Larger values imply better image quality.
5.1.2. Peak Signal-to-Noise Ratio (PSNR)
(12)PSNR=10log10L21/PQ∑i=1P∑j=1QRij-Fij2,where Rij, Fij denote, respectively, the (i,j)th pixel value of the reference image (ground truth image) and fused image and L is the count of gray levels in the image. PSNR value is higher if the ideal and fused images are more approximate. Higher value indicates better fusion.
5.1.3. Root Mean Square Error (RMSE)
RMSE between the reference image and the fused image can be calculated as(13)RMSE=1PQ∑i=1P∑j=1QRij-Fij2.
RMSE approaches zero when the reference and fused images are very approximate and it will increase when the similarity decreases.
The values of MI, PSNR, and RMSE of the fused images with various fusion methods are shown in Tables 2–4 and the best values are marked in bold. It can be seen that DTCWT2, DTCWT4, and DTCWT5 show better performance while being analyzed with MI. The PSNR values of LAP, DTCWT1, DTCWT2, and DTCWT3 methods are superior to the others. When calculated with RMSE, the best values belong to RAT, DTCWT3, and DTCWT4. Since DTCWT based fusion methods bear the best evaluation values in most situations, it can be concluded that the proposed fusion methods are superior to the existing pyramids and DWT methods while the fusion rules could be chosen in accordance with specific conditions, which will be studied in the future work.
Performance comparison of the standard images with MI.
Method
Images
Pep.
Cou.
Air.
Bal.
Lena
Flo.
GRAD
40.16
5.10
35.68
6.41
39.72
6.66
MOP
39.71
4.29
35.36
5.74
40.36
6.03
RAT
46.73
6.24
40.35
7.78
46.41
7.49
LAP
47.12
6.02
42.52
7.65
46.46
7.33
DWT
47.32
6.44
42.36
7.80
46.30
6.69
DTCWT1
49.66
6.98
42.95
8.27
47.46
7.32
DTCWT2
51.09
7.06
42.46
8.42
47.63
7.31
DTCWT3
50.12
7.04
42.06
8.39
47.63
7.68
DTCWT4
51.07
6.98
41.84
8.48
47.86
8.01
DTCWT5
52.11
7.02
44.12
8.50
48.30
7.66
Pep.: Peppers; Cou.: Couple; Air.: Airplane; Bal.: Balloon; Flo.: Flowers. The abbreviations here also apply to Tables 3, 4, and 9.
Performance comparison of the standard images with PSNR.
Method
Images
Pep.
Cou.
Air.
Bal.
Lena
Flo.
GRAD
27.77
30.60
28.63
29.15
29.27
27.13
MOP
29.07
30.55
28.16
29.86
30.85
28.22
RAT
28.14
28.80
26.69
30.35
28.82
27.20
LAP
41.78
46.34
37.21
44.03
39.62
38.74
DWT
41.09
45.28
34.87
42.14
38.94
34.24
DTCWT1
45.62
51.64
37.29
45.42
40.65
35.49
DTCWT2
46.41
52.13
35.96
45.16
40.32
35.03
DTCWT3
46.45
52.07
36.04
45.19
40.32
34.90
DTCWT4
46.36
51.75
35.83
45.17
40.05
34.79
DTCWT5
45.02
47.93
34.94
43.54
38.86
34.14
Performance comparison of the standard images with RMSE.
Method
Images
Pep.
Cou.
Air.
Bal.
Lena
Flo.
GRAD
6.15
4.75
5.27
5.75
5.33
6.11
MOP
5.42
3.27
4.95
4.96
4.59
5.55
RAT
0.20
1.09
1.22
1.39
1.24
2.23
LAP
1.67
0.89
2.16
1.07
1.34
2.05
DWT
1.61
0.89
2.46
1.32
1.70
2.75
DTCWT1
1.12
0.49
2.06
0.81
1.38
2.05
DTCWT2
0.91
0.44
2.11
0.71
1.40
2.05
DTCWT3
0.83
0.42
1.82
0.63
1.13
1.30
DTCWT4
0.93
0.48
2.13
0.64
1.59
1.16
DTCWT5
1.04
0.68
2.20
0.86
1.59
2.09
5.2. Experiments on Microscopic Images
In the second experiment, the fusion process is performed on two groups of images captured by a metallographic microscope. As described in Section 1, only the objects within the DOF of the objective lens of the microscope appear to be in focus. In reality, observed specimens within the field sometimes have rough surfaces which could not be completely captured in the DOF in one shot. A method is proposed to solve the problem in which several pictures partially focused are captured by adjusting the height of the lens and hence different parts are within the DOF. Then, fusion methods presented above are applied on the image sequence to obtain an entirely clear image.
Since metallographic microscopes are widely used in machinery, electronics, chip, chemical, precision instrument, and other industries for observing and analyzing surface quality of opaque matters, two examples, namely, the detection of Printed Circuit Board (PCB) and worn external turning tools, are presented in this paper with a metallographic microscope at 5x magnification and a CCD camera connected to a computer as our experimental devices. Fusion results are analyzed both visually and objectively.
In the first PCB experiment, since the components and their joints are uneven, it is hard to capture a totally clear image in one shot. But, with the methods proposed in this article, completely focused images can be acquired. Three image sequences of PCB are shown in Figures 5, 6, and 7, but, due to space reasons, only sequence 1 is displayed in detail. The PCB we used is shown in Figure 5(a) and the inspected area is marked in red blocks. In Figure 5, we can see three source images focused on board (Figure 5(b)), the solder paste (Figure 5(c)), and the component pin (Figure 5(d)) and ten fused images (Figures 5(e)–5(n)) using various fusion methods including GRAD, MOP, RAT, LAP, DWT, and DTCWT1 to DTCWT5. For better observation, a small part as shown in the red spotted line block in Figure 5(b) is magnified into Figures 5(o)–5(y), in which Figure 5(o) is extracted from the source image as a ground truth image and Figures 5(p)–5(y) are extracted from fused images. It is obvious that fused images with pyramids methods (Figures 5(p)–5(s)) show severe distortion. DWT method is better but contains lots of vertical light beams which do not exist in DTCWT method. DTCWT1 to DTCWT4 contain some degree of light dispersion but DTCWT5 shows perfect performance.
Source images and fused images of sequence 1 for PCB. (a) Tested PCB. (b) Source image 1. (c) Source image 2. (d) Source image 3. (e)–(n) Fused images with various fusion methods. (o) The magnified ground truth image. (p)–(y) The magnified fused images with various fusion methods.
Source image sequence 2 for PCB.
Source image sequence 3 for PCB.
In objective evaluation, we simply use PSNR and RMSE as our quantitative metrics since MI could not be used because each experiment has more than two source images. As it is impossible to obtain a completely clear image, we cut clear parts of source images as the ground truth image as shown in the red blocks in Figures 5–7. The fused images are also cropped at the same position for objective evaluation.
Values of PSNR and RMSE of these areas are shown in Tables 5 and 6 and the best values are marked in bold. It can be seen that DTCWT4 and DTCWT5 own the best performance when evaluated with PSNR while RAT, DTCWT3, DTCWT4, and DTCWT5 are better in RMSE evaluation. Generally, the results appear to be a good match to that in subjective evaluation. We can see that the proposed methods perform better. Fused images of the proposed methods reserve more details from source images and thus are clearer than other methods with less distortion. Using the proposed DTCWT fusion methods, we can obtain totally clear images of PCB for components failure detection, solder paste detection, and other applications.
Performance comparison of the PCB images with PSNR.
Method
Sequence 1
Sequence 2
Sequence 3
Part 1
Part 2
Part 3
Part 1
Part 2
Part 1
Part 2
GRAD
19.53
22.95
19.15
33.79
27.77
18.63
23.13
MOP
24.79
17.93
18.00
31.03
24.08
18.32
18.32
RAT
20.85
14.80
17.01
28.39
22.35
19.12
19.12
LAP
30.56
25.22
22.35
35.20
30.72
22.65
25.03
DWT
34.10
22.19
24.56
39.72
31.73
23.93
25.64
DTCWT1
37.33
24.17
31.67
45.66
36.68
24.68
28.90
DTCWT2
42.83
23.33
29.40
45.40
33.82
25.17
22.26
DTCWT3
33.34
20.53
26.03
43.66
33.86
21.92
23.18
DTCWT4
48.01
25.35
31.77
45.76
50.58
56.62
21.24
DTCWT5
42.22
25.74
29.16
48.58
33.58
59.39
30.98
Performance comparison of the PCB images with RMSE.
Method
Sequence 1
Sequence 2
Sequence 3
Part 1
Part 2
Part 3
Part 1
Part 2
Part 1
Part 2
GRAD
6.20
7.17
6.84
5.20
3.34
6.87
6.15
MOP
4.96
10.5
9.13
7.71
3.73
8.93
10.42
RAT
4.65
3.32
4.35
2.66
0.24
3.69
2.72
LAP
4.10
5.89
5.64
3.29
1.91
4.79
4.78
DWT
3.16
6.06
5.66
3.52
1.74
4.84
5.66
DTCWT1
2.18
4.63
2.72
2.32
0.94
4.35
4.54
DTCWT2
0.80
3.95
2.91
1.96
0.84
2.96
3.87
DTCWT3
0.61
3.48
2.09
1.06
0.65
2.57
2.96
DTCWT4
0.60
3.57
2.82
0.69
0.79
0.24
7.21
DTCWT5
1.25
5.33
4.20
2.84
0.74
0.19
4.32
For the second cutting tool experiment, image sequences for a worn external turning tool are shown in Figures 8–10 with the turning tool shown in Figure 8(a) and the inspected area marked in red blocks. Due to its worn surface, the turning tool seems to be partially focused and partially defocused in one image. We analyze the fusion results both subjectively and objectively, where fused images are not presented due to space reasons. The clear parts from different source images are cut as the ground truth images, as shown in the red blocks in Figures 8–10.
Source image sequence 1 for external turning tool.
Source image sequence 2 for external turning tool.
Source image sequence 3 for external turning tool.
PSNR and RMSE values of the fusion results are shown in Tables 7 and 8. We can draw the conclusion that DTCWT based methods also have outstanding performances in fusion of turning tool images.
Performance comparison of the turning tool images with PSNR.
Method
Sequence 1
Sequence 2
Sequence 3
Part 1
Part 2
Part 3
Part 1
Part 2
Part 1
Part 2
GRAD
21.23
16.78
24.91
20.93
30.23
27.70
20.84
MOP
21.05
15.93
20.54
18.32
30.09
28.49
18.71
RAT
17.29
13.37
17.16
16.73
24.33
23.91
16.79
LAP
27.22
18.93
23.26
19.81
35.64
34.55
21.73
DWT
28.89
19.31
23.73
22.66
36.78
39.02
22.56
DTCWT1
32.11
20.81
29.32
23.77
38.64
42.55
26.14
DTCWT2
34.90
20.49
24.22
20.85
39.32
44.31
23.21
DTCWT3
29.06
15.88
24.10
18.97
35.70
39.50
23.17
DTCWT4
39.91
48.72
25.34
41.54
63.63
53.18
24.39
DTCWT5
30.17
28.01
35.08
54.15
59.81
55.06
27.46
Performance comparison of the turning tool images with RMSE.
Method
Sequence 1
Sequence 2
Sequence 3
Part 1
Part 2
Part 3
Part 1
Part 2
Part 1
Part 2
GRAD
7.31
7.21
5.26
4.12
4.32
6.48
4.62
MOP
9.15
6.65
8.28
6.34
3.34
9.10
4.55
RAT
5.15
4.08
2.33
1.27
2.42
1.90
2.47
LAP
5.97
5.69
3.42
2.54
1.87
5.59
2.48
DWT
6.77
5.48
5.06
3.16
2.22
6.19
1.96
DTCWT1
6.20
4.39
2.24
2.80
2.01
4.22
1.34
DTCWT2
4.09
1.87
2.84
2.28
0.51
4.09
0.55
DTCWT3
3.71
2.16
1.82
2.21
0.89
3.44
0.57
DTCWT4
0.70
1.77
2.60
1.65
0.11
4.99
0.29
DTCWT5
4.73
1.76
2.80
0.34
0.18
5.93
0.31
Calculating time of the standard images (unit: s).
Method
Images
Pep.
Cou.
Air.
Bal.
Lena
Flo.
GRAD
0.617
0.251
0.663
0.266
0.539
0.246
MOP
2.240
1.116
2.567
0.774
2.586
0.790
RAT
0.449
0.252
0.355
0.211
0.354
0.220
LAP
0.601
0.249
0.440
0.239
0.437
0.228
DWT
6.335
1.746
5.902
1.640
5.833
1.628
DTCWT1
10.82
2.874
10.39
2.763
10.35
2.753
DTCWT2
10.81
2.798
10.33
2.758
10.33
2.759
DTCWT3
10.87
2.763
10.36
2.770
10.35
2.747
DTCWT4
10.90
2.803
10.40
2.761
10.29
2.764
DTCWT5
10.98
2.834
10.54
2.800
10.45
2.790
6. Time Consumption
This part discusses the calculation time of the above experiments. From Tables 9 and 10, it can be concluded that calculating processes using pyramids algorithm cost very little time while the DWT algorithm costs a little more time, and the proposed Q-shift DTCWT methods cost the longest time. It seems that different fusion rules utilized in this paper have little to do with the costing time.
Calculating time of PCB images and external turning tool (ETT.) images (unit: s).
Method
Images
PCB 1
PCB 2
PCB 3
ETT.1
ETT.2
ETT.3
GRAD
14.33
8.535
11.45
5.736
5.791
5.745
MOP
94.60
56.42
75.19
36.67
36.84
36.67
RAT
6.436
3.874
5.137
2.608
2.582
2.695
LAP
10.12
6.071
8.043
4.039
4.061
4.031
DWT
48.62
48.49
47.89
44.72
44.85
44.94
DTCWT1
91.14
87.00
87.51
81.84
81.73
82.06
DTCWT2
101.1
84.64
87.52
81.72
81.71
82.34
DTCWT3
90.47
85.16
87.76
81.75
82.14
82.03
DTCWT4
90.09
85.77
87.50
82.16
82.36
82.32
DTCWT5
92.96
90.37
89.44
83.36
83.27
83.77
7. Conclusion
This paper provides an effective method for fusing various images by using Q-shift DTCWT. Since DTCWT is approximately shift invariant and has six directions: (±15°, ±45°, ±75°), more than in DWT, it could preserve more detail information and edge information of source images, and the Q-shift resolution simplifies its filter construction process. Since fusion rule is another significant factor in image fusion, five different fusion rules are presented in this article and evaluated both visually and objectively. From the plenty of experimental data, we can draw the conclusion that Q-shift DTCWT methods are a little better than some other multiresolution fusion methods (using the same fusion rules), but, with low frequency regional based fusion rules or both low frequency and high frequency regional based fusion rules, the DTCWT methods have outstanding performances. The proposed methods are used on microscopic image fusion including PCB and worn external turning tool, and the results turn out to be consistent with the ideal experiments. The proposed methods were proven to cost more time than pyramids algorithm and the DWT algorithm, and it seems that using different fusion rules will not add the costing time. Future work will be done for founding better fusion rules and discussing whether different fusion rules are appropriate for different situations.
Competing Interests
The authors declare that they have no competing interests.
Acknowledgments
This research work was supported by the National Key Technology Support Program of China (Grant no. 2015BAF10B01) and Science and Technology Commission of Shanghai Municipality (Grants nos. 15111104002 and 15111106302).
BaiX.ZhangY.ZhouF.XueB.Quadtree-based multi-focus image fusion using a weighted focus-measure20152210511810.1016/j.inffus.2014.05.0032-s2.0-84907599964PiellaG.A general framework for multiresolution image fusion: from pixels to regions20034425928010.1016/s1566-2535(03)00046-02-s2.0-0242270987HallD. L.LlinasJ.An introduction to multisensor data fusion199785162310.1109/5.5542052-s2.0-0030735959LiuY.LiuS.WangZ.A general framework for image fusion based on multi-scale transform and sparse representation20152414716410.1016/j.inffus.2014.09.0042-s2.0-84922950106LiS. T.YangB.HuJ. W.Performance comparison of different multi-resolution transforms for image fusion2011122748410.1016/j.inffus.2010.03.0022-s2.0-78650564982MatsopoulosG. K.MarshallS.Application of morphological pyramids: fusion of MR and CT phantoms19956219620710.1006/jvci.1995.10182-s2.0-0029323845MiaoQ. G.WangB. S.Multi-sensor image fusion based on improved Laplacian pyramid transform2007279160516102-s2.0-35348997927SelesnickI. W.BaraniukR. G.KingsburyN. G.The dual-tree complex wavelet transform200522612315110.1109/MSP.2005.15501942-s2.0-33747800938LiS.KangX.HuJ.Image fusion with guided filtering20132272864287510.1109/tip.2013.22442222-s2.0-84878331557YinH.LiS.FangL.Simultaneous image fusion and super-resolution using sparse representation201314322924010.1016/j.inffus.2012.01.0082-s2.0-84885423284YangB.LiS.Pixel-level image fusion with simultaneous orthogonal matching pursuit2012131101910.1016/j.inffus.2010.04.0012-s2.0-80053987231DiH. W.LiuX. F.Image fusion quality assessment based on structural similarity20063557667712-s2.0-33745172158AliF. E.El-DokanyI. M.SaadA. A.Abd El-SamieF. E.Curvelet fusion of MR and CT images2008321522410.2528/pierc08041305ChenY.WangL.SunZ.JiangY.ZhaiG.Fusion of color microscopic images based on bidimensional empirical mode decomposition2010182121757217692094107610.1364/OE.18.0217572-s2.0-7804950506920941076RenG. C.ShiL.Medical image fusion algorithm based on 2v-SVM and consistency checking20104613199201YangY.HuangS. Y.GaoJ. F.QianZ.Multi-focus image fusion using an effective discrete wavelet transform based algorithm201414210210810.2478/msr-2014-00142-s2.0-84912559202SinghR.SrivastavaR.PrakashO.KhareA.Multimodal medical image fusion in dual tree complex wavelet transform domain using maximum and average fusion rules20122216817310.1166/jmihi.2012.10802-s2.0-84881236418LiX.ZhanX.A new EPMA image fusion algorithm based on contourlet-lifting wavelet transform and regional variance201051112001207WeiS.WangK.YuanG. L.A multi-focus image fusion algorithm in the complex wavelet domain2008135951957AslantasV.KurbanR.Fusion of multi-focus images using differential evolution algorithm201037128861887010.1016/j.eswa.2010.06.0112-s2.0-77957833232BalasubramaniamP.AnanthiV. P.Image fusion using intuitionistic fuzzy sets2014201213010.1016/j.inffus.2013.10.0112-s2.0-84901595802