This paper presents a novel Gabor phase based illumination invariant extraction method aiming at eliminating the effect of varying illumination on face recognition. Firstly, It normalizes varying illumination on face images, which can reduce the effect of varying illumination to some extent. Secondly, a set of 2D real Gabor wavelet with different directions is used for image transformation, and multiple Gabor coefficients are combined into one whole in considering spectrum and phase. Lastly, the illumination invariant is obtained by extracting the phase feature from the combined coefficients. Experimental results on the Yale B and the CMU PIE face database show that our method obtained a significant improvement over other related methods for face recognition under large illumination variation condition.
Natural Science Foundation of the Jiangsu Higher Education Institutions of China17KJB520021Jiangsu Government Scholarship for Overseas StudiesNanjing University of Information Science and TechnologyPriority Academic Program Development of Jiangsu Higher Education Institution1. Introduction
Face recognition has attracted much interest for its wide application. Although great progress has been made according to related researches [1, 2], many issues still remain unresolved, including varying illumination, pose, and expression problems. The varying illumination problem is intractable yet crucial and has to be dealt with. The illumination can change face appearance dramatically and thus will seriously affect the performance of face recognition system [3]. To address this problem, researchers have proposed many approaches, and these methods are mainly classified into three groups.
(1) Illumination Preprocessing. These approaches adopt image processing to remove lighting effects from face images to obtain illumination normalized face images [4, 5]. Tan and Triggs [6] proposed a preprocessing chain which combines Gamma correction (GC) and difference of Gaussian (DoG) with contrast equalization. It eliminates most of illumination effects while still preserving needed essential appearance details. Fan and Zhang [7] presented a homomorphic filtering (HF) based illumination normalization algorithm and obtained promising results. Recently, Lee et al. compensated illumination using orientated local histogram equalization (OLHE), which encoded rich information on the edge orientations [8]. Illumination preprocessing methods are simple, effective, and efficient. However, they could not resolve extreme uneven illumination variations completely [3].
(2) Face Modeling. Illumination variations are mainly generated from the 3D shape of human faces under various lighting directions. A generative 3D face model has been constructed to render face images with different poses and illumination. Belhumeur et al. [9, 10] proposed an illumination cone named generative model, which uses an illumination convex cone to represent face images set with changing illumination conditions under fixed pose. They first construct an illumination convex cone using a great deal of images with varying lighting and then use a low-dimensional linear subspace to represent the cone approximated. Basri and Jacobs [11] found that a 9D linear subspace could approximate the set of images of a convex Lambertian object with varying lighting very well. These methods need images of the same subject with varying lighting and 3D shape information for training. However, these needs could not be met in real world. Therefore, the application of these methods is limited.
(3) Illumination Invariant Extraction. This kind of approach is the mainstream, which tries to extract illumination-robust facial features. Many methods are based on the Lambertian illumination model. In that model, a face image f(x,y) under illumination conditions is generally regarded as a product f(x,y)=i(x,y)r(x,y), where r(x,y) is the reflectance component and i(x,y) is the illumination component at each point (x,y) [12]. The objective is to extract the reflectance component r(x,y), which is considered as the intrinsic information specific to each class. However, it is difficult to calculate the reflectance and the illuminance component from real images. A common assumption is that i(x,y) varies slowly and mainly lies in low frequencies, while rx,y can change abruptly and typically lies in high frequencies. Under this assumption, Jobson et al. [13] proposed the multiscale retinex (MSR) method which estimated the reflectance component as the ratio of the image and its low-pass version that served as estimate for the illumination component. Wang et al. [14] used a similar idea (with a different local filter, namely, the weighted Gaussian filter) in the Self Quotient Image (SQI), which was very simple and could be applied to any single image. However, the used weighted Gaussian filter can hardly keep sharp edges in low frequency illumination fields, and it needs experience and time to select proper parameter. Aiming at solving this problem, Chen et al. [15] replaced the weighted Gaussian filter by Logarithmic Total Variation (LTV) to improve SQI. In 2009, Zhang et al. [16] presented a wavelet-based illumination invariant method (WD), which extracted denoised high frequency component in wavelet domain as the reflectance component. Inspired by this, Cheng et al. [17] and Xie et al. [18] presented two similar illumination invariant extraction methods in the nonsubsampled Contourlet transform (NSCT) domain. In 2011, Chen et al. [19] utilized the scale invariant property of natural images to derive a Wiener filter approach to best separate the illumination invariant features from an image. Cao et al. [20] proposed a wavelet-based illumination invariant extraction approach while taking the correlation of neighboring wavelet coefficients into account in 2012. Recently, Song et al. [21] presented a novel illumination invariant, histogram-based descriptor, and Faraji and Qi [22] proposed a novel illumination invariant using logarithmic fractal dimension-based complete eight local directional patterns. Experiments show that these methods have achieved very good results. Chen et al. [23] revealed that the direction of the image gradient is insensitive to changes of illumination. Based on this, Zhang et al. [24] introduced the Gradientfaces method, which used the arctan of the ratio between y- and x-gradient of an image as Gradientfaces. Chen and Zhang [25] improved the Gradientfaces method by proposing multidirectional orthogonal gradient phase faces method.
Recent studies confirm that the phase also contains a lot of effective information for image feature extraction, comparing with the magnitude [26]. Based on this, Sao and Yegnanarayana [27] presented a 2D Fourier phase based face image representation, and Cheng et al. [28] presented a novel illumination invariant method, namely, multiscale principal contour direction (MPCD). Inspired by the above mentioned, based on Gabor wavelet’s excellent visual physiology background and its powerful ability as a feature descriptor, we present a novel illumination invariant extraction method based on the Gabor wavelet phase (GF) in this paper. We first preprocess the face image by using a homomorphic filtering (HF) based illumination normalization algorithm [7]. Then a set of 2D real Gabor wavelet with different directions is used for image transformation. Finally, multiple Gabor coefficients are combined into one whole in considering both spectrum and phase information and the illumination invariant is obtained by extracting the phase feature from the combined coefficients. The 2D symmetric real Gabor wavelet is chosen in our method, which aims not only at avoiding the complexity of complex calculations, but also at fitting the symmetry of the face image itself.
The rest of this paper is organized as follows. Section 2 presents the proposed method in detail. The experimental results and our conclusions are shown in Sections 3 and 4, respectively.
2. Algorithm Description
Researchers have found that Gabor functions have the capability of modeling simple cells in the visual cortex of mammalian brains [29]. Thus, image analysis using Gabor functions is similar to perception in the human visual system. Frequency and orientation representations of Gabor filters are similar to those of the human visual system, and they are particularly appropriate for texture representation and discrimination.
In recent years, the Gabor wavelet transform has been widely used as an effective element in face recognition [26, 30–34]. Gabor wavelet transform is insensitive to external environment factors such as illumination, facial expressions, gestures, and occlusion [35]. For this reason, it has been widely used to extract robust facial feature. Most existing Gabor feature-based methods usually use the Gabor magnitude features and discard the phase features. However, studies have shown that the phase information contains a number of effective image features, and it is insensitive to illumination variation. Inspired by this, the Gabor phase features are extracted as illumination invariants in this paper.
The proposed illumination invariant extraction method consists of three steps. Firstly, a homomorphic filtering based illumination normalization method [7] is used to preprocess the face images. Secondly, a set of 2D real Gabor wavelet with different directions is used for image transformation. Lastly, multiple Gabor coefficients are combined into one whole in considering both spectrum and phase information, and the illumination invariant is obtained by extracting the phase feature from the combined coefficients.
2.1. Illumination Normalization
We use the method presented in the literature [7], which combines homomorphic filtering and histogram equalization. In our paper, this illumination normalization method is called HF + HQ for short. This preprocessing greatly corrects the uneven illumination effects.
2.2. 2D Gabor Wavelet Transform
Different Gabor wavelets can be obtained by using different kernel functions. In order to avoid the complexity of complex calculations, and to fit the symmetry of the face image itself, the 2D symmetric real Gabor wavelet is chosen in our method. The kernel function used in this paper is the following.(1)Gx,y,θk,f=exp-12x′sx2+y′sy2cos2πfx′,x′=xcosθk+ysinθk,y′=ycosθk-xsinθk,where f is the frequency of the sinusoidal function, sx and sy represent the spatial scaling coefficient along x- and y-axes, respectively, and θk is the orientation of Gabor filter. θk is defined by this formula.(2)θk=πnk-1,k=1,2,…,n.
Here, n determines the number of the filter’s orientation, and we set n=8 in this paper. If sx and sy are selected, after the Gabor wavelet transform of an gray face image f(x,y), we have(3)Cx,y,θk,f=fx,y⊗Gx,y,θk,f.
Here, ⊗ indicates convolution of the two functions. Cx,y,θk,f is denoted as Cθk,f for short. All the Gabor wavelet transformed coefficients are denoted as follows.(4)Cθk,f,f=0,2,4,8,16,32;k=1,2,…,8.
Figure 1 illustrates the spectrograms of a face image under the same frequency (f=0) and 8 different orientations (θk).
The Gabor spectrogram of a face image.
2.3. Illumination Invariant Extraction
After 2D Gabor wavelet transforming, the set {Cθk,f} includes all the spectrum information under different orientations. In order to take the phase information into account, we define the complex wavelet coefficients as follows.(5)C-θk,f=Cθk,feiθk.
Then, summing up all the complex coefficients under the same frequency (f) aiming at reducing feature dimension, we have(6)Sf=∑θk=π/87π/8C-θk,f.
The phase feature is calculated by this formula.(7)Λfx,y=arctanImSfx,yReSfx,y.
Here, ReSfx,y and ImSfx,y are the real and imaginary part of Sfx,y, respectively. In this paper, the phase feature Λf(x,y) is considered as the illumination invariant. Figure 2 shows the illumination normalized face images and the obtained illumination invariant.
Examples of the HF + HQ processed images and the extracted illumination invariant.
3. Experimental Results
In this section, the performance of the proposed method (GF) is compared with the existing methods including MSR, WD, LTV, Gradientfaces, and MPCD using Yale B [36] and CMU PIE [37]. Firstly, we present different illumination invariants in image form. Then, we compare their recognition performance by using Eigenfaces under the same experimental conditions. According to the FERET testing protocol [38], the Tops 1 and 3 recognition rate are tested.
3.1. Comparison of the Different Illumination Invariants
To prove the efficiency of different methods, Figure 3 shows some original images in the Yale B and their corresponding illumination invariants. As can be seen from the images, the GF method has removed most effects of the illumination variation and greatly reduced the intraclass difference.
Examples of the different illumination invariants.
3.2. Experimental Results for the Yale B
The face database of Yale B has 10 different persons, and each person has 9 poses, and each pose is captured by 64 different illumination conditions. In our experiments, the frontal images are used. And, based on the angle of light source direction, these images are classified into five subsets. They are subset 1 (0–12°), subset 2 (13°–25°), subset 3 (26°–50°), subset 4 (51°–77°), and subset 5 (others) [39]. All images are cropped and rescaled to 192×168 pixels with strict alignment. The five images of each subset (each row) for one person are shown in Figure 4, and their illumination invariants are extracted by GF method.
Images of subsets 1–5 and their illumination invariants using GF.
Firstly, we select subset 1 as the training set and others as the testing set. As can be seen from Figures 5 and 6 the proposed method (GF) outperforms MSR, WD, and LTV, and GF obtains outstanding results similar to the Gradientfaces and the MPCD method. The average Top 1 recognition rate is nearly 99%.
Comparisons on the Top 1 recognition rate (the subset 1 used as the training set).
Comparisons on the Top 3 recognition rate (the subset 1 used as the training set).
Secondly, subset 4 is selected as the training set, and others are used as testing set. Figures 7 and 8 show the recognition rates. It is clearly seen that the performance of GF is far greater than others and achieves 100% recognition rate on each testing subset.
Comparisons on the Top 1 recognition rate (the subset 4 used as the training set).
Comparisons on the Top 3 recognition rate (the subset 4 used as the training set).
Thirdly, for the training set, we randomly choose 10 images for each person, namely, subset r, and the others are used for testing. To achieve a credible result, the result is averaged over 50 random splits. The experiment results are presented in Figures 9 and 10. It can be observed that the recognition rate of GF is higher than the other methods, and the performance is quite similar to that of the Gradientfaces and the MPCD, and it reaches a 100% recognition rate on every testing subset except subset 2.
Comparisons on the Top 1 recognition rate (the subset r used as the training set).
Comparisons on the Top 3 recognition rate (the subset r used as the training set).
The experiments are implemented using different training sets. It is clearly seen that the proposed method obtains excellent results under different conditions, and this demonstrates its robustness to illumination.
3.3. Experimental Results for the CMU PIE
The CMU PIE face database [37] contains images of 68 persons with various poses, illuminations, and expressions. In our experiments, the illumination subset (C27) is used, and C27 has 21 different illuminations for each person. All images used in our experiments are cropped and rescaled to 64×64 pixels. 21 different illumination images of a person on CMU PIE and their illumination invariants extracted by GF are shown in Figure 11.
Images of a person in C27 and their illumination invariants using GF.
The experiments on CMU PIE are divided into two sections. In Section 1, the first 3, 4, and 5 images of each person are selected as the training set and others as the testing set, respectively. Table 1 shows the recognition results of Section 1. In Section 2, we randomly choose 3, 4, and 5 images of each person as training set and others as testing set, respectively. To achieve a credible result, the result is averaged over 50 random splits. The recognition results of Section 2 are tabulated in Table 2. From Tables 1 and 2, it can be seen that the proposed method outperforms all other methods and consistently achieves a high recognition rate, which strongly shows its outstanding efficacy in relation to varying illumination.
Comparison on the recognition rate of different illumination invariants in Section 1.
Recognition rate (%)
Training set
Original
MSR
WD
LTV
Gradientfaces
MPCD
GF
Top 1
3
37.61
51.33
89.36
89.78
91.04
89.99
94.54
4
39.57
63.24
91.81
93.56
95.87
92.79
96.08
5
43.21
72.90
96.15
96.15
99.16
96.64
97.34
Top 3
3
47.20
61.20
95.52
94.82
97.13
95.73
99.44
4
50.49
71.99
97.48
96.99
99.02
97.55
99.51
5
54.41
79.90
98.81
98.32
99.86
99.30
99.93
Comparison on the recognition rate of different illumination invariants in Section 2.
Recognition rate (%)
Training set
Original
MSR
WD
LTV
Gradientfaces
MPCD
GF
Top 1
3
40.94
80.23
99.08
98.90
99.22
99.21
99.67
4
42.17
84.50
99.68
99.37
98.46
99.24
99.84
5
42.86
87.26
99.78
99.46
99.95
99.58
99.83
Top 3
3
53.60
87.85
99.66
99.55
99.93
99.75
99.93
4
55.59
91.16
99.89
99.79
99.98
99.87
99.96
5
56.38
92.75
99.93
99.80
99.98
99.93
99.96
Run time is also critical in real application. To evaluate the computational complexity of each method, the run time of processing a 168×192 pixel face image of each method is presented in Table 3. The hardware platform is 2.6 GHz P4 with 2G memory. Table 3 shows that the proposed method only needs 37 ms to process a face image, which shows that it can process face images in real time and thus it is able to handle large face databases. MSR, LTV, and MPCD are slower than our method.
Comparison on the run time of different illumination invariants.
Method
MSR
WD
LTV
Gradientfaces
MPCD
GF
Time (ms)
171
36
59
22
97
37
4. Conclusion
In this paper, we propose an efficient Gabor phase based illumination invariant extraction method. We first normalize face images using a homomorphic filter-based preprocessing method to preeliminate effects of the illumination changes. Then, a set of 2D real Gabor wavelet with different directions is used for image transformation, and multiple Gabor coefficients are combined into one whole in considering both spectrum and phase. Lastly, the illumination invariant is obtained by extracting the phase feature from the combined coefficients. The proposed method does not need 3D face shape information or a bootstrap for training. And the extracted illumination invariant contains more essential discriminant information while greatly reducing the effect of illumination changes at the same time. Experimental results show its effectiveness and robustness to different illumination variation.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work is supported by the Natural Science Foundation of the Jiangsu Higher Education Institutions of China under Grant 17KJB520021, Jiangsu Government Scholarship for Overseas Studies, Training Projects of Undergraduate Practice Innovation funded by Nanjing University of Information Science and Technology, and a Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institution.
ZhaoW.ChellappaR.PhillipsP. J.RosenfeldA.Face recognition: a literature surveyPhillipsP. J.ScruggsW. T.O'TooleA. J.FlynnP. J.BowyerK. W.SchottC. L.SharpeM.FRVT 2006 and ICE 2006 large-scale experimental resultsAdiniY.MosesY.UllmanS.Face recognition: the problem of compensating for changes in illumination directionShanS.GaoW.CaoB.ZhaoD.Illumination normalization for robust face recognition against varying lighting conditionsProceedings of the IEEE International Workshop on Analysis and Modeling of Faces and Gestures (AMFG '03)October 2003Nice, FranceIEEE15716410.1109/amfg.2003.1240838SavvidesM.KumarB. V.Illumination Normalization Using Logarithm Transforms for Face AuthenticationTanX.TriggsB.Enhanced local texture feature sets for face recognition under difficult lighting conditionsFanC.-N.ZhangF.-Y.Homomorphic filtering based illumination normalization method for face recognitionLeeP.-H.WuS.-W.HungY.-P.Illumination compensation using oriented local histogram equalization and its application to face recognitionBelhumeurP. N.KriegmanD. J.What Is the Set of Images of an Object under All Possible Illumination Conditions?GeorghiadesA. S.BelhumeurP. N.KriegmanD. J.From few to many: illumination cone models for face recognition under variable lighting and poseBasriR.JacobsD. W.Lambertian reflectance and linear subspacesHornB.JobsonD. J.RahmanZ.-U.WoodellG. A.A multiscale retinex for bridging the gap between color images and the human observation of scenesWangH.LiS. Z.WangY.Face recognition under varying lighting conditions using self quotient imageProceedings of the 6th IEEE International Conference on Automatic Face and Gesture Recognition (FGR '04)May 2004Seoul, Korea8198242-s2.0-4544274485ChenT.YinW.ZhouX. S.ComaniciuD.HuangT. S.Total variation models for variable lighting face recognitionZhangT.FangB.YuanY.Yan TangY.ShangZ.LiD.LangF.Multiscale facial structure representation for face recognition under varying illuminationChengY.HouY.ZhaoC.LiZ.HuY.WangC.Robust face recognition based on illumination invariant in nonsubsampled contourlet transform domainXieX.LaiJ.ZhengW.-S.Extraction of illumination invariant facial features from a single image using nonsubsampled contourlet transformChenL.-H.YangY.-H.ChenC.-S.ChengM.-Y.Illumination invariant feature extraction based on natural images statistics Taking face images as an exampleProceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011June 2011Colorado Springs, Colo, USA68168810.1109/CVPR.2011.59956212-s2.0-80052875951CaoX.ShenW.YuL. G.WangY. L.YangJ. Y.ZhangZ. W.Illumination invariant extraction for face recognition using neighboring wavelet coefficientsSongT.XiangK.WangX.-Y.Face recognition under varying illumination based on gradientface and local featuresFarajiM. R.QiX.Face recognition under varying illuminations using logarithmic fractal dimension-based complete eight local directional patternsChenH. F.BelhumeurP. N.JacobsD. W.In search of illumination invariants1Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '00)June 2000Hilton Head Island, SC, USA2542612-s2.0-0033719241ZhangT.TangY. Y.FangB.ShangZ.LiuX.Face recognition under varying illumination using gradientfacesChenX.ZhangJ.Illumination robust single sample face recognition using multi-directional orthogonal gradient phase facesZhangB.ShanS.ChenX.GaoW.Histogram of Gabor phase patterns ({HGPP}): a novel object representation approach for face recognitionSaoA. K.YegnanarayanaB.On the use of phase of the Fourier transform for face recognition under variations in illuminationChengY.WangC. L.LiZ. Y.HouY. K.ZhaoC. X.Multiscale principal contour direction for varying lighting face recognitionDaugmanJ. G.Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filtersWiskottL.FellousJ.-M.KrügerN.von der MalsburgC.Face recognition by elastic bunch graph matchingLiuC.WechslerH.Gabor feature based classification using the enhanced Fisher linear discriminant model for face recognitionZhangW.ShanS.GaoW.ChenX.ZhangH.Local gabor binary pattern histogram (LGBPHS): a novel non-statistical model for face representation and recognitionProceedings of the 10th IEEE International Conference on Computer Vision (ICCV '05)October 2005Beijing, China78679110.1109/ICCV.2005.1472-s2.0-33745953205XieS.ShanS.ChenX.MengX.GaoW.Learned local Gabor patterns for face representation and recognitionSerranoÁ.Martín De DiegoI.CondeC.CabelloE.Analysis of variance of Gabor filter banks parameters for optimal face recognitionGoswamiJ. C.ChanA. K.LeeK.-C.HoJ.KriegmanD. J.Acquiring linear subspaces for face recognition under variable lightingSimT.BakerS.BsatM.The CMU pose, illumination, and expression (PIE) databaseProceedings of the 5th IEEE International Conference on Automatic Face and Gesture RecognitionMay 2002Washington, DC, USA465110.1109/AFGR.2002.1004130PhillipsP. J.WechslerH.HuangJ.RaussP. J.The FERET database and evaluation procedure for face-recognition algorithmsDuS.WardR.Wavelet-based illumination normalization for face recognition2Proceedings of the IEEE International Conference on Image Processing, ICIP 2005September 2005Genoa, Italy95495710.1109/ICIP.2005.15302152-s2.0-33749605142