Thermal infrared (IR) images focus on changes of temperature distribution on facial muscles and blood vessels. These temperature changes can be regarded as texture features of images. A comparative study of face two recognition methods working in thermal spectrum is carried out in this paper. In the first approach, the training images and the test images are processed with Haar wavelet transform and the LL band and the average of LH/HL/HH bands subimages are created for each face image. Then a total confidence matrix is formed for each face image by taking a weighted sum of the corresponding pixel values of the LL band and average band. For LBP feature extraction, each of the face images in training and test datasets is divided into 161 numbers of subimages, each of size 8 × 8 pixels. For each such subimages, LBP features are extracted which are concatenated in manner. PCA is performed separately on the individual feature set for dimensionality reduction. Finally, two different classifiers namely multilayer feed forward neural network and minimum distance classifier are used to classify face images. The experiments have been performed on the database created at our own laboratory and Terravic Facial IR Database.
In the modern society, there is an increasing need to track and recognize persons automatically in various areas such as in the areas of surveillance, closed circuit television (CCTV) control, user authentication, human computer interface (HCI), daily attendance register, airport security checks, and immigration checks [
Wavelength ranges for different infrared spectrums.
|
Wavelength range |
---|---|
Visible spectrum | 0.4–0.7 |
Near infrared (NIR) | 0.7–1.0 |
Short-wave infrared (SWIR) | 1–3 |
Mid-wave infrared (MWIR) | 3–5 |
Thermal infrared (TIR) | 8–14 |
Thermal IR band is more popular to the researchers working with thermal images. Recently researchers have been using near-IR imaging cameras for face recognition with better results [
The proposed thermal face recognition system (TFRS) can be subdivided into four main parts, namely, image acquisition, image preprocessing, feature extraction, and classification. The image preprocessing part involves binarization of the acquired thermal face image, extraction of largest component as the face region, finding the centroid of the face region, and finally cropping of the face region in elliptic shape. The two different features extraction techniques have been discussed in this paper. The first one is to find LL band and HL/LH/HH average band images using Haar wavelet transform, and the total confidence matrix is used as a feature vector. The eigenspace projection is performed on feature vector to reduce the dimensionality. This reduced feature vector is fed into a classifier. The second method of features extraction technique is local binary pattern (LBP). As a classifier, a back propagation feed forward neural network or a minimum distance classifier is used in this paper. The block diagram of the proposed system is given in Figure
Schematic block diagram of the proposed system.
In the present work, unregistered thermal and visible face images are acquired simultaneously with variable expressions, poses, and with/without glasses. Till now 17 individuals have volunteered for this photo shots, and for each individual 34 different templates of RGB color images with different expressions, namely, (Exp1) happy, (Exp2) angry, (Exp3) sad, (Exp4) disgusted, (Exp5) neutral, (Exp6) fearful and (Exp7) surprised are available. Different pose changes about
Thermal face image and its various preprocessing stages.
Thermal face image
Corresponding grayscale image
Binary image
The binarization of 24-bit colour image is divided into two steps. In the first step, the colour image is converted into an 8-bit grayscale image using
If the gray value of any pixel
The foreground of a binary image may contain more than one object or components. Say, in Figure
Different connected neighborhoods.
4-connected neighbors
8-connected neighbors
A pixel to be connected to itself is called reflexive. A pixel and its neighbour are mutually connected is called symmetric. 4-connectivity and 8-connectivity are also transitive: if A is connected to pixel B, and pixel B is connected to pixel C, then there exists a connected path between pixels A and C. A relation (such as connectivity) is called an equivalence relation if it reflexive, symmetric and transitive. All equivalence classes of connected pixels in a binary image, is called connected component labelling. The result of connected component labelling is another image in which everything in one connected region is labeled “1” (for example), everything in another connected region is labeled “2”, and so forth. For example, the binary image in Figure
(a) Connected component, (b) Labeled connected components.
“Connected component labeling” algorithm is given in Algorithm
//LabelConnectedComponent(im) is a method which takes//one argument that is an image named if. ( neighbourhood) Create a new label for Mark e Choose one of the labels for (
Using “connected component labeling” algorithm, the largest component of face region is identified from Figure
The largest component as a face skin region.
Centroid has been extracted from the largest component of the binary image using
Normally, human face is of ellipse shape. Then, from the above centroid coordinates, human face has been cropped in elliptic shape using “Bresenham ellipse drawing” [
Cropped face region in elliptic shape.
The first method of feature extraction is discrete wavelet transform (DWT). The DWT was invented by the Hungarian mathematician Alfréd Haar in 1909. A key advantage of wavelet transform over Fourier transforms is temporal resolution. Wavelet transform captures both frequency and spatial information. The DWT has a huge number of applications in science, engineering, computer science, and mathematics. The Haar transformation is used here since it is the simplest wavelet transform of all and can successfully serve our purpose. Wavelet transform has merits of multiresolution, multiscale decomposition and so on. To obtain the standard decomposition [
As illustrated in Figure
// Im[1
Sketch map of the quadratic wavelet decomposition.
Let us start with a simple example of 1D wavelet transform [
Resolution, mean, and the detail coefficients of full decomposition.
Resolution | Mean | Detail coefficients |
---|---|---|
4 |
|
|
2 |
|
|
1 |
|
|
Thus, the one-dimensional Haar transform of the original four-pixel image is given by
Haar wavelet transform.
Transform rows
Transform columns
The pixels of LL2 image can be rearranged horizontally or vertically. So the image can be treated as a vector (called feature vector).
In the present work, wavelet transform is used on the elliptic shape face region once which divide the whole image into 4 equal sized subimages, namely, low-frequency LL band (approximate component) and three high-frequency bands (detailed components), HL, LH, and HH. Then the pixelwise average of the detail components is computed using
Next, a matrix called total confidence matrix
Mixing technique.
After calculating the total confidence matrices for all the images, each matrix is transformed into a horizontal vector, by concatenating the rows of elements in it. This process is repeated for all the images in the database. Let the number of elements in each such horizontal vector be
Principal component analysis (PCA) [
The second method of feature is local binary pattern (LBP). The LBP is a type of feature used for texture classification in computer vision. LBP was first described in 1994 [
Local binary pattern.
Artificial neural networks (ANNs) [
Recognition techniques based on matching represent each class by a prototype pattern vector. It places an unknown pattern in the class to which it is closest in terms of a predefined metric. The simplest approach is the minimum distance classifier [
Experiments have been performed on our own captured thermal face images at our laboratory and Terravic Facial Infrared database. In our Database, there are
In the first set of experiments, Haar wavelet is used to decompose the cropped face image once which produces 4 subimages as LL, HL, LH, and HH bands. Then the average of HL/LH/HH band subimages is computed using (
Recognition performance (own database) with varying numbers of eigenvectors and the values of
|
Recognition rate (%) | ||||
---|---|---|---|---|---|
10 eigenvectors | 20 eigenvectors |
30 eigenvectors | 40 eigenvectors | 50 eigenvectors | |
|
88.23 | 91.18 | 91.18 | 88.23 | 83.33 |
|
86.27 | 83.33 | 91.18 | 91.18 | 83.33 |
|
81.38 | 88.23 | 88.23 |
|
74.50 |
|
81.38 | 78.57 | 76.74 | 86.27 | 81.38 |
|
88.23 | 91.18 | 86.27 | 88.23 | 78.57 |
|
94.11 | 78.57 | 91.18 | 83.33 | 91.18 |
|
86.27 | 88.23 | 78.57 | 86.27 | 86.27 |
|
83.33 | 83.33 | 83.33 | 86.27 | 88.23 |
|
81.38 | 83.33 | 76.74 | 88.23 | 83.33 |
|
83.33 | 88.23 | 83.33 | 81.38 | 86.27 |
|
86.27 | 78.57 | 76.74 | 76.74 | 81.38 |
Recognition performance (benchmark database) with varying numbers of eigenvectors and the values of
|
Recognition rate (%) | ||||
---|---|---|---|---|---|
10 eigenvectors | 20 eigenvectors |
30 eigenvectors | 40 eigenvectors | 50 eigenvectors | |
|
83.33 | 86.27 | 80.39 | 86.27 | 75.49 |
|
86.27 | 86.27 | 75.49 | 89.22 | 86.27 |
|
89.22 | 83.33 | 89.22 | 83.33 | 80.39 |
|
89.22 | 83.33 | 86.27 | 80.39 | 83.33 |
|
86.27 | 80.39 | 86.27 | 89.22 | 89.22 |
|
94.11 | 83.33 | 92.15 | 89.22 | 89.22 |
|
80.39 | 78.57 | 86.27 | 92.15 | 88.22 |
|
80.39 | 86.27 | 89.22 | 86.27 | 83.33 |
|
89.22 | 83.33 | 86.27 | 80.39 | 78.57 |
|
89.22 | 78.57 | 80.39 | 89.22 | 83.33 |
|
80.39 | 80.39 | 83.33 | 75.49 | 83.33 |
Comparative study of recognition performance (own database) with varying numbers of eigenvectors and values of
Comparative study of recognition rate (performed on Terrivic Facial Thermal Database) with varying numbers of eigenvectors and values of
In the second set of experiments, the feature set was kept the same as those in the first set of experiments, but the classifier is chosen as minimum distance classifier. The recognition performance obtained on both the thermal face databases considered here is detailed in Table
Recognition performance (on own database and benchmark database) with minimum distance classifier and the value of
|
Recognition rate (%) | |
---|---|---|
Own database | Terravic Facial | |
Thermal Database | ||
|
89.22 | 94.11 |
|
86.27 | 86.27 |
|
86.27 | 83.33 |
|
83.33 | 86.27 |
|
86.27 | 83.33 |
|
80.39 | 80.39 |
|
80.39 | 78.57 |
|
80.39 | 86.27 |
|
83.33 | 86.27 |
|
83.33 | 86.27 |
|
86.27 | 80.39 |
Comparative study of Recognition performance (own database and benchmark database) with minimum distance classifier and the values of
In the third set of experiments, cropped face images are divided in to 161 subimages each of size 8 × 8 pixels. Then local binary pattern is used to extract features from each of the subimages which are concatenated in row wise manner. After performing PCA on the LBP features for dimensionality reduction, ANN and minimum distance classifier are used separately for recognition of the face images on the basis of the extracted features. The obtained recognition results are shown in Table
Recognition performance (own database and benchmark database) with varying numbers of eigenvectors, ANN, and minimum distance classifier.
Recognition rate (%) | ||||||
---|---|---|---|---|---|---|
ANN classifier |
Minimum distance | |||||
10 eigenvectors | 20 eigenvectors | 30 eigenvectors | 40 eigenvectors | 50 eigenvectors | ||
Own database | 86.27 | 83.33 | 86.27 | 86.27 | 83.33 | 89.22 |
Terravic Facial |
86.27 | 89.22 | 83.33 | 92.15 | 89.22 | 94.11 |
Recognition performance (own database and benchmark database) with varying numbers of eigenvectors, ANN and minimum distance classifier.
In this paper, a comparative study of thermal face recognition methods is discussed and implemented. In this study two local-matching techniques, one based on Haar wavelet and the other based on Local Binary Pattern, are analyzed. Firstly, human thermal face images are preprocessed and cropped the face region only, from the entire face images. Then above-mentioned two feature extraction methods are used to extract features from the cropped images. Then, PCA is performed on the individual feature set for dimensionality reducation. Finally, two different classifiers are used to classify face images. One such classifier is multilayer feed forward neural network, and another is minimum distance classifier. The experiments have been performed on the database created at our own laboratory and Terravic Facial IR Database. The proposed system gave higher recognition performance in the experiments, and the recognition rate was 95.09% for
The authors are thankful to a major project entitled “Design and Development of Facial Thermogram Technology for Biometric Security System,” funded by University Grants Commission (UGC), India, and “DST-PURSE Programme” at Department of Computer Science and Engineering, Jadavpur University, India, for providing necessary infrastructure to conduct experiments relating to this work.