Automatic authentication systems, using biometric technology, are becoming increasingly important with the increased need for person verification in our daily life. A few years back, fingerprint verification was done only in criminal investigations. Now fingerprints and face images are widely used in bank tellers, airports, and building entrances. Face images are easy to obtain, but successful recognition depends on proper orientation and illumination of the image, compared to the one taken at registration time. Facial features heavily change with illumination and orientation angle, leading to increased false rejection as well as false acceptance. Registering face images for all possible angles and illumination is impossible. In this work, we proposed a memory efficient way to register (store) multiple angle and changing illumination face image data, and a computationally efficient authentication technique, using multilayer perceptron (MLP). Though MLP is trained using a few registered images with different orientation, due to generalization property of MLP, interpolation of features for intermediate orientation angles was possible. The algorithm is further extended to include illumination robust authentication system. Results of extensive experiments verify the effectiveness of the proposed algorithm.
The need for personal identification has grown enormously in the last two decades. Previously, biometric identification using fingerprints or face images was restricted to criminal prosecution only. A few experts could serve the demand. With increased terrorist activities, stricter security requirements for entering buildings, and other related applications, need for automatic biometric machine-authentication systems is getting more and more important.
Recognizing people from face (face image) is the most natural and widely used method we human do always and effortlessly. Due to ease of collection without disturbing the subject, it is one of the most popular ways of automatic machine authentication. An excellent survey of face-recognition algorithms is available in [
In automatic face recognition, the first step is to identify the boundary of the face and separate it from the photographed image. Next, recognition algorithms extract feature vectors from the input (probe) image. These features are then compared with the set of such features stored in the database. The database (gallery image) contains same set of features already extracted and stored during registration phase for all persons required to be authenticated.
There are two classes of algorithms to extract features from the image—model based and appearance based. Model-based algorithms use explicit 2D or 3D models of the face. In model-based algorithms, geometrical features like relative positions of important facial components, for example, eyes, nose, mouth, and so forth, and their shapes are used as features. These features are robust to lighting conditions but weak for change in the orientation of the face. We use a subset of such features as “Angle-feature” in our previous work [
Though automated face recognition by computers for frontal face images taken under controlled lighting conditions is more or less successful, recognition in uncontrolled environment is an extremely complex and difficult task. Lots of researchers are trying to develop unconstrained face recognition system [
For most of the biometric applications, we need to authenticate a particular person in
Even though the person is same, the automatic authentication system may fail due to angle orientation, ambient lighting, age, make-up, glasses, expression of the face, and so forth which are different from the stored gallery image of the individual. It is said that about 75% of the authentication failure is due to the fact that angle of orientation of the probe face image is different from the stored image. It is impossible and very inefficient to store the images (i.e., image features) of an individual taken at all possible angles and at different illuminations in the gallery. But we need that information for correct recognition. In this work, we focus on angle-aware face recognition, and then the proposed algorithm is extended to include ambient light-aware face recognition. In the proposed angle- and illumination-aware face recognition, we store the available (training) information in a trained Artificial Neural Network. Retrieval of the features for any intermediate angle and illumination from the trained ANN is very efficient. The algorithm can be used in real time. We experimented with a benchmark database. Our system could achieve excellent results both for false-acceptance rate (FAR) as well as for false-rejection rate (FRR).
In the next section we briefly discuss related works on orientation and illumination robust face recognition. In Section
According to FERET and FRVT [ Single-view approach in which invariant features or 3D model based methods are used to produce a canonical frontal view from various poses. In [ Multiview face recognition is an extension of appearance-based frontal image recognition. Here, gallery images of every subject at many different poses are needed. Earlier works on pose invariant appearance based on multiview algorithms are reported in [ Class-based hybrid methods in which multiview training images are available during training but only one gallery image per person is available for recognition. The popular eigenface approach [
More recent methods to address pose and illumination are proposed in [
The simplest approach is to look for a feature which is invariant to variation of pose. But, till now such a feature is not found. Reference [
Our approach is to store multiple pose image features in a single trained MLP, so that both storage and searching for intermediate angles are efficient. We do not overload the database by adding features for the same face at different angles. We train an artificial neural network to store them all as a function of the orientation angle. Due to good generalization property of MLP, it can give feature values at intermediate angles and very efficiently too. Through experiments, we realized that geometrical features are fragile to angle variation. We used a subset of geometrical features to express the pose angle. The following important aspects were investigated while selecting the efficient angle features: low computational complexity to extract the angle feature, so that the algorithm can run real time the pose-angle feature contains enough information about the angle the feature values vary smoothly with angle variation, so that MLP can be trained easily and with little error.
The two main contributions regarding pose invariant face authentication, over our previous work [
Figure
Block diagram of registration and authentication phase.
If the number of cameras is
A person’s identification (ID) and the corresponding trained MLP (using her/his face image angle feature and image feature) are stored as a pair. Such ID-MLP pair forms gallery image “DATA BASE.” In the recognition phase, the individual’s face image (probe image) is presented with her/his ID. From gallery “DATA BASE” of MLPs, the particular trained MLP for the claimed ID is retrieved. Angle features from the image are extracted and used as input to that person’s MLP retrieved from the data-base. Image-feature from the probe image and that obtained as output from the MLP are compared. If the distance between two feature vectors are below some predefined threshold, the decision is accept, otherwise reject. The implicit assumption here in that the MLP would be able to deliver correct image feature for any intermediate face orientation due to its good interpolation (generalization) property.
In the following section, we will discuss how angle features are extracted from the face image. We will also show what angle features are finally selected for our system and why.
Angle feature should contain the information of the orientation angle of the face image. Geometrical features of a face image, which uses distances between important parts of the face and angle between connecting lines, are capable of expressing the orientation of the face image. We used cues from those approaches of feature extraction. In our previous work [
In the present work, we wrote algorithm to automatically identify the important points on the face. This facilitated working with a larger data set. Moreover, after filtering, we could always identify the eyebrows, eyes, nostrils, and mouth. All possible identifying points can be listed as two end points of left eyebrow, two end points of right eyebrow, two end points of left eye, two end points of right eye, nostril (sometimes two), and two end points of mouth. This is clear from the picture after binary conversion (the best result obtained with a threshold of 0.75), as shown in Figure
The important parts from the face image are separated as follows. At first minimum value filter is used. The minimum value filter emphasizes the part where the image is dark, because important parts on face are darker than that of surrounding skin. Through experiments, we ensured that this technique is effective to identify locations of eyebrows, eyes, nostrils, and mouth. After using the minimum value filter, binarization is performed to clearly identify important parts of the face. In addition to our targeted important parts of the face, hair also is filtered out. First the hair part is detected. Though it is an important element too to profile the face image, we do not use it. We delete the hair part and the background. We then identify eyebrows, eyes, nose, and mouth, with heuristic algorithm using knowledge of their relative positions. As we do not use eyebrows to create the angle-feature vector, eyebrows are also deleted after identification. Once both eyes, nose and mouth are located on the face image, we generate the angle features.
First we will give the details of the elements of angle-feature vector and then explain the rationality of choosing them. The angle feature vector is
Distance components of angle feature.
Description | Symbol |
---|---|
Distance between LE and RE |
|
Distance between LE and N |
|
Distance between RE and N |
|
Distance between LE and M |
|
Distance between RE and M |
|
Gradient components of angle feature.
Description | Symbol |
---|---|
Gradient of line joining LE and N |
|
Gradient of line joining LE and M |
|
Gradient of line joining RE and N |
|
Gradient of line joining RE and M |
|
All these features are easy to calculate and change more or less smoothly with angle variation. We did not include the distance between mouth and nose, the gradient of the line joining mouth and nose, and the line joining the two eyes. This is because these parameters do not change with angle change.
In order to ensure how our angle feature vector changes with change in the orientation of the face image, we plotted the Euclidean distance between angle vectors against the angle of orientation. It is shown in Figure
The plots were for all training samples. It shows the smooth changes, though nonlinear but monotonic. From this plot, we can ensure that our angle feature is suitably chosen, and an MLP could be trained in a small number of epochs. Of course, during registration period, this training will be done off-line, and a longer training time is permissible. At the time of authentication, the MLP will give out the face image feature, from the input angle feature, instantly. That will ensure real-time application.
In summary, compared to our previous work, we have improved our angle feature extraction technique not only by automating it but also by adding six more elements in the angle-feature vector to capture the angle of orientation information more faithfully. This also enables us to work with larger data set of face images.
The image feature captures the characteristic of the entire image, the spatial distribution of the pixel values. The most widely used method is eigenface, first proposed byTurk and pentlandin [
In our experiments 8 principal components, which carry 99% of the image information, were used. We further extended our experiments using independent components on image feature. As independent component feature of the image gave better results, in this paper we will only present those results.
Multilayer neural network, trained with error backpropagation, is used as a mapping function—to map an individual’s face orientation angle to his/her face image features for that particular angle. As angle feature vector consists of 12 elements, the MLP has 12 input nodes plus one bias node. We use a single hidden layer with 15 hidden nodes. Experiments were tried with different number of hidden nodes. The training is fast and quickly converges to very low MSE. Even with hidden nodes 10, it is possible to get low error after training, but we need more numbers of training epochs. The number of output nodes is eight, equal to the number of image features by using independent component analysis.
As already mentioned, we have separate MLP for every individual. For every registered individual, we have face images taken with orientation angle from −50 degrees to +50 degrees, at an interval of 5 degrees. In total, we have 21 image data for any individual. Out of the available 21 data, we use those taken at orientation −50, −40, −30, −20, −10, 0, +10, +20, +30, +40, and +50, that is, in total 11, for training the MLP. The rest 10 images, taken at angles −45, −35, −25, and so forth, were used for testing the trained MLP. Figure
In this work we also proposed an extension of our system to include correction for illumination variation. Two alternative systems are proposed shown in Figures
As already mentioned, the system consists of two stages—learning of MLP, that is, the registration phase, and using the learned MLP in the authentication stage.
When person “A” is to be registered, face photograph of person “A” is taken using multiple cameras set at different angles, as shown in Figure
Collection of several face images taken from equal distance but at different angles.
Images after different filtering steps.
Euclidean distance of (a) angle feature and (b) image feature.
The results of distances between self and non-self-face image features at different angles of orientation.
Block diagram of System I.
Block diagram of System II.
Registration phase which consists of taking the face image at different angles and use them to train an MLP.
In authentication phase, the person announces his/her identification and let the image be taken. The angle is arbitrary, depending on how the person poses in front of the camera. We assume this angle to be within −50 to +50 degrees. The mapping task of MLP is to interpolate. The layout of the authentication system is shown in Figure
Description of the authentication phase.
Here,
Compared to our previous work, in the present work the angle feature vector has changed, from 6 elements to 12 elements. The image feature vector is also changed from PCA to ICA, the number of elements remaining the same 8. As the number of input nodes is increased, we increased the hidden nodes to 16 for faster training. We used face image data, taken in same illumination condition, with orientation angle from −50 degrees to +50 degrees, taken at intervals of 5 degrees. Image data at intervals 10 degrees was used for training, and the intermediate is for testing. In total, face image of 15 individuals was used. Experiments were performed by varying the threshold in steps.
Experimental results, for the angle variation from −50 degrees to +50 degrees, are summarized in Figure
False rejection rate and false acceptance rate.
The image feature changes also with illumination condition. We did a preliminary experiment to investigate the image feature change with brightness and on the basis of our investigation proposed the robust systems for illumination variation presented in the earlier section.
To investigate the pattern of change, we varied the brightness of face image by steps of 4% (of the original brightness) to a level up to −80% of the original value. Here, maximum value of the brightness is considered to be 0%. The image features at different illumination levels are compared, in terms of Euclidean distance, with respect to the brightest image, that is, 0%. The results are summarized in Figure
Euclidean distance of image feature as brightness changes.
As shown, the Euclidean distances are larger with the decrease of brightness values. The nature of variation is easy to be learned by ANN. From this, we conclude that, we can extend the proposed system to be able to perform well in case of illumination variation too.
We compared our results for System I and System II. We use brightness of different image features at intervals of 4%, from −80% to 0%. The image features are the same. ICA features are used in Section
All the experimental results are summarized in Figure
Misidentification at different brightness levels.
Only angle | System I | System II | |
---|---|---|---|
0% | 0.090 | 0.182 | 0.127 |
–20% | 0.114 | 0.205 | 0.114 |
–40% | 0.287 | 0.199 | 0.172 |
Error rate due to changes in brightness.
In this work we have proposed an efficient technique for angle-aware face recognition and extended the same technique to take care of the effect of illumination variation. Though there are lots of works on angle invariant and illumination invariant face recognition proposed in the literature so far, there is a very few work in which same framework is used for taking care of both the problems simultaneously. Our proposed system can take care of angle variation from −50 degrees to + 50 degrees and at the same time 40 sets and different image feature set. The results are now reliable to work with larger data set. We used only one data set and currently are engaged in using other data sets for simulation experiments.
In this work, we considered the change in angle orientation in the horizontal plane, but orientation in the vertical plane may also vary and affect face recognition. We would like to extend our work to take care of the change in orientation in the vertical plane. Further experiments to work with more bench mark data sets are also our future target.