Image similarity and image recognition are modern and rapidly growing technologies because of their wide use in the field of digital image processing. It is possible to recognize the face image of a specific person by finding the similarity between the images of the same person face and this is what we will address in detail in this paper. In this paper, we designed two new measures for image similarity and image recognition simultaneously. The proposed measures are based mainly on a combination of information theory and joint histogram. Information theory has a high capability to predict the relationship between image intensity values. The joint histogram is based mainly on selecting a set of local pixel features to construct a multidimensional histogram. The proposed approach incorporates the concepts of entropy and a modified 1D version of the 2D joint histogram of the two images under test. Two entropy measures were considered, Shannon and Renyi, giving a rise to two joint histogram-based, information-theoretic similarity measures: SHS and RSM. The proposed methods have been tested against powerful Zernike-moments approach with Euclidean and Minkowski distance metrics for image recognition and well-known statistical approaches for image similarity such as structural similarity index measure (SSIM), feature similarity index measure (FSIM) and feature-based structural measure (FSM). A comparison with a recent information-theoretic measure (ISSIM) has also been considered. A measure of recognition confidence is introduced in this work based on similarity distance between the best match and the second-best match in the face database during the face recognition process. Simulation results using AT&T and FEI face databases show that the proposed approaches outperform existing image recognition methods in terms of recognition confidence. TID2008 and IVC image databases show that SHS and RSM outperform existing similarity methods in terms of similarity confidence.
Facial recognition technology will change society in many ways. Face recognition system is available nowadays. Face recognition has been used in many commercial and law enforcement applications [
People can recognize each other by the spectacular diversity of facial features and this is essential to the formation of complex societies. The face has the capability to send emotional signals, either voluntarily or involuntarily. The current biometric system technology reads faces as efficiently as humans do. Holy places use face recognition to track the presence of worshipers; retailers also use facial recognition technology to monitor thieves or to arrest suspects. Face recognition technology helps to verify the ID of the ride-hailing driver, verify the permits of tourists to enter tourist places, and let people buy things with a smile [
Face recognition is falling within the similarity of images where it is possible to recognize the face by finding the similarity and dissimilarity between the image stored in the database and the current image of the same person [
The measurement of image similarity is a significant point in the applications of the real world and several fields like optical character recognition (OCR), identity authentication, human-computer interfacing, surveillance, and other pattern recognition tasks [
To measure the similarity between two digital images, there is a simple method to calculate the similarity which is mean squared error. The advantage of mean squared error is easy to calculate, but, at the same time, it is not accurate for pattern recognition.
It is possible to use information-theory approach in image processing for analysis if we consider the image is a two-dimensional random variable, giving rise to the use of information-theoretic measures (such as joint histogram) to define similarity and recognition measures between images [
This paper handles the information-theoretic approach and includes the following sections: Section
There are several works that addressed the face recognition approach and images similarity measure by employing the information theory and entropy concepts. All previous work has solved a high level of challenges of the face recognition and image similarity to support this system to work in real time. The authors developed SSIM and explained the performance of SSIM by using some examples [
Zhang et al. (2011) presented a feature similarity index measure (FSIM) for image-quality assessment. Phase congruency was used as a primary feature in the feature similarity index measure, whereas the gradient magnitude was used as a secondary feature in the feature similarity index measure to compute the feature similarity index. Experiment results were on the six-benchmark image-quality assessment database. Later we will demonstrate that entropy metrics performance is much higher than FSIM [
The feature extraction using the discrete cosine transform (DCT) with the approach of illumination normalization in the domain of the logarithm is proposed by Arindam Kar et al. in 2013 [
In 2013 Darshana Mistry et al. used the concepts of entropy measure, joint entropy, and joint histogram to find the similarity between two digital images and test these measures on the brain images as a database [
In 2014, Lee et al. suggested a method for face recognition using Shannon entropy and fuzzy logic [
In 2015, Yulong Wang et al. introduced a MEEAR (Minimum Error Entropy-based Atomic Representation) framework for facial recognition system. MEEAR is based on the minimum error entropy (MEE) model to be more robust under noise condition [
Images similarity index based on entropy function and group theory is proposed by Y. G. Suarez et al. in 2015 [
In 2016 Q. R. Zhang et al. proposed the Improved Relative Entropy (IRE) method for face recognition approach. The IRE method is based on Shannon entropy and it is more accurate than Linear Discriminant Analysis and Locality Preserving Projections methods. The experimental results of IRE using CMU PIE and YALE B databases showed the high performance of the IRE versus LDA and LPP [
The system of emotion recognition based on facial expression is proposed by Y. D. Zhang et al. in 2016 [
To improve the kernel entropy component analysis (KECA), X. Ruan et al. in 2017 [
FRIQA (Full-Reference Image-Quality Assessment) is an algorithm proposed by Y. Ren et al. in 2017 [
In a recent development, the authors in [
The distance between two sets of various data points based on a given norm is called a “similarity measure.” If we have a dataset and a function that gives a large distance between this set and members of a database, except probably one member, then we have a similarity algorithm that can detect similarity between given data and members of a database. In this paper, two information-theory measures are designed based on entropies combined with a joint histogram of two images. Performance comparison is considered with well-known similarity and recognition measures. All the methods of recognition of the face image depend on the extraction of certain features of the images; the similarity shows the features of the statistical correlation or informatics correlation. To find the similarity between two images, several approaches are utilized; some are used for face and facial expression recognition. Here we present a brief description of well-known similarity and recognition measures for the sake of performance comparison, which is overviewed as follows.
The structural similarity index measure (SSIM) is one of the most popular metrics used to find the similarity between two images. Zhou Wang et al. proposed this measure in 2004 [
In 2011, Zhang et al. [
In 2017 NA Shnain et al. [
Zernike moments provide an efficient, rotation-invariant, and noise-resistant approach for image and face recognition, including the complicated effect of face expressions [
In this work we will use
In [
Researchers have proposed several similarity and recognition metrics used in image-processing field; each has its weaknesses and strengths. The most disturbing problem in image similarity for face recognition is the confusing high similarity given by a specific measure between the reference image and other images in the database.
In this paper, we propose novel information-theoretic similarity-recognition measures for image similarity and face recognition. The proposed measures reduce confusion when used in face recognition by giving a very small similarity between unrelated images. Information theory has already been applied to pattern recognition [
Entropy is the expected value of the information. Entropy has several applications in statistical mechanics, coding theory, statistics, and related areas. Emerging fields have also used entropy, such as image similarity [
Renyi entropy is another significant measure of information, given by
The main difference between Shannon entropy and Renyi entropy is the placement of the logarithm in the entropic equations, giving a flexible measure of the entropy as a result of the parameter
In a huge database for digital images like a face database, there might be identical histograms for very different images. This fact will be a problem when researchers want to compare images using a histogram as a distinctive feature. To solve this problem, Pass et al. [
A 2D joint histogram entry
Now we apply the entropy to measure the information held in the joint histogram that represents the joint probability of pixel cooccurrence. Note that both
One of the most difficult challenges for researchers in measuring image similarity for face recognition is that there is a high level of scepticism about the similarity between the reference image and test image in the same database, particularly when the image has low resolution or distortion in terms of illumination or background changes.
The differences in facial expressions and head poses for human faces often give rise to scepticism. Official government security systems do not rely entirely on face recognition systems, because the latter still suffer from challenges such as different facial expressions, illumination, and changing shape with age. However, a face recognition system can be very supportive of current routine security systems.
In this work, we have contributed to reducing these challenges regarding similarity of images, especially for the purpose of face recognition. We proposed new image similarity measures that can be utilized in face recognition. These measures are built using an information-theory approach; they proved to be very accurate in finding similarity between face images with more confidence than existing images similarity and image recognition measures. Our method is motivated by the problem of finding image similarity in large databases, where reduced confidence may open the door for big confusion.
The aim of this work is to provide metrics to find similarity between images for the purpose of face recognition; also, this can be used in case of nonface images. High performance and accuracy are the main features of proposed measures as compared to existing measures. Although other measures may have the ability to find the similarity between images (even for face recognition), the proposed measures have high confidence by giving almost a near-zero value in case of different images, while other measures give a nontrivial amount of similarity when comparing different images.
We have implemented the proposed measures on MATLAB and tested their performance against other measures as follows.
In this work, we used well-known face databases, AT&T and FEI [
Various face poses for a single person from the AT&T face database.
Various face poses for a single person from the FEI face database.
We divided the AT&T and FEI databases into two subgroups: testing group and the training group. In training group, we choose a random face image from the database to be a reference image, and then we select a different facial expression and pose from the testing group, for the same person as a challenging image to test the performance of measures in recognition and similarity.
On the other hand, there are several publicly available image databases in the image similarity community, including TID2008 and image and video-communication (IVC). Both are used here for algorithm validation and comparison. TID2008, as shown in Figure
Eight TID2008 reference images used for the test and comparison of image similarity measures.
Ten IVC reference images used for the test and comparison of image similarity measures.
For each reference image in the TID2008 and IVC databases, we use six complex distorted versions as image poses to test, compare, and prove that the proposed SHS & RSM outperforms the existing measures in terms of a recognition and similarity tests.
Note that although we obtained good results using this standard database, better results could be obtained using the Viola-Jones face detection algorithm [
Performance of the proposed measures has been tested against other efficient similarity and recognition metrics: SSIM, FSIM, FSM, ZESIM, and ZMSIM. The criterion for good performance is the amount of confusion in deciding whether an image belongs to a database or not. This confusion is measured by the difference in similarity produced (by a specific measure) between the reference image and the database images, with a focus on the best match and the second-best match. If a measure gives little difference in similarity between the same persons, then the confusion is high and the performance is low.
To evaluate the performance of the proposed SHS and RSM against SSIM, FSIM, ZSIM, ZMSIM, and the state-of-the-art FSM, we have to describe the experimental procedure in detail.
In this paper, we use four challenging datasets which are AT&T, FEI, IVC, and TID2008. AT&T and FEI are used to test the performance of all the measures in terms of face recognition, and TID2008 and IVC are used to test the performance of all the measures in terms of image similarity in the figures listed below.
Figure
Poses for person no. 17 in AT&T database. Reference pose is number 10 as indicated.
Face recognition using proposed SHS and RSM measures versus well-known measures. Reference pose is indicated in Figure
Figure
Poses for person no. 17 in AT&T database. Reference pose is number 3 as indicated.
Face recognition using proposed SHS and RSM measures versus well-known measures. Reference pose is indicated in Figure
Note that the measures have also been tested using different facial expressions with different illumination and different head pose in the same databases. Such cases represent the current challenges of any face recognition or image similarity measure. Results show more confidence in our proposed measurement.
In Figure
Poses for person no. 20 in AT&T database. Reference pose is number 10 as indicated.
Face recognition using SHS & RSM measures versus existing well-known measures. Reference pose is indicated in Figure
FEI database is used which represents the most face recognition challenges because it is containing a different facial expression with different illumination (white homogenous background) and different head pose (about 180 degrees).
Figure
Poses for person no. 17 in FEI database. Reference pose is number 08 as indicated.
Face recognition using SHS & RSM measures versus existing well-known measures. Reference pose is indicated in Figure
Figure
Poses for person no. 28 in FEI database. Reference pose is number 06 as indicated.
Face recognition using SHS & RSM measures versus existing well-known measures. Reference pose is indicated in Figure
As face recognition can be pose-dependent, we did averaging of similarity confidence measure for every pose in the AT&T dataset. The global average can be obtained as the mean of all these subaverages. Let
The global average similarity difference of best match and second-best match within all persons.
Measures | RSM | SHS | FSM | SSIM | ZESIM | ZMSIM | FSIM |
---|---|---|---|---|---|---|---|
Average Confidence, | 0.0239 | 0.0234 | 0.0192 | 0.0150 | 0.0119 | 0.0100 | 0.0066 |
The preparation of the database that is more suitable for this approach (e.g., in security applications) should take into consideration some important factors like lighting, expression, and viewpoint, while the reference image should consider the same factors.
It is clear that the proposed joint entropic-histogram measures give more confident decisions in face recognition and image similarity, whereas other measures, although they decide the proper person correctly, give low confidence in their decision.
Using a database with distorted images in the test of image similarity and image recognition measures is a real challenge to the proposed and existing measures. In this work, we tested the SHS & RSM on distorted images for the sake of image similarity and image recognition. The figures listed below show that the proposed methods are still superior versus others.
Figures
Performance of recognition measures using original image and distortion of the original image. (a) The reference image. (b) The distorted version of it. (c) Performance of SSIM, FSIM, FSM, ZESIM, ZMSIM, and SHS and RSM using TID2008 database.
Performance of recognition measures using original image and distort of the original image. (a) The reference image. (b) The distorted version of it. (c) Performance of SSIM, FSIM, FSM, ZESIM, ZMSIM, and SHS and RSM using TID2008 database.
Performance of recognition measures using original image and distort of the original image. (a) The reference image. (b) The distorted version of it. (c) Performance of SSIM, FSIM, FSM, ZESIM, ZMSIM, and SHS and RSM using IVC database.
Performance of recognition measures using original image and distortion of the original image. (a) The reference image. (b) The distorted version of it. (c) Performance of SSIM, FSIM, FSM, ZESIM, ZMSIM, and SHS and RSM using IVC database.
The difference in the values of the peaks of each measure is a new feature showing the high performance of the proposed measures (SHS & RSM). If the distance between the highest match and the second-best match is higher, that means the measure has better performance and vice versa; i.e., if the distance is less, that means the measure has been confused in deciding the best match by giving a nontrivial similarity between the different images. The new feature of recognition confidence can be very useful in security systems of big databases.
An ROC graph essentially shows the relationship between advantages (true positives) and disadvantages of the classifier (false positives). Tables
True positive rate (tpr) according to the threshold vector of confidence using AT&T database, with pose 10 (of each of the forty persons) as a reference image (included in the database).
Thr | SSIM | FSIM | ZESIM | ZMSIM | ISSIM | FSM | SHS | RSM |
---|---|---|---|---|---|---|---|---|
0.1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
0.3 | 1 | 0.275 | 1 | 0.875 | 0.975 | 1 | 1 | 1 |
0.5 | 1 | 0 | 0.275 | 0.075 | 0.05 | 1 | 1 | 1 |
0.7 | 0.15 | 0 | 0 | 0 | 0 | 1 | 1 | 1 |
0.8 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 |
0.9 | 0 | 0 | 0 | 0 | 0 | 0.075 | 1 | 1 |
0.95 | 0 | 0 | 0 | 0 | 0 | 0 | 0.275 | 0.85 |
False positive rate (fpr) according to the threshold vector of confidence using AT&T database, with pose 10 (of each of the forty persons) as a reference image (excluded from the database).
Thr | SSIM | FSIM | ZESIM | ZMSIM | ISSIM | FSM | SHS | RSM |
---|---|---|---|---|---|---|---|---|
0.1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0.3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0.5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0.7 | 0 | 0 | 0 | 0 | 0 | 0.05 | 0 | 0 |
0.8 | 0 | 0 | 0 | 0 | 0 | 0.15 | 0 | 0 |
0.9 | 0.3 | 0 | 0.875 | 0 | 0 | 0.3 | 0 | 0 |
0.95 | 0.5 | 0.025 | 1 | 1 | 0.1 | 0.6 | 1 | 1 |
ROC graphs for 8 similarity measures with 6 different confidence thresholds (bold for proposed measures). Graphs of the proposed measures are always above y=x curve (top-left corner, meaning more benefits than disadvantages).
This paper presented an efficient approach for face recognition and image similarity. The approach is based on an information-theoretic similarity measure derived using the entropy of a 1D version of the 2D joint histogram between two images. Two entropies have been used, Shannon and Renyi, giving rise to two measures: Shannon-Histogram Similarity (SHS) and Renyi Similarity Measure (RSM). The performance of RSM and SHS was tested against efficient existing similarity metrics feature-based similarity (FSIM), structural similarity (SSIM), and also Zernike-moments recognition approaches, specifically Zernike-Euclidean Similarity (ZESIM) and Zernike-Minkowski Similarity (ZMSIM) and the state-of-the-art FSM. A comparison with a recent information-theoretic ISSIM has also been considered. Experimental results showed superior performance for the proposed measures in terms of correct decisions with minimal confusion in face recognition and image similarity, using the AT&T and FEI face databases and TID2008, and IVC image databases. Confusion in recognition is introduced as a performance factor, measured as the difference between the similarity produced by the best match and that produced by the second-best match.
In this work, global face analysis has been applied, where the whole image is treated at once. Although good results were obtained using a standard database, difficulties may arise in practice. The Viola-Jones face detection algorithm and local analysis of face images played a significant role in improving face recognition. The authors intend to pursue this point in future works and extend their previous studies on local analysis to improve the performance of the measures defined above.
The data used to support the findings of this study are available from the corresponding author upon request.
The Authors declare that there are no ethical issues regarding this work.
The authors declare no conflicts of interest regarding this work.
All authors extensively discussed the contents of this paper and contributed to its preparation. Mohammed Abdulameer Aljanabi and Zahir M. Hussain have proposed and developed the model, performed experiments, and drafted the manuscript. Results analysis, mathematics check-up, and simulation revision of this manuscript were done by Songfeng Lu. All of the authors have contributed to the literature overview and modelling discussions.
This work is supported by the Natural Science Foundation of Hubei Province of China under Grant no. 2016CFB541 and the Applied Basic Research Program of Wuhan Science and Technology Bureau of China under Grant no. 2016010101010003 and the Science and Technology Program of Shenzhen of China under Grant nos. JCYJ20170307160458368 and JCYJCYJ20170818160208570.