A Digital Camera-Based Rotation-Invariant Fingerprint Verification Method

Fingerprint registration and verification is an active area of research in the field of image processing. Usually, fingerprints are obtained from sensors; however, there is recent interest in using images of fingers obtained from digital cameras instead of scanners. An unaddressed issue in the processing of fingerprints extracted from digital images is the angle of the finger during image capture. To match a fingerprint with 100% accuracy, the angles of the matching features should be similar. ,is paper proposes a rotation and scale-invariant decision-making method for the intelligent registration and recognition of fingerprints. A digital image of a finger is taken as the input and compared with a reference image for derotation. Derotation is performed by applying binary segmentation on both images, followed by the application of speeded up robust feature (SURF) extraction and then feature matching. Potential inliers are extracted from matched features by applying the M-estimator. Matched inlier points are used to form a homography matrix, the difference in the rotation angles of the finger in both the input and reference images is calculated, and finally, derotation is performed. Input fingerprint features are extracted and compared or stored based on the decision support system required for the situation.


Introduction
Image processing has many applications [1][2][3], including biometric registration and verification. Biometric registration and verification are performed to restrict unauthorized people from entering a specific building or wing, to keep track of users of service by registering their fingerprints in a database, to take automatic attendance of employees, and to store data along with biometric features in a database for issuing passports or national identity cards.
Biometric verification and registration using image processing techniques such as face detection, fingerprint detection, and retina or iris detection have been widely researched [4][5][6][7][8]. Face detection-based techniques [9,10] store facial features to be compared during the registration and verification process. Face verification and registration provides accuracy at the cost of high complexity and extensive allocation of system memory. Accuracy demands that issues such as scale difference, posture, illumination, and the opening and closing of the eyes and mouth be addressed, further increasing complexity.
Iris-based recognition and verification [7,11,12], on the other hand, is done by applying segmentation in the region of the iris to extract its radius. is is followed by feature extraction. ese techniques are used along with fingerprint verification at airports and other high-risk buildings to provide accurate identification. Iris-based recognition and verification provide good accuracy, but the required equipment is costly.
By using different approaches, such as combining fingerprint registration with odor analysis [21] and applying a ridgelet transform [22], researchers have improved existing algorithms. Complexity can be reduced by comparing a small number of features instead of entire images. However, these methods require the use of sensors (fingerprint readers) that are not always available. is can make performing the registration and verification process difficult.
Solutions that avoid the need of sensors have been proposed [24][25][26][27]. ese techniques focus on processing images captured using ordinary digital cameras. Hiew et al. [24][25][26] applied skin color-based segmentation followed by morphological processing to extract regions of interest. Short-time Fourier transform analysis (STFT) is then applied, followed by ridge orientation calculation and core point detection. e detection and matching of features such as bifurcations and ridges are not discussed. However, detection and matching can be performed using the same fingerprint techniques used in the algorithms applied to input obtained from sensors.
Sankaran et al. [27] proposed a method similar to that discussed in this paper that performs skin color segmentation followed by the region of interest (ROI) extraction. Image enhancement is applied followed by ScatNet-based feature extraction. Finally, feature-based fingerprint registration/matching is performed.
When features are extracted from a digital image of a fingerprint, scanners are not needed. However, orientation alignment is an issue that is inadequately addressed in the algorithms proposed by Hiew et al. and Sankaran et al. e fingerprint in the input image must be vertically aligned when conventional methods are used. When rotation of the finger is present, a fingerprint mismatch will occur even if features of that person are present in the database. erefore, this paper proposes a rotation and scale-invariant technique that takes an image captured using an ordinary digital camera as the input and applies skin colorbased segmentation to the input image and a reference image to extract regions of interest from both images. SURF [28] algorithm is applied to the segmented binary images, and the detected features from both images are matched. Inliers are extracted from the matched features using the M-estimator [29] for accurate calculation of the rotation angle using a homography matrix [30]. e input image is derotated to obtain an image with a vertically aligned finger. e vertically aligned input image is then subjected to image enhancement techniques, such as unsharp masking and contextual blockwise Fourier domain filtering. Otsu thresholding is applied for binarization. en, nonmaximal suppression is used to obtain ridges having a thickness of one pixel. Features such as bifurcations and ridge endings are extracted from the thin ridges to be stored in a database for registration or compared with the features of registered fingerprints for verification. Simulation results show that the proposed method provides an accuracy of 100 percent if the suggested parameter range is applied. is paper is organized as follows: Section 2 discusses the proposed methodology, Section 3 contains the results and discussion, Section 4 presents the conclusions and the direction of future work.

Methodology
e objective of this work is to propose a cost-effective rotation and scale-invariant alternative to the processing of fingerprint images obtained using scanners. A flowchart of the algorithm is shown in Figure 1. e major steps of the proposed algorithm are the derotation of the input image, image enhancement, binarization, thinning, minutiae extraction, and matching. e steps shown in the flowchart are explained in detail in this section.

Image Acquisition.
In this study, image acquisition was performed using several smartphones: Samsung Galaxy S2, iPhone 3s, Galaxy S6, and iPhone 8. Images of fingers were captured with the focus on their ridges.

Image Derotation.
A flowchart depicting the derotation process is shown in Figure 2. An image containing a finger aligned along the y-axis is selected as a reference image, as seen in Figure 3(a). Any input finger image requiring registration or verification is considered an input image and is subjected to the derotation process so that the fingerprint registration/matching process can be done smoothly. Figure 3(b) shows an example of an input image that requires rotational alignment for accurate feature matching.

Skin Color-Based Segmentation.
Both the input and reference images are subjected to adaptive skin color-based image segmentation [31]. e images shown in Figures 3(c) and 3(d) show the results of this segmentation on the reference and input images. e input image shown in Figure 3(b) is particularly complex because the color of the background is similar to the color of the skin. erefore, noise is detected as part of the finger. e white region in Figure 3(d) is the region of interest (ROI) and will be used in upcoming sections.

Input Image Scale Adjustment.
Both input and reference images usually are of different dimensions. erefore, the fingers in both images might have a huge scale difference. In that case, scale variance might produce problems while detecting and comparing features. In order to provide a scale-invariant derotation, scales of both input and reference fingers should nearly be the same. If the number of pixels belongs to the finger in the segmented reference image, it is denoted by N ref and if the number of pixels belongs to ROI, it is denoted by N input . Scaling factor is calculated using (1) as

Feature Extraction and Matching.
Intensity images are converted into binary images for time-efficient feature matching. Since finger shapes are all similar, features detected along their boundaries can provide better estimations of geometrical differences. For feature detection, the SURF [28] method is applied to the segmented binary images to obtain boundary features for both images. SURF is selected for its robustness and reasonable match rate for two same images with different angle of rotation [29]. After obtaining features, nearest neighbor-based feature matching by computing the pairwise distance between features is performed. en, feature matching provides matched features that contain truly matched inliers as well as erroneously matched outliers. Figure 3(e) shows all the matched features including both inliers and outliers.

Inlier Extraction and Derotation of Image.
To extract inliers from the matched points, an M-estimator [29] algorithm is applied. Figure 3(f ) shows the inliers extracted from all matched features using the M-estimator. It can be seen that extracted inliers represent true matches that can be used to estimate the angle of rotation between accurately matched inlier points.
Once several potential inliers are extracted, the angle of rotation between the matched inliers must be determined. A homography matrix [30] is calculated for this purpose. For the two sets of matched points, D � (x i , y i ) for the reference image and D ′ � (x i ′ , y i ′ ) for the input image. en, the relation between them is as given in the following equation: where D � e resultant matrix in (3) is of the form given in (4) as erefore, the angle of rotation can be calculated using h 1 2 or h 2 2 using (5) as where θ can be used to derotate the input image to obtain an image with a vertically aligned finger. Figure 3(g) shows the derotation of the input image after the calculation of θ � −176.29°. Figure 4 shows the alignment method applied to various input images showing fingers at different angles of rotation. It can be seen that the proposed algorithm predicted the angles of rotation accurately and that the fingers in the derotated images are aligned vertically irrespective of their initial alignment.

Image Enhancement.
Edge-based image enhancement is applied to the grayscale image to improve the visibility of ridges on the finger. Edge-based enhancement is performed in two stages. First, unsharp masking [32] is applied to the image. is algorithm subtracts an image from a blurred version of itself and enhances those regions which provide a good response after subtraction. To calculate the blurred image, a Gaussian filter is convolved with the original grayscale image using the following equation: where For the second stage of image enhancement, a contextual blockwise Fourier domain filter [33] is applied to the grayscale image by dividing it into a 16 × 16 overlapping block. ese blocks are then filtered in the Fourier domain using a frequency and orientation-selective filter with parameters based on the estimated local ridge orientation and frequency. e outcome of the Fourier domain-based enhancement is another image denoted as I enhanced . I enhanced is multiplied with a skin-segmented binary image so that only the region of interest (ROI) is processed further. Figure 5 shows the effects of this method on the derotated input image. e image in Figure 5(a) shows ridges that are not visible. e results of the unsharp masking can be seen in Figure 5(b) showing improved visibility of the ridges. Figure 5(c) shows the image once it has been converted to grayscale. Figure 5(d) shows that application of Fourierbased image enhancement further sharpens unclear edges and other details. Figure 5(e) shows the extracted ROI.

Image Binarization and Edge inning.
Features such as ridge endings and bifurcations can be accurately extracted when ridges having a thickness of one pixel are detected. To detect thin ridges, all ridges are first detected by applying Otsu thresholding [34] to the enhanced region of interest.
is will result in a binary image with ridge pixels represented in white and the remaining pixels in black. However, this will result in the detection of thick ridges. To limit the results to thin ridges, nonmaximal suppression [35] is applied. is method sets all the ridge pixels in a certain direction to zero if they do not have a maximum gradient value and retains ridge pixels with the maximal value. is results in conversion of the thick ridges to ridges with a thickness of only one pixel. Figure 6(a) shows Otsu thresholding being applied to the enhanced region of interest. e detected ridges are thick and thus cannot be used for feature detection. Application of nonmaximal suppression results in more desirable thin ridges, as seen in Figure 6(b).

Minutiae Extraction.
Minutiae are features that are used for biometric verification. Instead of storing fingerprint images in the database, minutiae are stored and compared. e most common method used for minutiae extraction is the crossing number (CN) [36].
For extracting minutiae, the 3 × 3 window shown in Figure 7 is moved across each pixel to calculate its crossing number using the following equation: where p i denotes the pixels around the center pixel at location (x, y). Using CN, the ridge pixel can then be classified as a ridge ending, bifurcation, or isolated point using Table 1. Figure 6(c) shows bifurcations and ridge endings represented by blue and red squares, respectively. For each bifurcation and ridge ending, the Euclidean distance D x,y between its spatial coordinate (x, y) and (x min ROI , y min ROI ), the normalized gradient value G x,y , and the respectively. e values donated by x min ROI and y min ROI are x-axis and y-axis points taken from two ROI points corresponding to the minimum x-axis and y-axis values. Figure 8 shows an example of calculation of (x min ROI , y min ROI ) that is represented by a green circle. is was determined using the minimum xaxis ROI point represented by a blue cross and the minimum y-axis ROI point represented by a red cross.

Fingerprint Registration and
Matching. Extracted minutiae are either stored or compared with minutiae already existing in the database as required by the use case. Suppose that the ridge endings extracted from the input image are represented by the set S in BF and that bifurcations are represented by S in RE as follows: where S in RE and S in BF are sorted in an ascending order according to the calculated Euclidean distance such that D in RE(k) ≤ D in RE(k+1) and D in BF(l) ≤ D in BF(l+1) for all k ∈ 1, 2, ..., N in RE and l ∈ 1, 2, ..., N in BF . ese sets must be compared with the stored features of a number of fingerprints N DB stored in the database, as follows: where S DBi RE and S DBi BF are also sorted in an ascending order according to the Euclidean distances D DBi REm and D DBi BFn where m ∈ 1, 2, ...., N DBi RE and n ∈ 1, 2, ...., N DBi BF . Suppose that S in RE and S in BF are to be compared with the stored features of the ith fingerprint having feature sets S DBi RE and S DBi BF . First, N RE min , N BF min , N RE max , and N BF max are to be determined using the following equations: where N in RE and N DBi RE are the number of ridge endings extracted from the input image and the ith image features stored in the database and N in BF and N DBi BF are the number of extracted bifurcations. e input fingerprint features are compared with the ith fingerprint features if the condition in (19) is justified: For matching, all 3D bifurcation and ridge ending points of the input fingerprint and the ith fingerprint from the database and input fingerprint are stored in M ref RE

Results and Discussion
MATLAB 2018a was used in this study. A value of σ � 2.5 was used, and a Gaussian window of size (2σ + 1) × (2σ + 1), T 1 � .06, and T 2 � 10°was used throughout the work.
ere was no dataset available for evaluation purposes, so features from the fingerprints of 40 people were stored in a database. Features obtained from digital images of the fingers of 100 people, including those of registered people, were compared with the features already stored in the database in order to evaluate the performance of the proposed algorithm. Every person provided eight images of their finger at approximately 90°, 135°, 180°, 225°, 270°, 315°, 0°, and 45°. e performance of the proposed method was evaluated by analyzing the performance of the derotation and matching methods. It was not possible to calculate the exact errors in the calculation of the rotation angles, so error calculation was done manually for ten random inputs for each set of rotation angles.
(23) Figure 9 shows a comparison of the percentages of true matches (TP) and mistaken mismatches (FN). It can be seen from Figure 9 that a value of 0.7 ≤ t match ≤ 0.8 successfully verified all 40 of the registered fingerprints irrespective of their angle of rotation. However, for values of t match > 0.8, mistaken mismatches occurred in the range from ∼(2% − 18%). True mismatches (TN) occurred in 100% of the cases, and mistaken matches (FP) occurred in 0% of all cases.

Conclusions and Future Work
is paper proposes a fingerprint detection and verification method that uses a digital image of a finger as an input instead of a sensor. e input fingerprint image can be of any resolution and depends on the quality of the digital camera. Often, the finger in the image is not vertically aligned. erefore, the input image is derotated by applying skin color-based binary segmentation, followed by downscaling or upscaling of the input image with a scale calculated using the segmented input image and a reference image. e angle of rotation of the input finger is then calculated by applying SURF-based feature extraction on both binary images. e features detected in both images are matched, and outliers are removed from the matched points using the M-estimator. e angle between the matched inliers is calculated using a homography matrix, and the input image is derotated by the calculated angle. After derotation, enhancement techniques including unsharp masking and Fourier-based blockwise enhancement are applied to increase the visibility of the ridges in the input image. Image enhancement is followed by the application of Otsu thresholding to segment the ridge regions, and then nonmaximal suppression is applied for edge thinning. A crossing number technique is used to extract ridge endings and bifurcations.
is is followed by the proposed Euclidean distance-based matching to detect a fingerprint match or mismatch based on a matching score. e proposed method is a novel method, and simulation results show that if the matching threshold is kept within a suggested range, accuracy of 100% can be achieved. Scar removal-based fingerprint detection using digital images and edge-preserving image interpolation techniques could extend the proposed method and is being considered for future work.

STFT:
Short-time Fourier transform analysis ROI: Region of interest SURF: Speeded up robust features M-estimator: Maximum-likelihood estimator CN: Crossing number TP: True positives TN: True negatives FP: False positives FN: False negatives.

Data Availability
e data used to support the findings of this study can be obtained by contacting the corresponding author through e-mail: dhlee77@hanyang.ac.kr.