Multimodal Biometrics for Person Authentication

Abstract

Unimodal biometric systems have limited effectiveness in identifying people, mainly due to their susceptibility to changes in individual biometric features and presentation attacks. The identification of people using multimodal biometric systems attracts the attention of researchers due to their advantages, such as greater recognition efficiency and greater security compared to the unimodal biometric system. To break into the biometric multimodal system, the intruder would have to break into more than one unimodal biometric system. In multimodal biometric systems: The availability of many features means that the multimodal system becomes more reliable. A multimodal biometric system increases security and ensures confidentiality of user data. A multimodal biometric system realizes the merger of decisions taken under individual modalities. If one of the modalities is eliminated, the system can still ensure security, using the remaining. Multimodal systems provide information on the “liveness” of the sample being introduced. In a multimodal system, a fusion of feature vectors and/or decisions developed by each subsystem is carried out, and then the final decision on identification is made on the basis of the vector of features thus obtained. In this chapter, we consider a multimodal biometric system that uses three modalities: dorsal vein, palm print, and periocular.

1. Introduction

Biometrics is a technology that uses physical and/or behavioral characteristics of people to identify them. Systems of this type implement two processes (Figure 1) [1]:

1. Enrollment

2. Authentication

Figure 1.

Biometric recognition system.

The physical features are fingerprints, hand geometry, handprint, facial image, iris, retina, and ear. Behavioral features are signature, lip motion, speech, dynamics of typing, hand movements, and gait.

The characteristics of effective biometrics are:

1. Unique features for each individual

2. Invariant traits over time (e.g., due to the effect of aging)

3. Features that are relatively easy to obtain (computational complexity small)

4. Precise algorithms enabling classification

5. Resistance to various types of attacks

6. Low cost

7. Ease of implementation

The security of the biometric system is usually assessed on the basis of some indicators. These are:

• False match rate (FMR). It belongs to the group of matching errors. This indicator is defined as the expected probability that the downloaded sample will be falsely matched to the template in the database, but it will not be the test user pattern. If the indicator is high, it means that there is a risk that an unauthorized person will be recognized as a system user.

• False rejection (FRR) is equivalent to the FMR. The difference between these indicators is that FMR refers to a single match, and the FRR refers to a situation where one or more attempts to match a sample to a template from the database may occur. The FRR error is referred to in the literature as type I error.

• False discrepancy (FNMR). This is the coefficient determining the probability that the sample taken will not be matched to the pattern in the database belonging to the user from whom the sample was taken. In biometric verification (1:1) systems, the indicator means that the sample has not been identified by a specific pattern, while in biometric identification systems (1:N), this indicator determines the probability that a given pattern will not be found in the database.

• The false acceptance factor (FAR) is equivalent to the FNMR indicator. The difference between him and FNMR is the same as between FRR and FMR.

• Equal error rate (EER). It is defined as the intersection of the FAR and FRR characteristics in the graph of the dependence of these errors on the threshold of sensitivity (t). This factor indicates the optimal sensitivity threshold at which the same number of people is incorrectly rejected and incorrectly accepted. The lower the EER error value, the better the biometric system is.

The FMR (FRR) and FNMR (FAR) parameters can also be represented by graphs (Figure 2):

• Receiver operating characteristic (ROC) curve showing the dependence of FNMR on FMR. You can use it to show the accuracy of the system.

• Detection error trade-off (DET) showing error rates on both axes, most often on a logarithmic scale. This curve is plotted for both matching errors and decisions (Figure 2).

Figure 2.

The graph of FAR, FRR, and EER in receiver operating characteristic (ROC) curve.

If we use only one biometric authentication system, the results obtained are not always good enough. Unimodal biometric systems using a single sensor have many limitations, such as lack of uniqueness, universality, and lack of interference level associated with the acquired data, as a result of which they are unable to provide the required level of identification/verification efficiency (Table 1). This is due to the fact that the reliability of the biometric modality applied is affected by the precision of a single biometric system (Table 2).

Name	Description
Distortion of the input biometric data	Distorted biometric data may prevent the correct alignment process with database templates, as a result of which users are incorrectly rejected or identified
Intra class variations	Biometric data obtained from the person during authentication may differ from the data used to generate the template during registration, thus affecting the matching process. The biometric template should have a small intra-class variance
Interclass similarities	Biometric features should be significantly different for different people and should ensure small similarities between classes in the feature space. There is an upper limit to the users who can be effectively distinguished by any biometric system. The capacity of the identification system cannot be arbitrarily increased for fixed sets of feature vectors and the matching algorithm. The biometric template should have large interclass variations
Non-universality	Obtaining accurate (useful) biometric data from the users is not always possible
Intruder attacks	Attacks of this type involve the manipulation of biometric features to avoid recognition. It is also possible to create artificial biometric patterns in order to accept the identity of another person

Table 1.

Limitations of unimodal biometrics.

Name	Description
Recognition accuracy	The multi-biometric system ensures greater accuracy and reliability thanks to many independent biometric features that are difficult to attack
Continuous monitoring	In case when one biometric modality is obstructed, other modalities of the multi-biometric system ensure correct user identification
Privacy	Multi-biometric systems provide greater resistance to certain types of loopholes and attacks. It is difficult and/or impossible to steal many biometric patterns (templates) stored in the biometric database
Biometric data enrollment	When biometric input data is unavailable or unacceptable by a biometric system, another biometric system modality may be used
Resistance on spoof attacks	Usually the attacker is not able to use many relevant (accurate) spoofed biometrics

Table 2.

Advantages of multi-biometric systems.

2. Multi-biometric systems

2.1 Types of multi-biometric systems

The multi-biometric system can be (Figure 3) (a) a multi-sensor system that allows obtaining data from various sensors using one biometric feature, (b) a system with multiple algorithms processing a single biometric feature, (c) a system consolidating multiple occurrences of the same body trait, (d) a system using multiple templates of the same biometric method obtained with the help of a single sensor, and (e) a multimodal system combining information about the biometric features of the individual to establish his identity [2, 3, 4].

Figure 3.

Types of multi-biometric systems.

2.2 Fusion levels

In multimodal biometric systems, there are a number of strategies (scenarios) for the fusion of biometric information:

• Data fusion from sensors. Data from various sensors form one vector. Fusion of information obtained from many different sensors for a single biometric feature.

• The fusion of feature vectors extracted from various biometric modalities for further processing. A merger of information obtained from several unimodal biometric systems that process different body characteristics of the same person (Figure 4a).

• Fusion at the decision level. The merger of decisions developed on the basis of information from different biometric modalities, and the resultant feature vector defines two main classes, i.e., rejection or acceptance (Figure 4b).

• Rank level fusion. The classifier determines the rank of each registered biometric identity. A high position is a good indicator of a good fit (Figure 4b).

Figure 4.

Levels of fusion. (a) Feature level fusion and (b) score/rank level fusion.

2.3 Related work

The fusion of biometrics modalities on different levels of multi-biometrics system is extensively studied in the literature (Table 3). For all that the merger at the level of feature vectors is relatively poorly discussed. The merger at this level includes the integration of feature vectors corresponding to many sources of information. Because the feature vectors contain more elements than the input biometric data, it is obvious that the merger at the feature vectors level will provide better authentication results. However, mergers at this level are difficult to implement in practice because (i) sets of features of many modalities may be incompatible, (ii) the combination of two feature vectors may result in a vector of features with very large dimensionality, and (iii) a complex comparing system is required.

Biometrics traits	Fusion methods	Description of the implementation method	References
Fingerprint and face	Feature	In [5], it was proposed to extract face and fingerprint characteristics invariant to the rotation and scaling of Zernike moments (ZM). On the basis of ZM, the fusion of facial features and fingerprints is realized. The RBF network implements the decision-making process. The accuracy rate is 96.55%. Testing result of authentication rate are FAR, 4.95%, and FRR, 1.12%	[5]
	Score	In [6], authors presented score level fusion technique using the SIFT features for the face and the minutiae features for fingerprint. Results are: FAR = 1.98%, FRR = 3.18%, and accuracy = 97.41	[6]
Fingerprint,; finger knuckle print,; finger vein; Finger shape	Feature	The multi-set canonical correlation analysis is used to fuse multiple feature sets. The feature based on MCCA achieves the recognition performance, with EER = 2.3900e-04	[1]
		With the help of the unified Gabor filter, fingerprint codes and finger vein codes are generated. The extraction of features is carried out by using a supervised local canonical correlation analysis (SLPCCA), and finally the NN-classifier is used	[7]
Fingerprint and iris	Score	In [8], authors propose a frequency approach to generate a unified homogeneous template for fingerprint and iris features. Scores generated from these templates are fused using the sum rule	[8]
Palm print and hand shape	Feature	Information from the face image and gait image are combined at the function level. Using the principal component analysis (PCA) method, facial features were obtained The result of multiple discrimination analysis (MDA) is gait energy image (GEI) Recognition rate results are 91.3%	[9]
Palm print and iris	Feature	In system described in [10], texture parameters are extracted based on Gabor filters.	[10]
		Fusion of the palm print features and iris features is based on the wavelets. Decision is obtained using kNN classifier. Recognition accuracy is 99.2% and FRR = 1.6%
		In [11], fusion method for the information of phases about the iris and palm utilizes a Baud limited image product (BLIP)	[11]
Finger knuckle and palm print	Feature	In this paper, feature extraction method for palm print is monogenic binary coding; for inner knuckle print recognition, two algorithms named ridgelet transform and SIFT are proposed. The extracted feature vectors are classified using SVM	[12]
Palm print and face	Feature	The PCA is used to extract features of palm and face images. Fusion technique concatenated the feature vectors of the face and palm modalities into one fused vector, and feature selection is performed.	[13]
Face and gait	Feature	Method is based on learning face and gait features in image transform spaces. Two methods are considered—PCA and LDA	[14]
Face and iris	Score	Multi-biometrics system using dual iris, visible and thermal face traits is considered. 1D Log-Gabor and Complex Gabor Jet Descriptor (CGJD) were used to extract feature vectors. Authors proposed a score level fusion algorithm	[15]
		The ordinal measures and local binary pattern (LBP) methods are proposed to extract features from iris and face regions, respectively	[16]
	Feature	Paper [17] presents the extractions of iris features based on 2D Gabor and facial features using the PCA method	[17]
Face and hand geometry	Feature	The 2D DCT is used to extract discriminant face features which are concatenated with hand geometric features. The resultant feature vector is classified using SVM	[18]
Face and ear	Scores	To match score level, fusion is proposed in [19]. Authors use Dempster-Shafer decision theory for each modality. Recognition rate is 95.53% with 4.47% EER	[19]
Ocular—iris and conjunctival vascular	Score	In [4], authors presented fusion of both iris and conjunctival vascular information. A weighted fusion method is proposed for each modality. The fusion resulted in an EER of 2.83%	[4]
Face, ear, and signature	Rank	In [20], the PCA and Fisher’s linear discriminant (FLD) methods in the face, ear, and signature, multimodal biometrics system is proposed. Local features are extracted from face, ear, and signature data. Features are matched using Euclidean distance. This system is using rank level fusion	[20]

Table 3.

Summary of works on multimodal biometric systems.

3. The proposed multi-biometric system

The multi-biometric system (dorsal vein + periocular + palm print) is presented in Figure 5.

Figure 5.

Considered multi-biometric system architecture.

In our proposed method, the first is preprocessing block including noise elimination, ROI detection and normalization, and contrast normalization. For all three modalities, noise elimination for an image $f\left(x,y\right)$ is performed using median filtering (2D MF) operation formulated as [21]:

$$\widehat{f}\left(x,y\right)={\mathit{\text{median}}}_{A_1}f\left(x,y\right)=\mathit{\text{median}}\left[f\left(x+r,y+s\right)\right]$$ E1

where $A_1$ is the MF window.

Next step in preprocessing phase is ROI detection and normalization (Figure 6). This operation is quite different for dorsal vein images, palm print images, and periocular images. For dorsal vein images, we use distance transform to detect the dorsal image center and build square ROI based on this center coordinates [22, 23]. The ROI design for palm print images is based on hand-specific points (finger valleys) and two angles [24]. The periocular region is detected based on the center of the iris. Using the conventional algorithm for detecting the iris, we determine the center of the iris and its diameter. The periocular area is a rectangle centered in the iris center [25, 26].

Figure 6.

ROI area for dorsal vein images (a), palm print images (b), and periocular images (c).

After the ROI detection, we perform image size normalization and apply the contrast normalization by using CLAHE algorithm. The image is divided into non-overlapping areas of equal size, and the histograms of each region are calculated. Next, the cutoff threshold for histograms is obtained, and each histogram is processed in such a way that its height does not exceed the cutoff threshold [21].

The sample input images after normalization operations and operations using the CLAHE algorithm are shown in Figure 7. Next processing blocks include feature extraction, feature selection, fusion, and classification.

Figure 7.

Images after normalization (size 150 × 150 pixels) and after applying the CLAHE algorithm.

3.1 Gabor feature extraction

In biologically inspired vision models, receptor fields exist that are the primary aspect of early visual processing in mammalian vision systems. Gabor functions are widely used in image feature analysis because they are similar to receptive field profiles in mammalian cortical simple cells. These fields are modeled using Gabor filters [27].

Imitation of mammalian vision systems (or some of them) in object recognition systems leads to their increased efficiency and plausibility. Object recognition systems that are inspired by the biological approach use filter banks, in particular Gabor filters (Figure 8) [28, 29, 30, 31, 32].

Figure 8.

2D functions and 2D Gabor filter.

The 2D Gabor filter family can be represented as expressed in Eq. (2):

$${\mathit{Gab}}_{\omega ,\theta }\left(x,y\right)=\frac{1}{2\pi {\sigma }_x{\sigma }_y}{G}_{\theta }\left(x,y\right)S_{\omega ,\theta }\left(x,y\right)$$ E2

where $G_{\theta }\left(x,y\right)=e^{-\left(\frac{{\left(\mathit{xcos\theta }+\mathit{ysin\theta }\right)}^2}{2{\sigma }_x^2}+\frac{{\left(-\mathit{sin\theta }+\mathit{ycos\theta }\right)}^2}{2{\sigma }_y^2}\right)}$ and $S_{\omega ,\theta }\left(x,y\right)=e^{i\left(\mathit{\omega xcos\theta }+\mathit{\omega ysin\theta }\right)}-e^{-\frac{{\omega }^2{\sigma }^2}{2}}$ .

The ${\mathit{Gab}}_{\omega ,\theta }\left(x,y\right)$ can be decomposed into a real $\mathfrak{R}\left\{{\mathit{Gab}}_{\omega ,\theta }\left(x,y\right)\right\}=\frac{1}{2\pi {\sigma }^2}{G}_{\theta }\left(x,y\right)\mathfrak{R}\left\{S_{\omega ,\theta }\left(x,y\right)\right\}$ and an imaginary.

$\mathfrak{I}\left\{{\mathit{Gab}}_{\omega ,\theta }\left(x,y\right)\right\}=\frac{1}{2\pi {\sigma }^2}{G}_{\theta }\left(x,y\right)\mathfrak{I}\left\{S_{\omega ,\theta }\left(x,y\right)\right\}$ parts (for ${\sigma }_x={\sigma }_y=\sigma )$ .

Gabor response images are obtained by convolution operation of multiscale and multi-orientation Gabor filters ${\mathit{Gab}}_{\omega ,\theta }\left(x,y\right)$ with the image f (x, y).

$$G_{\omega ,\theta }\left(x,y\right)=f\left(x,y\right)\ast {\mathit{Gab}}_{\omega ,\theta }\left(x,y\right)={\mathit{Mag}}_{\omega ,\theta }\left(x,y\right)e^{i{\mathit{Ph}}_{\omega ,\theta }\left(x,y\right)}$$ E3

$${\mathit{Mag}}_{\omega ,\theta }\left(x,y\right)=\sqrt{\mathfrak{R}{\left\{{\mathit{Gab}}_{\omega ,\theta }\left(x,y\right)\right\}}^2+\mathfrak{I}{\left\{{\mathit{Gab}}_{\omega ,\theta }\left(x,y\right)\right\}}^2},$$

$${\mathit{Ph}}_{\omega ,\theta }\left(x,y\right)=\mathit{arctan}\frac{\mathfrak{I}\left\{{\mathit{Gab}}_{\omega ,\theta }\left(x,y\right)\right\}}{\mathfrak{R}\left\{{\mathit{Gab}}_{\omega ,\theta }\left(x,y\right)\right\}},$$

where and $\ast$ is the convolution operator.

The Gabor filter responses for palm print image and dorsal vein image are shown in Figures 9 and 10, respectively.

Figure 9.

Imaginary part of the Gabor filter responses of a palm print image.

Figure 10.

Imaginary part of the Gabor filter responses of a dorsal vein image.

3.2 Periocular feature extraction by LBP

The periocular area contains the iris, eyes, eyelids, eyelashes, and partially eyebrows. The LBP method can be used to describe the texture of the periocular area, and the feature vectors contain LBP features. The operator of local binary patterns (LBP) was proposed by Ojala [33] as a texture descriptor.

LBP divides the image into non-overlapping blocks of the same size. Local image features are calculated for each block separately. For a set of pixels belonging to a given block, the LBP values are calculated and then a histogram is created. The feature vectors (histograms) of each block are combined to form a global vector of features of the entire image.

LBP analyzes the local neighborhood consisting of $g_p$ points located on a circle with radius R and surrounding the center point of $g_c$ and checks whether the points of $g_p$ are greater or lesser than the $g_c$ point value.

The LBP value of the $g_c$ point is specified as follows:

$${\mathit{LBP}}_{P,R}=\sum _{p=0}^{P-1}S\left(g_p-g_c\right)2^p$$ E4

where $g_p$ and $g_c$ are the luminance values of the neighborhood and center point, respectively.

The idea of this operator is presented in Figure 11.

Figure 11.

The basic idea of LBP approach.

For an image size $M\times N$ , the image descriptor is a histogram created from the LBP values:

$$H\left(k\right)=\sum _{i=1}^M\sum _{j=1}^{N}f\left({\mathit{LBP}}_{P,R}\left(i,j\right),k\right);k\in \left[0,K\right]$$ E5

$$\mathrm{f}\left(\mathrm{x},\mathrm{y}\right)=\left\{\begin{array}{c}1,\mathrm{x}=\mathrm{y}\\ 0,\text{otherwise}\end{array}\right.$$

where k is one LBP pattern and K is the maximal LBP pattern value (number bin of the histogram).

Using the LBP operator, we obtain $2^P$ different output values corresponding to $2^P$ different binary patterns created by P of neighboring pixels. Certain binary patterns contain more information than others, so we can only consider this subset of LBP values. Patterns of this subset are called uniform patterns. So we have a standard ${\mathit{LBP}}_{P,R}$ operator and an ${\mathit{LBP}}_{P,R}^{u2}$ operator.

Typically image is divided into n blocks and histograms of each block are concatenate into feature vector [34].

In the case ${\mathit{LBP}}_{P,R}$ operator, the histogram contains 256 bins. In the case $\mathit{\text{of}}{\mathit{LBP}}_{P,R}^{u2}$ operator, the histogram contains 59 bins (Figures 12 and 13).

Figure 12.

The original image (a) and image as a result of the LBP operator (b).

Figure 13.

The ${\mathit{LBP}}_{P,R}$ histogram (a), histograms of the n blocks (b), and the ${\mathit{LBP}}_{P,R}^{u2}$ histogram (c).

4. Feature reduction and data fusion

The multi-biometric system has been tested using certain parts of the following databases: PolyU palmprint [24], IIITD periocular database [25], and Bosphorus hand vein database [35]. We choose 20 subjects with 10 images per subject at random. From 10 images, 5 images are used for training and 5 for the testing.

The combination of feature vectors at this level is difficult to achieve in practice due to the combination of certain fundamentally different feature vectors that can result in a resulting vector of features with very large dimensionality. In a merger at the level of feature vectors, each individual modality process generates a feature vector. The fusion process combines these feature vectors into one vector.

For dorsal vein images and for palm print images, we perform the same image processing operations that the feature vectors have the same sizes. As a result of convolution operation of multiscale and multi-orientation Gabor filters with the input image, we get the Gabor response images. The feature vector has a very large size of (M x N x k x l) where M x N is the image size, k is the number of scales, and l is the number of orientations. In our case, for both dorsal vein images and palm print images, we get a feature vector containing (150 × 150 × 3 × 6) = 405,000 items. The images subjected to the Gabor filtration are rescaled with a scale factor of 0.1, which allows obtaining a vector of features with a size of 1 × 4050 elements.

For periocular images, the feature vector has a size of 36 × 59 = 2124.

Next we reduce dimensionality of these vectors used in PCA method (Figure 14 and Table 4) [5]. Separated features are normalized using zero mean and unit variance as

Figure 14.

Steps to image processing using PCA.

PCA algorithm

Organizing the training set of images; $T=\left[G_1,G_2,\cdots ,G_q\right]$; where q is the number of images in the training set

Calculating the average of the set T; $\mathrm{\Psi }=\frac{1}{q}\sum _1^q{G}_q$

Calculating; ${\mathrm{\Phi }}_i=G_i–\mathrm{\Psi }$

Calculating the covariance matrix C; $C=\frac{1}{q}\sum _1^q{\mathrm{\Phi }}_i{\mathrm{\Phi }}_i^t=A{A}^t$

The eigenvectors and corresponding eigenvalues are computed; $C{v}_i={\lambda }_i{v}_{i}i=1,\cdots ,q$

The eigenvectors and their corresponding eigenvalues are paired and ordered from high to low. Approximated image is calculated as; $\overline{G}=\mathit{vw}+\mathrm{\Psi }$; for $v=\left(v_1,v_2,\cdots ,v_k\right)$

Table 4.

PCA algorithm.

$${\overline{f}}_i=\frac{f_i-{\mu }_i}{{\sigma }_i}$$ E6

where ${\mu }_i$ and ${\sigma }_i$ are the mean value and standard deviation of the i-th feature, ${\overline{f}}_i$ is the normalized i-th feature vector.

Table 5 shows the recognition performance depending on the number of selected eigenvectors.

Modality	Number of the eigenvectors
	k = 40	k = 60	k = 80	k = 100
Dorsal vein	88	89.3	91.4	92.6
Palm print	88.7	89.3	90.6	92.8
Periocular	86	86.8	89	89.2
Dorsal vein + palm print	90.3	91.1	92.3	93.1
Dorsal vein + periocular	91.1	92	92.4	92.8
Palm print + periocular	90.7	91.4	91.8	92.1
Dorsal vein + periocular + palm print	93.2	94	94.5	95.3

Table 5.

Recognition rates [%] for different modality.

5. Conclusion

In this chapter, Gabor’s functions and LBP features are proposed for recognition in a multi-biometric system that uses three modalities: dorsal vein, periocular, and palm print. Using PCA method dimensionality feature vectors from these modality are reduced. Feature vectors are normalized and fused using concatenation operation. Based on the results, we suggest that multi-biometric system using the fusion of dorsal vein, periocular, and palm print images can offer recognition rate which the unimodal biometric system cannot.