Face Identification Using LBP-Based Improved Directional Wavelet Transform

1. Introduction

Researchers have devoted a substantial amount of effort in studying face identification methods in the context of computer vision, image processing, and pattern recognition due the wide acceptability of face as biometric [1]. Requirements for high recognition accuracy, high computational efficiency, and invariance to variations in facial expression, illumination, pose, and occlusions are the prominent challenges in face identification. The illumination problem comes from the fact that different illuminations can cause vast changes to the image of a subject’s face [2, 3]. Similarly, deviations in facial expressions along with head pose variation and occlusion can also lead to very unlike face images for the same subject. Moreover, face identification in the unrestrained environment is still a major challenge which greatly degrades the performance of various well-established methods. Additionally, there still exist challenges such as high dimensionality of feature data and intraclass variations occurring due to the effect of facial variations in restrained and unrestrained environments. A face identification method must be discriminatory for different subjects and invariant to numerous facial variations. Researchers have been extensively utilizing MRA methods and using various off–the-shelve designs of wavelet filters [4] for the implementation of isotropic 2-D DWT for facial feature extraction. Recently implemented 2-D DWT methods such as GHWFB [5] and TWFB [6] considers the handcrafted wavelet filters with additional features compared to off-the-shelve wavelet filters. But these methods do not achieve excellent results due to limited directions orientation and non-adaptation in facial feature selection.

Various local descriptors prominently the LBP [7] and weber local descriptors (WLD) [8] have been efficiently used for facial feature extraction. The constraint of the LBP-based feature extraction method is their noise intolerance and poor discrimination capability [8]. Recently, various non-adaptive MRA methods are applied as a pre-processing step before LBP feature extraction to improve the performance. The prominent methods are local Gabor binary patterns (LGBP) [9], Steerable Pyramid Transform (SPT)-LBP [10], Curvelet Transform-LBP (Curvelet-LBP) [11], Contourlet-LBP [12], and Wavelet Transform (WT) LBP [13]. Liu et al. [18] used hierarchical multi-scale LBP to create features of sparser coefficients and performed classification using sparse coding with the application of a greedy search approach. Wang et al. [19] combined the Gabor wavelet transform (GWT) and CLBP features and carried out the SRC to perform classification.

The aforementioned LBP-based MRA methods [9, 10, 11, 12, 13, 19] use non-adaptive directional transform which lacks the adaptive directional selectivity based on the image description. These methods also experience various issues, for instance, selection of transformed sub-bands, complex filter design, and the large dimension of the feature vector. Maleki et al. [14] proposed adaptive direction selection and applied directional lifting within the selected optimal direction and constructed a compact representation for adaptive MRA method. Due to such inherent characteristics, significant directional details for various face variations can be approximated by the detection of edges responsible for such variations [15].

For numerous facial variations, substantial directional details can be estimated by approximating the edges [16, 17] accountable for such variations which will considerably enhance the face identification performance which decides the basis of our method. The concept has been exploited in [15, 16, 17] for face recognition applications. This work extends the design of the adaptive directional scheme presented in [17] and presents an LBP-based IDW method to capture multi-resolution directional details from the face images. Subsequently in contrast to [17] where CLBP is used, LBP is applied to the generated IDW sub-bands to extract MRA-based local descriptive features.

The Implementation of the 2-D IDW using seven directions along with the quadtree partitioning (QTP) scheme [17] is explained in Section 2. A brief theory on LBP is described in Section 3. Further, the proposed facial feature extraction method is exhibited in Section 4. In Section 5, comparative results on the CASIA-WebFace and LFW face databases are demonstrated. Conclusions are highlighted in Section 6.

2. Implementation of 2-D improved directional wavelet (2-D IDW)

The fundamental concept of implementation of improved directional wavelet (IDW) is to carry out transform operations on a face image at a viable variety of possible directions while maintaining the properties of multi-resolution, localization, and isotropy intact. The authors in [17] considered a set of seven directions with a quad-tree partitioning scheme. Here we will provide a brief review of the work mentioned in [17].

The 2-D IDW being isotropic method performs a separable 1-D horizontal and vertical 1-D IDW on face image with variation in prediction and update steps where seven directions are considered in an adaptive direction scheme. While performing the 1-D IDW transform if non-integer sample arrives sub-pixel interpolation is performed.

Let xijbe a 2-D face image which is first horizontally sub-sampled to get even subsamples xeij=x2i+jand odd sub-samples xoij=x2i+1j. Next in the prediction step odd samples xoijare predicted from neighboring even samples with strong correlation along an optimal direction θ.The outcome of the prediction step and the generated high-pass signal Hijare described as [17],

Where Kpand 2Npare the length and coefficients of the prediction filter. Here, samples from six even rows are selected to conduct the prediction step [17]. Now in the update step, odd samples of Hijalong the same optimal direction θas used in (1) are selected to modify the even samples. The update step and the generated low-pass signal Lijare given as [17],

Where Kuand 2Nuare the coefficients and length of the update filter. Similarly, samples from six odd rows are selected to conduct the update step. The values of scaling factors are considered as gL=1.3416and gH=0.7071[15, 17].

Due to linear phase characteristics and large vanishing moments, Neville filters with the order as six are considered as the coefficients Kpand Ku[17]. The usage of Neville filters increases the approximation ability of 2-D IDW.

In contrast to the nine directions [15] and five directions [16], we also used seven pre-assigned directions to implement 2-D IDW [17].

These directions are used to confirm a strong correlation among samples and to extract directional MRA features from face images. It is to point out that signn−1tanθterm in (1) and (3) may not always locate integer samples and may not be present on the original image sampling grid [17]. So; sub-sample interpolation is conceded to compute intensity for such non-integer samples. To maintain perfect reconstruction lifting structure [4], the integer samples required to perform sub-sample interpolation for such non-integer samples at optimal direction θmust be even sampled. If optimal direction comes across the integer samples the value is computed by the nearby even sample otherwise the value of the non-integer sample is computed from the interpolation of the two nearby even samples.

To extract local edge details due to face variations that exist at different pixel regions, a quadtree partitioning (QTP) scheme is implemented to partition each face image into sub-blocks of distinct directional details. Each QTP sub-block will have the same direction. The improved QTP scheme provides an efficient direction assignment while implementing the prediction and update step.

Let each face image xijis applied with QTP to obtain non-overlapping sub-block xs. Also, consider the initial block size Sini, minimum block size as Sminand the Lagrangian multiplier as α. The energy summation of the prediction error (ESPE) for each block is computed as [17],

Where Fs,nijare the filtered responses obtained by applying the prediction filter Kpalong with the predefined directions θ. Bnis the number of bits spent on signaling the selection of directions. When a sample is predicted from the nearest samples, each candidate direction from (5) is checked and the direction with the smallest ESPE is ultimately selected. The optimal direction which gives the least value of ESPE is selected as,

The value of the lagrangian multiplier αdetermines the complexity of the QTP scheme and its value needs to be selected sensibly. Moreover, to detect the local edge details and to suit it to the adaptability of the IDW method, a face image needs to be segmented into partitions of clear orientation bias. To resolve this problem an improved QP scheme is proposed to suit the face identification problem as mentioned in [17]. The 1-D IDW can be simply extended to the 2-D IDW where second dimension lifting is yet again performed in the horizontal direction on high-pass signal Hijand low-pass signals Lijto generate four sub-bands i.e. LHij, LLij, HHij, and HLij.

3. Local binary patterns

The LBP [7] is estimated with sampling points xp∈0…P−1in the neighborhood of a center pixel xmicjcat a radial distance given by R[7],

Where tsdis a threshold function. The sampling points which does not fit within the center of a pixel are bilinearly interpolated [7]. Another extension of LBP is the uniform patterns and it is mapped from LBPP,Rto LBPP,Ru2[16], resulting in P×P−1+3feature dimension. After obtaining the LBP coded image, codes of the input image XLijpixels are formed into a histogram as a feature descriptor,

Where nis the number of different labels produced by the LBP operator. With the usage of LBP8,1u2, the feature dimension is 59 [16].

4. Implementation of LBP-based IDW method

We consider a resolution of 128×128pixels for face images of the selected databases and face preprocessing is performed on all the face images. Thus each one of the IDW sub-bands LLHLLHis of size 32×32pixels and each sub-band is divided into m=16regions with the size of each region as x×y=8×8pixels [16]. We applied LBP to each of the regions from each of the sub-bands LLHLLH. We used the uniform pattern LBP8,1u2[7] and NN classifier with Chi-square distance measure. This form an enhanced feature vector or descriptor EFV with a combined dimension as 59×m×3=2832[16]. The algorithm representing the proposed method is presented in Algorithm 1.

Algorithm 1: Face Identification using LBP-based IDW

Input: Test Image, Train image

Output: Rank-one recognition results of the feature vectors.

Algorithm:

Step 1: (Preprocessing)

 1.1. Consider the input face image X.

 1.2. Resize the image to the resolution of 128×128pixels.

Step 2: (Computation of IDW sub-bands)

for a number of decomposition levels do

 2.1. Quadtree partitioned the face image Xinto several non-overlapping sub-blocks.

 2.2. Estimate the value of ESPE using (6) for each sub-block.

 2.3. Estimate the optimal direction which gives the least value of ESPE using (7).

 2.4. Perform the prediction and the update steps as described in (1) and (3) in the selected directions in the selected sub-block.

 2.5. Obtain IDW sub-bands {LL, HL, LH, HH} and proceed with the LL sub-band for the next decomposition level.

Step 3: (LBP Computation)

 3.1. Consider the top-level {LL, LH, HL} sub-bands and divide each sub-band into non-overlapping regions Rkwith each of size 8×8pixels.

for each sub-band do

for each sub-block within the sub-band do

for each coefficient value within the sub-block do

 3.2. Compute the LBP8,1u2histogram features from each region Rkusing (8), (9), and (10).

 3.3. Concatenate all such LBP8,1u2multi-region histograms from each sub-band {LL, HL, LH} to form the histogram feature vectors LLl,k, HLl,k, and LHl,krespectively.

 end for end for end for

 3.4. Concatenate all the sub-band histogram features to form the final enhanced histogram feature vector EFV.

Step 4: (Dimensionality Reduction)

 4.1. Perform dimensionality reduction using LDA on the EFV feature vector.

 4.2. Save the reduced dimension train feature vector database to EFV_train.

Step 5: Repeat Step 1 to 3 and 4.1 on each test image to obtain the test feature vector EFV_test.

Step 6: (Identification)

 6.1. Compare test feature vector EFV_test against train feature vector EFV_train using the NN classifier using Chi-Square distance measure and calculate the Rank-one results in an identification process.

5. Experimental results

All experiments are performed using Matlab 2018a on a standard i5–3320 2.60 GHz machine with 8.0 GB RAM. Here we prove the effectiveness of the LBP based IDW feature extraction method for which a comparison is established with other LBP based non-adaptive MRA methods. In the experiments, we randomly select a few face images for training and rest for testing to obtain the recognition results.

The face identification performance of the proposed method is performed on CASIA-WebFace [21] and LFW [22] face Database with extreme facial variations where all the images are considered under unrestrained environment. Since, LBP-based IDW histogram features are extracted, comparative face identification methods include descriptive methods such as LBP [7], WLD [8], and SRC-GSLBP [18]. Besides, a comparison with few non-adaptive MRA-based LBP feature extraction methods such as LGBPHS [9], SPT-LBP [10], Curvelet-LBP [11], Contourlet-LBP [12], and GTCLBPSRC [19] is also performed for competitive analysis.

5.1 Experiments on the CASIA-WebFace face database

The CASIA-WebFace database [20] is a huge and complex face database. This database includes 494,414 face images of 10,575 subjects. Considering the subjects with only a few samples deters the recognition results. Thus 10,575 subjects are allocated in the decreasing order by the count of their images contained in the particular subject set. Here we consider only 9067 subjects which consist of at least 15 images. The remaining images of the rest of the 1508 subjects are discarded. Within this, we considered a subset of 600 subjects with 15 images per subject out of 9067 subjects. These subjects are specifically considered based upon their extreme facial variations. Face images are normalized and resized to128x128 pixels.

A random subset is constructed with T (T = 4, 5, 6, 7, 8) images of each subject for training and in every case remaining images for testing. Table 1 tabulates the Rank-one recognition results of various comparative methods. Since the images are collected from around the web with extremely unrestrained conditions, the results are less for all the methods. Due to extreme face variations which include illumination, expression, pose, occlusion, and age difference, this database imposes an immense challenge. Figure 1 depicts the trend of the rank one recognition rates for different comparative methods along with the proposed method for Casia-WebFace face database.

Method	Number of training samples per subject
	4	5	6	7	8
LBP [7]	14.07	16.06	18.24	21.81	24.14
WLD [8]	15.60	17.71	21.40	23.20	25.09
LGBPHS [9]	22.80	26.09	28.93	31.48	34.76
SPT-LBP [10]	25.22	30.10	32.20	34.52	36.12
Curvlet-LBP [11]	25.66	30.28	31.49	35.49	38.26
Contourlet-LBP [12]	27.22	31.33	32.49	36.55	40.60
SRC-GSLBP [18]	31.24	33.37	34.19	38.20	42.05
GTCLBPSRC [19]	31.25	32.55	34.42	38.63	42.30
ADWTLBP [15]	33.90	39.40	41.09	42.00	45.60
DIWTLBP [16]	34.65	40.36	41.78	42.17	45.81
IDW-LBP (Proposed Method)	35.89	41.39	42.20	43.31	46.12

Table 1.

Rank-One Recognition Results of different methods on the CASIA-WebFace face database (%).

5.2 Experiments on the labelled faces in the wild (LFW) face database

The LFW [21] is a large database that contains face images of 5749 famous personalities captured in an unrestrained environment with an extreme variation of background, pose, illumination, expression, and accessories. This makes it a challenging database for face identification. Here, we used the LFW-a database [22] which is an aligned version of the LFW database. For our experimentation purpose, we created a subset with 15 dissimilar images of 150 subjects from the LFW-a database. Each image is resized to 128×128pixels.

A subset with T (T = 4, 5, 6, 7, 8) images per subject is randomly selected to form a training set, and rest images per subject are selected to form the testing set. Rank-one recognition results of different comparative methods are tabulated in Table 2 . Since the images are selected in the unrestrained environment the Rank-one recognition results are also low in this database.

Method	Number of training samples per subject
	4	5	6	7	8
LBP [7]	24.07	30.16	38.22	42.81	44.14
WLD [8]	25.12	31.42	40.10	43.60	46.12
LGBPHS [9]	33.76	35.15	42.05	45.34	49.06
SPT-LBP [10]	34.90	40.20	43.62	45.93	50.61
Curvlet-LBP [11]	33.11	39.48	43.55	44.60	50.08
Contourlet-LBP [12]	35.09	40.25	45.52	46.41	53.32
SRC-GSLBP [18]	38.30	41.40	47.22	55.00	57.69
GTCLBPSRC [19]	40.72	42.80	50.30	58.59	60.23
ADWTLBP [15]	41.60	44.50	52.28	59.44	62.74
DIWTLBP [16]	40.80	44.89	53.03	60.18	63.04
IDW-LBP (Proposed Method)	40.77	45.64	54.80	61.01	64.42

Table 2.

Rank-One Recognition Results of different methods on the LFW face database (%).

Figure 2 depicts the trend of the rank one recognition rates for different comparative methods along with the proposed method for LFW face database.

6. Conclusion

This chapter discusses the recently developed implementation of interpolation-based ADWT with seven directions and an improved QTP scheme to extract directional MRA features from face images. LBP is applied to the selected top-level IDW sub-bands to extract the multi-region histogram-based local descriptive features. Experiments conducted on the complex face databases such as CASIA-WebFace and LFW database exhibit the efficacy of our proposed method. The identification results of our method are compared with various methods which include local descriptors such as LBP and WLD. Few LBP-based non-adaptive MRA methods are also utilized for a fair comparison as our method also falls into the category of MRA based methods.

LBP and WLD suffer from issues such as the large size of histogram features, extraction of only very local texture details, limited discriminative ability, and intolerance to noise.

It also is evident from Tables 1 and 2 that LGBPHS, SPT-LBP, Curvelet-LBP, and Contour-LBP methods provide lesser results against our proposed method. We examine progress over SRC-GSLBP, GTCLBPSRC which illustrates some effectiveness of usage of sparse features at the cost of increased complexity in implementation.

SPT-LBP also exhibits comparable performance to the proposed method but the feature selection is threshold dependent and necessitates the selection of sub-bands for efficient feature extraction. Curvelet-LBP uses only the LBP coded image of the approximation sub-band and mid-frequency sub-bands coefficients for feature generation and does not consider the multi-region information. Thus LGBPHS, Curvelet-LBP, Contourlet-LBP, and SPT-LBP are both memory and time exhaustive to extract the multiresolution and multi-orientation features due to selection and feature extraction from different sub-bands.

Moreover, these methods despite capturing the directional information lack the adaptation in selecting the directional details based on the image description and suffer from various issues such as the selection of sub-bands, high computational rate, and complex filter design. The GTCLBPSRC delivers close results to the proposed method for all the databases but at cost of additional computational time and due to the implementation of Gabor wavelet transform (GWT) which exhibits over-complete representation. We also examined comparable progress over SRC-GSLBP and GTCLBPSRC for all the databases which illustrate the effectiveness of sparse representation methods. The IDW method consists of benefits such as directional lifting and adaptation in the direction selection as per the characteristics of the images within a block of samples. Moreover, as a result of lifting based factorization, perfect reconstruction is also assured and the resultant multiresolution image is completely compatible with that of the conventional 2-D DWT multiresolution image. These facts effectively consider various edge manifolds that represent different face variations.

We also compared the proposed method with recently developed methods which also considers the facial descriptions in an adaptive MRA-based structure such as ADWT [15] and DIWT [16]. We applied a similar procedure to extract the LBP based features. The improvement in our method is visible owing to the adaptation of more directions as compared to DIWT [16] and application of sub-pixel interpolation in IDW which is absent in the ADWT method.

Thus, as per Tables 1 and 2 , it is verified that the IDW method exhibit high discrimination capability and offers excellent recognition results for a very complex database such as CASIA-WebFace and LFW databases which consists of facial variations with mild to intense pose, expression, and illumination variations. Experiments performed for an identification process verify that the proposed method excels with all the comparative methods.