Dimension comparison of three LBP features (
1. Introduction
Face recognition is one of the most broadly researched subjects in pattern recognition. Feature extraction is a key step in face recognition. As an effective texture description operator, Local Binary Pattern (LBP) feature is firstly introduced by Ahonen et al into face recognition. Because of the advantages of simplicity and efficiency, LBP feature is widely applied and later on becomes one of the bench mark feature for face recognition. The basic idea of LBP feature is to calculate the binary relation between the central pixel and its local neighborhood. The images are described with a multi-regional histogram sequence of the LBP coded pixels. Since most of the LBP pattern of the images are uniform patterns, Ojala et al, 2002 proposed Uniform Local Binary Pattern (ULBP). Through discarding the direction information of the LBP feature, they proposed the Rotation Invariant Uniform Local Binary Pattern (RIU-LBP) feature. The Uniform LBP feature partly reduces the dimension and retains most of the image information. RIU-LBP greatly reduces the dimension of the feature, but its performance in face recognition decreases drastically. This chapter mainly discusses the major factors of the ULBP and RIU-LBP features and introduces an improved RIU-LBP feature based on the factor analysis.
Many previous works also endeavored to modify the LBP features. Zhang and Shan et al, 2006 proposed Histogram Sequence of Local Gabor Binary Pattern (HSLGBP), whose basic idea was to perform LBP coding to the image in multi-resolution and multi-scale of the images, thereby enhancing the robustness to the variation of expression and illumination; Jin et al, 2004 handled the center pixel value as the last bin of the binary sequence, the formation of the new LBP operator could effectively describe the local shape of face and its texture information; Zhang and Liao et al, 2007a, 2007b proposed multi-block LBP algorithm (MB-LBP), the mean of pixels in the center block and the mean of pixels in the neighborhood block were compared; Zhao & Gao, 2008 proposed an algorithm for multi-directional binary mode (MBP) to perform LBP coding from four different directions; Yan et al, 2007 improved the robustness of the algorithm by fusing the mult-radius LBP feature; He et al, 2005 believed that every sub-block contained different information, and proposed an enhanced LBP feature. The original image was decomposed into four spectral images to calculate the Uniform LBP codes, and then the waterfall model was used to combine them as the final feature. In order to effectively extract the global and local features of face images, Wang Wei et al 2009 proposed LBP pyramid algorithm. Through multi-scale analysis, the algorithm first constructed the pyramid of face images, and then the histogram sequence in a hierarchical way to form the final features.
No matter how the ULBP features are modified, the blocking number, the sampling density, the sampling radius and the image resolution dominantly control the performance of the algorithms. They affect the memory consumption and computation efficiency of the final feature drastically. However, the values of these factors have to be pre-selected and in most previous work, their values are decided with some experience, which obviously ignores the influence degree of each factor and the experience values are hard to be generalized to other databases. In order to seek a general conclusion, in this chapter, we use statistical method of multivariate analysis of variance (MANOVA) to discuss the contribution of four factors for face recognition based on both ULBP and RIU-LBP features. Besides, we research the correlation of the factors and explore which factors play a key role in face recognition. We also analyze the characteristics of the factors; discuss the change of influence of factors for different LBP features. Based on the factor analysis, we propose a modified RIU-LBP feature.
The chapter is organized as follows. In Section 2, we introduce the LBP operators, the LBP features and the four major factors. In Section 3, we illustrate how the MANOVA is applied in exploring the importance of four factors and the results obtained for the two types of LBP features. Based on the above analysis results, an improved RIU-LBP algorithm is introduced in Section 4, which is a fusion of multi-directional RIU-LBP features. We summarize the chapter with several key conclusions in Section 5.
2. LBP features and factors
LBP feature is a sequence of histograms of blocked sub-images of face images coded with LBP operator. The image is divided into rectangle regions and histograms of the LBP codes are calculated over each of them. Finally, the histograms of each region are concatenated into a single one that represents the face image.
2.1. Three LBP operators
With the variation of the LBP operator, the obtained LBP features are of different computation complexity. Three types of LBP operators are compared in this chapter.
The basic LBP operator is formed by thresholding the neighborhood pixels into binary code 0 or 1 in comparison with the gray value of the center pixel. Then the central pixel is coded with these sequential binary values. Such coding denoted as
When there exist at most 2 times of 0 to 1 or 1 to 0 variation, the binary pattern is called a uniform pattern. The Uniform LBP operator
Rotation Invariant Uniform (RIU) LBP operator
where
2.2. Three LBP features
After the original face image is transformed into an LBP image with the LBP operators, the LBP image is blocked into
Due to different coding complexity, the above three LBP features
The general LBP | 14336 | 28672 | 57344 |
Uniform LBP | 3304 | 13608 | 13216 |
RIU-LBP | 560 | 1008 | 2240 |
2.3. The four factors of LBP feature
The blocking number, the sampling density, the sampling radius and the image resolution are four factors that determine the LBP features.
The blocking number and sampling density are two important initial parameters affecting the dimensions and arouse more attentions in previous research. In addition, the blocking number and the image resolution determine the number of pixels of each sub-image. It means how much local information of face contains in each sub-image. If the image resolution is
Among the four factors, the blocking number and the sampling density are two factors deciding the dimension of the LBP features. For an example shown in Table 1, in the case of different sampling density, the dimension of
For different values of the four factors, we compare the face recognition rate on a face database containing 2398 face images (1199 persons, each of 2 images) selected from the FERET database. Experimental results are evaluated with the curve of rank with respect to CMS (Cumulative Matching Score), meaning the rate of correct matching lower than a certain rank. The closer this curve towards the line
Figure 3 is the comparison of recognition rate for three LBP features. It can be observed that the Uniform LBP has very close performance with the basic LBP but of much lower dimension. While the recognition rate of the RIU-LBP feature decreases significantly due to the loss of direction information. The results indicate that it is good enough to adopt ULBP instead of the basic LBP feature and the direction information has major impact to the recognition rate.
Figure 4 compares the performance of the same kind of LBP feature with various values of the blocking numbers and sampling density. The comparison shows that higher sampling density and more blocking numbers could result in better performance. It demonstrates that the two factors affect not only the feature dimensions, but also the face recognition rate. Besides, the sampling radius determines the sampling neighborhood of sub-blocks. The image resolution and the blocking number determine the number of pixels of each sub-block.
Figure 5 compares the performance of the same kind of LBP feature with various values of the sampling radius and image resolution. The results show that higher resolution and larger sampling density are better for face recognition. Though, these two factors do not affect the feature dimensions, they have unneglectable impact on the recognition rate.
3. The importance of four factors
As described in the above, when we use LBP feature in face recognition, the blocking number, the sampling density, the sampling radius, the image resolution would affect the recognition rate in varying degrees.
In most present research, the values of these factors are selected according to experience in experiments. Ahonen, et al, 2004 compared different levels of the blocking number, the sampling density and the sampling radius based on the Uniform LBP feature. Under the image resolution of 130 * 150, they selected blocking number
However, more open problems could not be explained with the experienced values. How is the degree of impact of the four factors? Are they contributes the same in efficiency? Are there interactions between pairs of factors? How will the parameters affect the recognition? How to compare the performance of different LBP features? In order to solve these problems, we endeavor to compare four major factors for the two most typical LBP features, i.e. the ULBP and the RIU-LBP feature with MANOVA. Our purpose is to explore the contribution of the four factors for recognition and the correlation among them. The results of the studies to these problems provide important merits for the improvement of LBP features.
3.1. MANOVA
MANOVA is an extensively applied tool for multivariate analysis of variance in statistics. For our problem, the four variables waiting to be explored are the resolution of images, the blocking numbers, the sampling density and the radius of LBP operators for face recognition tasks. With MANOVA, we could identify whether the independent variables have notable effect and whether there exist notable interactions among the independent variables [12].
By denoting the four factors as follows:
And taking the face recognition rate as the dependent variable, the total sum of squares deviations
where
MANOVA belongs to the
3.2. Experiment design of factors
We use the same face database as described in Section 2.3 in MANOVA. As analyzed in Section 2.3, the general LBP feature has much higher dimensions than the ULBP feature but the performance is close, so we conduct the experiments for ULBP and RIULBP features.
We set each factor three or four different levels as shown in Table 2. Under the RIU-LBP features, 108 sets of experimental data were obtained. Under the ULBP features, 81 sets of experimental data were obtained (level of blocking number
Level1 | 4 | 1 | 35*40 | |
Level2 | 8 | 2 | 70*80 | |
Level3 | 16 | 3 | 140*160 | |
Level4 | — | — | — |
3.3. Analysis of the factors of RIU-LBP
3.3.1. The significance and interaction
We firstly analyze the independent influence of four factors and significance of interaction. Table 3 shows the results based on RIU-LBP feature. The row of Table 3 is in descending order by
The first part of Table 3 shows the independent effect of the four factors. The value of
Df | Sum Sq | Mean Sq | |||
3 | 1.509 | 0.503 | <0.001 | ||
2 | 0.464 | 0.232 | 415.919 | <0.001 | |
2 | 0.381 | 0.190 | 341.202 | <0.001 | |
2 | 0.048 | 0.024 | 42.772 | <0.001 | |
6 | 0.067 | 0.011 | 20.058 | <0.001 | |
4 | 0.043 | 0.010 | 19.239 | <0.001 | |
6 | 0.021 | 0.004 | 6.300 | <0.001 | |
6 | 0.011 | 0.003 | 4.729 | 0.002 | |
6 | 0.007 | 0.002 | 2.923 | 0.027 | |
6 | 0.006 | 0.001 | 1.700 | 0.134 | |
Residuals | 68 | 0.038 | |||
Total | 107 | 2.594 |
3.3.2. Analysis of levels of each single factor
For each single factor, we also design the MANOVA for each pair of levels. The
Table 4 is the analysis result based on four levels of blocking numbers. From the second column it can be seen that the more the blocking number is, the higher the mean of recognition rate. For blocking numbers, the
Level | Mean | ||||
0.477 | 1.000 | <0.001 | <0.001 | <0.001 | |
0.679 | <0.001 | 1.000 | 0.016 | 0.002 | |
0.754 | <0.001 | 0.016 | 1.000 | 0.410 | |
0.777 | <0.001 | 0.002 | 0.410 | 1.000 |
Through the first part of the analysis, we know that the sampling density is of the least affect among four factors. The pair-wise results for sampling density in Table 5 show that there is no significant difference for various levels since the
Level | Mean | 4 | 8 | 16 |
4 | 0.642 | 1.000 | 0.61 | 0.61 |
8 | 0.689 | 0.61 | 1.000 | 0.89 |
16 | 0.684 | 0.61 | 0.89 | 1.000 |
Table 6 and Table 7, respectively are the analysis result based on the three levels of sampling radius and image resolution. Although these two factors would not affect the feature dimension, but from previous discussion we already know that they also affect recognition rate. If we select the appropriate parameter values of these factors, it could help to improve the recognition rate.
For the resolution of image, there exists significant difference between 35*40 and 140*160. The significance between 35*40 and 70*80 is larger than that between 70*80 and 140*160 as shown in Table 6. Because the clearer the images are, the more useful information could be extracted. But at the same time, we should also consider that the higher the resolution, the larger the storage space is required. For the massive database, high-resolution images would increase the storage difficulties and need to spend more time to load image information. So according to the requirement of applications, we could select the lower resolution images to reduce storage space with acceptable recognition rate.
Level | Mean | 35*40 | 70*80 | 140*160 |
35*40 | 0.591 | 1.000 | 0.007 | <0.001 |
70*80 | 0.692 | 0.007 | 1.000 | 0.252 |
140*160 | 0.732 | <0.001 | 0.252 | 1.000 |
In Table 7, we observe that both level 2 and level 3 are more important than level 1 for the sampling radius. However, there is no significant difference between level 2 and level 3.
Level | Mean | 1 | 2 | 3 |
1 | 0.580 | 1.000 | 0.001 | <0.001 |
2 | 0.705 | 0.001 | 1.000 | 0.452 |
3 | 0.730 | <0.001 | 0.452 | 1.000 |
3.4. Analysis of factors of ULBP feature
In comparison with RIU-LBP feature, ULBP feature keeps the order of neighborhood coding and thus bears more direction information. Whether the analysis result with the RIU-LBP feature could be applicable to the ULBP features? Similar to the RIU-LBP, we perform MANOVA analysis to ULBP.
3.4.1. The significance and Interaction
Table 8 is the results for independent factors and their interactions. It shows that the four factors have significant effects independently for recognition rate. From large to small, the order of impact is respectively the image resolution, the blocking number, the sampling radius and the sampling density. The influence of sampling density is still the minimal. Compared with the RIULBP features, the image resolution is of the most notable effect for
Df | Sum Sq | Mean Sq | |||
2 | 0.0788 | 0.0394 | <0.001 | ||
2 | 0.0672 | 0.0336 | <0.001 | ||
2 | 0.0217 | 0.0108 | 90.517 | <0.001 | |
2 | 0.0107 | 0.0053 | 44.824 | <0.001 | |
4 | 0.0061 | 0.0015 | 12.680 | <0.001 | |
4 | 0.0043 | 0.00108 | 9.030 | <0.001 | |
4 | 0.0042 | 0.00106 | 8.856 | <0.001 | |
4 | 0.0018 | <0.001 | 3.862 | 0.008 | |
4 | 0.0011 | <0.001 | 2.321 | 0.070 | |
4 | 0.0007 | <0.001 | 1.575 | 0.196 | |
Residuals | 48 | 0.0057 | |||
Total | 80 | 0.2026 |
ULBP feature instead of the blocking number. But for ULBP feature, the
3.4.2. Analysis of levels of each single factor
Similarly, we also analyze the difference among various levels of factors for ULBP feature, and the results are summarized in Table 9-12.
Based on Table 9, there are significant differences between the three levels of blocking numbers, and the
Level | Mean | |||
0.735 | 1.000 | <0.001 | <0.001 | |
0.781 | <0.001 | 1.000 | 0.004 | |
0.805 | <0.001 | 0.004 | 1.000 |
As in Table 10, the three levels of sampling density are not significantly different. The average recognition rate is the highest when the sampling density is 8, which indicates that high sampling density is not necessary.
Level | Mean | 4 | 8 | 16 |
4 | 0.759 | 1.000 | 0.120 | 0.420 |
8 | 0.788 | 0.120 | 1.000 | 0.420 |
16 | 0.776 | 0.420 | 0.420 | 1.000 |
For sampling radius, there is no apparent difference between level 2 and 3. When the sampling radius is 2, the mean of recognition rate is the highest.
Level | Mean | 1 | 2 | 3 |
1 | 0.751 | 1.000 | 0.025 | 0.025 |
2 | 0.786 | 0.025 | 1.000 | 0.899 |
3 | 0.785 | 0.025 | 0.899 | 1.000 |
Lastly, for the most prominent factor, i.e. the image resolution, there is no prominent difference between 70 * 80 and 140 * 160.
Level | Mean | 35*40 | 70*80 | 140*160 |
35*40 | 0.730 | 1.000 | <0.001 | <0.001 |
70*80 | 0.791 | <0.001 | 1.000 | 0.410 |
140*160 | 0.800 | <0.001 | 0.410 | 1.000 |
3.5. More analysis on the blocking number
Based on the MANOVA analysis for both RIULBP and ULBP feature, we can conclude that the more the blocking number is, the higher the recognition rate will be. But should its value goes to the up-limit of 1 pixel per block? And what is the suitable blocking number? We do more experiments to analyze these problems for RIU-LBP features.
We pick two more levels of the blocking number, i.e.
(With sampling density 8, sampling radius 2 and image resolution 70*80)
We extend the experiments to 35*40 and 140*160 image resolution cases. The recognition rate of blocking number
Based on the analysis result of ULBP and RIU-LBP feature, we could take the following steps in setting the parameters of the four factors. Although different setting might be necessary for the specific applications, the basic rule is that more bits should be assigned to the blocking numbers and less for the sampling density. Moreover, the appropriate blocking number should be selected in consideration of the image resolution together, and then to choose the proper value for sampling density and sampling radius.
35*40 | 0.460 | 0.689 | 0.750 | 0.751 | N/A | |
70*80 | 0.567 | 0.743 | 0.817 | 0.830 | 0.784 | |
140*160 | 0.581 | 0.770 | 0.831 | 0.847 | 0.849 |
3.6. Summary
We comprehensively analyze the four factors of the ULBP and RIU-LBP features and many useful conclusions are drawn. Firstly, the blocking number is the main factor that influences the recognition rate, which indicates that the contribution of local features in face recognition is more important than the global features. Secondly, the effect of sampling density for the recognition rate is small, but it severely affects the feature dimension. At the same time, such results mean the feasibility of using low sampling density to acquire features of high recognition rate. In addition, the sampling density and the sampling radius decide the setting of the LBP operator, but they have much less obvious affect to the recognition rate compared with the blocking numbers and the image resolution. These conclusions demonstrate that the complex encoding of the LBP operator is not important in face recognition. Finally, the interactions between the factors are of less effect to recognition rate compared with the independent factors.
4. Fusion of multi-directional RIU-LBP
The difference between the ULBP feature and the RIU-LBP feature lies in the way of LBP coding. The latter totally abandons the directional information and hence of much lower dimension but of less precision in face recognition. However, if introducing the direction information into the RIU-LBP feature in a linear complexity, a new feature of much lower dimension could result in similar precision as the ULBP feature.
4.1. Multi-directional RIU-LBP
The dimension explosion of the LBP features is mainly aroused by the sampling density. The basic LBP operator adopts an ordered way to code the variation in each direction around one pixel, thus it has to cost
First, we split
For the two division settings shown in Fig.7, the dimension of
With such simple way of feature fusion, the
4.2. Performance analysis
We perform the experimental analysis to the multi-directional RIU-LBP feature with the same face database described in Section 2.3. We also compare the proposed multi-directional RIU-LBP feature
Though all three types of LBP features have adopted exactly the same neighborhood in LBP coding, the curves in Fig.8 and Fig.9 prove the promising application of the proposed multi-directional RIU-LBP features in comparison with the uniform LBP feature and the RIU-LBP
feature. The RIU-LBP feature abandons the direction information of intensity variation around pixel. The uniform LBP feature uses a complex ordered coding way which bears of course more direction information of intensity variation. However, both work no better than the proposed algorithm.
In one hand, the experiments reveal that the direction information of intensity variation is very important and useful in face recognition. In another hand, the proposed algorithm reserves such direction information through feature fusion. With much lower dimensional in comparison with the uniform LBP feature, the proposed algorithm gains very close and even better precision.
5. Conclusion
In this chapter, we first perform a thorough analysis of the four factors of the ULBP features and the RIU-LBP features. From a statistical point of view, we use MANOVA to study four factors the blocking numbers, the sampling density, the sampling radius and the image resolution that affect the recognition rate. Based on the analysis results, a multi-directional RIU-LBP, a modified LBP feature, is proposed through fusing the RIU-LBP features of various initial angels. Several conclusions are drawn as follows: 1) For the RIU-LBP features and the ULBP features, the impact of the blocking number is more important than the other factors. For example, for the RIU-LBP feature with constant values of other factors, when the blocking number changes from 7 *8 to 14 * 16, the recognition rate increased 9 percents. This result indicates that the accuracy of face recognition strongly relies on the localized LBP histograms. 2) The sampling density has little contribution to the recognition rate and high sampling density could not help to achieve high recognition rate. So complex LBP coding is not necessary. (3) The correlation between pairs of the factors is not important. (4) The multi-directional RIU-LBP feature preserve intensity variation around pixels in different directions at the cost of linearly increasing of feature dimension instead of the power 2 way of ULBP feature. Experiments not only show that the proposed scheme could result in better recognition rate than RIU-LBP feature after reserving the direction information in feature-level fusion, but also prove that with much lower dimension of features, the proposed multi-directional RIU-LBP feature could result in very close performance compared with the ULBP feature. In all, through the statistical analysis of the importance of four factors, an effective feature is proposed and the future directions to improve the LBP feature are presented.
Acknowledgments
This work is supported by the National Natural Science Foundation of China under Grant No.60605012, the Natural Science Foundation of Shanghai under Grant No.08ZR1408200, the Open Project Program of the National Laboratory of Pattern Recognition of China under Grant No.08-2-16 and the Shanghai Leading Academic Discipline Project under Grant No.J50103.
References
- 1.
Ahonen T. Hadid A. Pietik°ainen M. 2004 Face Recognition with Local Binary Patterns .469 481 ,978-3-54021-984-2 Prague, Czech Republic, May 11-14, 2004 - 2.
Ojala T. Pietikäinen M. Mäenpää T. 2002 Multiresolution Gray-scale and Rotation Invariant Texture Classification with Local Binary Patterns .24 7 (August 2002),971 987 ,0162-8828 - 3.
Zhang W. C. Shan S. G. Zhang H. M. Chen J. Chen X. L. Gao W. 2006 Histogram Sequence of Local Gabor Binary Pattern for Face Description and Identification . ,17 12 (December 2006),2508 2517 ,1000-9825 - 4.
Jin H. L. Liu Q. S. Lu H. Q. Tong X. F. 2004 Face Detection Using Improved LBP under Bayesian Framework.306 309 ,0-76952-244-0 Kong, China, December 18-20, 2004 - 5.
Zhang L. Chu R. F. Xiang S. M. Liao S. C. 2007 Face Detection Based on Multi-Block LBP Representation .11 18 ,978-3-54074-548-8 Seoul, Korea, August 27-29, 2007 - 6.
Liao S. C. . Zhu X. X. . Lei Z. . Zhang L. Li S. Z. 2007 Learning Multi-scale Block Local Binary Patterns for Face Recognition .828 837 ,978-3-54074-548-8 Seoul, Korea, August 27-29, 2007 - 7.
Zhao S. Gao Y. 2008 Establishing Point Correspondence Using Multidirectional Binary Pattern for Face Recognition .1 4 ,978-1-42442-174-9 Tampa, Florida, USA, December 8-11, 2008 - 8.
Yan S. C. Wang H. Tang X. O. 2007 Exploring Feature Descriptors for Face Recognition. ,I629 I632 ,1-42440-727-3 Hawaii, USA, April 15-20, 2007 - 9.
He L. H. Zou C. R. Zhao L. Hu D. 2005 An enhanced LBP feature based on facial expression recognition. 3300 3303 ,0-78038-741-4 China, September 1-4, 2005 - 10.
Wang W. Huang F. F. Li J. W. Feng H. L. 2009 Face Description and Recognition by LBP Pyramid .21 1 (January 2009),94 100 ,1003-9775 - 11.
Ojala T. Pietikinen M. Harwood D. 1996 A Comparative study of texture measures with classification based on featured distributions . Journal of ,29 1 (January 1996),51 59 0031-3203 - 12.
Ahonen T. Hadid A. Pietik°ainen M. 2006 Face Description with local binary patterns: Application to Face Recognition. Journal of ,28 12 (December 2006),2037 2041 ,0162-8828 - 13.
Chen C. J. 2008 Decision Level Fusion of Hybrid Local Features for Face Recognition. Proc. International Conference Neural Networks and ,199 204 ,978-1-42442-310-1 Najing, China, June 7-11, 2008 - 14.
Xie S. F. Shan S. G. Chen S. L. Meng X. Gao W. 2009 Learned local Gabor patterns for face representation and recognition . Journal of ,89 12 (December 2009),2333 2344 ,0165-1684 - 15.
Wang W. Huang F. F. Li J. W. Feng H. L. 2008 Face description and recognition using multi-scale LBP feature .16 4 (April 2008),696 705 ,0100-4924 X - 16.
Proc. Transmission & Distribution Conference and Exposition,Moitre D. Magnago F. Using M. A. N. O. V. A. Methodology in. a. Competitive Electric. Market under. Uncertainties 1 6 16 1-42440-287-5 Venezuela, August 15-18, 2006