Local Patterns for Face Recognition

Abstract

The main objective of the local pattern is to describe the image with high discriminative features so that the local pattern descriptors are more suitable for face recognition. The word “local” represents the measured image with the subregion and is the key in this chapter. Regardless of the techniques proposed, the local pattern is one of the most interesting areas in face recognition. The local facial descriptor is a local pattern that generates the descriptor by considering the subregion of an image. Techniques based on various combination methods from the local facial descriptors are not unusual. This chapter is concerned primarily to help the reader to develop a basic understanding of the local pattern descriptors and how they apply to face recognition. We begin to describe the outline of the local pattern in face recognition and its relative facial descriptors. Next, we give an introduction to the popular local patterns and establish examples to demonstrate the process of each method. To the end of this chapter, we conclude those methods with a discussion of issues related to the properties of the local patterns.

1. Introduction

Due to the intelligence security monitoring is more popular in recent years, the automatically recognizing face is needed for various visual surveillance systems, for example the accessing control system for personal or company to verify the legal/illegal people, policing system for identifying the thief and the robber who presents the illegal behavior in public or private space. To construct an efficient face recognition system, the facial descriptor with discriminated characteristic is required.

The facial descriptor refers to the process of extracting the discriminative features to represent a given face image. Numerous methodologies are proposed to recognize face and those can be classified as global and local facial descriptors. The global facial descriptor describes the facial characteristics with the whole face image, such as principal component analysis (PCA) [1, 2] and linear discriminant analysis (LDA) [3, 4]. PCA converts the global facial descriptor from high dimension to low dimension by using the linear transform methodology to reduce the computational cost. Linear discriminant analysis (LDA) also called the Fishers Linear Discriminant is similar to PCA, while it is a supervised methodology. Although the global facial descriptor can extract the principal component from the facial images, reduces the computational cost, and maintains the variance of the facial image, the performance is sensitive to the change of the environment, such as the change of light.

The flexibilities of the local facial descriptors are better than the global facial descriptors because they successfully and effectively represent the spatial structure information of an input image. A well local facial descriptor generates discriminative and robust features to achieve good recognition results with computational simplicity. In this chapter, we represent a number of approaches in the local facial descriptor including the local binary pattern (LBP), local derivation pattern (LDP), local tetra pattern (LTrP), local vector pattern (LVP) and local clustering pattern (LCP).

2. Local pattern descriptor

A local pattern considers the variations of subregion in an image, which is also called a micropattern. In this section, we introduce the basic and several popular techniques of local pattern descriptor for facial recognition.

2.1. Local binary pattern

Local binary pattern (LBP) [5] is designed to describe the texture in a local neighborhood is an invariant texture measure and has been various comparative studies, such as fingerprint recognition [6], face recognition [7], and license plate recognition [8]. The main characteristics of LBP are: (1) highly discriminative capability (2) and computational efficiency.

The basic LBP encodes the pixels of an image by thresholding $3\times 3$ neighborhood of each pixel with the given referenced pixel $X_c$ and concatenates the results to form a binary expression, as shown in Figure 1. The equation of basic LBP operator is formulated as follows:

$${\mathit{LBP}}_P\left(X_c\right)=\left\{f_1\left(I\left(X_p\right),I\left(X_c\right)\right)\right\}\left|{}_{p=1,2,\dots ,P}\right.$$ E1

Figure 1.

Example of 8-neighborhood surrounding a referenced pixel $X_c$ .

where $X_p,p=1,\dots ,8$ is a neighborhood of the referenced pixel $X_c$ in the local subregion of an image $I$ . $f_1\left(\cdot ,\cdot \right)$ is the coding scheme which decides the binary number of each neighborhood, called the threshold function and this can be expressed as

$$f_1\left(I\left(X_p\right),I\left(X_c\right)\right)=\left\{\begin{array}{l}1,\mathit{\text{if}}\left(I\left(X_p\right)-I\left(X_c\right)\right)\ge T_1\\ 0,\begin{array}{ll}& \end{array}\mathit{\text{else}}\begin{array}{ll}\begin{array}{ll}\begin{array}{ll}\begin{array}{ll}\begin{array}{ll}\begin{array}{l}\end{array}& \end{array}& \end{array}& \end{array}& \end{array}& \end{array}\begin{array}{l}\end{array}\end{array}\right.$$ E2

where $p$ is the index of the neighborhoods which is surrounding the referenced pixel $X_c$ . and $T_1$ is the threshold. $f_1\left(\cdot ,\cdot \right)$ represents the gradient variation between a given referenced pixel $X_c$ and its neighborhoods. In practical, the threshold $T_1$ can be set to $0$ , if $f_1\left(I\left(X_p\right),I\left(X_c\right)\right)=0$ , it means the neighborhoods $X_p$ have higher gradient information compared with referenced pixel $X_c$ . Figure 2 is an example of generating an LBP micropattern. Figure 2 demonstrates that LBP is generated by using Eqs. (1) and (2) from $X_1$ to $X_8$ and encodes the binary pattern of a give reference pixel $X_c$ as 10011111. Figure 3 demonstrates the spatial distribution of the example of LBP as shown in Figure 2 in one-dimensional. In Figure 3, the neighborhoods, which are encoded as 1, are arranged on the right of reference pixel $X_c$ , and the others are arranged on the left of reference pixel $X_c$ . The distance is the gradient variant between reference pixel and its neighborhoods as shown in Figure 3.

Figure 2.

Example of generating LBP for local region.

Figure 3.

Spatial distribution of example of LBP.

Furthermore, to address the problem of the textures at different scales, there are some followers which extend to use neighborhoods with various scales [9, 10]. To compare with basic LBP, the local neighborhoods are evenly spaced on a circle centered at the reference pixel $X_c$ , and the formulation of Eqs. (1) and (2) is re-formulated as follows:

$${\mathit{LBP}}_{P,R}\left(X_c\right)=\left\{f_1\left(I\left(X_{p,r}\right),I\left(X_c\right)\right)\right\}\left|{}_{p=1,2,\cdots ,P;r=1,\mathrm{..}R}\right.$$ E3

$$f_1\left(I\left(X_{p,R}\right),I\left(X_c\right)\right)=\left\{\begin{array}{l}1,\begin{array}{ll}& \end{array}\mathit{\text{if}}\left(I\left(X_{p,R}\right)-I\left(X_c\right)\right)\ge T_1\\ 0,\begin{array}{ll}& \end{array}\mathit{\text{else}}\begin{array}{ll}\begin{array}{ll}\begin{array}{ll}\begin{array}{ll}\begin{array}{ll}\begin{array}{l}\end{array}& \end{array}& \end{array}& \end{array}& \end{array}& \end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\end{array}\right.$$ E4

where $r$ is the radius between the referenced pixel $X_c$ and its neighborhood pixels $X_p$ . Figure 4 illustrates examples of circular neighborhoods with any radius and number of sampling points. The neighbor that does not fall in the center of a pixel is estimated by using bilinear interpolation.

Figure 4.

Circularly symmetric neighborhoods sets for different $\left(P,R\right)$ . (a) (P=4,R=1), (b) (P=8,R=1), (c) (P=8,R=2).

2.2. Local derivative pattern

LBP is a nondirectional first-order local derivative pattern of images and fails to extract more detailed information, such as the directions between neighborhoods and referenced pixel, and the high-order gradient information. Local derivative pattern (LDP) can be considered as an extension of LBP with directional high-order local derivative pattern [11]. To encode the $n^{\mathit{th}}$ -order LDP, the ${\left(n-1\right)}^{\mathit{th}}$ -order local derivative variations with various distinctive spatial relationships along $0^{\circ }$ , ${45}^{\circ }$ , ${90}^{\circ }$ , and ${135}^{\circ }$ directions are used. The first-order derivatives of the referenced pixel $X_c$ along $0^{\circ }$ , ${45}^{\circ }$ , ${90}^{\circ }$ , and ${135}^{\circ }$ directions can be written as

$$I_{0^{\circ }}^{\text{'}}=I\left(X_c\right)-I\left(X_4\right)$$ E5

$$I_{{45}^{\circ }}^{\text{'}}=I\left(X_c\right)-I\left(X_3\right)$$ E6

$$I_{{90}^{\circ }}^{\text{'}}=I\left(X_c\right)-I\left(X_2\right)$$ E7

$$I_{{135}^{\circ }}^{\text{'}}=I\left(X_c\right)-I\left(X_1\right)$$ E8

where $I$ is a given image, $X_c$ is the referenced pixel and $X_p$ , $p=1,2,3,4$ are the neighborhoods of $X_c$ as shown in Figure 1. Then, the second-order LDP can be encoded as,

$${\mathit{LDP}}_{\alpha }^2\left(X_c\right)=\left\{f_2(I_{\alpha }^{\text{'}}\left(X_c\right),I_{\alpha }^{\text{'}}\left(X_1\right)),f_2{I}_{\alpha }^{\text{'}}\left(X_c\right),I_{\alpha }^{\text{'}}\left(X_2\right),\cdots ,f_2{I}_{\alpha }^{\text{'}}\left(X_c\right),I_{\alpha }^{\text{'}}\left(X_p\right)\right\}\left|{}_{p=1,2,\cdots P}\right.$$ E9

where $\alpha$ is derivative direction at referenced pixel $X_c$ along $0^{\circ }$ , ${45}^{\circ }$ , ${90}^{\circ }$ , and ${135}^{\circ }$ directions and $f_2\left(\cdot ,\cdot \right)$ is the binary coding function which describes the spatial relationship between referenced pixel $X_c$ and its neighborhoods $X_p$ in various derivative directions, and that can be expressed as

$$f_2\left(I_{\alpha }^{\text{'}}\left(X_c\right),I_{\alpha }^{\text{'}}\left(X_p\right)\right)=\left\{\begin{array}{l}1,\begin{array}{ll}& \end{array}\mathit{\text{if}}\begin{array}{l}\end{array}{I}_{\alpha }^{\text{'}}\left(X_c\right)\cdot I_{\alpha }^{\text{'}}\left(X_p\right)\le 0\\ 0,\begin{array}{ll}& \end{array}\mathit{\text{if}}\begin{array}{l}\end{array}{I}_{\alpha }^{\text{'}}\left(X_c\right)\cdot I_{\alpha }^{\text{'}}\left(X_p\right)>0\end{array}\right.\begin{array}{ll}& \end{array}p=1,2,\cdots ,P$$ E10

The spatial relationship between two pixels includes the conditions of turning and monotonically increasing/decreasing and be coded as 1 and 0 in LDP, respectively.

Finally, the second-order LDP is defined as the concatenation of the four directional LDPs

$${\mathit{LDP}}^2\left(X_c\right)=\left\{{\mathit{LDP}}_{\alpha }^2\left(X_c\right)\left|\alpha =0^{\circ }, {45}^{\circ }{90}^{\circ }{135}^{\circ }\right.\right\}.$$ E11

The ${\left(n-1\right)}^{\mathit{th}}$ -order derivatives along $0^{\circ }$ , ${45}^{\circ }$ , ${90}^{\circ }$ , and ${135}^{\circ }$ directions are calculated by modifying Eqs. (5)–(8) and be re-formulated as

$$I_{0^{\circ }}^{n-1}=I_{0^{\circ }}^{n-2}\left(X_c\right)-I_{0^{\circ }}^{n-2}\left(X_{4,R}\right)$$ E12

$$I_{{45}^{\circ }}^{n-1}=I_{{45}^{\circ }}^{n-2}\left(X_c\right)-I_{{45}^{\circ }}^{n-2}\left(X_{3,R}\right)$$ E13

$$I_{{90}^{\circ }}^{n-1}=I_{{90}^{\circ }}^{n-2}\left(X_c\right)-I_{{90}^{\circ }}^{n-2}\left(X_{2,R}\right)$$ E14

$$I_{{135}^{\circ }}^{n-1}=I_{{135}^{\circ }}^{n-2}\left(X_c\right)-I_{{135}^{\circ }}^{n-2}\left(X_{1,R}\right)$$ E15

Then, the $n^{\mathit{th}}$ -order LDP, ${\mathit{LDP}}_{\alpha }^n\left(X_c\right)$ , in $\alpha$ derivative direction at referenced pixel $X_c$ is expressed as

$${\mathit{LDP}}_{\alpha }^n\left(X_c\right)=\left\{f_2(I_{\alpha }^{n-1}\left(X_c\right),I_{\alpha }^{n-1}\left(X_{p,R}\right))\right\}\left|p=1,2,\cdots ,P\right.;R=1,\alpha =0^{\circ },{45}^{\circ },{90}^{\circ },{135}^{\circ }$$ E16

An example of high-order derivative is shown in Figure 5. Figure 5(a) is the original value of image, Figure 5(b) is the first-order derivative in $0^{\circ }$ direction by using Eq. (5), and Figure 5(c) is the second-order derivative in $0^{\circ }$ direction by using Eq. (12) with the value in Figure 5(b).

Figure 5.

Example of high-order derivative in $0^{\circ }$ direction. (a) Original values (b) First-order (c) Second-order.

Figure 6 demonstrates an example to encode the second-order LDP in $0^{\circ }$ direction. To encode the second-order LDP, the results of first-order derivatives are needed. Taking the bit 1 as an example, the results of first-order derivatives of referenced pixel $X_c$ and the neighborhood $X_1$ are $\left(7-3\right)=4$ and $\left(4-9\right)=-5$ , respectively. The spatial relationship between neighborhood $X_1$ and referenced pixel $X_c$ is turning $\left(7-3\right)\times \left(4-9\right)=-20\le 0$ . Therefore, we encode the bit 1 as 1 by Eq. (10). Similarly, the spatial relationship between referenced pixel $X_c$ and neighborhoods pixels $X_p$ , $p=4,5,7,8$ presents the turning and be encoded as “1”. The reset of neighborhoods pixels is encoded as “0”. The second-order LDP in $0^{\circ }$ direction, ${\mathit{LDP}}_{P,R=1,\alpha =0^{\circ }}^2$ , is encoded as “10011011”. According to the same encoding process, the results of second-order LDP in ${45}^{\circ }$ , ${90}^{\circ }$ and ${135}^{\circ }$ are ${\mathit{LDP}}_{P,R=1,\alpha ={45}^{\circ }}^2=01110100$ , ${\mathit{LDP}}_{P,R=1,\alpha ={90}^{\circ }}^2=11100001$ , and ${\mathit{LDP}}_{P,R=1,\alpha ={135}^{\circ }}^2=10011101$ , respectively. Finally, ${\mathit{LDP}}_{P,R=1}^2=10011011011101001110000110011101$ with 32-bit is generated by concatenating the four 8-bit LDPs with various derivative directions.

Figure 6.

Example to encode second-order LDP in $0^{\circ }$ direction.

Figure 7 demonstrates the spatial distribution of example of LDP in $0^{\circ }$ direction in one-dimensional. In Figure 7, the evaluation results of LDP in $0^{\circ }$ direction are normalized into the region of $\left[-8\begin{array}{l}\end{array}8\right]$ , the neighborhoods that are encoded as 1 are arranged on the left of 0, and the others are arranged on the right of 0. The distance is the magnitude of gradient variant between reference pixel $X_c$ and its neighborhoods.

Figure 7.

Spatial distribution of example of LDP in $0^{\circ }$ direction.

2.3. Local tetra pattern

Local tetra pattern (LTrP) [12] adopts the concepts of LBP and LDP which extends the spatial relationship from one-dimensional to two-dimensional. LTrP uses two high-order derivative directions with four distinct values to encode the micropattern for extract more discriminative information. The $n^{\mathit{th}}$ -order LTrP is derivative from ${\left(n-1\right)}^{\mathit{th}}$ -order derivatives along $0^{\circ }$ and ${90}^{\circ }$ which can be written as

$$I_{0^{\circ }}^{n-1}=I_{0^{\circ }}^{n-2}\left(X_{h,R}\right)-I_{0^{\circ }}^{n-2}\left(X_c\right)$$ E17

$$I_{{90}^{\circ }}^{n-1}=I_{{90}^{\circ }}^{n-2}\left(X_{v,R}\right)-I_{{90}^{\circ }}^{n-2}\left(X_c\right)$$ E18

where $X_c$ is the referenced pixel, $X_{h.R}$ and $X_{v.R}$ horizontal and vertical neighborhoods of referenced pixel $X_c$ , respectively; $R$ is the distance between reference pixel $X_c$ and its neighborhood; $I_{0^{\circ }}^{n-2}\left(\cdot \right)$ , $I_{{90}^{\circ }}^{n-2}\left(\cdot \right)$ are the $\left(n-2\right)$ -order derivatives in $0^{\circ }$ and ${90}^{\circ }$ directions, respectively; $I_{0^{\circ }}^{n-1}\left(\cdot \right)$ , and $I_{{90}^{\circ }}^{n-1}\left(\cdot \right)$ are the $\left(n-1\right)$ -order derivatives in $0^{\circ }$ and ${90}^{\circ }$ directions, respectively. Then, the direction of the referenced pixel $X_c$ can be expressed as the quadrant representation and be defined as

$$I_{\mathit{Dir}}^{n-1}\left(X_c\right)=\left\{\begin{array}{l}1,\begin{array}{l}\end{array}{I}_{0^0}^{n-1}\left(X_c\right)\ge 0\begin{array}{l}\end{array}\mathit{\text{and}}\begin{array}{l}\end{array}{I}_{{90}^0}^{n-1}\left(X_c\right)\ge 0\\ 2,\begin{array}{l}\end{array}{I}_{0^0}^{n-1}\left(X_c\right)<0\begin{array}{l}\end{array}\mathit{\text{and}}\begin{array}{l}\end{array}{I}_{{90}^0}^{n-1}\left(X_c\right)\ge 0\\ 3,\begin{array}{l}\end{array}{I}_{0^0}^{n-1}\left(X_c\right)<0\begin{array}{l}\end{array}\mathit{\text{and}}\begin{array}{l}\end{array}{I}_{{90}^0}^{n-1}\left(X_c\right)<0\\ 4,\begin{array}{l}\end{array}{I}_{0^0}^{n-1}\left(X_c\right)\ge 0\begin{array}{l}\end{array}\mathit{\text{and}}\begin{array}{l}\end{array}{I}_{{90}^0}^{n-1}\left(X_c\right)<0\end{array}\right.$$ E19

where $I_{\mathit{Dir}}^{n-1}\left(X_c\right)$ describes the direction of the referenced pixel $X_c$ along $0^{\circ }$ and ${90}^{\circ }$ directions with quadrant. Then, the $n^{\mathit{th}}$ -order tetra pattern of referenced pixel $X_c$ , ${\mathit{\text{LTrP}}}_{P,R}^n\left(X_c\right)$ , is encoded as

$${\mathit{\text{LTrP}}}_{P,R}^n\left(X_c\right)=\left\{f_3(I_{\mathit{Dir}}^{n-1}\left(X_{p,R}\right),I_{\mathit{Dir}}^{n-1}\left(X_c\right))\right\}\left|{}_{p=1,2,\cdots ,P;R=1}\right.$$ E20

where $f_3\left(\cdot ,\cdot \right)$ is the coding function which describes the referenced pixel $X_c$ with four quadrants and be written as

$$f_3\left(I_{\mathit{Dir}}^{n-1}\left(X_{p,R}\right),I_{\mathit{Dir}}^{n-1}\left(X_c\right)\right)=\left\{\begin{array}{ll}{I}_{\mathit{Dir}}^{n-1}\left(X_{p,R}\right),& \mathit{\text{if}} I_{\mathit{Dir}}^{n-1}\left(X_{p,R}\right)\ne I_{\mathit{Dir}}^{n-1}\left(X_c\right)\\ 0& \mathit{\text{else}}\end{array}\right.$$ E21

Figure 8 illustrates the coding scheme of Eq. (21), if the quadrant of the referenced pixel $X_c$ is as same as its neighborhood, the corresponding bit of tetra pattern is assigned to be “0”, otherwise, the bit is assigned to be the same as the neighborhood. Then, the tetra patterns are decomposed into three binary patterns as follows:

$$\begin{array}{l}{\mathit{\text{LTrP}}}_{P,R}^n\left|{}_{{\displaystyle \overline{\mathit{Dir}}}}\right.=f_4\left({\mathit{\text{LTrP}}}_{P,R}^n\left(X_c\right)\right)\left|{}_{{\displaystyle \overline{\mathit{Dir}}},p=1,2,\cdots P}\right.\\ f_4\left({\mathit{\text{LTrP}}}_{P,R}^n\left(X_c\right)\right)\left|{}_{{\displaystyle \overline{\mathit{Dir}}}}\right.=\left\{\begin{array}{l}1,\begin{array}{l}\end{array}\mathit{\text{if}}\begin{array}{l}\end{array}{\mathit{\text{LTrP}}}_{P,R}^n\left(X_c\right)={\displaystyle \overline{\mathit{Dir}}}\\ 0,\begin{array}{l}\end{array}\mathit{\text{else}}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\end{array}\right.\end{array}$$ E22

Figure 8.

Illustration of coding LTrP micropattern.

where $\overline{\mathit{Dir}}$ contains four quadrants except the quadrant of the referenced pixel $X_c$ and $f_4\left(\cdot ,\cdot \right)$ is a coding function to generate the three binary patterns. Similarly, the three tetra patterns are encoded according to the abovementioned procedure for the rest directions of the referenced pixel. Therefore, the four tetra patterns with 12 8-bit binary patterns are generated. Moreover, the 13th 8-bit binary pattern is considered which is the magnitudes of horizontal and vertical first-order derivatives and be calculated by the following equation,

$${\mathit{\text{LTrP}}}_{P,M}={\left.f_5\left(M\left(X_p\right)-M\left(X_c\right)\right)\right|}_{p=1,2,\cdots P}$$ E23

$$M\left(X_i\right)=\sqrt{{\left(I_{0^{\circ }}^1\left(X_p\right)\right)}^2+{\left(I_{{90}^{\circ }}^1\left(X_p\right)\right)}^2}$$ E24

$$f_5\left(M\left(X_p\right)-M\left(X_c\right)\right)=\left\{\begin{array}{l}1,\begin{array}{l}\end{array}\mathit{\text{if}}\begin{array}{l}\end{array}M\left(X_p\right)-M\left(X_c\right)\ge 0\\ 0,\begin{array}{l}\end{array}\begin{array}{l}\end{array}\mathit{\text{else}}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\end{array}\right.$$ E25

where $M\left(X_p\right)$ is the magnitudes of horizontal and vertical first-order derivatives and $f_5\left(\cdot ,\cdot \right)$ is a coding function to generate the binary patterns of the magnitude. Figure 9 demonstrates an example of the second-order LTrP which takes a subregion as shown in Figure 5(a) as an example. The quadrant of referenced pixel $X_c$ is 4, which is assigned by using Eq. (18) with the first-order derivatives in $0^{\circ }$ and ${90}^{\circ }$ directions. Similarly, the quadrants of each neighborhood of referenced pixel $X_c$ are 2, 1, 1, 3, 3, 3, 3, 2, respectively. We take the neighborhood pixel $X_1$ as an example, the quadrants of $X_c$ and $X_1$ are 4 and 2, respectively, which is not the same. Thus, the corresponding bit of the LTrP is assigned to be “2” as shown in Figure 9. Similarly, the remaining bits of the LTrP are encoded by using the same procedure and the complete LTrP can be expressed as ${\mathit{\text{LTrP}}}_{P,R}^2=21133332$ . Then, the tetra pattern is decomposed into three 8-bit binary pattern according to Eq. (22). To generate the first 8-bit binary pattern, the tetra pattern with symbol “1” is set to be “1”, and the rest symbols of tetra pattern are set to be “0”. Then, we obtain the first 8-bit binary pattern “01100000”. Repeatedly, we generate the other 8-bit binary patterns “10000001” and “00011110” by considering the tetra pattern values “2” and” 3″, respectively. Finally, the 12 8-bit binary pattern is obtained by concatenating the rest tetra patterns with three directions (1, 2, and 3) of referenced pixel. The additional binary pattern is obtained from the magnitude and be encoded as “10111110″.

Figure 9.

Example of coding second-order LTrP micropattern.

2.4. Local vector pattern

Local vector pattern (LVP) [13] is inspired by local binary pattern (LBP) which is sample and intuitive. To compare with LBP and LDP, LVP further considers the neighborhood relationship with various distances from different directions and the relationship between various derivative directions.

LVP is a micropattern in high-order derivative space which considers the direction value in encoding procedure, as shown in Figure 10. The derivative direction vector of the referenced pixel $X_c$ , $V_{\beta ,D}\left(X_c\right)$ , with various directions and distance are formulated as

$$V_{\beta ,D}\left(X_c\right)=I\left(X_{\beta ,D}\right)-I\left(X_c\right)$$ E26

Figure 10.

Neighborhoods pixels of $V_{\beta ,D}\left(X_c\right)$ with various distance along different directions.

where $I$ is a local subregion of an image, $\beta$ is the index of angle (direction), and $D$ is the distance between referenced pixel $X_c$ and its neighbors. $V_{\beta ,D}\left(X_c\right)$ is the derivative vector of the referenced pixel $X_c$ along the $\beta$ direction with $D$ distance. Figure 10 demonstrates the distance between $X_c$ and its neighbors are 1, 2, and 3 and are marked with green, blue and yellow, respectively.

The LVP, ${\mathit{LVP}}_{\beta }\left(X_c\right)$ , in $\beta$ derivative direction at referenced pixel $X_c$ is encoded as

$${\mathit{LVP}}_{P,R,\beta }\left(X_c\right)=\left\{f_5\left(V_{\beta ,D}\left(G_{p,R}\right),V_{\beta +{45}^{\circ },D}\left(G_{p,R}\right),V_{\beta ,D}\left(G_c\right),V_{\beta +{45}^{\circ },D}\left(G_c\right)\right)\right\}\left|{}_{p=1,2,\cdots P,R=1}\right.$$ E27

where $f_5\left(\cdot ,\cdot \right)$ is the coding function which can be formulated as

$$\begin{array}{l}{f}_5\left(V_{\beta ,D}\left(G_{p,R}\right),V_{\beta +{45}^{\circ },D}\left(G_{p,R}\right),V_{\beta ,D}\left(G_c\right),V_{\beta +{45}^{\circ },D}\left(G_c\right)\right)=\\ \left\{\begin{array}{l}1,\begin{array}{ll}& \mathit{\text{if}}\begin{array}{l}\end{array}{V}_{\beta +{45}^{\circ },D}\left(G_{p,R}\right)-\left(\frac{V_{\beta +{45}^{\circ },D}\left(G_c\right)}{V_{\beta ,D}\left(G_c\right)}\times V_{\beta ,D}\left(G_{p,R}\right)\right)\end{array}\ge 0\\ 0,\begin{array}{ll}& \mathit{\text{else}}\end{array}\begin{array}{llll}\begin{array}{llll}\begin{array}{llll}\begin{array}{llll}\begin{array}{llll}& & & \end{array}& & & \end{array}& & & \end{array}& & & \end{array}& & & \end{array}\end{array}\right.\end{array}$$ E28

Finally, the LVP of referenced pixel $X_c$ is defined as the four 8-bit binary patterns, as shown in the following,

$${\mathit{LVP}}_{P,R}\left(X_c\right)=\left\{{\mathit{LVP}}_{P,R,\beta }\left(X_c\right)\left|\beta =0^{\circ },{45}^{\circ }{90}^{\circ }{135}^{\circ }\right.\right\}$$ E29

To extend the discriminative of 2D spatial structures, LVP integrates four pairwise directions ( $0^{\circ }-{45}^{\circ },{45}^{\circ }-{90}^{\circ },{90}^{\circ }-{135}^{\circ },{135}^{\circ }-0^{\circ }$ ) of vector to form a 32-bit binary pattern for each referenced pixel $X_c$ .

The coding function of LVP is a weight vector of dynamic linear decision function which is a comparative space transform (CST) and addresses the two-class problem in pattern recognition. The dynamic linear decision function, $\mathit{CST}\left(X_{p,R}\right)$ , can be formulated as

$$\mathit{CST}\left(X_{p,R}\right)=w{\left(X_c\right)}^T\cdot v\left(X_{p,R}\right)$$ E30

where $w{\left(X_c\right)}^T$ and $v\left(X_{p,R}\right)$ are the weight vector and pairwise direction value of the neighborhoods which are surrounded by referenced pixel $X_c$ in two different directions. The formulations of $w\left(\cdot \right)$ and $v\left(\cdot \right)$ can be expressed as,

$$w\left(X_c\right)=\left(1,\frac{V_{\beta +{45}^{\circ },D}\left(X_c\right)}{V_{\beta ,D}\left(X_c\right)}\right)$$ E31

$$v\left(X_{p,R}\right)={\left(V_{\beta +{45}^{\circ },D}\left(X_{p,R}\right),V_{\beta ,D}\left(X_{p,R}\right)\right)}^T$$ E32

where the first term of $w\left(\cdot \right)$ is to describe the original value of neighborhood pixel $X_{p,R}$ at $\left(\beta +{45}^{\circ }\right)$ direction and the second term is the transform ratio which compares the derivative value of the neighborhood $X_{p,R}$ in $\beta$ direction to that of in $\left(\beta +{45}^{\circ }\right)$ direction surrounds around the referenced pixel $X_c$ . $v\left(\cdot \right)$ is the augmented pattern which presents the pairwise direction values of vector of neighborhood pixel $X_{p,R}$ . Then, Eq. (30) can be rewritten as,

$$\mathit{CST}\left(X_{p,R}\right)=w{\left(X_c\right)}^T\cdot v\left(X_{p,R}\right)=V_{\beta +{45}^{\circ },D}\left(X_{p,R}\right)-\frac{V_{\beta +{45}^{\circ },D}\left(X_c\right)}{V_{\beta ,D}\left(X_c\right)}\times V_{\beta ,D}\left(X_{p,R}\right)$$ E33

We take the example of the local subregion of an image as shown in Figure 5(a) to illustrate the encoding process of generating first-order LVP, as shown in Figures 11 and 12. Figure 11 illustrates the first-order LVP of the referenced pixel $X_c=7$ in $\beta =0^{\circ }$ direction. In Figure 11, we calculate the pairwise derivative direction vector of the referenced pixel $X_c$ to form the 2D spatial structures, as shown in Figure 12. In Figure 12, the pairwise derivative direction vectors $V_{\beta ,D}\left(X_c\right)$ and $V_{\beta +{45}^{\circ },D}\left(X_c\right)$ are indicated as x- and y-axis, respectively, in which, $\beta =0^{\circ }$ and $D=1$ . The first-order derivative direction value of referenced pixel $X_c$ and its neighborhoods in directions $\beta =0^{\circ }$ and $\beta +{45}^{\circ }={45}^{\circ }$ are shown in Figure 13. Then, we calculate the transform ratio $\frac{V_{\beta +{45}^{\circ },D}\left(X_c\right)}{V_{\beta ,D}\left(X_c\right)}=\frac{-1}{4}=-0.25$ which is used to transform the $\beta$ -direction value of the neighborhoods to comparative space $\beta +{45}^{\circ }$ -direction. The CST value of neighborhood pixel $X_1=4$ of referenced pixel $X_c=7$ is evaluated according to Eq. (33) ( $\mathit{CST}\left(X_{1,1}\right)=V_{{45}^{\circ },1}\left(X_{1,1}\right)-\frac{V_{{45}^{\circ },D}\left(X_c\right)}{V_{0^{\circ },D}\left(X_c\right)}\times V_{0^{\circ },1}\left(X_{1,1}\right)=-1-\frac{-1}{4}\times -5=-2.25$ ). Then, the first corresponding bit of the 8-bit binary codes of ${\mathit{LVP}}_{P,R,0^{\circ }}\left(X_c\right)=01100100$ is encoded by using sign function. Similarly, the rest of LVPs with various pairwise directions are ${\mathit{LVP}}_{P,R,{45}^{\circ }}\left(X_c\right)=10101011$ , ${\mathit{LVP}}_{P,R,{90}^{\circ }}\left(X_c\right)=11100001$ , and ${\mathit{LVP}}_{P,R,{135}^{\circ }}\left(X_c\right)=00101101$ . The four binary pattern LVPs are concatenated to generate ${\mathit{LVP}}_{P,R}\left(X_c\right)=01100100101010111110000100101101$ .

Figure 11.

Example of first-order LVP in $\beta =0^{\circ }$ direction.

Figure 12.

Comparative space transform (CST) in encoding first-order LVP in $\beta =0^{\circ }$ direction.

Figure 13.

The first-order derivative direction value. (a) $\beta =0^{\circ }$ ; (b) $\beta +{45}^{\circ }={45}^{\circ }$ .

2.5. Local clustering pattern

Local clustering pattern (LCP) [14] is designed to solve the problems in face recognition: (1) to reduce feature length with low computational cost and (2) to enhance the accuracy for face recognition. To generate the local clustering pattern, four phases have to be considered: (1) to generate the local derivative variations with various directions; (2) to project the local derivative variations with various directions on the pairwise combinatorial directions in the rectangular coordinate system; (3) to transform the coordinate from the rectangular coordinate system into the polar coordinate system; and (4) encoding the facial descriptor which is local clustering pattern, as a micropattern for each pixel by applying the clustering algorithm. The details are described in the following subsections: local clustering pattern (LCP) and coding scheme.

2.5.1. Local clustering pattern

Taken a subregion image $I\left(X\right)$ as an example, as shown in Figure 1, in which $X_c$ is the referenced pixel and $X_p,p=1,\cdots ,8$ are the adjacent pixels around $X_c$ . LCP firstly generates the first-order derivatives of $X_c$ , $I_{\alpha }^{\text{'}}\left(X_c\right)$ , in various directions and can be written as

$$I_{\alpha }^{\text{'}}\left(X_c\right)=I_{\alpha }\left(X_p\right)-I_{\alpha }\left(X_c\right)$$ E34

where $\alpha$ is the derivative direction including $0^{\circ }$ , ${45}^{\circ }$ , ${90}^{\circ }$ , and ${135}^{\circ }$ directions. Then, the LCP is generated by integrating the pairwise combinatorial directions of the derivative variations, $0^{\circ }-{45}^{\circ }$ , ${45}^{\circ }-{90}^{\circ }$ , ${90}^{\circ }-{135}^{\circ }$ and ${135}^{\circ }-0^{\circ }$ , in polar coordinate system. The generation of LCP in pairwise combinatorial direction can be expressed as,

$${\mathit{LCP}}_{\alpha }\left(X_c\right)=\sum _{n=1}^N{f}_{r,\theta }\left(I_{\gamma ,D}^{\text{'}}\left(X_p\right),I_{\gamma ,D}^{\text{'}}\left(X_c\right)\right)\times 2^{n-1}\left|{}_{\gamma \in \left\{\alpha ,\alpha +{45}^{\circ }\right\},N=8}\right.$$ E35

where $f_{r,\theta }\left(\cdot ,\cdot \right)$ is the coding scheme and $D=1,2,3$ is the distance between referenced pixel $X_c$ and its adjacent pixels $X_p$ , as shown in Figure 10. The coding scheme is executed in the polar coordinate system, and the formula can be formally defined as follows,

$$f_{r,\theta }\left(I_{\gamma ,D}^{\text{'}}\left(X_p\right),I_{\gamma ,D}^{\text{'}}\left(X_c\right)\right)\left|{}_{\gamma \in \left\{\alpha ,\alpha +{45}^{\circ }\right\}}\right.=\left\{\begin{array}{ll}0,& \mathit{\text{if}} I_{\gamma ,D}^{\text{'}}\left(X_p\right)\begin{array}{l}\end{array}\mathit{\text{and}}\begin{array}{l}\end{array}{I}_{\gamma ,D}^{\text{'}}\left(X_c\right)\in C_i\\ 1,& \mathit{\text{else}}\end{array}\right.$$ E36

where $C_i$ is the cluster center. Finally, the LCP at referenced pixel $X_c$ , $\mathit{LCP}\left(X_c\right)$ , is combinatorial of the four 8-bit binary patterns LCPs, and can be formally as

$$\mathit{LCP}\left(X_c\right)=\left\{{\mathit{LCP}}_{\beta }\left(X_c\right)\right\}\left|{}_{\beta =0^{\circ },{45}^{\circ },{90}^{\circ },{135}^{\circ }}\right..$$ E37

2.5.2. Coding scheme

In this subsection, we further discuss the coding scheme in LCP which is considered as the problem of classification. The coding scheme of LCP is executed in the polar coordinate system based on the characteristics of the derivative variations in the pairwise combinatorial directions.

First, four combinations of the derivative variations in the pairwise directions are utilized in LCP, including $0^{\circ }-{45}^{\circ }$ , ${45}^{\circ }-{90}^{\circ }$ , ${90}^{\circ }-{135}^{\circ }$ , and ${135}^{\circ }-0^{\circ }$ . The coordinate of the pairwise combinatorial directions of the derivative variations is in the rectangular coordinate system (RCS). To consider the magnitude and orientation between pairwise combinatorial directions, the coordinate is transformed from the rectangular coordinate system (RCS) into the polar coordinate system (PCS) by calculating the magnitude $\left(m\right)$ and orientation $\left(\theta \right)$ for each pair directions of derivative variations. The magnitude $\left(m\right)$ and orientation $\left(\theta \right)$ of $X_p$ are calculated as

$$m_{\gamma }\left(X_p\right)=\sqrt{{\left(I_{\gamma ,D}^{\text{'}}\left(X_p\right)\right)}^2+{\left(I_{\gamma +{45}^{\circ },D}^{\text{'}}\left(X_p\right)\right)}^2}\left|{}_{\gamma \in \alpha }\right.$$ E38

$${\theta }_{\gamma }\left(X_p\right)=\text{arctan}\frac{I_{\gamma +{45}^{\circ },D}^{\text{'}}\left(X_p\right)}{I_{\gamma ,D}^{\text{'}}\left(X_p\right)}\left|{}_{\gamma \in \alpha }\right.$$ E39

where $\frac{-\pi }{2}<{\theta }_{\gamma }<\frac{\pi }{2}$ is normalized to $0^{\circ }˜{360}^{\circ }$ .

The feature vectors $\mathbf{v}$ are $m_{\gamma }$ and ${\theta }_{\gamma }$ coordinate in the polar coordinate system and can be written as

$$\mathbf{v}={\left[m_{\gamma }\left(X_n\right),{\theta }_{\gamma }\left(X_n\right)\right]}^T$$ E40

where $\gamma \in \alpha$ and $n=1˜9$ are the pixels in the subregion image $I\left(X\right)$ including the referenced pixels and its adjacent pixel in the polar coordinate system.

LCP is ensemble of several decisions from the results of clustering. Each clustering result is considered as a problem of a two-class case, whose center vector $\mathbf{C}$ is written as

$$\mathbf{C}={\left[C_1,C_2\right]}^T$$ E41

where $C_1$ and $C_2$ are the two-class centers, in which $C_1$ is also the center of $X_c$ . To classify the feature vectors $\mathbf{v}$ in sub-image $I$ , we randomly initialize two-class centers $\mathbf{C}$ and adopt the k-means clustering algorithm for classification. The clustering procedure is repeated T times to find the cluster two-class centers ${\mathbf{C}}_{\mathbf{i}}$ that have the highest probability $P\left({\mathbf{C}}_{\mathbf{i}}\left|\mathbf{v}\right.\right)$ .

The adjacent pixels of the reference pixel $X_c$ are encoded as the following equation,

$$C\left(m_{\gamma }\left(X_p\right),{\theta }_{\gamma }\left(X_p\right)\right)\left|{}_{\gamma \in \alpha }\right.=\left\{\begin{array}{l}0,\begin{array}{l}\end{array}\mathit{\text{if}}\begin{array}{l}\end{array}{X}_p\in C_1\\ 1,\begin{array}{l}\end{array}\begin{array}{l}\end{array}\mathit{\text{else}}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\begin{array}{l}\end{array}\end{array}\right.$$ E42

where $C_1$ is the cluster center which includes $X_c$ .

2.5.3. Example

The local subregion of an image as shown in Figure 5(a) is taken as an example to illustrate the encoding process of generating first-order LCP, as shown in Figure 14. First, LCP calculates the first-order derivatives along $0^{\circ }$ and ${45}^{\circ }$ directions as shown in Figure 13. Then, the coordinates of referenced pixel $X_c$ and its neighborhoods are translated from rectangular coordinate system (RCS) into polar coordinate system (PCS). The results of coordinate translation are shown in Figure 14. After that, the clustering technique is applied to find the centers of two clusters, as indicated as the hollow rectangles with red and purple colors, respectively. Only $X_5$ belongs to the second class, the rest pixels belong to the first class. Then, the corresponding bit of the 8-bit binary codes of ${\mathit{LCP}}_{P,R,0^{\circ }}\left(X_c\right)=00001000$ .

Figure 14.

Example of the LCP takes Figure 5(a) as an example (the derivative variations along $0^{\circ }$ and ${45}^{\circ }$ ).

3. Comparison

In this section, we discuss the characteristics of the local patterns descriptors as mentioned. The local binary pattern (LBP) generates the local facial descriptor by comparing the gray value between referenced pixel and its adjacent pixels for each pixel in the face image. The texture information, such as spots, lines and corners, in the images is extracted. Although LBP considers the spatial information to generate the local facial descriptor, it omits the directional information and is sensitivity when light is slightly changed.

The local derivation pattern (LDP) analyzes the turnings between referenced pixel and its neighborhoods from the derivative values. The derivative values with four directions are considered to generate the local facial descriptor in the high-order derivative space. However, the turnings between referenced pixel and its neighbors are discussed in the same derivative direction.

The local tetra pattern (LTrP) utilized the two-dimensional distribution with derivative values in four quadrants to describe the texture information and that can extract more discriminative information. Although LTrP considers the derivative variations with two dimensions, there exist two problems: (1) the dimension of facial descriptor and (2) the sensitivity of the features. To compare with LBP and LDP, the dimension of facial descriptor of LTrP is high. The features of LTrP in the four quadrants of the rectangular (or Cartesian) coordinate system are altered when illumination is changed.

The local vector pattern (LVP) designs the comparative space transform (CST) and that is associated with the pairwise directions of vector to encode the micropatterns. Comparing LVP with LBP, LDP, and LTrP, LVP not only successfully extracts distinctive information but also reduces the feature length. However, its computational cost is higher than LBP and LDP.

The local clustering pattern (LCP) derivatives the local variations with multidirections and that are integrated to form the pairwise combinatorial direction. To generate the discriminative local pattern, the features of local derivative variations are transformed into the polar coordinate system by generating the characteristics of magnitude ( $m$ ) and orientation ( $\theta$ ). LCP generates the discriminative local clustering pattern with low-order derivative space and low computational cost which are stable in the process of face recognition. The summarization of each method is demonstrated in Table 1.

Methods	Information used	Distribution of coding scheme	Feature Length
LBP	Original values	One dimensional	$8$
LDP	High-order derivative values	One dimensional	$8\times 4$
LTrP	High-order derivative values	Two dimensional	$8\times 13$
LVP	High-order derivative values	Two dimensional	$8\times 4$
LCP	High-order derivative values	Two dimensional	$8\times 4$

Table 1.

Comparison of various methods.

In Table 1, we analyze these methods with three indicators: (1) information used, (2) distribution of coding scheme, and (3) feature length. The indicator of the information used presents the information which is used in facial descriptor generation. LBP uses the original values such, as gray value; LDP considers the single high-order derivative values; LTrP uses both horizontal and vertical high-order derivative values; LVP uses the high-order derivative values and be described as the vector representation; the high-order derivative values are utilized in clustering process of LCP.

The distribution of coding scheme is to present how many directions of used information are considered in coding at each time. LBP and LDP generate the micropattern by considering a single direction at each time, for example, LDP generates the micropatterns of one direction at a time and then integrates the results of each direction to form the facial descriptor; LTrP considers two-direction information, horizontal and vertical, when coding; LVP and LCP use the pairwise combinatorial directions.

The feature length is to demonstrate the feature length of each micropattern. LBP considers eight neighborhoods and its feature length is 8; LDP further considers four directions including $0^{\circ }$ , ${45}^{\circ }$ , ${90}^{\circ }$ , and ${135}^{\circ }$ , its feature length is $8\times 4=32$ bits, in which “8” is the number of neighborhood of referenced pixel and “4” is the number of derivative directions; the feature length of LTrP is $8\times 13=8\times \left(3\times 4+1\right)=104$ bits, where “8” is the number of neighborhood of referenced pixel, “3” is the number of the binary patterns in a tetra pattern, “4” is the number of the tetra patterns, and “1” number of the binary pattern which is obtained from the magnitude; the feature length of LVP and LCP is $8\times 4=32$ bits, where “8” is the number of neighborhood of referenced pixel, and “4” is the number of pairwise combinatorial directions.

4. Summary

The principal object of this chapter is to present the local pattern descriptors for understanding and accessing the facial descriptor in face recognition. The concept of local pattern is sample and intuitive, and the extended techniques of the basic local pattern are widely used in various areas. A partial listing of local pattern descriptors includes local binary pattern (LBP), local derivative pattern (LDP), local tetra patterns (LTrP), local vector pattern (LVP) and local clustering pattern (LCP) are widely applied to variety of image processing problems such as object detection, object recognition, image retrieval, fingerprint recognition, character recognition, face recognition, license plate recognition. Since it is impractical to cover all the approaches of local pattern descriptor in a single chapter, the basic and popular techniques included are chosen for their value in introducing and clarifying fundamental concepts in the field.