Facial Emotion Recognition Feature Extraction: A Survey

Abstract

Facial emotion recognition is a process based on facial expression to automatically recognize individual emotion expression. Automatic recognition refers to creating computer systems that are able to simulate human natural ability of detection, analysis, and determination of emotion by facial expression. Human natural recognition uses various points of observation to make decision or conclusion on emotion expressed by the present person in front. Facial features efficiently extracted aid in improving the classifier performance and application efficiency. Many feature extraction methods based on shape, texture, and other local features are proposed in the literature, and this chapter will review them. This chapter will survey some recent and formal feature expression methods from video and image products and classify them according to their efficiency and application.

1. Introduction

Recent research in Computer Science is more driven by constructing a solution smarter product (hardware and software). Computing is becoming ubiquitous and pervasive, with human at the center. Devices are attaining more ability to average human intelligent actions and interactions. Human beings are basically emotional and affective. They express their emotion in many ways and they require emotion expression in their natural interaction no matter what they interact with (human, machine, or nature) [1]. Together with the objective of having human-centered digital solutions, affective computing aims to endow computers with the ability of sensing, recognition, and expressing emotion [2, 3].

Automatic emotion recognition is one of the recent research trends in Artificial Intelligence, especially in the field of Machine Learning. Based on scientific ground, emotion recognition is about a mapping from feature space to emotion descriptors or label space. This feature space is built from different identified cues extracted from an original element, which is the subject of study [4]. These cues seem to help distinguish two different situations or cases during a classification task and minimize differences within elements of the same class.

In order to recognize human affect state automatically, some of the steps studied and worked on consist of data collection, data preprocessing, feature extraction, and emotion recognition, as represented in Figure 1.

Figure 1.

Emotion recognition processes.

Data collection as the first step in automatic recognition consists of reassembling raw data from different sensors according to the work at hand, that is, the modality to study or its application [5]. This chapter is about acquiring a video with a recognizable human face and expressing emotion. Collected data are tarnished with many noises and unwanted details that need to be removed [6, 7]. Data preprocessing generally involves data cleaning, normalization (or standardization), and missing data processing. Cleaned data serve as basic space for extraction of main features, which convey more information for an expected pattern. The feature extraction step consists of representing data in a digital form to present to a filter. It draws out the values, which are more informative and nonredundant for a future easy learning process and quick generalization.

It is very important to extract an effective facial representation from all considered facial images for any effective facial expression recognition system. The resulting representation should preserve indispensable information possessing distinguished contrast power and stability, which lessens within-class variations of expressions whereas expands between-class variations [8]. Extracted features aid in emotion classification [9]. At this level, two procedures are done: training a classifier and testing it. Emotion classification is the last step, resulting in the process of classification of a new case into its category using the trained classifier. Classification performance is greatly subjective to the quality of information contained in the expression representations [10]. Thus, the step of feature extraction has a great influence on the classification outcome.

This chapter contains a global point of view on feature extraction, and different types of facial expression recognition feature extraction methods are detailed in the following chapters.

2. Feature extraction

Features are also called attributes or input variables. Feature extraction consists in draw out the feature relating to the modality. The precision of the most relevant feature for extraction in emotion recognition research is still an open topic [11, 12, 13]. However, the often-studied modalities are face expression, speech, body motion, hand gestures, and physiological signals. They are the representation of the data and can be in binary form, categorical, discrete, or continuous. Feature extraction is subdivided into two processes, that is, features construction and feature selection.

2.1 Feature construction

The feature construction consists of determining the good data representation, according to the domain specifications and measurements availability [13]. The extracted features are proper to modalities and an interesting task. In emotion recognition, feature extraction focuses on cues that convey better the affect expression. Actually, referring to human natural emotion or intention expression and perception, there are many studies that have proved some frequently observed units to convey useful information for emotion categorization.

Table 1 represents a summary of the frequently observed and studied units for feature extraction, according to the recording methods or the study of interest.

Modality		Units
		Basic	Intermediates
Face expression		Eyes, eyebrows, nose, mouth	Action Units, pupil
Speech		Linguistic	Word, multi-word, phrases, sentences, documents
		Paralinguistic	Pitch intensity of utterances, bandwidth, duration, voice quality, Mel frequency Cepstral coefficients (MFCC)
Body	Head gestures	Head position
		Head movement
	Hand Gestures	Shape
		Motion; keystrokes
	Body motion	Spinal column	Neck, chest and abdomen
		DOF body	Symmetrical arms
		Body center mass	Movement of body center of mass
		Joints	Degree of joint rotation
Physiologic		Hearth; Brain; Limbs; Blood	Electrocardiogram (ECG); breath rate; electro-dermal activity (EDA); electro-myogram (EMG)

Table 1.

Modality and extracted features.

Table 1 presents a summary of the list of the combinations and cues considered according to modality in the study. Modality means any human body parts that can be used to express emotion. In affect detection, some basic units encompass other intermediate units. This list relates to the most cited elements in the literature. The modalities are defined as the main objectively observed entities, which convey most information about emotion expression. Basic units are the small elements of the whole modality and can stand for an independent study [14, 15, 16, 17, 18]. Intermediate units are more detailed than the basic units. These unity measurements produce multiple feature values, which constitute the vector feature of the modality [19]. The features construction can be manually processed and/or complemented by automatic feature construction methods [20, 21].

Recently, the research in feature extraction techniques has ended up by proposing some automatic feature extraction tools and algorithms. Some examples are given in Table 2.

Toolkit	Modality	Feature extracted/functionality	Brief description
PRAAT [22]	Audio	Duration, F0, Range, Movement, Slope, Energy features	PRAAT (a system for doing phonetics)
FEELTRACE [23, 24]	Audio	Labeling	Allowing the emotional dynamics of speech episodes to be examined.
OpenEAR [25]	Audio	Signal Energy, Loudness, Mel-/Bark-/Octave-Spectra, MFCC, PLP-CC	openEAR provides efficient (audio) feature extraction.
OpenSMILE [26]	Audio	Signal Energy, Loudness, Formants, Mel-/Bark-/Octave-Spectra, MFCC, PLP-CC, Pitch, Voice quality (Jitter, Shimmer), LPC, Line Spectral, Pairs(LSP), Spectral, Shape description	It is an open source toolkit, for feature extraction in machine learning and data mining [27]
EyesWeb [28]	Body	Quantity of motion, cue, Contraction index of the body, velocity, Acceleration, fluidity of the hand’s, barycenter	Open software for extended Multimodal Interaction.
Luxand FSDK 1.7	Face	Action units	Facial recognition software [29]
ANVIL [30]	Audio	Annotation tool in a multimodal dialog	Free for research purposes [31]

Table 2.

Some automatic feature extraction tools.

In Table 2, the toolkit column corresponds to the name given to the tool or algorithm in the literature. The modality column means the channel conveying needed information. The listed tools are mostly available online and free of charge, and are compatible with the most popular platforms, such as Windows, Linux, and Macintosh. The references within the table are the work that has utilized the tool or the reports of the authors.

The step of feature construction builds a feature set which is full of some unnecessary or superfluous data. In order to clean that feature set, a feature selection is necessary to prepare a proper dataset useful in the learning process.

2.2 Feature selection

The step of features selection mainly aims to select some features, which are more relevant and explanatory to the study in view. The feature construction creates thousands of features that require an important amount of storage and slows down the training process, the curse of dimensionality. The feature selection uses a data reduction method to eliminate irrelevant and redundant information to a sufficient minimum dimension. The main objective is to get attributes with a large distance between classes and small variance in the same class [7].

The step of feature extraction success affects the training process, recognition accuracy, and application efficiency. It constitutes a subject of study on its own, and it is the subject of the present work, extensions are limited to facial feature extraction.

3. Facial expression feature extraction

Facial feature extraction is all about exactly localizing different features on the face, which include the detection of eyes, brows, mouth, nose, chin, etc. [32]. Facial features are often subdivided into appearance or transient features and geometric or intransient features [10, 33, 34]. Local appearance-based methods extract appearance changes of the face or a region of the face, while geometric features express the shape of the facial components (eyebrows, eyes, mouth, etc.) and the location of prominent points of the face (comers of the eyes, mouth, etc.).

3.1 Geometric feature extraction

3.1.1 Facial feature points (FFP)

The shape and location-related features could be achieved using Active Appearance Methods (AAM) [35]. It has been used to label 68 facial feature points (FFPs) as related in the work of Wu et al. [36]. Facial feature points are visible marks in facial images or points that constitute interesting part of images, such as eye centers, nose tip, mouth corners, and other salient facial points. They are often used as a reference or for measurement. Figure 2 represents an example of the FFPs extracted based on the AAM alignment and the corresponding animation parameters, and this figure is extracted from the work of Wu et al. [36].

Figure 2.

Example of facial feature points labeled using AAM alignment [36].

Facial feature points are also referred to as facial points, fiducial facial points, or facial landmarks [37]. The points shown in Figure 2 can be concatenated to represent a shape x = (x₁, · · ·, x_N, y₁, · · ·, y_N)^T, where (x_i, y_i) denotes the location of the i-th point and N is the number of points (here Figure 2, N equals 68). The FFPs are grouped into Facial Animation Parameters (FAPs), to facilitate the normalization among people. Every FAP limits a segment of a key distance on the face. The AAM was initially developed in the work of Cootes and Taylor [35], and has presented strong promise in multiple technologies of facial recognition technologies, including in recognizing emotions by its ability to both aid in beginning face-search algorithms and feature extraction based on texture and shape [38].

3.1.2 Facial affective coding systems (FACS)

Other works consider the Facial Affective Coding System (FACS) and define the Active Unities (AUs) as the facial muscle action [39, 40]. Facial action unit research studies the movement of facial muscles [41] and describes facial movement changes. Based on the work of Ekman Paul and Friesen [42], Facial Action Coding System (FACS) contributes as one of the most representative methods for facial expression application in measurement technology. Action units can precisely extract facial expressions, but they are less applied in facial expression recognition because of their exact positioning. Figure 3 represents some examples of Action Unities.

Figure 3.

Examples of action Unity description and muscles involved.

In Figure 3, the examples display the considered action unities detected on facial images. Those action units are randomly chosen for illustration. The description is about facial muscle movement or portrayal. Muscles indicate the action done on the facial muscles or the whole head. The emotion expression corresponds to an ascertained combination of some specific action unities, and Table 3 represents some examples of possible combinations, their description in facial muscles, and the corresponding emotion expression.

Action Unities combination	Description	Emotion
4 + 5 + 7 + 23	Brow Lowerer, Upper Lid Raiser, Lid Tightener, Lip Tightener	Anger
9 + 15 + 16	Nose Wrinkler, Lip Corner Depressor, Lower Lip Depressor	Disgust
1 + 2 + 4 + 5 + 7 + 20 + 26	Inner Brow Raiser, Outer Brow Raiser, Brow Lowerer, Upper Lid Raiser, Lid Tightener, Lip Stretcher, Jaw Drop	Fear
6 + 12	Cheek Raiser, Lip Corner Puller	Happiness/Joy
1 + 4 + 15	Inner Brow Raiser, Brow Lowerer, Lip Corner Depressor	Sadness
1 + 2 + 5 + 26	Inner Brow Raiser, Outer Brow Raiser, Upper Lid Raiser, Jaw Drop	Surprise

Table 3.

Example of action unities combination for emotion analysis.

In Table 3, the combinations of Action Unities are referred to the work of the visual book of group iMOTIONS. For more details, we refer to the above-mentioned review [36] and the work in Refs. [39, 40, 41, 42] and references therein.

3.2 Appearance-based features

Local appearance descriptors in the literature are mostly the LBPs (Local Binary Pattern) and its derived, the Local Direction Number pattern (LDN) and the Edge-Oriented Histogram. Local appearance-feature-based methods are used because of their close descriptor of the appearance.

3.2.1 Local binary pattern (LBP)

Local Binary Pattern (LBP) [43] method is a texture operator mostly used in computer vision and image processing applications, such as in object detection, object tracking, face recognition, and fingerprint matching [44, 45, 46]. It is a good operator for real time and very high frame rate applications. The LBP computes features for each image pixel; therefore, real-time extraction of LBP features requires considerable computational performance. It was proposed for a texture analysis [29], and it is insensitive to illumination changes and has an extension to rotation invariant [31].

An LBP feature is a binary vector obtained from a neighborhood around the current image pixel. The basic LBP operator is the 3*3 neighborhood pixels, which is called LBP 8, 1, that is, there are nine pixels with one center and eight neighborhood pixels. The value of the LBP feature is the result of the thresholding of every pixel’s luminance against the center pixel’s luminance. It is equal to 1 if the difference is positive and to 0 otherwise. The resultant binary number is computed by concatenating all the above binary codes in a clockwise direction, beginning from the top-left one, as shown in Figure 4, and the corresponding decimal value is used for labeling [45]. The obtained numbers are known as Local Binary Patterns or LBP codes.

Figure 4.

LBP operator.

The basic operator of 3*3 neighborhoods is small to capture dominant features with large-scale structures. Later on, Ojala et al. [47] proposed an advanced operator, which is proficient to deal with texture at different scales by using neighborhoods of different sizes. A set of sampling points is evenly spaced on a circle centered at the current pixel to label and define a local neighborhood. A bilinear interpolation permits interpolation of the points that do not fall within the pixels, thus allowing to use a radius of any size and to have any number of sampling points in the neighborhood. Some examples are illustrated in Figure 5.

Figure 5.

Examples of the extended LBP operator.

Figure 5 represents an example of LBP extended operator with the circular (8, 1), (16, 2), and (24, 3) neighborhoods.

Given a pixel at (x_c, y_c), for an extended LBP _{(P, R)} operator with P sampling points neighborhood on a circle of radius R, the LBP can be computed as follows in decimal form:

$${\mathit{LBP}}_{\left(P,R\right)}\left(x_c,y_c\right)=\sum _{P=0}^{P-1}s\left(i_P-i_c\right)2^P$$E1

where i_c and i_p are gray-level values of the central pixel and its neighborhood, the P is the number of surrounding pixels in the circle neighborhood with a radius R, and the function is defined as follows:

$$s\left(x\right)=\left\{\begin{array}{lll}1& \mathit{\text{if}}& x\ge 0\\ 0& \mathit{\text{if}}& x<0\end{array}\right.$$E2

The LBP _{(P, R)} operator produces 2^P different output values, corresponding to 2^P different binary patterns formed by P pixels in the neighborhood. That makes the extended LBP sensitive to image rotation, and in order to deal with it, a rotation invariant LBP was proposed and is computed as follows:

$${\mathit{LBP}}_{P,R}^{\mathit{ri}}=min\left\{\mathit{ROR}\left({\mathit{LBP}}_{P,R},i\right)|i=0,1,\dots ,P-1\right\}$$E3

where $\mathit{ROR}\left(u,i\right)$ executes a circular by bit right shift on the P-bit number u*i times. This operator computes occurrence statistics of individual rotation invariant patterns corresponding to certain micro-features in the image. It is a good operator for real time and very high frame rate applications.

LBP is invariant against monotonic gray-scale variations and has extensions to rotation invariant texture analysis. In the work of Ojala et al. [47], it was shown that there are patterns containing more information than others do and they were called “uniform patterns” denoted ${\mathit{LBP}}_{\left(P,R\right)}^{U2}$. In fact, it is possible to use a subset of 2^P binary pattern to represent the image’s texture. Uniform local binary patterns are the patterns containing at most two bitwise transitions from 0 to 1 or vice versa when the corresponding bit string is considered circular. For example,

•00000000 (0 transitions).

•01110000 (2 transitions).

•11,001,111 (2 transitions).

•11,001,001 (4 transitions).

•01010011 (6 transitions).

In natural images, LBP is uniform. The uniform value can be found using the equation below:

$${\mathit{LBP}}_{P,R}^{u2}=\left\{\begin{array}{cc}\sum _{p=0}^{P-1}s\left(i_p-i_0\right),& U\left({\mathit{LBP}}_{P,R}\right)\le 2\\ P\left(P-1\right)+2& \mathit{\text{otherwise}}\end{array}\right.$$E4

where

$$U\left({\mathit{LBP}}_{P,r}\right)=\left|s\left(i_{P-1}-i_c\right)-s\left(i_0-i_c\right)\right|+\sum _{p=1}^P\left|s\left(i_P-i_c\right)-s\left(i_{p-1}-i_c\right)\right|$$E5

If U ≤ 2, it is a uniform LBP otherwise it is nonuniform LBP. The LBP space dimension is reduced from 2^P to P*(P-1) +2 output values. Figure 6 represents an example of uniform and nonuniform patterns.

Figure 6.

Uniform and nonuniform patterns.

However, there have been different improvements in the LBP operator performance, such as improvement of its discriminative capability [48, 49, 50, 51, 52, 53], enhancement of its robustness [54, 55], selection of its neighborhood [56, 57, 58], extension to 3D data [59, 60, 61], and combination with other approaches [62, 63, 64, 65]. For more details, we refer to the survey done by Huang et al. [29].

3.2.2 Local directional numbers pattern (LDN)

A Local Directional Numbers Pattern (LDN) is proposed in the work of Rivera et al. [66]. It is a face descriptor that enables to acquire structural information and the intensity variations of the face texture. LDN descriptor extracts features by analysis of all eight (08) directions at every pixel position with a compass mask and generates a code from the analysis of its directional information. From all directions, the top positive and top negative directions are chosen to return a significant descriptor for different textures with similar structural patterns.

Local Directional Number Pattern (LDN) is a six-bit binary code. The resulting feature describes the local primitives, including different types of curves, corners, and junctions, more stably and more informative. They allow making differences in intensity changes in the texture. Figure 7 represents an example of LDN code computation, and it is proposed in the work of Rivera et al. [66].

Figure 7.

LDN coding.

The produced code represents information on the texture structure and intensity transitions of each pixel of the input images. The LDN descriptor permits to use of the information of the entire neighborhood, instead of using sparse points. In the coding scheme, LDN code is generated by analyzing the edge response of each mask, representing edge significance in its respective direction, and combining the dominant directional numbers.

Edge responses are not equally important; a high negative or high positive value signals a prominent dark or bright area. The encoding of these outstanding areas is based on the sign information, the top positive directional number represents the three most significant bits in the code and the top negative the three least significant bits. The masks are shown in Figure 8; they take names of basic and secondary directions. The code is defined as:

Figure 8.

Kirsch edge response masks.

$${\mathit{LDN}}_{\left(x,y\right)}=8i_{x,y}+j_{x,y}$$E6

where (x, y) is the central pixel of the neighborhood to encode, and ix,y and jx,y are directional number maximum positive and minimum negative responses, respectively, which are defined by:

$$\begin{array}{c}{i}_{x,y}=\underset{i}{argmax}\left\{{\mathrm{\Pi }}^i\left(x,y\right)\left|0\le i\le 7\right.\right\}\\ j_{x,y}=\underset{j}{argmin}\left\{{\mathrm{\Pi }}^j\left(x,y\right)\left|0\le j\le 7\right.\right\}\end{array}$$E7

where Πⁱ is the convolution of the original image, I, and the i^th mask, defined by Πⁱ = I*Mⁱ.

This approach allows us to distinguish intensity changes (e.g., from bright to dark and vice versa) in the texture that otherwise will be missed most evident directions descriptor uses the information of the whole neighborhood, it does not use sparse points for its computation as it is for LBP. LDN translates the directional information of the face’s textures (i.e., the texture’s structure) in a compact way, producing a more discriminative code.

3.2.3 Edge orientation histogram

Edge Orientation Histogram (EOH) engenders a feature set extracted based on the gradient of the pixels that correspond to edges of an image. It is used as a descriptor in classification or detection tasks. These descriptors rely on the abundance of the information of edge and are invariant to global illumination [67, 68, 69]. The edge is computed by filtering the gray-scale image using the Sobel operator. Five operators provide information about the strength of the gradient in five particular directions, as represented in Figure 9.

Figure 9.

Sobel mask for five directions [70].

Figure 9 represents the Sobel mask for five directions; in (a) it is the vertical direction, (b) it is the horizontal direction, (c) and (d) are the diagonals directions, and (e) it is the non-direction case. The gradient pixels are classified into β images corresponding to β orientation ranges; they are also designated as bins. Therefore, a pixel in bin k_n∈β contains its gradient magnitude if its orientation is inside β’s range, otherwise it is null. Integral images are now used to store the accumulation image of each of the edge bins. Figure 10 represents the Edge Orientation Histogram.

Figure 10.

Edge orientation histogram [70].

Though these two feature extraction approaches are mostly present in the literature but there are other works that considered hybrid approach [71]. A hybrid method means to at the same time use of appearance features and other shape features to make them complementary. Mixing these two types of features will improve classifier performance.

3.3 Feature extraction method classification

The facial expression recognition rate is more influenced by the basic features used for classifier training. From different works on facial feature extraction research, there are two main categories of feature extraction methods as mentioned above: geometric based and appearance based. In this work, we propose Table 4 for a classification.

Feature category	Feature details	Techniques	References	Applications
Geometric based	FFP	Active Appearance Model (AAM)	Ratliff & Patterson [38]	Texture and Shape
		Active Shape Model (ASM)	Iqtait et al. [72]	Shape
	FACS	Holistic spatial analysis based on PCA, Feature-based approach and Facial motion analysis	Tian et al. [39]	Action Units
		Convolutional Experts Constrained; Local Model (CE-CLM) and Histograms of Oriented Gradients (HOG	Yang et al. [40]	Geometric and appearance features
Appearance based	LBP	Improved LBP (Mean LBP)	Jin et al. [48],	Effects of central pixels
			Bai et al. [49]
		Hamming LBP	Yang and Wang [50]	Decrease of error rate caused by noise disturbances
		Extended LBP	Huang et al. [51]	Deals with variations of illumination
		Completed LBP	Guo et al. [53]	Better texture classification for rotation invariant
		Local Ternary Patterns	Tan and Triggs [54]	Discriminant and less sensitive to noise in uniform regions
		Soft LBP	Ahonen and Pietikäinen [55]	Robust to noise and output continuous according to input
		Elongated LBP	Liao and Chung [56]	New feature Average Maximum Distance Gradient Magnitude (AMDGM)
		FMulti-Block LBP	Liao and Si [57]	More robust and consider integral image
		Three/Four Patch LBP	Wolf et al. [58]	Improves multi-option identification and same/not-same classification
		3D LBP	Fehr [59]	Texture analysis in 3D
		Volume LBP	Zhao and Pietikäinen [61]	Combines motion and appearance
		LBP and SIFT	Heikkilä et al. [63]	Tolerance to lighting changes, robustness on even image areas, and computational efficiency
		LBP and Gabor wavelet	Zhang et al. [73]	No need training procedure to build the face model
			He et al. [62]
		LBP Histogram Fourier	Ahonen et al. [65]	Rotation invariant image descriptor
	EOH	Haar wavelet and EOH	Gerónimo et al. [67]	Object change in cluttered environments
		EOH for smile	Timotius and Setyawan [69]	Discriminate lip to depict a smile
	LDN	LDN basic	Rivera et al. [66]	Directional information of the face textures
		LDPv	Kabir et al. [10]	texture and contrast information of facial components

Table 4.

Classification of different feature extraction methods.

From this classification in Table 4, there are mainly two categories of feature extraction: appearance based and geometric based. Different cited methods or techniques are used for facial expression feature extraction as well as other feature extraction-related work [68]. Among geometric feature extraction, the active appearance model is mostly used combined with the principal component analysis method to reduce the vector dimension for efficient application in real time. Among the appearance-based feature extraction, the local-based pattern algorithm is the mostly found in the literature and highly expended.

In recent work, in view of collecting enough features to enhance facial expression recognition rate by including more details, researchers propose hybrid methods [71, 72].

4. Conclusions

Automatic facial emotion recognition is a recent research trend that is applied in many areas, such as security, health, education, and social interaction. Facial feature extraction is one of the crucial steps in order to get a good and quick classifier at the end. In view of getting a performant classifier firstly, facial feature representation has to distinguish different individuals well and at the same time tolerate that there can be minor variation within-class members. It should be easy to be extracted from the basic facial images to speed up further processing; all that demands is that the final sample space must stay in a low dimensional space to reduce classification complexity.

This work pictures different methods used in facial feature extraction and their best usage. It can serve as a reference and guide to researchers in facial expression recognition. Hereby, cited methods are mainly applied to 2D images and but works considering 3D mage are also related. Actually, as devices are getting smarter and averaging natural perception, it is a judiciary that the corresponding software development follows.