Feature Extraction for Emotion Recognition: A Review

Abstract

With the increasing role of Artificial Intelligence and ubiquitous computing paradigms, a stage has arrived where human being and machine is interacting seamlessly. However, the users may face issues while interacting with these systems. A more simple and reliable interaction with machines will be possible by recognizing user’s emotions. In order to develop a system that can respond effectively to user’s emotions can be modeled by utilizing the electroencephalogram (EEG) as a bio-signal sensor. The emotion recognition plays a vital role in the area of Human Computer Interaction (HCI) and Brain Computer Interaction (BCI) to provide good interaction between brain and machine. The emotion recognition comprises of three major phases feature extraction, feature selection and classifiers. The present chapter provides an overview of feature extraction techniques utilized by researchers in frequency domain analysis, time domain analysis and time-frequency domain analysis. The chapter also discusses the process, issues and challenges for feature extraction in EEG, the application area of the EEG.

1. Introduction

The world in the verge of paradigm change in the field of Intelligent Systems: moving from an era in which people control gadgets to one in which autonomous devices, capable of self-management and aware of their environmental and situational context [1]. As per the definition of Ubiquitous Computing conceived by Mark Weiser [2], the world is approaching to a level of automation and computing where human and computers are interacting with each other naturally without the awareness of users. Ironically, one of the major issue is growing complexity with this paradigm shift is that user find it difficult to interact with such systems [3]. As a result, it’s crucial to identify all potential interaction modalities and organize them according to problem domain. For example personalized interaction is highly required in computer- mediated interaction like virtual reality to maintain user’s interest and engagement with the cognitive activity. Task engagement includes both the user’s cognitive activity and engagement but it also requires an understanding of user’s emotional transfer. Therefore, the physiological computing system can be used to provide insights into the cognitive and emotional processes involved in completing tasks [4]. In particular, the display of level of brain activity by processing EEG signals may be of benefits when integrated with input modality [5]. ER can be performed with the help of physiological and non-physiological signals [6].

Facial expressions, speech, voice, actions etc., are the non-physiological signals that could not contribute very precisely for ER [7]. EEG signals have proved strong implications in finding emotions states [8]. The classification accuracy for emotion recognition is higher for EEG signals and the model can also the changes in mood too [9].

2. Broad area description

With the advancement of technology HCI is providing the support to BCI. Providing machines with the capacity to understand and discover the various emotional states of users can be of essential significance for the next generation. Endowing digital devices with logical reasoning abilities about user affective context will provide the facility to detect the present state of feelings. Such as signs of feeling low, frustrated, fear and allow the machine to react in a more intellectual and empathetic way [10]. As a result, this collectively along with other HCI traits like consistency, flexibility, and usability may give the base to produce more clever and more adaptive interface [11]. Now a day’s various input modalities have been utilized to gather the input data for emotion recognition. The very first of them is audiovisual based communication along with eye gazed, facial expression, speech evaluation, etc. The second physiological measure is sensor based signals along with EEG signals; galvanic skin response and electrocardiogram can also be used [12].

3. What is emotion?

The emotion can be the affective aspect of consciousness or mental reaction or can be states a strong feeling towards a particular object and it can have some associated physiological and behavioral changes in voice, expression or mood, etc. The signals of emotions can be categorized into two categories physiological signals and non-physiological signals [13].

The physiological reactions can be the increment or decrement the value of EEG signals, body temperature, heartbeats, blood pressure, breathing rate, etc. the studies states the more impact of physiological reactions on women in comparison with the men [13]. The non-physiological reactions include the expressions, action, voice etc. Here physiological signals are difficult to ignore during data acquisition stage and provides more accurate result [14].

The researchers have proposed various emotion models based on discrete model and dimension model theory [6]. The theory proposed by P. Ekman has six primary emotional states surprise, happiness, fear, anger, disgust, and sadness [15]. The other complex emotions are secondary one that is the composition of these basic emotions. P. Lang proposed a dimensional model called valence- arousal model [16]. The model maps emotions into two-dimensional space where valence represents positive and negative conditions and arousal represents the intensity of human emotions. This model maps different emotions at same place, which results in difficulty to distinguish. The model has been represented in Figure 1. To overcome this issue, A. Mehrabian has proposed model by adding dominance dimension to the valence-arousal model [18]. This model is called Pleasure-Arousal-Dominance (PAD) model.

Figure 1.

Arousal valence based emotion model [17].

4. Characteristics of EEG signals

The EEG different frequency bands are related to conscious human activities [19]. Based on the frequency of EEG signals, the signals are divided into five categories, alpha, beta, gamma, theta and delta.

Delta signal has frequency of range 0.5–4 Hz and occurs in the state of unconsciousness such as dreamless, deep sleep. Theta signals occur with frequency of 4–8 Hz and appear in the state of sub consciousness like sleepiness, dreaming etc. Alpha waves arise at frequency 8–13 Hz frequency when in consciousness human is in relaxed state. Alpha waves have higher oscillatory energy in neutral and negative emotions than beta and gamma waves.

Beta waves occur when human mind is highly concentrated and active with frequency 13–30 Hz. The active state of mind can be dictated by taking average power ratio of beta and alpha waves. Gamma waves occur at very high frequency greater than 30 Hz and show the hyperactivity of brain. As the results suggest, that emotional EEG is more evoked in lower frequency band in comparison to higher frequency band. Similarly negative emotions have higher intensity and wider distribution than positive ones [20].

5. Features of EEG signals

EEG signal is a weak physiological signal, which is highly utilized by researchers for emotion recognition due to the high accuracy of results [6]. For emotion recognition, the features of EEG signals categorized in categories spatial domain features, time domain features, frequency domain features and time-frequency domain features.

5.1 Spatial domain features

Spatial analysis is distribution of electrical signals at different regions of mind during acquisition of EEG signals. Different regions of brain respond in a different manner to emotions [21], considering how spectral, spatial and temporal aspects complement each other. To consider both spatial-temporal and spatial-spectral characteristics parallel streams are created.

5.2 Time domain features

EEG signals are often recorded in the form of time domain which is statistical in nature. Time domain analysis is commonly performed by using histogram analysis or statistical methods [6]. However the time-domain features include EEG signals with less information loss. But due to complex form of EEG signals, there is no standard unified method for analysis. So the analysts need to be rich in knowledge and experience.

5.3 Frequency domain features

Frequency domain features are obtained after converting the original time domain signal using Fourier Transformation. The aim of frequency domain is to find the frequency information of signals along with power characteristics of various frequency bands.

5.4 Time-frequency domain features

The time-domain and frequency domain signals are merged to examine EEG signals more accurately. As convergence process of signals from time domain to frequency domain does not lost the time information.

Time- frequency analysis has the ability to completely reflect the distinctive information in EEG signals. Continuous wavelet transforms (CWT) and discrete wavelet transform (DWT) are the two primary forms of wavelet transformation, which separate the low-frequency component of the signals. It is more advantageous than the conventional approaches for dealing nonlinear and non-stationary signals.

6. Methodology

The Emotion Recognition using EEG signals process is comprises of four phases as shown in Figure 2.

1. EEG Signal Acquisition—The collection of EEG signals can be divided into two methods—invasive and non-invasive. Here, the data collected using invasive method has higher signal to noise ration while non-invasive has utilization in BCI. The collection of EEG signals can be divided into two methods- invasive and non-invasive. Here, the data collected using invasive method has higher signal to noise ration while non-invasive has utilization in BCI. The first part comprises of four ways to collect EEG signals [22]; the first technique is electroencephalogram i.e. is graph obtained by recording and amplifying the brain’s signals. The signals are collected with the help of electrodes which are fixed in scalp. The second method is Electrocorticogram (ECoG). In ECoG, the recording is performed by surgically implanted electrodes. ECoG provides better spatial resolution and accuracy. Another method of signal acquisition is Functional Magnetic Resonance Imaging (fMRI), is a neuroimaging technique. fMRI reflect the oxygen saturation and blood flow level through the measurement of MR signals. Depth electrode is the fourth method of EEG signal detection. Deep Brain Simulation (DBS) is performed by placing electrodes at specific regions of brain to deliver current.

2. Pre-processing—the pre-processing phase comprises of removal of artifacts, thresholding of the output, amplifying the signals, edge detection and signal averaging etc. [23].

3. Feature Extraction—the feature selection phase is used to find a feature vector. Here, feature denotes a distinctive or characteristic measurement from a segment of pattern which plays a useful part in classification. For linear analysis of one-dimensional signals for either frequency or time domain, the various method has been discussed below:

1. Fast Fourier Transformation (FFT) method

   The fast Fourier Transformation (FFT) is a simple and fast way of Discrete Fourier Transformation (DFT). FFT utilized to filter signals from the time domain to the frequency domain. FET is good for stationary signals. However the overall EEG signals are not stationary in nature but for particular band it can be utilized.

   Welch’s method is one of the methods for transforming EEG signals using Fourier transformation [24]. The data sequencing is applied to data windowing, where data sequence x_i (n) and Fourier transformation is represented as

   $$x_i\left(n\right)=\left(\frac{1}{L}\right)\sum _{k=1}^{L}X\left(k\right){{\omega }_L}^{-\left(n-1\right)\left(k-1\right)}$$E1
   $$X\left(k\right)=\sum _{n=1}^{L}x\left(n\right){{\omega }_L}^{\left(n-1\right)\left(k-1\right)}$$E2

2. Wavelet Transformation (WT) method

   WT plays a vital role in the field of diagnostic and recognition, which compresses the time-varying signal, into few parameters that represents the nature of signal [25]. The model uses time-frequency domain with a variable size windows to provide more flexibility to the systems. WT can be categorized in two ways continuous WT and discrete WT.

3. Eigenvectors

   The eigenvector methods are utilized to calculate signals frequency and power from artifact ruled measurements. The essence of the techniques is to correlate the corrupted signals using Eigen decomposition [26]. The techniques that employed eigenvector are MUSIC method, Pisarenko’s method and minimum norm method.

4. Time-frequency distribution

   The Time-frequency domain is utilized with the stationary and noiseless signal, that’s why windowing process is required for pre-processing. The various methods are used for TFD model like Short-time Fourier Transform (STFT), wavelet packet transformation (WPT) and Hilbert-Huang transformation (HHT).

5. Autoregressive method

   Autoregressive (AR) method is advantageous for short data segment analysis. It also limits the loss of spectral leakage and provides better frequency resolution. Yule-Walker method and Burg’s method are used in AR model for spectral estimation.

Figure 2.

Emotion classification process for EEG signals [6].

The comparison of various models along with their advantages and disadvantages has been tabulated below in Table 1.

Method name	Advantages	Disadvantages	Analysis method	Suitability
Fast Fourier transform	Good tool for stationary signal processing It is more appropriate for narrowband signal, such as sine wave It has an enhanced speed over virtually all other available methods in real-time applications	Weakness in analyzing nonstationary signals such as EEG It does not have good spectral estimation and cannot be employed for analysis of short EEG signals FFT cannot reveal the localized spikes and complexes that are typical among epileptic seizures in EEG signals FFT suffers from large noise sensitivity, and it does not have shorter duration data record	Frequency domain	Narrowband, stationary signals
Wavelet transform	It has a varying window size, being broad at low frequencies and narrow at high frequencies It is better suited for analysis of sudden and transient signal changes Better poised to analyze irregular data patterns, that is, impulses existing at different time instances	Needs selecting a proper mother wavelet	Both time and freq. domain, and linear	Transient and stationary signal
Eigenvector	Provides suitable resolution to evaluate the sinusoid from the data	Lowest eigenvalue may generate false zeros when Pisarenko’s method is employed	Frequency domain	Signal buried with noise
Time frequency distribution	It gives the feasibility of examining great continuous segments of EEG signal TFD only analyses clean signal for good results	The time-frequency methods are oriented to deal with the concept of stationary; as a result, windowing process is needed in the preprocessing module It is quite slow (because of the gradient ascent computation) Extracted features can be dependent on each other	Both time and frequency domains	Stationary signal
Autoregressive	AR limits the loss of spectral problems and yields improved frequency resolution Gives good frequency resolution Spectral analysis based on AR model is particularly advantageous when short data segments are analyzed, since the frequency resolution of an analytically derived AR spectrum is infinite and does not depend on the length of analyzed data	The model order in AR spectral estimation is difficult to select AR method will give poor spectral estimation once the estimated model is not appropriate, and model’s orders are incorrectly selected It is readily susceptible to heavy biases and even large variability	Frequency domain	Signal with sharp spectral features

Table 1.

Comparison of various feature extraction techniques [26].

As we have discussed, the various techniques for feature extraction of EEG signals, Table 2 representing a comparative analysis for frequency resolution and spectral leakage from where one can easily conclude that AR method is utilized where one need to avoid spurious features.

1. 4. Classification- the final stage of EEG signals processing is classification. Machine learning classifiers are hugely deployed to use features to predict the corresponding class. The classification is performed in three categories: positive, neutral and negative [28]. Algorithms like Support Vector Machine (SVM), Naive Bayes (NB), Random Forest (RF), etc. Are hugely used by researchers due to their simplicity and accuracy [6]. But these techniques never consider the temporal information associated with EEG signals.

Method	Frequency resolution	Spectral leakage
FFT	LOW	HIGH
WT	HIGH	LOW
AR	HIGH	LOW

Table 2.

Comparison of frequency resolution and spectral leakage of feature extraction techniques [27].

Recently deep learning algorithms like Convolution Neural Network (CNN), Recurrent Neural Network (RNN), Artificial Neural Network (ANN), etc. are also applied by researchers for relatively high accuracy [29].

7. Applications of EEG signals

Few application of emotion detection using EEG signals has been discussed below:

1. Medical diagnosis—the signals collected using EEG can be used to diagnose various diseases related to brain like brain edema, Parkinson’s disease, epilepsy scots, etc.

2. Education—studies has shown that there are some emotional states that are helpful for learning [17]. So by given different educational task and acquire signal during them can help in studying the impact on human.

3. Video gaming—video games are to entertain player and making them attached and involved [30]. By analyzing the mood and mind-set of users, the video games can engage users emotionally, which may result in good participation rate.

The application are not restricted to these three categories only, they are also applicable in Brain Computer Interaction (BCI), Patient Care, Driving Autonomous Car, etc.

8. Solutions and recommendations

In this chapter, we have presented different machine learning algorithm and techniques and their comparison that are used for emotion recognition. In emotion recognition there is no direct approach that suite for all kind of applications. There are multiple factors that affect the choice of machine learning algorithm. The usefulness of emotion recognition and its application has been discussed. Machine algorithms are used to analyze data used for classification. The algorithm basically filter data into categories, which is achieved by providing a set of training examples, each set marked as belonging to one or the other of the two categories.

9. Future scope and conclusion

The future of emotion recognition is very bright. Emotion recognition is already an incredibly powerful tool that helps to solve really hard classification of emotions problems. With the use of emotion recognition, a very hard problem can be solved and system can be work as per the mood and feelings of users.