Mapping and Timing the (Healthy) Emotional Brain: A Review

Abstract

The study of the emotional processing in the brain began from a psychological point of view in the last decades of the 19th century. However, since the discovery of the electrical background of mental activity around 1930, a new scientific way of observing and measuring the functioning of the living brain has opened up. In addition, Functional Magnetic Resonance Imaging (fMRI) has given neuroscientists a (literally) deeper instrument to perform such measurements. With all this technological background, the last decades have produced an important amount of information about how the brain works. In this chapter, we review the latest results on the emotional response of the brain, a growing field in neuroscience.

1. Introduction

The study of emotions deals with the physiological and psychological correlates of subjective experiences that are evident to conscious human beings. Emotions are present and influence our lives and even our perception of reality, making the scientific approach to their study, which has only begun in relatively recent decades, very difficult.

According to [1], the pseudoscience of phrenology brought the critical idea of the physical distribution of psychological functions in the brain, opening the door to modern neuroscience that has largely corroborated this assumption.

It is widely assumed that emotions are the subjective representations of naturally evolved primarily neural circuits and functions that helped surviving since the very first complex animals [2, 3]. This has two main consequences: on the one hand, the physical localization of emotional circuits is hidden in the ancient brain (the limbic system, the amygdalae, and other inner regions). On the other hand, these regions are largely connected to more developed areas, such as the cortex or the cerebellum. Therefore, not only should external stimuli trigger automatic motor responses, but cognitive information can be critical as well as a “brake” on these autonomous reactions (implemented in the cerebellum) and can produce more flexible and adaptive responses.

Although much research in this field focuses on damaged brains, this review covers the healthy brain that responds to emotional stimuli under laboratory conditions.

1.1 History

The connection between the physical processes of the brain and its biomarkers has been assumed since the late 19th century [4].

In 1929, German psychiatrist Hans Berger developed the novel method of Electro-Encephalography (EEG), opening a disruptive and scientific way of studying the processes of the living brain. Although a vast and unexplored field was opened, the first results using EEG to measure emotions did not occur until the 1960s [5]. However, interest in emotional studies still had to wait some years, till the mid-70s’ when some researches began to appear [6, 7].

Since then, the same basic experimental setup has been replicated in research until today: a subject connected to the EEG, or to new tomography technologies (as in [8]), is exposed to different stimuli while his brain activity is recorded.

Figure 1 shows the historic timeline developing this research field.

Figure 1.

A chronology of major events associated with the development of human brain imaging, from [9], adapted with permission.

Positron Emission Tomography (PET) and Magnetic Resonance Imaging (MRI), two new techniques to access information inside the brain, and not only at scalp level, were developed in 1975 and 1979 respectively, and began to yield significant results in the 1980s (see, for example, [10]).

1.2 Brain atlases and areas’ references

The brain began to be mapped according to its anatomical differences in 1909 by Brodmann [11], who defined 52 regions that modern neuroscience considers extraordinarily accurate for those years. In fact, today most neuroscientific works still provide Brodmann’s nomenclature to specify the areas of activation.

However, it has been shown that it is not precise enough to evaluate some functional characteristics of the brain, so the Montreal Neurological Institute proposed in the 1990s a more modern division of the human brain [12], with 1 mm³ templates organized in a system of coordinates (X, Y, Z). Today, this brain atlas is considered to be a standard.

The main regions in the brain are depicted in the Figure 2.

Figure 2.

Basic brain anatomy. 1: Brain stem. 2: Limbic system. 3: Cerebellum. 4: Cerebrum. 5: Occipital lobe.6: Temporal lobe. 7: Parietal lobe. 8: Frontal lobe.

1.3 Emotional maps

Emotions are subjective feelings, but they must be quantified in some way to allow a methodical study. Since the 1980s, there have been two main approaches in the field of emotion research: the categorical approach and the dimensional approach.

The dimensional approach considers that emotions are organized along a few psychological dimensions. Step by step, a consensus was established on the representation of emotions around a two-dimensional plane, shown in Figure 3.

Figure 3.

Emotional map from [13], adapted with permission.

The debate about the separation of the emotional features of valence and arousal flows over the correlation of these two variables. Barret, for example, found weak correlation between them [14], and Lang supports this idea, inferring that some neural circuits are similarly engaged by motivationally relevant cues independently of the valence, while there may be some other hedonic circuits to discriminate valence [3, 15]. However, other researchers have found contradictory results [16], specifically with respect to valence and arousal of negative stimuli.

This paradigm presents another issue since Miller’s studies [17]. It seems that there is a distortion in the linearity of this space: a negativity bias (for equal amount of positive or negative stimulus, the negative one produces higher responses) and a positive offset (in neutral scenarios, there is a predisposition to appetitive responses).

Combinations of different scales have been used to provide representational spaces with more dimensions and, allegedly, higher accuracy [16]. Examples of these rating scales are the Bivariate Evaluation and Ambivalent Measures (BEAM), described in [18] and the three-dimensional space proposed in [19], which distinguishes between tension arousal and energy arousal.

As an example of categorical approach, we can find the Self-Assessment Manikin (SAM): SAM is a non-verbal pictorial assessment technique that directly measures the valence, arousal, and dominance associated with a person’s affective reaction to a wide variety of stimuli [20].

There is an important body of evidence supporting cross-cultural stability in the perception of emotions [21, 22], and the pleasant-unpleasant dimension seems to exists in all cultures [23]. Ekman considers a few categories of innate and universal emotions (happiness, sadness, anger, fear and disgust) from which all other emotions can be derived [24]. The categorical approach that considers some discrete emotional maps related to basic adaptive problems, has been shown to be cross-cultural or even cross-species (for a review, please refer to [25]).

1.4 Emotional stimuli

As stated in the History section, the setup of most of neuroscientific experiments involves stimuli to elicit emotions (or any other response) in the subject under study.

In addition, different stimuli trigger different and specific areas of the brain, so the choice of stimuli is crucial for the information expected to be retrieved from the experiment.

We will present the most used ones and some points about their effectiveness.

1.4.1 Emotion elicitation techniques

Out of a total of 248 articles, al-Nafjan gathers the type of stimulus used in Table 1.

Technique	Number of Articles	Domain (Medical, Non-Medical)
Visual-based elicitation using images	88	26%, 73.9%
Prepared task	43	25.6%, 47.4%
Audio-visual elicitation using short film video clips	38	18.4%, 81.6%
Audio-based elicitation using music	29	17.2%, 82.8%
Multiple techniques	19	26.3%, 73.9%
Other	17	11.7%, 88.2%
Imagination techniques/memory recall	10	20%, 80%
Social interactions	4	25%, 75%

Table 1.

Emotional stimuli used according to their nature, from [26].

Why using images?

Psychologists have already shown that images have strong effects on the emotions of human beings [16]. An important result found in the literature states that simple images generate better emotional responses than complex scenes [3, 27].

Please refer to [28] for a deeper review.

Why using music?

The role of music in producing emotional responses is widely accepted and is one of its defining features [29]. Moreover, this has been proven to be cross-cultural [30, 31], making it a very stable and reliable way to provoke emotions in subjects.

When using audio-visual stimuli, it is important to take into account the predominance of image over sound [32], in case of ambiguity or emotional conflict.

Why using words?

It is well known that the brain dedicates exclusive resources, organized hierarchically, to word and language processing, such as the areas of Broca and Wernicke, which demonstrates their importance in evolution and survival. It has subsequently been found that words and language stimuli function as emotional triggers [33, 34].

Others

Among the “other” stimuli in Table 1, researchers have used olfactory [35, 36] or food [37] stimulation, for example.

1.4.2 Databases

To ease the replication, comparison and contrast of theories and results, many research institutions and authors have developed databases with normalized emotional stimuli, which are publicly available. They have labeled stimuli according to different paradigms, and they have been tested. Some of the most used ones are the following:

• Surrey Audio-Visual Expressed Emotion (SAVEE) Database: Audio-visual clips with male actors in different emotions [38].

• International Affective Picture System (IAPS): This database offers a “large set of standardized, emotionally-evocative, internationally-accessible, color photographs that includes contents across a wide range of semantic categories” [39].

• International Affective Digital Sounds (IADS): The same institution and researchers have published the International Affective Digitized Sound system (IADS), with similar structure, labeling and testing parameters [40].

• Affective Norms for English Words (ANEW): The word-based version of the previous couple of databases [41].

• Affective Norms for English Text (ANET): In the case of using text extracts, this database “provides normative ratings of emotion (pleasure, arousal, dominance) for a large set of brief texts in the English language for use in experimental investigations of emotion and attention” [42].

• The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): This is a multimodal database of emotional speech and songs, labeled following a discrete emotional space, with neutral stimuli included [43].

• The Montreal Affective Voices (MAV) consist of a set of short vocal interjections expressing anger, disgust, fear, pain, sadness, surprise, happiness, sensual pleasure, and neutrality [44].

As we have seen, the databases cover language, images, sounds and combinations thereof.

2. Methods

Nowadays, almost all neuroscientific studies and findings are based on two non-invasive, biomarkers-free technologies: Electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI).

2.1 EEG

The EEG is based on the evidence that massive clusters of neurons fire at the same time when they work synchronized, producing tiny voltage changes around them (in the range of millivolts to microvolts). The EEG can be measured directly on the surface of the brain surface and scalp. In both cases, the system has the following elements:

• Electrodes: conductive elements sensitive to voltage variations.

• Amplifier: low-noise, band-filtered amplifier to scale the small voltage measured.

• Register: Analog or (nowadays) digital recording of the transformed signals, together with time stamps, position information and other contextual information.

Important advantages of this technique are its setup (some equipment is portable) and its price (actually, the cheapest of the techniques exposed here). In the scalp version, it is non-invasive and very safe for the subject.

The main advantage of this technique is the time resolution, around the millisecond, which measures very rapid changes in scalp potentials. Because of this, only the EEG technique allows phase measurements, synchronization computations, spectral analysis, or other time-related processing.

In the EEG, the first representational information found was the so-called Event-Related Potentials (ERPs), signals produced as a reaction to a stimulus, typically within a few hundred milliseconds to several seconds. These signals have proven to be stable between users and experiments, and some of them, as the P300 (positive peak 300 ms after stimulus onset), are universally known and used.

The EEG allows frequency analysis, and its signals can be transformed into bands. The spectral power (also called Power Spectral Density -PSD-), i.e., the amount of energy in each band-, can be used to obtain important information from the raw EEG data.

Since 1977 [45], researchers have proposed a new approach, combining the ERPs and the bands, called Event-Related Synchronization (ERS) and Desynchronization (ERD), to measure instantaneous responses to stimuli in specific bands, as shown in Figure 4.

Figure 4.

ERD/ERS detection, from [46], adapted with permission.

The main counterparts of this technique are the volume conductance effect that makes it difficult to locate internal potential sources [47], and the limitation of recording only signals on the surface of the brain (if no electrodes are placed inside the brain), which also limits the measurement of internal sources. To partially address this limitation, some novel techniques recreate inner sources from their fingerprint on the scalp voltage through complex algorithms, such as the Low Resolution Electromagnetic Tomography (LORETA) first proposed in [48], and other “reverse problem methods” summarized in [49, 50].

In addition, EEG registers suffer from artifacts (from electrical power networks, lighting, muscle movements …) that must be removed or filtered out before the recordings can be interpreted. Although some automatic approaches have been proposed, this is a craft task that many researchers still perform manually. Another source of noise is the impedance of the electrodes (as they are transductors between the scalp and the wires), which must be kept low enough to accurately measure extremely low scalp voltages, typically below 50 Ω. This requires the application of a conductive gel, cleaning with electrodes with alcohol, washing the hair before the experiment, etc.

Finally, the EEG system, as a voltage recorder, needs a reference. This does not change the relative voltage distribution on the scalp, but depending on the choice, it can lead to different absolute measures.

Another important process of standardization of EEG measurements has been the definition of electrode positions, which must be constant between studies to allow replication and falsifiability. Depending on the number of electrodes, different standard configurations are defined, the most used being, in the case of 32, the 10–20 International configuration of Figure 5.

Figure 5.

The 10–20 international EEG configuration.

The names and positions of the electrodes are defined in this standard and applied elsewhere. For a larger number of electrodes some other standards can be found (see, for example, [51]).

The correlation of EEG signals with emotions is well established, as stated in [26]: “ We found that the majority of the 130 articles used event-related potentials, whereas 48 articles used Frontal EEG asymmetry in their analysis, six articles used event-related desynchronization/synchronization, and four articles used steady-state visually evoked potentials”.

2.2 fRMI

fMRI began in the 1980s, and soon produced extremely novel results. fMRI measures differential activations of brain regions [52] according to de-oxy-hemoglobin distribution.

fMRI requires a massive magnet (typically around a few tesla), which makes the setup extremely space demanding and expensive. Besides, it cannot be used with metallic components (nor implanted in the subject’s body) so the presentation of the stimuli must be deviated with reflective screens, remote speakers, etc.

The temporal resolution of fMRI is poor, in the range of seconds, which makes it useless to record rapid changes or reactions to the stimuli. However, the main positive aspect is the spatial resolution and the real three-dimensionality of the recording, which generates a map of voxels (volumetric units of information) of very few mm³ if a high temporal resolution is not needed (in fact there is a trade-off between these two parameters; for example, for a voxel size of 3x3x5mm³, the sampling rate falls to about 2 s [53]). Unlike the EEG registering technique, fMRI has the difficulty of mapping different brains (of different participants) in a canonical brain in which the activations and regions can be represented. This forces a spatial transformation to standard geometries that implies loses in spatial resolution [53].

The functionality of the MRI is given, among others, by the Blood Oxygen Level Dependent (BOLD) imaging, which measures differences in oxygenated blood flowing through the brain (since oxy-hemoglobin and de-oxy-hemoglobin have different magnetic susceptibility), correlated with neural activation.

BOLD techniques have the temporal limitations of the physiological processes on which they are based (see [53] for more details). In most studies, fMRI data are statistically processed to generate a meaningful representation of changes, in so-called Statistical Parametric Maps (SPM), yielding to images as that shown in Figure 6.

Figure 6.

SPM in which the color of pixels is representative of its p-value and, thus, the statistical significance of its activation or deactivation when two or more tasks are compared. From G.Konstantina, CC BY-SA 4.0, via Wikimedia commons.

2.3 Simultaneous measuring and comparison

Since both EEG and fRMI are based on related physiological processes, it is easy to find correlations between them.

These two non-invasive techniques for exploring the interior of the living brain are not mutually exclusive, and both have advantages and disadvantages. Therefore, both are used in neuroscience research today.

Table 2 summarizes the main characteristics of each one.

Technique	Temporal Resolution	Spatial Resolution	Portability
EEG	High	Low	Mid
fMRI	Low	High	Low

Table 2.

Features comparison between EEG and fMRI.

In 1996, Gerloff and others [54] combined fMRI and EEG for the first time to evaluate the co-registration of both techniques applied to the primary motor cortex and the sensory cortex.

Over the past decade, several studies using both techniques have been proposed to, among other, find new large-scale brain networks [55], examine some specific networks [56, 57, 58] or even provide neurofeedback to assist in the regulation of some circuits [59].

Babayan et al. [60] have recently published a large database with combined EEG and fMRI data from 227 healthy participants.

Unfortunately, the joint use of both techniques has its drawbacks: the signal-to-noise ratio (SNR) can be degraded [61] and interferential artifacts can be generated, as shown in [62].

For more in-depth in brain data imaging, please refer to the handbook [63].

3. Measuring emotions

The neuronal correlates of emotions present features and effects in various dimensions that interact in the living brain.

To help understand the results collected in the scientific literature on emotions, we will divide the findings into different categories, although they are mixed and sometimes inseparable.

3.1 Timing

Studies dealing with the temporal signals created by the emotional processing are, most of the time, based on EEG recordings. The reason is, as explained, the temporal resolution of this technique.

Typically, the measurement of an EEG signal follows the scheme of Figure 7.

Figure 7.

Typical EEG signal during a stimulus-based experiment, from [64], adapted with permission.

When recording the brain’s reaction to stimuli, it is important to define a control group or baseline with which to compare activations or deactivations. Koelstra [65] proposes 5 seconds prior to stimuli as such a baseline.

What happens in the reference period is not uninteresting when emotions are studied from a neuroscientific approach. It has been proven that if in this period the index of asymmetry (the difference in the global activation in each hemisphere, calculated as the left over the right) is high, the subject will present a bias towards positive stimuli, and vice versa as far as fear is concerned [66].

After the stimulus onset, Wei defines a time range of [0.5–4] s as the temporal space in which emotional signals appear [13]. This period has been divided by some authors into three sections: Early [400–1100] ms, Middle [1000-3000] ms, Late [3000–5000] ms [32].

For example, it is widely established that a high positive ERP, in the range of 200–300 ms, widely known as P300, is elicited by emotional stimuli (such as emotional words) compared to neutral ones [67, 68, 69, 70, 71]. This ERP appears in the occipital-temporal regions with an arousal-related amplitude (independent of the valence) compared to neutral stimuli [72].

As already mentioned, the emotional response is mediated or modulated by different neural systems. The P300 has proven to be a modulator of emotional processing regardless of valence when presenting emotional versus neutral pictures [5, 67, 73, 74, 75]. These effects were seen in both pleasant and unpleasant pictures [5, 75, 76].

One of the most stable and reliable neural signatures of emotional processing is the so-called Late Positive Potential (LPP), which appears after 1 second of the stimulus presentation, and can be traced for up to 6 seconds in the central-parietal region [77].

It has been shown that LPP appears with both emotional pictures or words, with an amplitude that depends on the arousal intensity [67, 70, 71], being higher in emotional (both positive and negatives) images compared to neutral ones [78] and not habit-forming [67, 79, 80], although it may decrease somewhat with repetition [15]. Another interesting feature of this signal is its ability to appear with very short exposures to visual stimuli (down to 25 ms) [3].

The LPP is independent of the characteristics of the stimuli as realism/symbolism, complexity/simplicity, etc. [27, 67] and therefore very reliable: “The late positive potential evoked by picture stimuli is a reliable, replicable index of their motivational relevance” [3], correlated with the self-reported arousal [3].

For all these reasons, the LPP has been labeled as the “motivational significance” of a stimulus [81].

The LPP has been localized in the central-parietal region, but also in the secondary visual processing sites in the lateral occipital cortex [82] with visual stimuli.

Summarizing the findings of time-analysis of EEG signals correlated with emotional processing, we can say that the P300 and LPP track emotional processes [78]. There is a golden rule that says that valence is processed before arousal [83], since it has been shown that the early ERP components are correlated with valence [28, 84]. In contrast, the long-term ERP components are correlated with the arousal [85, 86, 87].

Hajcak et al. [67] illustrate the temporal and spatial evolution of the different signals.

3.2 Mapping

The fMRI has shown the processing cores in the inner regions of the brain, such as the limbic system. In terms of arousal, it was found that the area of greatest response in the brain is the amygdalae, a couple of little clusters of nuclei belonging to the limbic system in the temporal lobes, on the internal part of the brain. Regarding the role of the amygdalae in emotional processing, survival instincts, memory, etc. Lang et al. found that this area responds to the intensity of emotional stimuli, and has a central role in enhancing sympathetic reactivity to such stimuli [15]. But the amygdalae do not react independently of the valence of the stimuli: the preferred stimuli selectively activated the right amygdala, in relation to aversive ones in some experiments [88, 89].

Valence has also been correlated with specific limbic neural circuits closely connected to the amygdalae: the mesolimbic reward system, in which the nucleus accumbens (NAc) is particularly relevant in the processing of reward and pleasure evoking stimuli (the reward, motivation and addiction circuits) [90, 91]. Another study extends this list of central processing centers to the Ventral Tegmental Area (VTA) and the hypothalamus, working together as a tripartite network that manages the responses to the emotional aspects of music [92]. Another reward network component is the ventral striatum, which, along with the cingulate cortex, has also shown correlations with the arousal of positive emotions when listening to music [93, 94].

The relation between the mesolimbic networks and some frontal regions (as the Orbito-Frontal Cortex (OFC) and the Interior Frontal Cortex (IFC)), more in charge of cognitive processing, has led some researchers to establish a close relationship between “affective” and “cognitive” processing involved in music listening [92]. Another hypothesis is that the interactions between the OFC and the NAc may be related to the control of emotions [95].

Overall, although it belongs to the cognitive cortex, the role of the OFC in the emotional processing is beyond doubt, and is supported by many studies dealing with music [32, 92, 96, 97], images [98, 99] or decisions [95]. Close to the OFC, the IFC has also been considered relevant, producing a bilateral activation when listening to music, according to [92].

Other important cortex cores for emotional processing are the parietal and temporal areas. The centro-parietal area has shown an activation proportional to the arousal of emotional pictures in the first moments (in a range of 300 to 700 ms) [3, 92]. Positive centro-parietal signals [300–6000 ms] have shown valence independence [3]. Furthermore, the link between the frontal and right parieto-temporal areas with the arousal of a stimulus has been also established [100].

It is worth mentioning the anterior insula, belonging to the temporal lobe, which has been thoroughly studied and defined as a relay between the limbic (specifically the human mirror neuron system) and the motor system (in the cortex) [31, 101], and may be the physiological support for subjective states, like pain, hunger, heart rate perception or emotional awareness [102, 103, 104].

Early lateralization has been found to be correlated with the valence of sounds [32], and this effect does not exist with neutral sounds. One of the first findings in this field is due to Schwartz [7], who found a lateralization in the brain activity.

Back to the surface of the cortex, there are areas specially engaged in emotional processing, measured with both EEG and fMRI.

In the first case, there is a discussion about which electrodes are the most representative of the undergoing emotional processes. For example, we can find the work of Wei et al., who proposes moving from F1, F2, T3 and T4 to F1, F2, F7 and F8 respectively (shifting the registering area from bilateral front-temporal to pre-frontal medial areas), obtaining much better predictions [13]. This change is also proposed independently (and partially) by Lin et al., stating that fronto-central electrodes are specially relevant when measuring theta asymmetry (F7-F8 and FC3-FC4) as a correlate of arousal [105].

By focusing on synchronizations between different areas, it has been found that there is a phase synchronization between frontal and right temporo-parietal areas depending on the valence and the energetic arousal [106]. In another study [107], a beta-band synchronization was found between the pre-frontal and posterior areas when observing high-arousal images. Finally, unpleasant images caused a phase synchronization in the gamma band according to [108]. Please refer to [109] for further details about synchronization.

We have shown the interactions of the limbic system with the cognitive areas of the brain, in relation to images, sounds or decisions. But we have also found some interactions with other unrelated areas, mainly the motor areas, when empathy in involved. It seems that in the processing of emotions, many different and specialized areas need to interact to account for such a subjective experience. This cross modality has been studied in depth. For example, [32] shows that emotional sounds modulate visual primary cortex (P1). In the same region, relationships have been found between emotional processing cores and visuo-spatial and visuo-motor regions [110], or even premotor regions including the intra-parietal sulcus and the ventral premotor cortex [111]. The ventral premotor and the posterior parietal cortex were elicited during the observation of Classical and Renaissance sculptures suggesting, as Di Dio state, “motor resonance congruent with the implied movements portrayed in the sculptures.” [88].

For a final summary, please refer to Table 1 in [87], which provides a detailed review of the EEG spatial correlates of emotions.

4. Final considerations and conclusions

It has been shown how the way we react to emotional situations depends on very different and scattered areas in the brain. Both initial reactions and the later dependencies lay on different systems [112]. Reward calculation and empathy have been identified as being involved in fear or appetite reactions, and are also the most primitive, but they interact with more evolved areas of the cortex that deal with visual and auditory processing, decision making and even motor activation.

Many aspects of the emotional brain remain open. For instance, global or synchronized processing networks beyond the cortical surface have yet to be described, as the limitations of fMRI and non-invasive EEG do not allow this unknown field to be addressed.

Furthermore, it is not clear whether and how gender affects emotional processing. Although responses to musical stimuli, recorded with EEG, have shown no significant gender differences in brain frontal regions [100], gender differences have been observed during verbal learning tasks [113], in emotional networks in adolescents [114] and with esthetic stimuli producing bilateral parietal activation in women, but lateralized in men [115].

Other limitations are due to the stimuli used. Real life involves interaction with various external and internal sources of information that modulate our feelings, and laboratory conditions barely address such complex situations.

This branch of neuroscience is not new (compared to neuroscience itself), but still presents many open doors to be explored.

The way emotions mobilize resources in the brain seems to be large and deep, and many other functions (such as memory) depend on it, showing their pre-eminent position to survive and behave in human (and many other animals’) life.