Emotion Recognition – Recent Advances and Applications in Consumer Behavior and Food Sciences with an Emphasis on Facial Expressions

Abstract

For decades, the study of emotions has been the center of attention in research and practice. Based on relevant literature, this paper focuses on the subject of measurement, and provides a structured overview of common measurement tools by distinguishing between methods of communication and observation. Given the authors’ field of competence, presentation pursues a consumer behavior and food sciences perspective. Furthermore, the paper devotes attention to automatic facial expressions analysis technology which advanced considerably in recent years. Three original empirical examples from the authors’ range of experience reveal strengths and weaknesses of this technology.

1. Introduction

1.1 Intended contribution

For decades, emotions have been a hot topic in multiple scientific disciplines such as Psychology (cf. the seminal work of Darwin published first in 1872). They are said to be an integral part of human nature while contributing to behavior control but were sometimes thought to bias rational thinking and behavior. Theories on the study of emotions are manifold. For example, Scherer’s ([1], p. 697) view is very broad in that he defines emotions as an “episode of interrelated, synchronized changes in the states of all […] organismic subsystems in response to the evaluation of an external or internal stimulus.” Consequently, his component process model of emotions indicates a comprehensive conception by pointing to the organismic human subsystems (central nervous system, neuro-endocrine system, autonomic nervous system, and somatic nervous system) which become active in response to the evaluation of occurring stimuli and induce an emotion in turn: (i) cognitive component (appraisal); (ii) neurophysiological component (bodily symptoms, arousal); (iii) motivational component (action tendencies); (iv) motor expression component (facial and vocal expression); and (v) subjective feeling component (emotional experience). In an ideal world of science, the researcher would need to measure all components. However, Scherer ([1], p. 709) concedes that “comprehensive measurement of emotion has never been performed and is unlikely to become standard procedure in the near future.”

Whereas Scherer’s [1] component process model provides a thorough and multi-facet view on emotions, this chapter—while focusing on measurement issues—pays tribute to practicability. As discussed later, measurement procedures employed empirically, typically focus on a single aspect of emotions only. Given this view, this paper intends to provide a structured overview of common tools that enable the measurement of emotions. Furthermore, the focus lies on measurement procedures in consumer behavior and food science. In particular, special attention is devoted to automatic facial expressions analysis technology which advanced considerably in recent years.

The remainder of this book chapter is structured as follows. The next subsection provides a theoretical foundation from the literature which delivers insights into the concepts of emotional measurement procedures. Section 2 is central for this chapter and presents a variety of means for measuring emotions. It is structured according to methods of communication and of observation. Section 3 offers three empirical examples that employ automatic facial expressions analysis technology for capturing customers’ emotions while being exposed to commercials or tasting food products. These examples underpin the strengths and weaknesses of the employed means of technology. Section 4 concludes by providing recommendations for research and practice.

1.2 Basic considerations on emotions

The classical view on emotions is a categorical and dimensional theory. It claims that only a limited number of basic emotions exist. These emotions, in turn, are described as inborn reactions that universally apply to all cultures [2, 3]. Ekman and colleagues suggested that facial expressions (cf. a motor expression component of emotions) can be categorized into a small number of basic emotions like happiness, fear, anger, surprise, disgust, contempt, and sadness. The theory of constructed emotions offers a different point of view [4, 5] and contradicts the classical view in many aspects: First, emotions are not inborn reactions, but arise from basic components. Second, emotions are not universal, but vary from culture to culture. Third, they are not triggered, but are expressed by the individual. Fourth, emotions emerge as a combination of the physical properties of the body, a flexible brain that wires itself to any environment it develops in, and the culture and education, which form that environment.

Consensus exists that emotions are the results of certain stimuli that do not last very long as compared to feelings. Thus, emotions are often considered as “short-term affective responses to the appraisal of particular stimuli” ([6], p. 191). In relation to that, the appraisal theory claims that emotions are elicited when an event is being evaluated, which contributes to an important goal of the individual [7, 8, 9]. The connotation of the emotion is positive when the concern is advanced and negative when the concern is impeded. This theory is in line with Rolls’ [10] conceptualization which defines emotions as states elicited by rewards and punishments. In this sense, emotions regulate distance and nearness. For example, the emotion of disgust elicits avoidance behaviors (distance), while positive emotions such as happiness promote approach behaviors (nearness). According to Gibson ([11], p. 54), feelings last in contrast to emotions longer and are referred to as psychological arousal states with “interacting dimensions related to energy, tension and pleasure (hedonic tone) […] and may be more covert to observers”. (Sources: Matthews & Deary, 1998; Rolls, 2007).

The appraisal theory follows a cognitive approach and focuses on the mind’s organization of conscious and unconscious knowledge, and on the fundamental questions of how emotions are caused and what their effects are. Oatley and Johnson-Laird [12] describe three cognitive theories of emotion: (1) the action-readiness theory, (2) the core-affect theory, and (3) the communicative theory. Ad (1), the action-readiness theory holds that emotions are built from elements that are not itself emotions. Frijda and Parrott [13] label them as “basic emotions,” viewed as states of readiness for certain actions, giving priority to a particular goal. Ad (2), the core-affect theory postulates two stages in generating an emotion: level of arousal and valence level (pleasure–displeasure) [1, 14]. Ad (3), the communicative theory claims that emotions relate to communication within the brain and amongst individuals [9] and that distinct basic emotions have evolved as adaptations in social mammals.

Another important consideration is whether emotions are conscious or unconscious by nature. One stream of literature describes an emotion as the conscious subjective experience that accompanies affective states created by bodily sensations [6, 15]. However, several more recent studies point to the existence of unconscious emotions. In particular, a person is not aware of these emotions when explicitly asked to report them [16].

James [17] and Lange [18] represent the premise that emotional experiences are produced by sensing peripheral bodily changes like heart rate or tensions in the skeletal muscles, which are referred to as somatic markers [19]. This leads to the—for many contra-intuitive—situation, that an individual is not running away, because (s)he is scared, but rather is scared because of (s)he is running away. Many objections have been raised against this theory, which are summarized by Rolls [10].

In conclusion, emotions appear on at least three dimensions. First, they are experienced by individuals in a distinct subjective manner. This is the most common and widely known aspect of emotions—everybody knows, how disgust, sadness or happiness, etc. feel. Second, emotions are often connected with changes in physiology, mainly in reactions of the autonomic nervous system. Sweating or perceiving a strange stomach/gut feeling are two examples for physiological reactions. Third, emotions have a behavioral aspect, which means that emotions can change the behavior of an individual such as facial expressions, posture, walking speed, speech, or gestures [20, 21]. Irrespective of the question, which of these dimensions appears first, second, and last, and how they interact, these three dimensions can be used and have been used (subjective experience, physiology, and behavior) to characterize the emotional state of an individual.

2. Selected measures of emotions

Table 1 provides a structured overview of the most appropriate measurement approaches developed in the last decades (see for instance Coppin and Sander [22] for an alternative survey). On the first level, we distinguish whether these methods employ communication or observational techniques (upper or lower panel of Table 1). When using means of communication, subjects provide information about their emotional experience verbally in their own words or by responding to scales or by commenting on pictures or photographs. Clearly, this is a conscious process. These methods are therefore based on introspection and self-reports.

Means of communication
Degree of structure	Degree of disguise
	Undisguised	Disguised
Unstructured	Personal interviews	Content analysis of diaries
	Think-aloud technique	Associative networks
		Zaltman metaphor elicitation technique (ZMET)
Structured	Verbal scales	Picture / photo scale
	Differential Emotions Scale (DES)	Self-Assessment Manikin Technique (SAM)
	Pleasure/Arousal/Dominance scale (PAD)	PrEmo-instrument ©
	Emotions Profile Index (EPI)	EmoSensor
	Positive And Negative Affect Schedule (PANAS)	Emotive Projection Test (EPT)
	Consumption Emotion Set (CES)	Implicit Association Test (IAT)
	Temporal Dominance of Sensations (TDS)
	EsSense Profile ®
Means of observation
Method of administration	Setting
	Contrived laboratory setting	Near-to-life/real-life setting
Human	Facial expressions (FAST, FACS)	Facial expressions (FAST, FACS)
	Body movements	Body movements
		Mystery shopping
Technical equipment	Facial ElectrocMyoGraphy (fEMG)	Voice pitch analysis
	Automatic Facial Expressions Analysis (AFEA)	Automatic Facial Expressions Analysis (AFEA)
	Electro Dermal Response (EDR)	Neurophysiological measures using wearables (EDR, pupillary dilation, heart rate)
	Electro EncephaloGraphy (EEG), Positron Emission Tomography (PET), functional Magnetic Resonance Imaging (fMRI)
	Program analyzer

Table 1.

Overview of emotional measurement procedures.

Observational techniques have been employed or developed to avoid possible biases of introspection-based self-reports (i.e., social-desirable response behavior). Observations can be used in an implicit way, which means that the measurement outcomes reflect the construct under investigation (e.g., emotions) in an automatic manner based on processes that are uncontrolled, unintentional, goal-independent, purely stimulus-driven, autonomous, unconscious, efficient, and fast. Physiological, neurological, and behavioral reactions deliver implicit measures, in contrast to explicit measures, which are controlled, intentional, goal-dependent, not only stimulus-driven, conscious, slow, and potentially intrusive [23]. Data from implicit measures are said to have more external validity and therefore, contribute more to the understanding and prediction of human behavior in real life.

Dijksterhuis [24] suggests three criteria for the evaluation of the implicitness of a method. Does the subject (1) need to think about him−/herself to answer; (2) know that (s)he is tested; (3) know about the research question? The more yes-answers are provided the less implicit, the more no-answers are stated the more implicit the evaluated method is. Columns of Table 1 (reflecting the second level of classification) refer to the latter by distinguishing between degrees of disguise, setting, respectively. Typically, subjects are neither informed about the research agenda nor are always aware of the conducted study in a disguised or real-life setting (such a situation would be called probiotic). Ethical issues concerning research integrity have to be considered in such cases to a great extent, but these considerations are beyond the scope of this article.

The third level (rows of Table 1) categorizes measures of emotions according to a more technical aspect degree of structure (i.e., degree of standardization imposed on the questions asked and the answers permitted) or the method of administration (human vs. technical equipment).

2.1 Methods using means of communication

All methods employing means of communication use the ability of humans for introspection, and reporting about the result of introspection, such as emotions in terms of language, pictures, or photos. Implicitly they are thus following (some variant of) appraisal theory. Scholars favoring, for example, a biologic theory, accentuate limitations referred to as cognitive bias inherent in self-report measures. There are enormous differences in the test design on how individuals are enabled to communicate the results of their introspection. As an aside, qualitative methods introduced in the following two subsections have been adapted for emotion measurement but were originally developed for a broader spectrum.

2.1.1 Unstructured, undisguised communication methods

Personal interviews The personal interview, with a more or less structured conversation, is led by an instructed interviewer. Within a free-response format, the researcher asks participants to respond with freely chosen labels or short expressions that best characterize the nature of the emotional state they experience when being confronted with a certain external or internal stimulus (cf. [1]). A guideline helps to lead the interview. Personal interviews score highly with respect to flexibility but negatively on issues such as the subjects’ potential problems communicating personal responses with appropriate expressions, individual differences in the range of their active vocabulary, and biasing influences of the interviewer on the communication process. In addition, the recruiting of subjects might be challenging since the process needs to be executed (and recorded) in a quiet surrounding and takes a considerable amount of time. Making the data amenable to quantitative analysis requires categorization which is a labor-intensive process. Scherer [1] offers the Geneva Affect Label Coder (GALC) which attempts to recognize 36 affective categories commonly distinguished by words in natural language.

Think-aloud technique Subjects are asked to perform a specific task (for the present situation for instance looking at a certain video clip or website; walking through a store and finding a certain product on the shelf; tasting a certain meal) and to articulate whatever comes into their mind (in particular their perceived feelings and emotions) as they complete the task. These verbalizations of subjects’ cognitive processes are recorded by the researcher (if present) or on a technical device (e.g., a voice recorder) and its content is analyzed thereafter. We emphasize the dynamic component of this method because protocols have to be connected with the various stages and aspects of the task to find out which particular emotions they elicited at which point in time.

2.1.2 Unstructured, disguised communication methods

Content analysis of diaries Traditionally, diary studies request subjects to self-report certain behaviors or activities over a longer period of time which seems to contradict the definition of emotions as a short episode in time (i.e., contrasting feelings, cf. subsection 1.2). A special format of diaries [25], event-based diaries or in-situ loggings, asks participants to log information in the situation they occur. The situation in turn is defined by the researcher as an event (e.g., consuming a certain product), a usage scenario (e.g., engaging with a certain product), or making a selfie in a certain location, searching for photographs about past events or consumption experiences, etc. To emphasize the short-term characteristic of emotions, reporting might be executed by audio/video devices, hand-held computers, or means of social media rather than by traditional paper diaries. Spontaneity of responses is important to mitigate potential rationalization. The interpretation of reports requires diligent (content) analysis.

Associative networks This method [26] builds upon the well-established theory that human memory might be viewed at as possessing a network structure consisting of nodes (representing stored information of an object) and interconnecting links (representing the strength of the association between thereby connected objects). For the present case, the researcher is interested in the arousing potential of a stimulus (e.g., an ad, a food product) on different emotions (e.g., surprise). Thus, when using this technique, free associations with stimulus words or stimulus images are entered on a sheet of paper in a list. By entering the list, the order (and the spontaneity) of the associations is automatically recorded. After that, subjects evaluate the recorded words in another column of this sheet of paper (as having a negative, neutral or positive meaning in the considered context). In the next step, subjects are asked to assign as many as possible of the previously recorded words to already predefined categories. This has the advantage that the categorization does not have to be carried out subsequently by the researcher. The categorization scheme has to be developed within a pre-study or existing schemes (e.g., GALC) might be adopted for the present application.

ZMET Zaltman Metaphor Elicitation Technique [27] is a patented technique of marketing research, which aims at determining conscious and unconscious thoughts, feelings, and emotions by investigating the symbolic and metaphorical answers of the tested individuals. Starting point of this technique is the collection of pictures that represent the thoughts and emotions of a study participant at their premises. These pictures help to discover the often-unconscious thoughts and feelings, whose structure is investigated in the following interviews. In essence, it is a sophisticated combination of an event-based diary (collection of pictures at the subjects’ home), an in-depth personal interview (at the researcher’s lab) conceptually based on neural network brain structures.

2.1.3 Structured, undisguised communication methods

There is a huge variety of scales intending to measure emotions. We restrict our presentation to a few starting with scales conceptualized for general, rather than psychological, purposes; DES might be classified as a discrete emotions approach, PAD, EPI, PANAS as dimensional approaches. Subsequently, we provide three scales which are targeting consumer behavior (CES), food sciences (TDS, EsSense Profile ®).

DES The main idea behind Izard’s [28] Differential Emotions Scale rests on the existence of 10 basic emotions and the assumption that language-based categories correspond to unique emotion-specific patterns (cf. action-readiness theory). Thirty items comprise the DES: three adjectives per basic emotion (e.g., for anger “enraged,” “angry,” and “mad”). Respondents are asked to describe their emotional state by (dis)approving to each item on a rating scale. These ratings are then aggregated yielding a score per basic emotion.

PAD Mehrabian and Russell [29] are pioneers in the field of environmental psychology. They propose a SIR (stimulus–intervening processes–response) model of consumer behavior in which emotional (intervening) variables play an important role. Based on previous work, they suggest describing emotions by their position in a three-dimensional space formed by the dimensions of Pleasure, Arousal, and Dominance. They develop a scale with six items for each dimension. The items are framed as a semantic differential (e.g., “happy/unhappy” for pleasure).

EPI The dominance dimension of the PAD is criticized by scholars, amongst others because of the lack of empirical support for this construct (and practicability issues as a three-dimensional space is difficult to present in a two-dimensional figure). Many theorists, therefore, limit their models to the two dimensions of valence and arousal (slightly renaming the pleasure dimension; cf. core-effect theory). Plutchik’s [30] Emotions Profile Index is representative for these approaches. This profile consists of 62 pairs of properties (e.g., “affectionate vs. cautious”) and subjects are asked to choose which of the two alternatives applies for them. Finally, responses are aggregated and represented in a circumplex, which is conceptually very similar to his wheel of emotions (with eight basic emotions and three different levels of intensity). As an example, opposing emotions are displayed at opposing positions of the circumplex, like joy versus sadness (assumed to possess opposing valence but similar arousal). Scherer’s [1] Geneva Emotion Wheel is conceptually very similar but with two exceptions: he proposes 16 basic emotions arranged according to the dimensions’ conduciveness and coping potential. In fact, it turns out, that changing dimensions corresponds to a 45° rotation of the axes of the circumplex.

PANAS The Positive And Negative Affects Schedule [31] can also be viewed as a dimensional approach, however, these dimensions are not related to the constructs from above. The dimensions only distinguish between positive and negative emotions. Each dimension is described by 10 adjectives (e.g., “active” for the positive, “afraid” for then negative dimension) and Likert-framed response categories.

CES Based on extant literature and in particular, extensive empirical studies, Richins [32] developed the Consumption Emotion Set. Conceptually it is similar to the DES but adopted to the purpose of consumption-related emotions. The scale encompasses 17 different categories with two or three descriptors each (e.g., for anger “frustrated,” “angry,” and “irritated”), in sum 47 items with a rating response format.

TDS Jager et al. [33] adopted the Temporal Dominance of Sensation method to emotions, which is intended to measure the dynamics of food-related emotions during consumption. They use 10 emotional attributes and participants have to rate the dominance of these 10 emotions while eating. For instance, in the case of chocolate, this method results in the temporal description of emotions elicited by the experience of chocolate during oral processing. In the first seconds “interested” might be dominant, followed by the emotions “energetic” and “happy,” in the end also “loving,” “calm,” and “guilty” might be found dominant.

EsSense Profile ® Literature originating from the field of (clinical) Psychology is particularly interested in negatively valenced emotions (corresponding with some psychical illness). The circumplex model implicitly implies some symmetry between positively and negatively valenced emotions (as does, e.g., the PANAS scales with 10 items each). Naturally, in a consumption context, positive emotions play a more important role. The EsSense Profile ® (King and Meiselmann [34]) pays tribute to this focus: the profile consists of 25 positive (e.g., “glad”), 3 negative (e.g., “bored”), and 11 unclear (e.g., “eager,” “daring,” “tame”) items. Subjects describe their emotional state by (dis)approving to each item on a rating scale. Aggregated (over respondents) ratings are displayed in a radar chart or consolidated by multivariate techniques (factor or cluster analysis). EsSense Profile ® has been validated and gained influence in sensory science. In response to the critique regarding the scale (very subtle differences between the verbal emotion descriptions, which require high cognitive capabilities and articulateness from respondents), shorter versions of EsSense Profile ® have been proposed [35].

However, in all methods employing verbal scales it is certainly not clear whether the respondents’ experienced emotions are measured or only their more or less vague associations with emotions elicited by the stimulus, which makes of course an enormous difference. Despite all reservations and problems of self-report questionnaires, Cardello and Jaeger [36] recommend scales as the default measuring method for emotions.

2.1.4 Structured, disguised communication methods

A general issue with verbal descriptions of emotions (as used in scales) is, that emotions are not always easily expressed with words and there also exist differences across cultures and languages in the emotion lexicon [5]. For that reason, alternative scales based on pictures rather than on verbal descriptors have been developed.

SAM Inspired by the PAD scale Lang [37] establishes the Self-Assessment Manikin scale which offers respondents visual response categories, that is, differently shaped pictograms. For the dimension pleasure, the “friendliness of the face” of the manikin is varied; for arousal indicated “body movements” and for dominance the “size” of the manikin. Positive experiences are reported in that subjects quickly complete and easily, intuitively and unambiguously understand the response format, however, some authors criticize the validity of this scale.

PrEmo-instrument © The Product Emotion Measurement Instrument [38] also employs visual response categories but these pictograms appear as a cartoon character and are not static but animated on a computer screen. Animation visualizes changing intensity of seven positive (desire, pleasant surprise, inspiration, amusement, admiration, satisfaction, fascination) or seven negative emotions (indignation, contempt, disgust, unpleasant surprise, dissatisfaction, disappointment, boredom). Gutjar et al. [39] apply PrEmo © in a food consumption context but the rather small number of emotions may not be sufficient for the description of the various emotions elicited by product categories like foods.

EmoSensor This scale, developed in cooperation between academics (Gröppel-Klein et al. [40]) and the market research institute GfK, offers photographs representing individuals (of different ages, gender, cultural provenience) engaged in different activities as visual response categories when asking respondents to assess their emotional response to a certain stimulus (for instance an ad). Verbal labeling attached to the photographs (e.g., joy) turned out to be advantageous. The scale consists of three photographs for each of the 22 different emotions. Subjects select the photo which best reflects their emotional state. The authors emphasize that facial expressions might be ambiguous in some cases and that, therefore, testing for reliability and validity was of crucial importance.

EPT The Emotive Projection Test is a special type of thematic apperception test (TAT). The TAT, also known as “picture interpretation technique,” shows ambiguous scenes and by describing such scenes one can learn more about the participant’s emotions, motivations, and personality. The EPT was developed by Köster, Mojet, and Van Veggel [41] and consists of 30 pictures with neutral or ambiguous facial expressions. When participants are asked to which state emotion is portrayed in each picture (using the “check all that apply” technique), they project their own emotions, moods, and feelings into the faces on the pictures. Participants ascribe characteristics to these faces that depend largely on their own emotions. Therefore, this test is an implicit (i.e., third person) test. It was used, for example, to study the effect of flowers on the mood of restaurant visitors [41] and the effect of vanilla in yogurt on emotional responses [42].

IAT One of the most often referenced tests in Psychology is the implicit association test, developed by Greenwald et al. [43]. Like the associative network concept, the IAT is based on the idea that the human brain is structured as a neural network with highly related content being more closely connected in this net than loosely connected content. Subjects have to solve a set of easy association tasks on a computer by pressing either of two answer-keys; their reaction time is recorded. Given the network structure of the brain, it is thus easier for participants to react with the same answer-key on associated elements than using the other key and therefore they are quicker in answering to associated elements than to not associated elements. Whereas the IAT was developed for general purposes (i.e., associations in memory), stimulus material shown on the screen can be adopted for the measurement of the emotional elicitation potential of a certain object.

2.2 Methods using means of observation

2.2.1 Observational methods with human administration in a laboratory setting

Facial expressions Emotions are communicated through facial expressions in everyday life [44] by non-verbal means [45]. Twenty face muscles create a wide array of facial expressions and muscles on the head allow meaningful movements of the jaw and neck [46]. Humans have the impression that the interpretation of facial expressions occurs mainly automatically, immediately and without any effort humans know what a certain facial expression means—joy, disgust, or surprise. In contrast, the objective description and quantification of facial expressions as emotions remain a difficulty.

Two general approaches exist to analyze facial expressions in an objective way, judgment- and anatomically-based methods [47]. Whereas for the former a coder directly classifies facial expressions as a certain emotion, for the latter (s)he measures the movement or activity of specific facial muscles and then relates the resulting activity pattern to the identified emotion. Human coders can be very accurate in judging facial reactions, both for anatomically as well as judgment approaches provided, that they are extensively trained. The analysis is considerably time-consuming and expensive.

In this context, Ekman and Friesen [48] developed the Facial Affect Scoring Technique (FAST), a judgment-based method, and the Facial Action Coding System (FACS), an anatomically based method. Since inspecting certain facial muscles thoroughly takes time, subjects’ facial expressions are video recorded and their separate anatomical movements (i.e., “action units”) are identified using a slow-motion playback of the video. The coding schemes assist in transferring the identified action units into related emotional expressions. Marketing applications analyzed, among others, facial expressions of customers at the point of sale, or when interacting with salespersons, or when being exposed to commercials.

Only few published studies make use of human coders to investigate emotions elicited by food. Zeinstra et al. [49] found that children’s facial expressions, analyzed by human coders, are a good indication for disliking but not for liking of juices. Similarly, consumers showed more negative emotions just before tasting presumably insect-based chips compared to consumers who were expecting to taste protein enriched chips [50].

Body movements Body movements, including posture and gesture, can also be used as motor expressions of emotional states. Coding schemes (e.g., Berner System [51]) have been developed at about the same time as FAST and FACS. However, established relationships between bodily behaviors and emotions are less pronounced. Body movements are much more affected by culture, personal style, and context than facial expression. On the one hand, Weinberg [52] concludes that while facial expressions signal the type of emotional state, bodily expressions signal their intensity (e.g., a person who is feeling angry might gesture more forcefully, while a person who is feeling sad might gesture less). In a similar vein, individuals might more easily suppress or mask their true emotions with respect to facial expressiveness than with respect to body movements because they occur unconsciously to an even greater extent that facial movements. On the other hand, exemplary findings suggest that a person who is feeling confident might stand tall with their shoulders back, while a person who is feeling anxious might slouch or hunch their shoulders. In any case, the interpretation of body movements is highly complex and—to the best of the authors’ knowledge—a comprehensive manual (with respect to emotional states) has not been developed so far and as a consequence, a software for automatic emotional pattern recognition based on bodily behaviors is missing.

2.2.2 Observational methods with human administration in a real-life setting

Facial expressions body movements For these methods, the setting does not make a substantial difference for measurement (and we thus refer to the previous subsection). In general, a laboratory setting better controls for environmental conditions and better recording, thus increasing internal validity, a real-life setting reduces or even excludes reactive behavior of subjects and increases external validity.

Mystery shopping Mystery shopping is rooted in ethnographic research and may be described as a special form of participating observation. In more detail, a trained individual, known as a mystery shopper, poses as a customer in order to evaluate the performance of a business [53]. Mystery shopping is not typically used for the purpose of measuring emotions but rather to evaluate customer service, sales, and other aspects (e.g., general attitudes of persons involved in an encounter) of a business’ performance. To apply for emotion measurement, this individual needs to be trained in judgment-based expertise (see sub-Section 2.2.1) on relating facial or bodily expressions of customers and employees (e.g., during a sales or service interaction) to their emotional states. The short-term character of emotions, the unfeasibility of recording the interaction, and of taking notes limits the suitability of mystery shopping for emotion measurement.

2.2.3 Observational methods using technical equipment in a laboratory setting

fEMG The most direct and sensitive method to measure facial reactions is Facial ElectroMyoGraphy, where surface electrodes are attached to the face and the muscles’ activities are amplified and displayed on a monitor [54]. Two face muscles are of special importance: the zygomaticus major (responsible for the expression of smiling) indicates positive valence and intensity of emotions, the corrugator (responsible for frowning) negative valence. A considerable limitation of fEMG is the application of electrodes to the face. This can be rather intrusive to the subject, it obviously limits the implicitness of the study and might bias results.

Applications are reported from fine arts but rarely from consumer behavior. For tasting food products or looking at pictures of food products, studies using fEMG have shown that the activity of selected facial muscles correlates with self-reported hedonic responses [55, 56].

AFEA Automatic Facial Expression Analysis systems have become widely available in the last decade. AFEA systems employ a software to automatically analyze facial reactions using judgment-based and anatomically approaches from video and image sources. In comparison to the use of trained human coders, this approach reduces time and costs of emotion investigations substantially.

Most AFEA systems conduct a similar 3-step strategy for the classification of facial expressions: (1) Face acquisition—identification of the face, its position, and orientation, (2) Feature extraction—two general approaches are used; either the whole face is processed holistically, or specific areas of the face are selected and processed individually. (3) Classification: based on a complex model, facial expressions are classified using either a direct classification of emotions or an anatomically based coding scheme (e.g., activation of muscles or muscle groups).¹ Furthermore, sophisticated algorithms (deep learning systems) take gender and age into account. In a laboratory setting, initial calibration of the measurement device adopted to the subjects to be observed might also account for cultural characteristics (e.g., facial expressions of East Asians); in a real-life setting, algorithms might identify such peculiarities instantaneously.

In marketing, AFEA has been applied in many different situation (e.g., analyzing emotional potential of commercials, shop windows, web designs, stationary retailing facilities). In food sciences, AFEA analyzed emotions elicited by a wide range of taste or smell stimuli and food products, including basic taste solutions [57], beverages [58], confectionary [59], and also full meals [60].

One of the advantages of analyzing facial reactions is the temporal nature of the measurement and the fact that it does not bias the subjects’ natural behavior. Therefore, this methodology fits perfectly into the recent efforts to test food products in real-life situations and to investigate the effect of context on food perception and emotions with the intention to enhance external validity [60].

A general limitation is that only a few “basic” emotions are strongly associated with facial expressions, considerably fewer than used in explicit survey approaches (cf. subsection 2.1 and the number of emotions encompassed in the different scales; [34]), which potentially results in less fine-grained insights. Another disadvantage of AFEA so far is the fact, that small variations of facial expressions are not categorized into different emotions. For example, ironic or nervous grinning cannot be differentiated from a happy smile, although the emotion the underlying emotion is very different.

There are also some food-specific limitations. When emotions are measured during food consumption, oral processing including biting, chewing, and swallowing causes certain facial movements, which in turn might bias the simultaneous analysis of emotion-based facial expressions. Besides, both human coders and AFEA require high-quality video recordings of the subjects. Additionally, head movements or faces concealed by cutlery, cups, glasses, or even beards can negatively influence the quality of the results. Section 3 also demonstrates potential limitations by providing empirical examples.

EDR Electro dermal response is a measure of the electrical conductance of the skin and is widely used as a neurophysiological measure of emotional arousal. EDR is often quantified using a device called a skin conductance sensor, which consists of two electrodes that are placed on the skin (usually on the fingers or palms of the hand) to detect the electrical conductance of the skin. The sensor sends a small electric current through the skin and measures the resistance of the skin to the current. Boucsein [61] emphasizes on the multidimensionality of arousal which impedes direct interpretation of skin conductance amplitude. Complementary data collection is required to distinguish whether emotional arousal refers to the affect dimension (i.e., flight/fight) or the preparatory dimension (i.e., readiness of behavioral action). EDR is regarded as a reliable and easy-to-use instrument which does not depend on the subjects’ cultural origin.

EEG Electro encephalography is a technique that measures the electrical activity of the brain. It uses electrodes placed on the scalp to detect and record the electrical impulses of neurons in the brain. These activities (amplitude and frequency of brain waves) in turn indicate different emotional states, however, special expertise is required to interpret results from such a measurement procedure. Furthermore, high-precision (and hence costly) instruments are required for exact identification of the brain region in which the neural activities take place. A more practical disadvantage might be subjects’ (in particular women’s) reluctance to accept electrodes on the scalp and thus potentially damage hairdressing.

PET, fMRI Positron emission tomography and functional magnetic resonance imaging are both medical imaging techniques. PET uses radioactive tracers, fMRI magnetic field, and radio waves to produce detailed images of the brain. For the present utilization changes in brain activity associated with different emotional states are of interest (e.g., the amygdala is known to be central for emotional processing). PET, fMRI are invasive methods and their high costs limit applications for emotion measurement.

Neural measures such as EEG, PET, and fMRI share in common that they are costly, require special expertise for executing and exploiting, are restricted to small samples and might suffer from highly controlled experimental conditions. However, they deserve credit due to the fact that they enabled the validaftion of more easy-to-use neurophysiological methods [61].

Program analyzer The program analyzer implements the concept of Mehrabian and Russell [29] that individuals approach emotionally positively but avoid emotionally negatively perceived environments (see also Rolls [10] in subsection 1.2). In more detail, subjects are exposed to certain stimuli (e.g., a commercial) and requested to move a slider toward themselves, away, respectively, and thereby continuously express emotional valence. Whereas the program analyzer was originally developed in 1937 by Lazarsfeld and Stanton to record people’s moment-to-moment reactions to radio programs [62], nowadays, this audience measurement instrument has been used for testing ads (see [63]). Neibecker [64] verified reliability and validity of this procedure.

2.2.4 Observational methods using technical equipment in a real-life setting

Voice pitch analysis is a technique that involves measuring and analyzing pitch, volume, intonation, and speaking tempo of a person’s voice, which can be used as an indicator of their emotional state. For example, a person’s pitch may become higher when they are excited or happy, and lower when they are sad or angry. Accent or deep-throat might impede exploration. Voice pitch analysis can be done using software (e.g., PRAAT) that analyzes audio recordings of a person’s voice. We note that recordings may be done in real-life (background noise might cause distortion in this case) and laboratory settings.

AFEA For automatic facial expression analysis, the setting does not make a substantial difference (and we thus refer to the previous subsection for some methodological explications). Commercial software (e.g., FaceReader provided by Noldus, EmoScan used by GfK, software provided by iMotions) and academic software (e.g., by Fraunhofer-Gesellschaft [65]) is readily available for such an analyses (see [66] for a respective overview). Technology advanced considerably such that remote analysis became feasible. Subjects in front of a laptop (or even their cellular telephone in a fixed position), and either the built-in camera or a camera mounted on the screen scans their face while viewing content.

Wearables There are several options which are available for determining the neurophysiological component of emotional arousal. Here we list those which do not require a laboratory setting. EDR (see subsection 2.2.3) equipment might be stored in a bag which then has to be carried by subjects when experiencing real-life settings (e.g., walking along the aisles of a supermarket). Pupillary dilation (the increase in the size of the pupils) can also be used as a physiological measure of emotional arousal. The size of the pupils is controlled by the sympathetic nervous system, and is known to increase during certain emotional states (such as fear, anxiety, and excitement). Pupillary dilation can be measured using infrared light (for instance as an aside when conducting mobile eye-tracking). Heart rate is still another physiological measure of emotional arousal. The heart rate is controlled by the autonomic nervous system, and is known to increase during certain emotional states. Heart rate can be measured in many different ways (for instance as an aside when conducting EDR). Results of both methods might be affected by other factors such as lighting / physical activity, age, and medications.

3. Examples illustrating automatic facial expressions analysis technology

The following examples aim to illustrate potential shortcomings associated with recent technological advances in facial recognition. The authors urge researchers and marketers to carefully consider raw data in line with statistical results to avoid misleading interpretations. For instance, pure emotion tracking over time (when analyzing commercials) might not necessarily yield causal relationships between the emotions expressed by the presenter and those aroused for the viewers. Besides, different measurement tools might not always result in equivalent outcomes and finally following the results without a critical reflection could lead to a wrong interpretations of the respondents’ actual intentions.

3.1 Measuring emotions of a commercial

Recently, advertising and market research agencies started promoting their business portfolio by pointing to their capacity to determine the emotional impact of commercials.² Methods to conduct such kind of analyses differ, but frequently either the emotional appearance of the endorser (for instance a celebrity) or the emotional response of a sample of respondents (or both) are tracked continuously over the whole duration of the spot. Most commonly in such a setup, AFEA technology is employed. This example aims to demonstrate that measurements of emotions can only be viewed as an initial step because the interpretation of such data requires special competence. Some market research agencies collected data of emotional evaluations of a broad range of ads, established certain benchmarks or developed artificial intelligence-based diagnostics tools. These resources in turn enable providing management recommendations to the agencies’ clients.

For this purpose,³ we cooperated with a business (Austrian bakery Felber) and analyzed a commercial posted on social media channels (i.e., Facebook)⁴ during the pandemic crisis in 2020. This bakery is a midsized company (with approximately 440 employees and about 50 shops located in Vienna⁵). In the commercial spot, the co-owner approached customers of hardware stores (as hardware stores were allowed to reopen retailing services after the first lockdown) to visit shop-in-shop outlets located in some of these hardware stores. The presenter frontally faced the camera (i.e., the audience), her face looked lighthearted, she smiled frequently and presented a variety of different products (e.g., bread, buns, bread rolls). She spoke in a dialect with the aim to target do-it-yourselfers by making use of ambiguous wording and wordplays (e.g., “You do your home yourself and Felber offers homemade bakery”). This unique and authentic presentation went viral and was heavily discussed on social media (while some people liked the spot very much, others disapproved her wordplay to a great extend).

Our analysis is based on three sources of information. (1) Content of the ad. The ad lasted about 1 min; during the first 16 s, the co-owner presented the reopening of her shops in hardware stores as a present for the upcoming Easter holidays. During the main body of the spot (seconds 17–46) she presented six different products (each for about 5 s) using humorous speech. In the last part, she essentially asked spectators to visit her stores and to purchase. (2) Emotional expressions of the presenter. The FaceReader (of Noldus) measured basic emotions (anger, disgust, fear, happiness, sadness, surprise) of the presenter with five observations per second. This fine-grained data pay regards to Scherer’s [1], p. 702) view that “events, and particularly their appraisal, change rapidly […] the emotional response pattering is also likely to change rapidly as a consequence.” (3) Emotional expressions of a sample of subjects. A sample of 31 subjects watched the video and their emotions during the exposure were measured with the same granularity as indicated earlier by employing the online version of the Noldus FaceReader (data collection took place during the pandemic period which did not allow physical contacts between researchers and subjects). Data were aggregated over respondents resulting in a single trajectory for each of the six basic emotions. In order not to overload the presentation, we concentrate on the presentation of results with respect to anger and happiness which reflects the discussion about this spot in social media ([1], p. 707, also emphasizes the dominant role of these two emotions).

Figure 1 exhibits trajectories of anger and happiness aggregated over subjects and trajectories for the presenter. The horizontal axis reflexes the time shows, the three stages of the ad, and the time slots (1, …, 6) of presenting the six products. The vertical axis is to be interpreted as intensity of measured emotions—with a domain between 0 and 1. Aggregated trajectories are much smoother because averaging balanced individual differences. In accordance with the intention of this spot, happiness is more pronounced than anger. The presenter emphasized the demonstration of her products by happy facial expressions most of the time (cf. trajectory’s peaks). Subjects’ happiness increased during the second part of the spot continuously but decreased during farewell. Anger does not play a role because its trajectories stayed almost constant (at a low level for all subjects, near zero for the presenter).

Figure 1.

Emotional trajectories during exposure to the ad.

Table 2 offers descriptive statistics for the emotional trajectories. Average values for anger are much lower than for happiness and this also applies for standard deviations. The latter is most pronounced for happiness of the presenter. Whereas these figures nicely mirror the situation depicted by Figure 1, we are further interested whether there is a causal relationship between the emotions expressed by the presenter and those aroused for the viewers. Or put differently, did the presenter’s facial expressiveness evoke subject’s emotions? From a pure data analytic standpoint, correlation coefficient might shed some light on such an echoing effect (emotional contagion). Table 2 reports that there is no significant correlation for anger. Due to the short temporal distances between subsequent observation (five observations per second) time series of emotions ($X_{\mathit{ti}},Y_{\mathit{ti}}$; $t$ denotes time, $i$ type of emotion) are autocorrelated and, consequently, correlation coefficients based on the raw data ($X_{\mathit{ti}},Y_{\mathit{ti}})$ might be misleading. Accounting for autocorrelation, we also computed correlation coefficients based on the first differences ($∆X_{\mathit{ti}}=X_{\mathit{ti}}-X_{t-1i},∆Y_{\mathit{ti}}=Y_{\mathit{ti}}-Y_{t-1i}$) which turned out to be very small (cf. Table 2). There might be some response latency of subjects because their reactions might not occur instantaneously. Thus, we repeated this procedure by lagging emotions from the presenter (i.e., $X_{t-si},s=1,2,\dots )$but again did not find significant correlations. Consequently, inferences purely based on statistics would state that the presenter’s facial expressiveness did not impact respondents’ emotions (which contradicts Figure 1 displaying increasing happiness during exposure to the ad). Of course, there might be other drivers of subjects’ emotions (such as the verbal content of the presenter’s presentation, her voice, the displayed products, etc.) besides her facial expressions. Nevertheless, we aim to emphasize that pure emotion tracking over time (when analyzing ads) might fall short in some aspects despite the fact that this is a powerful instrument prima facie.

	Anger		Happiness
	Mean	Standard deviation	Mean	Standard deviation
All subjects, $Y_{\mathit{ti}}$	0.051	0.006	0.186	0.087
Presenter, $X_{\mathit{ti}}$	0.002	0.003	0.121	0.156
$\mathit{\text{Corr}}\left(X_{\mathit{ti}},Y_{\mathit{ti}}\right)$	−0.041		0.411
$\mathit{\text{Corr}}\left(∆X_{\mathit{ti}},∆Y_{\mathit{ti}}\right)$	−0.033		0.093

Table 2.

Descriptives of emotional trajectories for the ad.

3.2 Reliability of automatic facial expressions analysis

Measurement procedures based on observational methods employing a technical equipment usually score highly on aspects of reliability as potential errors due to human intervention can be prevented. The typical user of technical measurement equipment (purchased from a professional supplier) accepts the instrument provided and does not question issues of implementation. Whereas the basic idea of an AFEA (as described above) is generally accepted, implementation might vary, however. The given example highlights potential difficulties resulting therefrom, as recent literature points to the limited number of “independent peer-reviewed validation studies” ([66], p. 10). To date, prior research focuses mainly on “deliberately posed displays”, as compared to naturalistic expressions.

For the present case, we compare results achieved by analyzing identical stimuli employing naturalistic expressions by two distinct, commercially available instruments (iMotions, Noldus⁶). According to their websites⁷ both providers explain that the recognition software works in basically three steps: (1) the position of the face is framed within a box; (2) facial landmarks such as eyes and eye corners, brows, mouth corners, the noise tip, etc. are detected; and (3) based upon these key features classification algorithms identify action unit codes and emotional states. These algorithms make use of artificial neural networks which have been trained on large databases. These databases might vary in targeting special groups of interest (e.g., East Asians, elderly, children) and as pointed out “you might get slightly varying results when feeding the very same source material into different [classification] engines” (iMotions, p. 21).

The designs of this and the previous example have in common that short commercials (with a rather dominant endorser presenting a product or service) are used as objects of investigation, in particular, the emotional facial states of the endorsers are of interest. In this case, 13 different video clips serve as stimuli to be analyzed with AFEA (all of approximately the same length, 60 seconds). Importantly, presenters exhibit naturalistic expressions in a controlled setting. Specifically, the study employed stimuli (high-quality resolution and professional lightning) under controlled conditions by depicting full frontal shots with a neutral background and a frontal head orientation. No accessories (i.e., eyeglasses) were used to assure optimal conditions for facial recognition. The content of the commercials is not relevant in the present case, because we compare results of different measurement procedures applied to the same data. The two measurement instruments analyzed faces of these 13 presenters and provided average (calculated over the duration of the ad) amounts of emotional expressiveness for all 6 basic emotions every time: $X_{\mathit{ij}}^{\left(m\right)},$ with $m$ is the type of measurement {iMotions, Noldus}, $i$ is the type of emotion {anger, disgust, fear, happiness, sadness, surprise}, and $j$ is the type of presentation {1, 2, …, 13}.

Table 3 presents (selected) descriptive statistics. Focusing on presentation 1, columns 2 and 3 (rows 3–8) contrast emotional displays as measured by the two procedures (figures for both types of measurement are to be interpreted as intensity with a domain between 0 and 1). We observe considerable differences, which underpins recent findings by Dupré et al. [66], who compare eight different facial recognition tools and found that “there was considerable variance in recognition accuracy ranging from 48% to 62%” (p. 1). To emphasize this point, row 11 exhibits correlation coefficients (calculated over the type of emotion) for presenter 1. Because of the small number of observations, the nonparametric correlation coefficient due to Spearman has been included but even for this metric coherence is rather modest. Due to lack of space, Table 3 only presents results for presentation 1 but results for the other presentations yield similar findings. The right part of Table 3 demonstrates this claim. Rows 3–8 show correlation coefficients (calculated over presentations) for each emotion. Results are quite discouraging, in particular for the emotion of anger (correlations are even negative). Row 11 displays the range (over presentations) of correlation coefficients (calculated over the type of emotions). Whereas in some cases results of measurement seem to be consistent (as documented by large correlation coefficients), for others this does not seem to be the case since even negative correlations can be observed⁸.

	Presentation $j=1$		$\underset{j}{\underbrace{\mathit{\text{Corr}}}}\left(X_{\mathit{ij}}^{\left(1\right)},X_{\mathit{ij}}^{\left(2\right)}\right)$
Emotions $i$	Measurement iMotions	Measurement Noldus	Pearson	Spearman
Anger	0.001	0.003	−0.275	−0.310
Disgust	0.095	0.111	0.064	0.474
Fear	0.060	0.093	0.690	0.508
Happiness	0.079	0.256	0.583	0.630
Sadness	0.003	0.031	0.416	0.616
Surprise	0.650	0.050	0.787	0.630
	$j=1$		Range over $j$
	Pearson	Spearman	Pearson	Spearman
$\underset{i}{\underbrace{\mathit{\text{Corr}}}}\left(X_{\mathit{ij}}^{\left(1\right)},X_{\mathit{ij}}^{\left(2\right)}\right)$	−0.103	0.600	(−0.240;0.973)	(−0.131;0.941)

Table 3.

Descriptives of displayed emotions for 13 different presentations.

Whereas it is widely known that measurement of facial expressions might suffer from subjects’ mascara, eyeglasses, knitting their eyebrows, etc. the amount of discrepancy identified in the current context is not very promising, given the claim of technology providers that the software is measuring the same phenomenon (and in the same units). We aim to explicitly emphasize that we neither favor one of the two approaches over the other nor aim to provide a recommendation at this point. Searching the relevant literature, however, we found several papers confirming validity of the Noldus software (e.g., [67]).

We definitively want to draw attention to this unfortunate state of affairs because results of investigations using AFEA technology might depend on the employed measurement approach (an unpleasant situation for a researcher). Recent studies show that “human observers clearly outperformed all automatic classifiers in recognizing emotions from both spontaneous and posed expressions” ([66], p. 11). One possible explanation for the discrepancies between human and computer-automated coding procedures as well as major differences between software providers might be ascribed to “the quality and quantity of data available to train computer-based systems […], most current automatic classifiers have typically been trained and tested using posed or acted facial behavior” ([66], p. 11). Thus, we urge for the undertaking of clear efforts to consolidate these measurement procedures by considering reliability (or even validity) of this technology in a naturalistic setting.

3.3 Example from food science (self-reports vs. explicit AFEA vs. implicit AFEA)

Studying facial expressions in an implicit way might enhance the external validity of gained data, which means that such results might be better in explaining and predicting the behavior and experience of humans in real life. Therefore, a study in the test booths of the sensory Laboratory at University of Natural Resources and Life Sciences with 99 subjects compared data from an implicit face reading design with explicit data from willingly expressed facial expressions and self-reported liking. Judging products via self-reported liking by means of a 9-point hedonic scale is a widespread method in sensory science and we wanted to test whether facial expressions measured by AFEA (FaceReader 5 by Noldus Information Technology) and self-reported liking yield similar or dissimilar information.

The testing procedure in a nutshell was as follows: The subjects got instructions about the test on the notebook in front of them, but in this first phase subjects were not aware, that they were videotaped during the test. Throughout the experiment, Compusense® five (Compusense Inc.) software was used to present the questionnaire, to guide the participants through the testing procedure and for data collection. After mounting the electrodes of the autonomic nervous system measuring device on the fingers, a short familiarization phase took place, in which subjects got used to their slight pressure on the finger and generally came to rest. During the whole testing procedure subjects were video-recorded by the camera of the notebook; thus, subjects did not recognize being video-recorded. Then subjects were asked to drink a sample of juices from a shot glass (banana, orange, mixed vegetable, grapefruit, sauerkraut). A short time period after swallowing the juice was used to measure the implicit facial expressions via FaceReader. Then subjects were requested to raise their hand and to show how they liked the juice by making an appropriate face, which served as the explicit measure of their emotions. Afterwards they were also asked to rate the hedonic impression of the juice on a 9-point hedonic scale.

Highly significant differences between juice samples were detected in the implicit and in the explicit facials reactions measurements. Disgusted, happy, neutral, and sad were significant in the implicit approach and angry, disgusted, happy, neutral, and also sad in the explicit approach (Table 4).

Implicit facial expression	Explicit facial expression
angry	angry ***
disgusted ***	disgusted ***
happy ***	happy **
neutral ***	happy ***
sad **	sad *
scared	scared
surprised	surprised

Table 4.

Implicit and explicit facial expressions.

Note: Significant differences are marked as * (p < 0.05 and > 0.01), ** (p < 0.01 and > 0.001), *** (p < 0.001).

The depiction of the coherence between explicit facial expression and self-reported liking is as expected (Figure 2). Disgusted and sad are highest for low liking ratings and low for high liking and vice versa is true for happy. When it comes to the relation of implicit facial expressions and self-reported liking (Figure 3) a remarkable aspect becomes evident. As expected, disgust and sadness are high when self-reported liking is low, but it is salient, that also happiness is high when liking is low, whereas happiness is not high for the medium to high ratings between 4 and 9 of the 9-point hedonic scale. Our hypothetical explanation is, that the FaceReader was not able to differentiate between various forms of smiling, which humans are easily able to perceive and interpret correctly as a specific, different emotions. The implicit smile subjects were showing in this study was not the expression of happiness, but it was probably a smile of rejection and embarrassment, maybe also in some cases a kind of nervous and amused laughter about the unexpected disgusting and strange sauerkraut juice we were serving to them. So, this example documents one of the limitations of an AFEA-system. Following the results of this study without critical reflection one could decide to design juice products which elicit implicit reactions detected as happy by the AFEA, which in fact do not really reflect happiness but rejection, embarrassment, ironic or nervous smile reactions to an unexpected or strange stimulus.

Figure 2.

Relation between self-reported liking and the intensity of emotions implicitly measured by a FaceReader.

Figure 3.

Relation between self-reported liking and the intensity of willingly expressed facial emotions measured by a FaceReader.

There is no doubt that AFEA systems are progressing quickly. New statistical methods allow better insights into the temporal development of emotions and new deep-learning algorithms have been developed to better deal with partial concealments. However, we want to draw the attention to the many shades of emotions AFEA systems are struggling with. The meaning of variants of facial expressions and the muscle activation patterns might also be different in various cultures. Thus, without a better and more accurate and reliable categorization of facial expressions into meaningful emotions results of AFEA systems could be misleading.

4. Conclusions

This chapter provided the reader with a comprehensive overview on emotional measurements procedures. Given constrained space we limited our presentation to the basic ideas of these methods. Pointing again to Scherer’s [1] component process model we emphasize that, typically, a certain method only considers one emotional component and likely falls short in accounting for others. Before deciding on a certain type of measurement, the researcher should thus determine these dimensions of the construct emotions which are of particular relevance in the current situation. Even better would be a triangulation of different methods in order to increase validity and reliability—validity of a method should never be taken for granted but always assessed in the context of the research question.

Rapid recent developments of automatic facial expressions analysis encouraged devoting special attention to AFEA. New statistical methods allow better insights in the temporal development of emotions, new algorithms have been developed to better cope with partial concealments (e.g., deep learning algorithms). Advances in software technology increased applicability, user-friendliness, processing speed and variety of results provided. Given our own experience (partly presented in Section 3), however, we again caution against the uncritical acceptance of communicated results (even from well-established research companies) but rather urge more research and efforts to increase validity and consistency of results achieved by different procedures which claim to measure the same construct when applied to identical data.