Open access

Design, Implementation and Evaluation of a Multimodal Sensor System Integrated Into an Airplane Seat

Written By

Bert Arnrich, Cornelia Kappeler-Setz, Johannes Schumm and Gerhard Trooster

Submitted: 26 October 2010 Published: 13 June 2011

DOI: 10.5772/18237

From the Edited Volume

Sensor Fusion - Foundation and Applications

Edited by Ciza Thomas

Chapter metrics overview

2,478 Chapter Downloads

View Full Metrics

Keywords

1. Introduction

Air travel has become the preferred mode of long-distance transportation for most of the world’s travelers. People of every age group and health status are travelling by airplane and thus the airplane has become part of our environment, in which passengers could benefit from assistive support. In this regard, the European research project SEAT has investigated sensor technologies to provide assistive support related to the health and well-being of airplane passengers. Since the main interaction point between a passenger and the airplane is the seat, a seat-integrated sensor system was developed to measure health and affect-related signals of a passenger. The measured signals include the electrocardiogram (ECG), electrodermal activity (EDA), skin temperature, respiration as well as movement of the passenger. In this chapter we describe the design, implementation and evaluation of the seat-integrated sensor system. In particular we highlight two approaches of sensor fusion in order to appraise the signal quality in an airplane scenario and to identify the passenger’s affective state.

In the first part, we show how the design of the seat-integrated sensor system is influenced by the trade-off between sensor comfort and signal quality: To achieve the acceptance and hence the use of the system, the sensors need to be attached in a comfortable and non-obtrusive way or even be totally integrated into the seat. On the other hand, a comfort-optimized sensor placement usually limits the signal quality. We argue that not only the development of comfortable and reliable sensor technology but also the quality appraisal of the data generated by the sensors needs to be addressed. Artifact detection through sensor fusion is presented and the working principle is shown in a feasibility study, in which normal passenger activities were performed. Based on the presented method, we are able to identify signal regions in which the accuracies for detecting the heart-rate is 88% compared to 40% without any artifact removal [Schumm et al., 2010].

In the second part, we will explain another sensor fusion approach in the context of emotion recognition. Previous work on emotion recognition from physiology has rarely addressed the problem of missing data. However, data loss due to artifacts is a frequent phenomenon in practical applications. Discarding the whole data instance if only a part is corrupted results in a substantial loss of data. To address this problem, we investigated two methods for handling missing data: imputation and reduced-feature models using ensemble classifier systems. The five emotions amusement, anger, contentment, neutral and sadness were elicited in 20 subjects by film clips. The following six physiological signals were recorded by means of the seat-integrated sensor system and an external recording device: ECG, electromyogram (EMG), electrooculogram (EOG), EDA, respiration and finger temperature. Results show that classifier fusion increases the recognition accuracy in comparison to single classifiers using imputation by up to 14%. We were able to analyze 100% of the data even though only 47% of the data was artifact free. Since more artifacts are expected in the “field” than in the laboratory, the proposed methods are especially beneficial for practical applications, e.g. in the airplane [Setz et al., 2009].

Advertisement

2. Design and implementation of a seat-integrated multimodal sensor system

Since air travel has become the preferred mode of long-distance transportation, the airplane will soon be part of our environment in which people may need assistive support. The European research project SEAT aims at extending existing airplane seats with new sensor technologies in order to assess and improve health and well-being of the passengers. A first step in this direction is the reliable and unobtrusive recording of relevant physiological signals. In order to achieve the acceptance and hence the use of the system, the sensors need to be attached in a non-obtrusive way or even be totally integrated into the environment. Thus, conventional sensors and measurement locations, such as wet electrodes at the chest for cardiovascular monitoring, are not feasible due to the lack of acceptance by the passengers.

Figure 1.

Airplane seat with integrated unobtrusive sensors [Schumm et al., 2010]

In order to record health and affect-related signals of a passenger in an unobtrusive way, the following signal modalities were measured by sensors integrated into an airplane seat: electrocardiogram (ECG), electrodermal activity (EDA), skin temperature and respiration (see Figure 1). The comfort aspect was targeted by (i) incorporating a contact-less ECG measurement system; (ii) using dry electrodes at the fingers instead of wet electrodes at the chest; (iii) combining ECG, EDA and temperature sensors into one setup in order to reduce the number of electrodes; and (iv) incorporating the respiration measurement into the safety belt of the airplane seat. A main concern of the system design was to make sure that signal disturbances can be detected. Since passenger’s movements induce signal disturbances, additional sensors for measuring the movements of the fingers and the contact pressure at the back rest were integrated into the seat. These additional sensors, referred to as artifact sensors, are able to measure movement patterns that provoke artifacts, which influence the signal quality of the physiological signals.

2.1. Sensor system

For the ECG measurement, two measurement systems were incorporated into the seat. The first system consists of a contactless capacitive ECG system developed by RWTH Aachen University [Steffen et al., 2007]. This system, referred to as “Contactless-ECG”, allows measuring the ECG as unobtrusively as possible. It was incorporated in the backrest of the airplane seat and measures the ECG capacitively without direct skin contact. Since the Contactless-ECG system is sensitive to body movements, additional pressure sensors were incorporated into the backrest of the seat in order to measure the contact pressure between the body and the backrest. In order to find adequate locations for the pressure sensors, a feasibility study was conducted. The contact pressure at the back of the seat was recorded with a pressure mat, simultaneously with the contactless ECG signal. Based on a visual inspection of both signals, it was decided to place one pressure sensor at the top and another one at the bottom of each ECG electrodes. The second ECG system, referred to as ‘‘Finger-ECG’’, measures the ECG at the index finger of both hands. The used dry electrodes that are fixed to the fingers provide a higher user comfort compared to wet electrodes attached to the chest. Due to the direct skin contact, this system is clearly more obtrusive than the ‘‘Contactless-ECG’’ but also more reliable. However, finger movements can also evoke artifacts in the ECG signal. In order to spot these finger movements, the finger stripes for fixating the electrodes are equipped with 3-axis accelerometer sensors.

Since the EDA measurement requires direct skin contact, the EDA is recorded at the index and middle finger as proposed in literature [Boucsein, 1992]. The implemented measurement principle is referred to as an exosomatic quasi constant voltage method. Hereby, a constant voltage is applied to the electrode at the index finger, leading to a current flowing through the skin to the other electrode. This current is measured and thereby the

Figure 2.

Combining the measurement of EDA and ECG at the fingers [Schumm et al., 2010]

skin conductance can be assessed. In order to reduce the overall number of electrodes attached to the fingers, a novel concept for combining the measurement of EDA and ECG at the fingers was developed: the electrode at the index finger of the left hand was used for both measurements, ECG and EDA. As a result, the number of electrodes could be decreased from four to three (see Figure 2). For the ECG measurement, the high offset voltage at the left index finger, caused by the parallel EDA measurement, had to be considered.

For the measurement of the respiration, common approaches require a belt strapped around the chest. In our setting, a textile resistive sensor was incorporated in the seatbelt. During the breathing cycle, the sensor is periodically stretched and the resulting resistance change is measured.

For the measurement of the skin temperature, a commercially available sensor was chosen. Due to the small size of the sensor (5mm x 9.5mm x 9.1mm), it was incorporated beside the left electrode of the EDA measurement and does therefore not further decrease the perceived comfort.

2.2. Artifact detection through sensor fusion

Artifacts lead to signal disturbances and thereby might lead to a wrong interpretation of extracted features from the physiological signals. The proposed multi-modal sensor system allows artifact detection. The artifact sensors are used to spot regions of artifacts and consequently the corrupted sequences are not considered for further processing. In Fig. 3 the working principle is shown.

Figure 3.

Working principle of artifact removal through sensor fusion [Schumm et al., 2010]

For the Contactless-ECG and the respiration signals, movements of the upper body lead to artifacts. The Contactless-ECG system delivers good signal quality as long as a constant contact pressure exists between the upper body part and the back of the seat. Movements of a person like leaning forward change the elongation of the respiration sensor without being related to breathing. This information can be measured using pressure sensors incorporated into the back of the seat. By summing up the four pressure signals, a single artifact signal referred to as “contact pressure” is obtained. If no contact or fast changes due to movements of the upper body signal are detected, the signal is marked as corrupted. In Figure 4, a signal example of the contactless ECG and the contact pressure is shown. It can be observed that changes in the contact pressure coincide with disturbances in the contactless ECG signal. In particular, it can be seen that the R-peaks, which are the most characteristic features of the ECG signal, are no longer visible during periods of varying contact pressure. As a result, it would not be possible to calculate the heart rate in those periods.

Figure 4.

Changes in the contact pressure due to upper body movements lead to a temporary disturbance of the contactless ECG signal [Schumm et al., 2010]

For the EDA and ECG measurements at the fingers, movements of the fingers or the hand can evoke artifacts. In order to spot these finger movements, the finger stripes of both index fingers are equipped with 3-axis accelerometers. The three dimensions of both accelerometers are added up to form a single artifact signal referred to as “finger movement”. In Figure 5, a signal example of the finger ECG and the corresponding artifact signal is shown. It can be observed that the ECG signal is corrupted in regions with strong finger movements. Similar to the contactless ECG, it can be seen that the R-peaks are no longer visible during periods of movements.

In order to evaluate the performance of the proposed artifact detection through sensor fusion, we first define a quality measure to appraise the signal quality before and after artifact removal. The Quality Index (QI) describes the correctness of a characteristic feature extracted from the ECG. As already mentioned, the most characteristic feature of the ECG is the R-peak. We therefore compare the R-peaks extracted from the contactless ECG with the R-peaks extracted from a simultaneously recorded ground truth ECG signal. The ground truth signal was recorded from the chest using wet electrodes. For each detected R-peak in the ground truth ECG, we searched for a single corresponding R-peak in the contactless ECG, within a time interval of 150ms. The overall quality measure is defined as the ratio of correctly identified R-peaks in the contactless ECG divided by the total number of R-peaks detected in the ground truth ECG signal.

Figure 5.

Movements of the fingers evoke artifacts in the finger ECG measurement [Schumm et al., 2010]

The benefits of artifact removal were investigated in a feasibility study. A test subject was sitting in the seat while performing typical airplane activities such as entertainment, working, reading, sleeping and eating. Each activity was performed for 10 minutes. During these activities, the contactless ECG and the ground truth ECG were measured synchronously. After data recording, the quality measure was calculated before and after artifact removal. The resulting quality measures are presented in Table 1. First, it can be observed that the quality measures are substantially higher for the calm activities (entertainment and sleeping) in comparison to the more active activities (working, reading and eating). This reflects an inherent constraint of the contactless ECG system: it only delivers good signal quality if a good and constant contact pressure exists between the upper part of the body and the back of the seat. Second, it can be observed that regions with high activity are almost totally discarded by the artifact removal method: for working and reading all the data is discarded whereas for reading the remaining signal length is only 7%.

Entertainment (%) Working (%) Reading (%) Sleeping (%) Eating (%)
Quality measure before artifact removal 71 16 32 63 17
Quality measure after artifact removal 88 - 47 97 -
Remaining signal length 38 0 7 35 0

Table 1.

Quality measure before and after artifact removal for typical airplane activities

Taking all activities together, almost 84% of the data is discarded while the mean quality measure (weighted by the remaining signal length) rises from 40% to 88%. If the remaining signal length found in our study is scaled up to a 12h flight, 2.8h of the Contactless-ECG data is expected to be almost artifact free and would be available for computation of health indicators.

Advertisement

3. Sensor fusion in emotion recognition

Applications for emotion recognition are predominantly found in the field of Human-Computer Interaction (HCI). By including emotions, HCI shall become more natural, i.e. more similar to human-human interactions where information is not only transmitted by the semantic content of words but also by emotional signaling in prosody, facial expression and gesture. In recent years several research groups have employed pattern recognition methods in order to automatically detect different affective states of a subject. Modalities which have been used to detect affective states include facial expression [Busso et al., 2004], speech [Neiberg et al., 2006] and physiological signals [Kim & André, 2008].

Previous work on emotion recognition from physiology has rarely addressed the problem of missing sensor data. In general, a multi-modal data set is recorded, e.g. simultaneous recordings of ECG, EDA and respiration. If a single modality fails, the entire data instance, i.e. all the remaining signal modalities, are usually discarded. This results in a substantial amount of unusable data for classifier training. Moreover, predicting emotions during run-time becomes impossible, if any of the signal modalities that were used to train the classifier fails. Since data loss due to artifacts is frequently encountered not only in our airplane scenario but also in many practical applications, missing values represent a serious problem that needs to be addressed but has so far gained little attention in emotion recognition from physiology. We therefore investigated two methods for handling missing data: imputation and reduced-feature models using ensemble classifier systems.

Figure 6.

Emotions to be recognized in the 2-dimensional emotion model of arousal and valence [Setz et al., 2009]

3.1. Emotion elicitation

The emotions to be recognized were chosen according to the well-known 2-dimensional emotion model of arousal and valence often used in emotion recognition studies [Cowie et al., 2001]. One emotion in each quadrant plus neutral were selected as shown in Figure 6: amusement (high arousal, positive valence), anger (high arousal, negative valence), contentment (low arousal, positive valence), sadness (low arousal, negative valence) and neutral (medium arousal, zero valence).

For each emotion and the neutral state one film clip has been chosen since films are capable of eliciting strong emotional responses under highly standardized conditions which enables replication studies [Rottenberg et al., 2007]. Twenty (12 male, 8 female) participants with a mean age of 28.6 years were recruited for the experiment. After sensor attachment the subjects had 10 minutes to rest. Afterwards, the five film clips were presented to each subject in a constant order. Between two film clips a recovery time of 3 minutes was introduced. During the experiment the following physiological signals were recorded with the seat-integrated sensor system explained above: ECG, EDA, respiration and finger temperature (see Figure 7). In addition, two sensor modalities were simultaneous recorded with an external device: vertical component of the Electrooculogram (EOG) and the Electromyogram (EMG) of the muscle between mouth and eye (see Figure 8).

Figure 7.

ECG and EDA measurements at the hand, respiration measurement in the seat belt [Setz et al., 2009]

Figure 8.

Subject wearing video glasses to watch emotion eliciting film clips. Vertical EOG and EMG measurement [Schumm et al., 2010].

3.2. Emotion recognition

From all six physiological signals, a total of 53 features were extracted. In a first step, the following erroneous signal modalities were detected: (i) EDA signal reached saturation, (ii) erroneous R-peak detection from the ECG due to motion artifacts, and (iii) eye blinking is not visible in the EOG due to dry skin. This resulted in 96% correct data for EOG, 78% for EDA, 64% for ECG and 100% for the remaining modalities. The percentage of data containing only valid features of all modalities amounted to 47%. In order to tackle the low amount of data containing only valid features, we investigated imputation of missing data in combination with two classifier fusion approaches as outlined in the following.

Single classifier with imputed features: Missing feature values in the training data are imputed by the mean value of the available training instances belonging to the same emotion class. All signal modalities are used to train a classifier in order to discriminate the emotion classes. Missing features in the test data are imputed by the mean value of the entire training feature vector independent of the class membership of the test sample.

Fusion of classifiers with imputed features: Missing feature values are imputed in the same way as for the single classifier. For each signal modality, a separate classifier is trained. The decisions of those classifiers are fused according to majority or confidence voting. In majority voting, the class that receives the highest number of votes is chosen. In confidence voting, the class of the classifier with the highest confidence is chosen.

Fusion of classifiers with reduced-feature models: Missing feature values are not imputed. Like above, a separate classifier is trained for each signal modality. In case a missing feature is encountered in a certain signal modality, this modality is not used for training the classifier. In the test set decisions of all the available classifiers are fused according to majority or confidence voting.

Fusion Method Classifier Imputation Accuracy 5 classes Accuracy Arousal Accuracy Valence
None LDA Yes 45.0 62.5 71.3
Majority LDA Yes 41.0 62.5 68.8
Confidence LDA Yes 50.0 63.8 72.5
Majority LDA No 41.0 67.5 70.0
Confidence LDA No 47.0 61.3 73.8
None QDA Yes 35.0 57.5 60.0
Majority QDA Yes 49.0 58.8 63.8
Confidence QDA Yes 45.0 60.0 63.8
Majority QDA No 49.0 58.8 68.8
Confidence QDA No 48.0 60.0 68.8

Table 2.

Comparison of classifier fusion methods and single classifiers with and without imputation for LDA and QDA.

As classifiers, Linear and Quadratic Discriminant Analysis (LDA and QDA) were investigated. The performance of the classifier fusion methods in comparison to single classifiers with and without missing value imputation are shown in Table 2.

It can be observed that classifier fusion always yields a considerable benefit in comparison to single classifiers using all features (with imputation) for discriminating the 5 emotion classes. A maximum increase in accuracy of 14% was observed when comparing an QDA ensemble classifier system to a single classifier using imputation. When comparing the ensemble classifiers using imputation with the corresponding ones using reduced-feature models, the reduced-feature model ensembles perform once better, twice equally and once worse. Based on these result, it is difficult to decide whether to use imputation or not. The two strategies seem to be competitive. However, using no imputation is computationally less expensive and might be preferred in practical applications. Another interesting observation is that confidence voting always performs better then majority voting for LDA, while the tendency is reversed for QDA. The results for discriminating arousal and valence again indicate that classifier fusion outperforms single classifiers which use all the features (with imputation). When comparing the ensemble classifiers using imputation with the corresponding ones employing reduced-feature models, the reduced-feature model ensembles perform once better, once worse and twice equal for arousal. Considering valence, reduced-feature models yield better results in all four cases.

Advertisement

4. Conclusion

Due to the evolution of long-distance transportation, the European Commission has identified the airplane as part of our environment in which passengers could benefit from assistive support. The European research project SEAT aims at extending existing airplane seats with new sensor technologies in order to assess and improve health and well-being of the passengers. A first step in this direction is the reliable and unobtrusive recording of relevant physiological signals. In the first part of this chapter we have presented our approach to incorporate sensor technology into an airplane seat aiming at unobtrusively measuring physiological signals. The trade-off between passenger’s comfort and signal quality was identified to be an important issue to achieve the acceptance and hence the use of the system. The proposed multi-modal sensor system allows automatic artifact detection of physiological signals through sensor fusion. For validation purposes, we proposed a quality measure that appraises the signal of interest based on a ground truth signal. In a feasibility study the benefits of artifact removal were investigated. During typical airplane activities, the contactless ECG and the ground truth ECG were measured synchronously and the quality measure was calculated before and after artifact removal. It could be shown that the quality measures are substantially higher for the calm activities in comparison to the more active activities. Taking all activities together the mean quality measure rises from 40% to 88%. If the remaining signal length after artifact removal is scaled up to a 12h flight, 2.8h of the contactless ECG data is expected to be almost artifact free and would be available for computation of health indicators.

Since data loss due to artifacts is frequently encountered not only in our airplane scenario but also in many practical applications, in the second part of this chapter we have investigated two methods for handling missing data: imputation and reduced-feature models using ensemble classifier systems. In our emotion experiment, more than half of the data would have been lost if no strategy to handle missing values had been employed. With the proposed methods we were able to analyze 100% of the data. Classifier fusion has shown to substantially increase the recognition accuracies. A maximum increase in accuracy of 14% was observed when comparing an ensemble classifier system to a single classifier using imputation. Whether majority or confidence voting performs better depends on the underlying classifier.

Advertisement

Acknowledgments

This project is partly funded by the EU research project SEAT (http://www.seat-project.org), contract number: 030958. All views here reflect the author’s opinion and not that of the European Commission.

References

  1. 1. [Boucsein, 1992] Boucsein W. 1992 Electrodermal Activity. Plenum Press, New York.
  2. 2. Busso et al. 2004 Busso; Carlos; Deng, Zhigang; Yildirim; Serdar, Bulut; Murtaza; Lee; Chul Min; Kazemzadeh; Abe; Lee; Sungbok; Neumann; Ulrich; & Narayanan; Shrikanth. 2004. Analysis of emotion recognition using facial expressions, speech and multimodal information. 205 211 of: ICMI ‘04: Proceedings of the 6th international conference on Multimodal interfaces. New York, NY, USA: ACM.
  3. 3. Cowie et al. 2001 Cowie; R., Douglas-Cowie; E., Tsapatsoulis; N., Votsis; G., Kollias; S., Fellenz; W., & Taylor, J.G. 2001. Emotion recognition in human-computer interaction. Signal Processing Magazine, IEEE, 18(1), 32-80.
  4. 4. Kim André 2008 Kim, Jonghwa; & André, Elisabeth. 2008. Emotion Recognition Based on Physiological Changes in Music Listening. IEEE Trans. Pattern Anal. Mach. Intell., 30(12), 2067-2083.
  5. 5. Neiberg et al. 2006 Neiberg, Daniel, Elenius, Kjell, & Laskowski, Kornel. 2006. Emotion Recognition in Spontaneous Speech Using GMMs. In: Proceedings of the 9th ISCA International Conference on Spoken Language Processing (Interspeech).
  6. 6. Rottenberg et al. 2007 Rottenberg, J.; Ray, R. R.; & Gross, J. J. 2007. The handbook of emotion elicitation and assessment. New York: Oxford University Press. Chap. Emotion elicitation using films, page 9 28 .
  7. 7. Schumm et al. 2010 Schumm, Johannes; Setz, Cornelia; Bächlin, Marc; Bächler, Marcel; Arnrich, Bert, & Tröster, Gerhard. 2010. Unobtrusive Physiological Monitoring in an Airplane Seat. Personal and Ubiquitous Computing, 14(6), 541-550.
  8. 8. [Steffen et al., 2007] Steffen, M.; Aleksandrowicz, A., & Leonhardt, S. 2007. Mobile Noncontact Monitoring of Heart and Lung Activity. Biomedical Circuits and Systems, IEEE Transactions on, 1(4), 250 –257.
  9. 9. Setz et al. 2009 Setz, C.; Schumm, J.; Lorenz, C.; Arnrich, B. & Tröster, G. Using Ensemble Classifier Systems for Handling Missing Data in Emotion Recognition from Physiology: One Step Towards a Practical System. Proceedings 2009 International Conference on Affective Computing & Intelligent Interaction ACII 2009 2009 187 194

Written By

Bert Arnrich, Cornelia Kappeler-Setz, Johannes Schumm and Gerhard Trooster

Submitted: 26 October 2010 Published: 13 June 2011