Noncontact Monitoring of Vital Signs with RGB and Infrared Camera and Its Application to Screening of Potential Infection

Abstract

In recent years, much attention is being paid to research and development of technology that provides noncontact measurement of vital signs, i.e., heart rate, respiration, and body temperature, which are important for understanding the state of a person’s health. As technology for sensing biological information has progressed, new biological measurement sensors have been developed successively. There have also been reports regarding methods for measuring respiration or heart rate using pressure sensors, microwave radar, air mattresses, or high-polymer piezoelectric film. The methods have wide-ranging applications, including systems for monitoring of elderly people, identification of sleep apnea, detection of patients suspected to have an infectious disease, and noncontact measurement of stress levels. In this chapter, the principles behind noncontact measurement of respiration and heartbeat using infrared/RGB facial-image analysis are discussed, along with the applications for such measurement in the detection of patients suspected to be suffering from infectious diseases.

1. Introduction

Much attention is being paid to research and development of technology that provide noncontact measurement of vital signs, such as heartbeat, respiration, and body temperature, which are important for understanding the state of a person’s health. In the past, a practice called “悬丝诊脉 (suspension examination)” was documented in China’s Tang Dynasty, 1400 years ago. Prominent physician Sun Simiao used strings to perform pulse diagnosis from the radial artery pulse without touching the patient’s body, demonstrating that minute movements on the body’s surface and changes in facial coloring have been used to obtain biological information and perform diagnoses since long ago. In recent years, as technology for sensing biological information has progressed, new biological measurement sensors have been developed successively. There have also been reports regarding methods for measuring respiration or heartbeat using pressure sensors [1], microwave radar [2], air mattresses, or high-polymer piezoelectric film. The methods have wide-ranging applications, including systems for monitoring elderly people [3], identification of sleep apnea [4], detection of patients suspected to have an infectious disease [5, 6], and noncontact measurement of stress levels [7]. In this chapter, the principles behind noncontact measurement of respiration and heartbeat using infrared/RGB facial-image analysis are discussed, along with the applications for such measurement in the detection of patients suspected to be suffering from an infectious disease.

2. Principles of measuring respiratory/heart rate using infrared/RGB facial-image analysis

The blood vessels throughout the body are broadly categorized as arteries or veins, and there is a difference in the manner in which the volume of blood flowing through each blood vessel type changes. Blood flow volume through veins changes little, whereas blood flow volume through arteries varies according to the pulse. Further, one of the properties of oxygenated hemoglobin in the blood flowing through arteries is that it easily absorbs light from a specific wavelength. Because of these properties, when the skin is exposed to continuous light of a specific wavelength, the reflected light changes according to variation in blood flow volume, and the pulse waveform can be obtained by continuing to measure that reflected light [8]. The pulse wave obtained here is referred to as a volume pulse wave.

The simplicity and noninvasiveness of obtaining measurements with a photoplethysmography (PPG) sensor have led to the PPG sensor’s application as a tool for monitoring health [9, 10]. In PPG sensors, light is emitted from a dedicated light source (wavelengths 660 and 940 nm), and the volume pulse wave is obtained by measuring the reflected light with photodiodes. In this study, heartbeat was measured with an RGB camera by substituting ambient light for the dedicated light source and an RGB camera for the photodiodes and then measuring the form of the volume pulse wave in the same way as in PPG. Because of the degree of skin exposure and ease of detection, the heartbeat waveform was measured from the face. Additionally, respiration was measured by using thermography to detect nasal-region temperature changes associated with respiration. During exhalation, warm air from inside the lungs is released and it increases the temperature in the nasal region, whereas during inhalation, cool air from the external environment is breathed in and it lowers the temperature in the nasal region. The respiration waveform can be obtained by using an infrared camera to measure such nasal-region temperature changes associated with respiration. Figure 1 shows the principles of measuring respiratory/heart rate using infrared/RGB facial-image analysis.

Figure 1.

Principles of measuring respiratory/heart rate using infrared/RGB facial-image analysis.

3. Evaluation of the accuracy of measuring respiratory/heart rate using infrared/RGB facial-image analysis

3.1. Experimental protocol

The experiment was conducted with ten healthy university students. An infrared/RGB camera was set up 50 cm in front of the measurement subjects, and the subjects were instructed to sit and maintain a resting state. Images were taken for 30-s periods when at rest and after exercise (ergometer exercise: 70 rpm, 100 W, 2-min duration). Furthermore, measurements were also taken simultaneously using thoracoabdominal respiration sensors and an electrocardiogram as references for respiratory rate and heart rate, respectively (Figure 2).

Figure 2.

Experimental protocol.

The nasal area was manually extracted from the infrared images taken (150 × 150 pixels). During respiration, the temperature of the nasal area changes as air moves in and out. These temperature changes over time were recorded to create a waveform. A band-pass filter (0.17–0.42 Hz) was applied to this waveform by a signal processer. Then, the waveform was normalized, and the autocorrelation function was calculated. Last, the respiratory rate was calculated based on the peak interval of the autocorrelation function calculated earlier. Meanwhile, the visible images taken with the RGB camera were converted to a single color (green), and an area centered on the nose (150 × 180 pixels) was extracted. The formula below was applied to the extracted image to calculate luminance value Y [11]:

$$Y=\left(00.5870\right)\left(\begin{array}{l}R\\ G\\ B\end{array}\right)$$ E1

The luminance values calculated were recorded over time to create a waveform. A band-pass filter (0.83–2.5 Hz) was applied to this waveform. Afterward, the same type of processing as in respiratory rate calculation was performed to calculate the heartbeat.

3.2. Results

Figure 3(a) shows the respiration-heartbeat curves obtained from the thoracoabdominal respiration sensors and infrared images. Figure 3(b) shows the heartbeat curves obtained from the ECG and visible images. They demonstrate that both respiration and heartbeat are similar to the curves obtained from the references.

Figure 3.

Respiration curves obtained from thoracoabdominal respiration sensors and infrared images; heartbeat curves obtained from ECG and visible images. (a) The respiratory rate signal obtained from IR camera. (b) The heartbeat signal obtained from RGB camera.

Next, a Bland-Altman analysis was applied to all 20 pieces of data obtained from each subject at rest and after exercising, and the accuracy was verified (Figure 4). The results for respiratory rate were as follows: a correlation coefficient of 0.99 (p < 0.01), an average difference of −0.03 (breaths per minute) between the respiratory rates obtained from the infrared images and thoracoabdominal respiration sensors, and a 95% confidence interval of −1.04 to 0.9 (breaths per minute). The results for heart rate were as follows: a correlation coefficient of 0.96 (p < 0.01), an average difference of −0.27 (beats per minute) between the respiratory rates obtained from the visible images and ECG, and a 95% confidence interval of −6.1 to 5.5 (beats per minute). These results demonstrate that highly accurate measurement of respiratory rate and heart rate are possible, regardless of how those values fluctuate (respiratory rate: 9.2–23.9 breaths per minute, heart rate: 47.9–93.9 beats per minute).

Figure 4.

Plot of respiration/heart rate from Bland-Altman analysis.

4. Applications in infectious-disease screenings for respiratory/heart rate measurement technology using infrared/RGB facial-image analysis

In recent years, much importance has been placed on developing new quarantine systems to use against emerging infectious diseases and epidemics involving new strains of viruses. From past outbreaks of severe acute respiratory syndrome (SARS) to the H1N1 flu virus, a variety of infectious diseases have spread widely, causing damage to countries around the world. Although some of those infectious diseases are in the process of being eradicated, there are also fears about outbreaks of reemerging and emerging infectious diseases. In these conditions, it is essential to take preventive measures at national borders at the time when a person enters the country, and quarantine inspection at airports and the like includes interview-based medical exams using questionnaires and fever checks with thermography. However, because the former method relies on voluntary cooperation from people entering the country, false statements are a possibility, and the latter method has its own limits, for example, it does not cover passengers who have taken antifebrile medication. To overcome these challenges in quarantine inspections, Matsui and Sun et al. are developing infectious-disease screening systems using multiple biosensors that measure vital signs to identify symptomatic people [5, 6]. As a result of being infected, not only body temperature but also heart and respiration rates increase. The system automatically detects infected individuals within 15 s by a discriminant function using measured vital signs. Heart and respiration rates are determined using a microwave radar by noncontact way, and facial skin temperature is monitored by a thermographic camera. By using these three parameters, the detection accuracy of the system improved ranged from 81.5 to 98.0% in case-control studies. This is notably higher compared to the conventional screening methods using only thermography.

Considering future practical applications for infectious-disease screening systems, it would be beneficial to build infectious-disease screening systems powered by image analysis using infrared cameras [12]. There are four reasons for this. (1) Infrared cameras are already used at airport quarantine stations around the world and are extremely versatile. (2) It is possible to measure vital signs by simply changing a camera’s software. (3) When vital signs can be measured, and symptomatic people identified by taking images with just a camera, the need for quarantine can be determined using existing equipment without the need to bring in new devices. (4) There is a lower risk of secondary infection caused by contact with equipment, such as measurement devices.

A clinical evaluation was conducted to confirm the accuracy of identifications made by an infectious-disease screening system using infrared/RGB facial-image analysis (Figure 5). We used an infrared RGB camera to measure facial surface temperature, respiratory rate, and heart rate over a period of 10 s in a symptomatic group of 16 people (average age 36.6 ± 14.1) diagnosed with influenza A (seven people) or influenza B (nine people) by a physician. The same measurements were taken in a healthy group of 22 people (average age 43 ± 18.7). The identification accuracy was verified by conducting a logistic regression analysis of the symptomatic and healthy groups based on the subjects’ body surface temperature, respiratory rate, and heart rate measured using image analysis:

$$\mathrm{Z}\left(x_1,x_2,x_3\right)=\text{log}\left(\frac{p_i}{1-p_i}\right)=-203.27+0.49x_1+0.36x_2+4.68x_3$$ E2

Figure 5.

The infectious disease screening system using infrared/RGB facial-image analysis.

Here, $\mathrm{Z}\left(x_1,x_2,x_3\right)=\text{log}\left(\frac{p_i}{1-p_i}\right)$ is the identification score, $x_1$ is respiratory rate, $x_2$ is heart rate, and $x_3$ is the body surface temperature. The results of the identification were as follows: false negatives in two subjects produced misclassification, sensitivity was 87.5%, specificity was 100%, positive predictive value (PPV) was 100%, and negative predictive value (NPV) was 91.7%.

The goal of this study was to develop an infectious-disease screening system using infrared/RGB facial-image analysis. Furthermore, to make this system a reality, we verified the accuracy and clinically evaluated measurement of respiratory rate and heart rate using infrared/RGB images. The results confirmed that measuring respiratory rate and heart rate using infrared/RGB analysis makes highly accurate measurement possible.

5. Conclusion

This chapter introduced the principles behind noncontact measurement of respiration and heartbeat using infrared/RGB facial-image analysis, and the applications for such measurement in the detection of patients suspected to have an infectious disease were explored. Looking toward future practical applications, elimination of noise caused by body movement and a face/nose tracking function for automatic measurement of heart and respiratory rates will likely be added, improving the accuracy and stability of measurements, and we can expect this technology to be applied in a variety of ways.

Acknowledgments

This research was supported by a Grant-in-Aid for Young Scientists (16K16363) founded by the Ministry of Education, Culture, Sports, Science and Technology of Japan.

Conflict of interest

The authors declare no conflicts of interest.