The respiratory index (
Abstract
The human walking behaviour can express the physiological information of human body, and gait analysis methods can be used to access the human body condition. In addition, the respiratory parameters from pulmonary spirometer are the standard of accessing the body condition of the subjects. Therefore, we want to show the correlation between gait analysis method and the respiratory parameters. We propose a vision sensor-based gait analysis method without wearing any sensors. Our method proposed features such as D′p, V′p and γυ to prove the correlation by classification and prediction experiment. In our experiment, the subjects are divided into three levels depending on the respiratory index. We run classifying and predicting experiment with the extracted features: V′p and γυ. In the classifying experiment, the accuracy result is 75%. In predicting experiment, the correlations of predicting the forced expiratory volume in 1 s (FEV1) and forced vital capacity (FVC) are 0.69 and 0.67, respectively. Therefore, there is a correlation between the pulmonary spirometer and our method. The radar system is a tool using impulse to record the moving of the subjects’ chest. Combining the features of radar system with our features improves the classification result from 75 to 81%. In predicting FEV1/FVC, the correlation also improves from 25 to 42%. Therefore, cooperating with radar system improves the correlation.
Keywords
- gait analysis
- classification
- prediction
- pattern recognition
- feature extraction
1. Introduction
Walking behaviour can express information of human body-like pathological symptoms. For example, Parkinson’s disease patients are characterized by special pace rhythm [1].
People increase the respiratory ventilation when they are walking or exercising. However, those people who suffer from chronic obstructive airway disease (COAD) cannot increase their respiratory ventilation quick enough to maintain the exercising behaviour. Consequently, they change their behaviour such as walking slowly so that they can maintain their respiratory ventilation. We can perform gait analysis on COAD patients because their walking behaviours are different with normal people when they are exercising because of their respiratory function.
Chronic obstructive pulmonary disease (COPD) is one condition of COAD. Nowadays, there are many chronic diseases in our daily life. COPD is one of them. COPD is a chronic airway disease characterized by progressive going downhill of the breathing functions [2]. One characteristic of COPD disease is the decreasing of forced expiratory volume in 1 s (FEV1) because of the obstructive airway [2]. Depending on their disease severity, they have different walking behaviours. Therefore, gait analysis can be used to judge the COAD patients’ airway condition by observing their walking behaviour. However, it is difficult to collect the data of COAD patients without medical staff. Without clinical data, we cannot verify the correctness of our gait analysis algorithm.
By cooperating with Shuang-Ho Hospital in New Taipei, Taiwan, we set up an experiment. We film the side view of the subjects when they are performing a 6-min brisk walking test. By gait analysis, we can extract the features from walking behaviour such as pace distance and walking speed variation. However, in order to obtain that physiological information, we may need to wear sensors or markers on the subjects. Our method does not need to wear any sensors on the subjects.
In gait analysis, it is common to wear markers or sensors to record walking behaviour. In the experiment [1], the subjects need to wear a recorder on the ankle so that it could record the stride interval. In another experiment [3], the subjects also need to place a designed insole with 12 sensors into their own shoes.
However, there are some drawbacks of using makers or sensors. Firstly, it is inconvenient and uncomfortable of attaching them on human body and might affect the normal walking. Secondly, some sensors are heavy or hard to use for the elderly. Thirdly, some sensors have the electromagnetic interference that might affect and harm human body. In addition to the sensor problem, it is hard to tell a subject suffering from COAD disease or not by a single experiment. Without a complete examination, it is hard to judge whether the subjects are COAD patients or not. Consequently, we decide to access the respiratory function. By the pulmonary spirometer, we can obtain the tested subjects’ respiratory data. Nowadays, the parameters from pulmonary spirometer are the standard to access the respiratory function.
2. Related work
2.1. Chronic obstructive pulmonary disease
COPD is one condition of COAD and we introduce some studies of COPD disease. The research of the World Health Organization (WHO) and the global initiative for chronic obstructive lung disease revealed that nowadays the COPD is the fourth cause of death in the United States. For the entire world, the COPD can be considered the fifth cause of death [4]. In another study, the COPD can be considered the fourth leading cause of death in the world [5] and is projected to be the fifth by 2020 as a worldwide burden of disease [6].
Generally speaking, chronic disease is gradually rising because of the ageing population and changing habits. The treatment for COPD patients is pulmonary rehabilitation programmes, including when patients are discharged at home [7]. Economic analyses have shown that over 70% of COPD-related health-care expenditures result from emergency room visits and hospital care for exacerbations; this translates to >$10 billion annually in the Unites States [8]. COPD patients have a lower physical activity level than healthy peers [9]. The reduced level of physical activity also related to an increased risk [10]. Liao et al. [11] proposed a review that concentrated on describing wearable devices for measuring physical activity level in COPD patients. In [12], the authors evaluated a method for detecting an exacerbation onset in COPD patients. They used data collected through a pulse oximeter, permitting an easy way to the cohort of patients composed of elderly people affected by COPD. The study [13] provided a system, which offers an effective platform for the satisfaction of the clinical and the patient’s needs in the area of early diagnosis of patient’s health status. The portable system aims at the effective management in the health status of the patients who are suffering from COPD.
2.2. Gait analysis
Gait analysis plays an important role in accessing human’s walking behaviour and it aims to extract biomechanics information. The obvious disease on lower limbs is Parkinson’s disorder. In [14], they provided a feasible method image marker to measure gait with little skin movement. Then, they performed quantitative analysis to extract gait parameter. According to their analysis, the joint angle, rotation angle of lower limbs, stride velocity and stride length have significant difference between Parkinson’s disease patients and non-diseased subjects. The study [1] demonstrated that the gait variability in terms of statistical parameters of stride interval such as STC
The above methods all use markers or sensors to get interest points and then use gait analysis. With the advances in smart phones, Susu Jiang et al. [16] used a smart phone with an accelerometer and a gyroscope to collect human walking gait data in daily life. However, one smart phone only records one feature data. If we want to acquire many interest point data, testers need to tie many smart phones on the body, which might be inconvenient for walking.
Now, we proposed a gait analysis algorithm without any wearable marker or sensor. In addition, we also collect clinical COAD and control data to verify the correlation between pulmonary spirometer and our gait analysis method.
3. Vision sensor-based gait analysis
To avoid wearing sensors on the tested subjects, we propose a vision sensor-based gait analysis method. This method is composed of four parts: input video frame decomposition, pre-processing, feature extraction and gait analysis. The input video takes the side view of the subjects when they are performing a 6-min brisk walking test. Then, the input video is decomposed into individual frames for further processing.
In the pre-processing part, there are three components to extract the silhouette of the tested subject: background subtraction, shadow removal and connected component labelling (CCL). The background subtraction subtracts the background image by the current video frame to capture the moving object. Because the human shadow cannot be removed by background subtraction, we adopt Jamie and Richard’s method [17] to solve the shadow problem. Then, there are some noises that cannot be removed, so we use the connected component-labelling method to reduce the noises and obtain the maximum object.
In the feature extraction part, there are two steps to obtain the desired features: segmentation and feature extraction. In the segmentation part, we have to find the central of gravity (COG) first, and use the COG point to perform body segmentation and extract legs of the subjects. Then, we can get the features such as pace distance and pace velocity in the feature extraction part.
In the gait analysis part, we divide whole subjects into two groups as the Bad and Good by consulting the proposed respiratory index formula. We use support vector machine (SVM) to perform classification and use adaptive network-based fuzzy inference system (ANFIS) to perform prediction.
3.1. Pre-processing
3.1.1. Background subtraction
We take the first frame of the input video as the background image. The background subtraction method is shown in Eq. (1). The components
3.1.2. Shadow removal
After subtract background image, some interferences still exist. The result of background subtraction is shown in Figure 1. The human shadow is viewed as foreground and we need to remove it. We consult Jamie and Richard’s [17] method to solve the shadow problem. The method is divided into two parts: (1) brightness distortion and (2) chromatic distortion.
There is a threshold
In the chromatic distortion part, we calculate the distance in RGB space between
In the same way, we also set a threshold
After finishing these two parts above, we combine these two images and the background subtraction result together to have a result without shadow.
3.1.3. Connected component labelling
Connected component labelling is used to detect the connected components and label them. These components have their own label number. Figure 3 shows an example. Figure 3(a) shows three different regions and (b) shows the labels of the regions. We keep the largest group as our result. In this case, the region of label 3 is reserved and discards other regions.
Though we have an image without shadow, some noises exist. We perform connected component labelling method so that we can have an image without these noises. Figure 4 shows the flow of pre-processing.
3.2. Feature extraction
After the pre-processing, we get the complete target silhouette. In this section, we separate legs from extracted silhouette by segmentation. Then, we find out the gait features such as pace distance and pace velocity in the feature extraction part.
3.2.1. Segmentation
In the segmentation part, we need to find the central of gravity first. Then, we use the extracted silhouette to get the contour by edge detection in order to build the distance map (DM). We can separate legs from human silhouette by DM.
We find the central of gravity from the extracted human silhouette by Eq. (7). After finding the COG of the whole body (COG
We find the Nlegs, Nb2(l), Nb2(r) three points from DM as shown in Figure 6. Connecting Nb2(l) and Nb2(r) divides human body into top and down two parts. Connecting (COG
3.2.2. Gait features
In this part, we extract gait features such as pace distance, pace time and pace velocity. Figure 8 shows a pace cycle model and we can extract pace distance (
3.3. Gait analysis
To verify that there exists a correlation between pulmonary spirometer and our system, we perform classification and prediction experiments. There are two groups, namely bad and good, which are classified by the parameters from pulmonary spirometer and it becomes our classification standard. We take support vector machine for classification and take adaptive network-based fuzzy inference system for prediction. Here, we introduce the tools: SVM and ANFIS.
3.3.1. Support vector machine
SVM comes from Vapnik’s statistical learning theory [18] and it is a machine-learning method, which can be a powerful tool for learning from data and for solving classification problem [18]. In a two-group classification problem such as our study (Bad/Good), the target is to find the
3.3.2. Adaptive network-based fuzzy inference system
ANFIS was presented by Jang in 1993 [19]. Adaptive network-based fuzzy inference system can construct an input-output mapping based on human knowledge by a hybrid-learning algorithm. The fuzzy inference system is employed with adaptive network. ANFIS contains a five-layer forward neural network to construct the inference system.
Input space is mapped to a given membership function (MF). By membership function, the input becomes a degree between 0 and 1. With different membership functions and the number of membership functions, the results are different. Figure 10 shows the ANFIS structure with two inputs and four rules.
The study [20] explained the function of each layer. In Layer(1), the outputs are the membership function degree of the inputs, which are given by Eqs. (10) and (11)
Layer(2) involves fuzzy operations. ANFIS fuzzifies the inputs by using AND operation. The label _ means that they perform simple multiplier. Equations (12) and (13) can show the output of Layer(2)
In Layer(3), the label
Layer(4) is the product of the normalized data which can be represented as Eq. (15). The parameters
Layer(5) implements sum of all inputs such as Eq. (16)
4. Proposed gait features
We propose some gait features from
The mean of distance for section
From
5. Clinical experiment environment
5.1. Experiment set-up and flow
The experiments are run at Shuang-Ho Hospital in New Taipei, Taiwan. We film the side view of the subjects when they perform the 6-min brisk walking test. We set up a green curtain to exclude the interferences such as the movement of other people from our experiment. We film the walking subjects using Nikon P330 digital camera.
Firstly, the therapists ask the subjects’ profile including height, weight and age. Secondly, by using pulmonary spirometer, we can get respiratory parameters such as FEV1 and FVC data about the subjects. Thirdly, before the experiment starts, the therapist helps the subjects wear a pulse oximeter on the index finger, which is used to measure the oxygen and pulse. Fourthly, the subjects need to take a 2-min break so that the pulse oximeter can record the oxygen and pulse in normal condition. Fifthly, when the walking test begins, the subjects should walk along the trail as fast as possible. While the subjects start their walking test, we film the side view of the subjects. Sixthly, after the 6-min walking test, the subjects use the pulmonary spirometer to measure the respiration parameter after exercising again.
5.2. Data collection
We run the experiments from September 2014 to July 2015. In the experiments, there are 60 subjects who aged between 24 and 91 years. Among these 60 subjects, there are 48 men and 12 women.
There are two rooms: the subjects walk from the right room to the left one, then turnaround to walk into the right room. When the subjects walk to the other side of the trail, they need to turnaround and continue to walk along the trail. They decrease their walking speed so that they can turnaround easily when they are close to the border. To avoid recording those slowdown steps, we abandon those steps and keep normal steps. Taking Figure 13 as an example, there are six steps in this walking trail. We just consider steps 1 and 6 as the normal steps and abandon steps 2–5.
Depending on the respiratory index that comes from Eq. (19), these subjects are divided into three levels: level 1 (the worst respiratory function), level 2 (poor respiratory function) and level 3 (normal respiratory function). Table 1 shows the respiratory index used to classify the three levels. We call the respiratory index as
Group | |||
---|---|---|---|
REX | 0.7 | [0.7,1.65] | 1.65 |
The main item of REX formula is postFEV1 and other items are used to adjust it. The three items of
The FEV1 is the volume that has been exhaled at the end of the first second of forced expiration. The FVC is the forced vital capacity that is used for the determination of the vital capacity from a maximally forced expiratory effort. The IC is the inspiratory capacity that is the sum of inspiratory reserve volume and tidal volume. The
6. Experimental results
In this chapter, we use the features that we get from Chapter 4 to perform our experiment. There are three experiments: classification with support vector machine, prediction with adaptive neural fuzzy inference systems and cooperating with radar system.
In the first experiment, we choose one feature from
6.1. Classification with support vector machine
Support vector machine is one of the most widely used machine-learning algorithms for classification problems [22].
We group the subjects of level 1 and level 2 into the Bad group, and the subjects of level 3 belong to the Good group. There are 32 subjects in the Bad group and 28 subjects in the Good group. Those subjects who belong to the Good group are marked with blue triangles and those subjects who belong to the Bad group are marked with red circles.
1 | 2 | 3 | 4 | 5 | 6 | 7 | |
Accuracy | 0.55 | 0.55 | 0.55 | 0.56 | 0.56 | 0.58 |
According to the above chapter, we divide all steps into
Level | ||
47.72 | 117.4 | |
50.49 | 129.2 | |
From Table 3, the subjects of level 3 have larger
Level | ||
3.16 | 11.2 | |
3.48 | 11.9 | |
The subjects who have better respiratory function have larger
In addition, people have lower variance after they exercise. Figure 17 shows the pace distance of a person in three different conditions: normal walking, walking after a short run and walking after a long run. The variation of walking after a long run is smaller than others.
The
Therefore, the
Level | |
0.93 | |
0.96 | |
Input features | ||
Accuracy | 0.66 |
The
The accuracy of input features (
6.2. Prediction with adaptive neural fuzzy inference systems
We utilize adaptive neural fuzzy inference system to help us predict the parameters from pulmonary spirometer. The ANFIS system comes from the toolbox of Matlab.
Because we only collect about 60 cases so far, it is not enough for ANFIS to perform prediction. In order to increase the training samples, we adopt
In ANFIS, it is important to choose a correct membership function. In addition, we also need to choose the input sections. In the experiment, we use six different membership functions including trapmf, gbellmf, gaussmf, gauss2mf, pimf and dsigmf. Figure 22 shows the membership functions we use in our prediction experiment. The inputs of the experiment are
We try to predict three different parameters that come from the pulmonary spirometer: post
Correlation | MSE |
Regression slope | |
---|---|---|---|
Trapmf | 0.226 | 0.040 | 0.139 |
Gbellmf | 0.203 | 0.041 | 0.127 |
Gauss2mf | 0.195 | 0.040 | 0.112 |
Pimf | 0.215 | 0.039 | 0.125 |
Dsigmf | 0.210 | 0.039 | 0.121 |
FEV1-FVC is an index which is used to access the severity of airway obstruction. The lower value means that the airway obstructs severely. Table 7 shows our prediction results and we use [2 2] as the input sections. The
FEV1 is also a parameter to access the respiratory function. The higher FEV1 value means that the subjects have better respiratory function. Consequently, we also predict the FEV1 value. Table 8 shows the prediction results and we use [3 2] as using 3 nodes for the first input and 2 nodes for the second input in the ANFIS system. The features
Correlation | MSE |
Regression slope | |
---|---|---|---|
Gbellmf | 0.668 | 0.186 | 0.827 |
Gaussmf | 0.640 | 0.193 | 0.774 |
Gauss2mf | 0.386 | 1.968 | 1.264 |
Pimf | 0.560 | 0.365 | 0.880 |
Dsigmf | 0.352 | 2.692 | 1.321 |
The correlation and regression slope of predicting
Correlation | MSE |
Regression slope | |
---|---|---|---|
Trapmf | 0.660 | 0.079 | 0.621 |
Gbellmf | 0.647 | 0.094 | 0.693 |
Gauss2mf | 0.432 | 0.249 | 0.654 |
Pimf | 0.192 | 0.458 | 0.354 |
Dsigmf | 0.156 | 1.151 | 0.469 |
Table 10 shows the best results of predicting
Predicting target | Correlation | MSE |
Regression slope |
---|---|---|---|
0.251 | 0.167 | ||
FEV1 | 0.140 | ||
FVC | 0.076 | 0.646 |
6.3. Cooperating with radar system
In this section, we combine the features of radar system with our features
In the SVM experiment, there are still 32 subjects in the Bad group and 28 subjects in the Good group. Because we use (
Target (features) | Correlation | MSE |
regression slope |
---|---|---|---|
0.251 | 0.167 | ||
0.071 | |||
0.428 | 0.054 | 0.534 | |
FEV1, ( |
0.746 | ||
FEV1, (ΔAmp,Δβratio) | 0.474 | 0.190 | 0.39 |
FEV1, ( |
0.675 | 0.020 | |
FVC, ( |
0.646 | ||
FVC, (ΔAmp,Δβratio) | 0.129 | 0.217 | 0.13 |
FVC, ( |
0.517 | 0.190 |
In the ANFIS experiment, we also predict
Radar system improves our analysis results on both SVM classification and predicting the parameter
7. Conclusion
We propose a vision sensor-based gait analysis method without wearing any sensor on human body. In our approach, the proposed gait features analyse the subjects’ respiratory function. We also perform a clinical experiment on COAD patients and normal people with our vision sensor-based gait analysis method. With the extracted features,
In addition, by combining the features of radar system (ΔAmp and Δ
References
- 1.
Yunfeng Wu, Sridhar Krishnan. Statistical Analysis of Gait Rhythm in Patients with Parkinson's Disease. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2010; 18 (2):150–158. DOI: 10.1109/TNSRE.2009.2033062 - 2.
Ming-Feng Wu, Chao-Ling Chen, Chih-Yu Wen, and Jeng-Yuan Hsu. Design of Pervasive Rehabilitation Monitoring for COPD. IEEE Sensors Journal. 2013; 13 (11):4413–4422. - 3.
Howell AM, Kobayashi T, Hayes HA, Foreman KB, and Bamberg SJM. Kinetic Gait Analysis Using a Low-Cost Insole. IEEE Transactions on Biomedical Engineering. 2013; 60 (12):3284–3290. - 4.
Fabbri LM and Hurd SS. Global Strategy for the Diagnosis, Management and Prevention of COPD: 2003 update. European Respiratory Journal. 2003; 22 (1):1–2. - 5.
World Health Organization. The World Health Report 2000. 2000. - 6.
Murray CJ and Lopez AD. Mortality by Cause for Eight Regions of the World: Global Burden of Disease Study. Lancet. 1997; 349 (9061):1269–1276. - 7.
ATS/ERS Task Force on Pulmonary Rehabilitation. An Official European Respiratory Society Statement on Physical Activity in COPD. European Respiratory Journal. 2014; 44 (6):1521–1537. - 8.
Sullivan SD, Ramsey SD, and Lee TA. The Economic Burden of COPD. Chest. 2000; 117 :5S–9S. - 9.
Vorrink SN, Kort HS, Troosters T, and Lammers JW. Level of Daily Physical Activity in Individuals with COPD Compared with Healthy Controls. Respiratory Research. 2011;12-33. DOI: 10.1186/1465-9921-12-33. - 10.
Gimeno-Santos E., et al. Determinants and Outcomes of Physical Activity in Patients with COPD: A Systematic Review. Thorax. 2014; 69 (8):731–739. - 11.
Shu-Yi Liao, Roberto Benzo, Andrew L. Ries, and Xavier Soler. Physical Activity Monitoring in Patients with COPD. Journal of the COPD Foundation. 2014; 1 (2):155–165. - 12.
Mario Merone, Leonardo Onofri, Paolo Soda, Claudio Pedone, Raffaele Antonelli Incalziand, and Giulio Iannello. Early Experiences in COPD Exacerbation Detection. In: IEEE International Symposium on Computer-Based Medical Systems, pp. 406–410. - 13.
Bellos CC, Papadopoulos A, Rosso R, and Fotiadis DI. Identification of COPD Patients' Health Status Using an Intelligent System in the CHRONIOUS Wearable Platform. IEEE Journal of Biomedical and Health Informatics. 2014; 18 (3):731–738. - 14.
Wen L, Qian J, Hu X, Shen L, Wu X, and Yu C. Gait Measurement and Quantitative Analysis in Patients with Parkinson's Disease for Rehabilitation Assessment. In: Robotics and Biomimetics (ROBIO), IEEE International Conference on. IEEE; 2013. Shenzhen, China. pp. 286–291. - 15.
Chung PC, Hsu YL, Wang CY, Lin CW, Wang JS, and Pai MC. Gait Analysis for Patients with Alzheimer'S Disease using a Triaxial Accelerometer. In: IEEE International Symposium on Circuits and Systems. IEEE; 2012. Seoul, Korea. pp. 1323–1326. - 16.
Susu Jiang, Bofeng Zhang, Guobing Zou, and Daming Wei. The Possibility of Normal Gait Analysis Based on a Smart Phone for Healthcare. In: Green Computing and Communications, IEEE and Internet of Things, IEEE International Conference on and IEEE Cyber, Physical and Social Computing. IEEE; 2013. Beijing, China. pp. 2235–2240. - 17.
Jamie Schiel and Richard Green. Adaptive Human Silhouette Extraction with Chromatic Distortion and Contour Tracking. In: International Conference on Image and Vision Computing New Zealand (IVCNZ 2013); Wellington. IEEE; 2013. pp. 288–292. - 18.
Vladimir N. Vapnik. The Nature of Statistical Learning Theory. New York, NY: Springer; 1995. DOI: 10.1007/978-1-4757-3264-1 - 19.
Jang JSR. ANFIS: Adaptive-Network-Based Fuzzy Inference System. IEEE Transactions on Systems, Man and Cybernetics. 1993; 23 (3):665–685. - 20.
Al-Hmouz A, Shen J, Al-Hmouz R, and Yan J. Modeling and Simulation of an Adaptive Neuro-Fuzzy Inference System (ANFIS) for Mobile Learning. IEEE Transactions on Learning Technologies. 2012; 5 (3):226–237. - 21.
Cleveland Clinic Foundation. 2003. Available from: http://www.clevelandclinicmeded.com/medicalpubs/diseasemanagement/pulmonary/pulmonary-function-testing/March 2015. - 22.
Jie Xu, Yuan Yan Tang, Bin Zou, Zongben Xu, Luoqing Li, and Yang Lu. The Generalization Ability of Online SVM Classification Based on Markov Sampling. IEEE Transactions on Neural Networks and Learning Systems. 2015; 26 (3):628–639.