Summary of the studies.
Abstract
The aim of this study is to explore the usage of statistical learning methods on wearable devices and realize an experimental study for recognition of human activities by using smartwatch sensor data. To achieve this objective, mobile applications that run on smartwatch and smartphone were developed to gain training data and detect human activity momentarily; 500 pattern data were obtained with 4‐second intervals for each activity (walking, typing, stationary, running, standing, writing on board, brushing teeth, cleaning and writing). Created dataset was tested with five different statistical learning methods (Naive Bayes, k nearest neighbour (kNN), logistic regression, Bayesian network and multilayer perceptron) and their performances were compared.
Keywords
- statistical learning
- activity recognition
- wearable devices
- smartwatch
- Bayesian networks
1. Introduction
The usage of wearable technology is increasing rapidly, and the effects of user healthcare are enormous. Today's smart devices have more built‐in sensors than before. Wearable sensors are small devices which are carried by people, while they are performing daily activities. These sensors such as an accelerometer, microphone, GPS and barometer record the physical condition of person such as location change, moving direction and moving speed. Latest smartphones and smartwatches have many wearable sensors as built‐in [1, 2]. Because of equipped with various on‐board sensors, smartphones and wrist‐worn devices such as smartwatches are being extensively used for activity recognition in recent studies [3]. With the popularity of the smartwatches, wrist‐worn sensor devices will become an increasingly important tool in personal health monitoring [4]. Statistical learning methods are generally used in activity recognition studies. Statistical learning refers to a set of tools for modelling and understanding complex datasets. It is a recently developed area in statistics and blends with parallel developments in computer science and, in particular, machine learning [5].
The aim of this chapter is to investigate the usage of statistical learning methods on wearable devices and carry out a case study for recognition of human activities with accelerometer data of smartwatch by using statistical learning methods. This chapter is organized as follows: related works are described in detail in Section 2. Then, overview of statistical methods is mentioned in Section 3. Next, human activity recognition with smartwatches is explained in Section 4. Finally, Section 5 concludes the chapter.
2. Related works
When examining the literature, various studies are found statistical learning methods with wearable devices. Wang et al. [6] imagined a user typing on a laptop keyboard while wearing a smartwatch. The accelerometer and gyroscope data, which obtained from Samsung Galaxy Live, were used as training data, and processed through a sequence of steps, including key‐press detection, hand‐motion tracking, character point cloud computation and Bayesian modelling and inference. Shoaib et al. [3] carried out to recognize of different living activities by using a smartphone and a smartwatch simultaneously and evaluated their effectiveness in recognizing human activities. They used J48, kNN and SVM (support vector machines) to recognize 13 various activities. da Silva and Galeazzo [7] presented the development of a system based on computational intelligence techniques and on an accelerometer to perform, in a comfortable and non‐intrusive manner, the recognition of basic movements of a person’s routine. Three different computational intelligence techniques were evaluated in order to search for the best performance of the recognition of the movements executed by the watch user. Chernbumroong et al. [8] studied classification of five human activities by using only accelerometer data and two learning algorithms: Artificial Neural Networks and Decision Tree C4.5. Scholl and van Laerhoven [9] presented a feasibility study with smokers wearing an accelerometer device on their wrist over the course of a week to detect their smoking habits based on detecting typical gestures carried out while smoking a cigarette. The Gaussian method was used as a classifier. Dong et al. [10] described a new method that uses a watch‐like configuration of sensors to continuously track wrist motion throughout the day and automatically detect periods of eating. Accelerometer and gyroscope sensor data were used in this study. Ramos‐Garcia and Hoover [11] developed a Hidden Markov model (HMM) and compared its recognition performance against a non‐sequential classifier (kNN), using a set of four actions (rest, utensiling, bite and drink). Trost et al. [12] compared the activity recognition rates of an activity classifier trained on acceleration signal collected on the wrist and hip. Features were extracted from 10 seconds windows and inputted into a regularized logistic regression model. Guiry et al. [1] investigated the role of smart devices including smartphones and smartwatches which can play in identifying activities of daily living. The activities examined include walking, running, cycling, standing, sitting, elevator ascents, elevator descents, stair ascents and stair descents. Data from this study were used to train and test five well‐known statistical machine learning algorithms: C4.5, CART, naïve Bayes, multilayer perceptrons and finally support vector machines. Mortazavi et al. [4] introduced a framework for platform creation (e.g. accelerometer only system versus accelerometer and gyroscope) and machine learning of some activities, which can be especially useful in the emerging market of smartwatches. Random forests, decision trees, Naive Bayes and SVM methods were compared. Khan et al. [13] implemented a smartphone‐based HAR scheme in accordance with these requirements. Time domain features were extracted from only three smartphone sensors, and a nonlinear discriminatory approach was employed to recognize 15 activities with a high accuracy. Evaluations were performed in both offline and online settings. Dadashi et al. [14] carried out detection of important breaststroke swimming events automatically by using Hidden Markov model (HMM) and wearable sensors. Parkka et al. [15] used accelerometers and gyroscopes attached to ankle, wrist and hip to estimate intensity of physical activity. Data from common everyday tasks and exercise were collected with 11 subjects. Shen et al. [16] tracked the 3D posture of the entire arm—both wrist and elbow—using the motion and magnetic sensors on smartwatches. Bieber and Peter [17] studied behaviour analysis using 3D sensor data and learning techniques and obtained sufficient results. Bao and Intille [18] developed an algorithm and evaluated to detect physical activities from data acquired using five small biaxial accelerometers worn simultaneously on different parts of the body. Kim et al. [19] developed an application by using sensor signals from smartphone and smartwatch. Summary of the literature is given in Table 1.
Ref No. | Author | Year | Detection | Device | Sensors | Methods |
---|---|---|---|---|---|---|
[16] | Shen et al. | 2016 | Arm posture | Smartwatch | Accelerometer, gyroscope, compass | Hidden Markov model |
[6] | Wang et al. | 2015 | Typing on a laptop keyboard | Samsung Gear Live | Accelerometer gyroscope | Bayessian Inference |
[3] | Shoaib et al. | 2015 | Smoking, eating, typing, writing, drinking coffee, talking, walking, jogging, biking, walking upstairs and downstairs, sitting, standing | Smartphone and smartwatch | Accelerometer gyroscope | Support vector machine, k nearest neighbour, J48 decision trees |
[12] | Trost et al. | 2014 | Lying down, sitting, standing, walking, running, basketball and dancing | ActiGraph GT3X+ | Accelerometer | Logistic regression |
[1] | Guiry et al. | 2014 | Walking, running, cycling, standing, sitting, elevator ascents, elevator descents, stair ascents, stair descents | Samsung Galaxy Nexus smartphone, Motorola MotoActv smartwatch | Accelerometer, magnetometer, gyroscope, GPS, light, pressure. Smartwatch only accelerometer | C4.5, CART, Naive Bayes, multilayer perceptron and support vector machine |
[4] | Mortazavi et al. | 2014 | Bicep Curls, crunches, jumping jacks, push‐ups, shoulder lateral raises | Samsung Galaxy Gear | Accelerometer, gyroscope | Random forests, decision Trees, SVM and Naive Bayes |
[13] | Khan et al. | 2014 | 16 different subject | LG Nexus 4 Smartphone | Accelerometer, pressure, microphone | Artificial neural network, Support vector machines and Gaussian mixture model |
[10] | Dong et al. | 2014 | Period of Eating | iPhone 4 | Accelerometer gyroscope | Naive Bayes |
[11] | Ramos‐Garcia and Hoover | 2013 | Gesture recognition | Wrist‐worn accelerometer and gyroscope | Accelerometer gyroscope | Hidden Markov model, k Nearest neighbour |
[14] | Dadashi et al. | 2013 | Breaststroke swimming temporal phases | IMU wearable sensor | Accelerometer, gyroscope | Hidden Markov model |
[7] | da Silva and Galeazzo | 2013 | Walking, running, sitting, standing, lying, climbing stairs, coming down stairs and working on computer | Ez‐430 Choronos | Accelerometer | Multilayer perceptron, k nearest neighbour, support vector machine |
[9] | Scholl and van Laerhoven | 2012 | Cigarette smoking | Hedgehog | Accelerometer | Gaussian classifier |
[8] | Chernbumroong et al. | 2011 | Sitting, standing, lying, walking, running | Ez‐430 Choronos | Accelerometer | Artificial neural network, decision tree |
[17] | Bieber and Peter | 2008 | Walking, running, cycling, and resting | Bosch 3D‐acceleration sensor | Accelerometer | SVM, Bayesian nets and decision trees, J48 |
[15] | Parkka et al. | 2007 | ironing, vacuuming, walking, running, cycling on exercise bicycle | Kionix accelerometer, XV‐3500 gyroscope | Accelerometer, gyroscope | Pearson linear correlation |
[18] | Bao and Intille | 2004 | 20 different subject | ADXL210E accelerometers (On Body) | Accelerometer | Decision table, IBL, C4.5, naïve Bayes |
3. Overview of statistical learning
Statistical learning contains a large number of unsupervised and supervised tools for inferencing from data. In general terms, supervised statistical learning is employed as a statistical model to estimate or predict an output using relevant inputs in various areas such as public policy, medicine, astrophysics and business. In unsupervised statistical learning, learning of relationships and structure of data is possible without supervising the output [5]. In this chapter, supervised statistical learning methods (Naive Bayes, logistic regression, Bayesian network, k nearest neighbour (kNN) and multilayer Perceptron) are used for activity recognition.
The Naive Bayes method is applied to learn and represent probabilistic information from data with clear and easy understanding by using supervised learning tasks in which classes are known in training phase, in which prediction of classes is realized in the test phase [20]. Multilayer perceptron is a feedforward structure of artificial neural networks because the output of the input layer and all intermediate layers is submitted only to the higher layer. Here ‘layer’ means a layer of perceptrons. The number of hidden layers and the number of perceptrons at each hidden layer are not limited [21]. In kNN, the whole of the calibration data set is used as a classification model. In other words, kNN does not create a different model from calibration data set due to its non‐parametric construction. In the same multidimensional hyperspace, a test set is used as the calibration set for classification. From the new test set object to the calibration objects, the K nearest neighbours are computed. The smallest length using a chosen norm is called as ‘nearest’ [22]. Logistic regression is used to describe and test suppositions about associations between class variable and other related predictor variables by estimating probabilities using a logistic function. Logistic regression can be binomial, ordinal or multinomial [23]. One of the probabilistic graphical models is Bayesian networks. In Bayesian networks, the knowledge about a vague subject is showed as graphical structures. In particular, variables are represented as nodes in the graph, whereas probabilistic dependencies among the variables are represented as the edges. The values of the edges in the graph can be calculated by using known computational and statistical methods [24]. The model structure of the Bayesian Network used for the research in the case study is shown in Figure 1. Variables are standard deviations and averages of
4. Case study: activity recognition on smartwatches using statistical learning methods
In this study, activity recognition is performed by using accelerometer sensor data. Accelerometer measures the acceleration force in m/s2 that is applied to a device on all three physical axes (
Figure 3 shows amplitude change of accelerometer
Accelerometer signals of smartwatch are utilized for activity detection by using statistical learning methods. Figure 4 represents the flowchart of activity recognition which includes collecting data, feature selection, classification and development of smartwatch application steps. Information about these steps is given in the following sub‐sections.
4.1. Collecting data and feature selection
Smartwatch application has only one push button. This button serves to begin and end collecting the sensor data. Figure 7 shows structure of storing sensor data to smartphone internal storage. The collected sensor data are transferred to the connected smartphone and stored in smartphone internal memory as CSV format with the desired label name. In order to start the data collection process, the user writes performing activity name to mobile phone application and press the ‘Begin’ button on smartwatch application. During the data collection, smartwatch must be located on the wrist.
For training statistical methods, raw sensor data are collected on nine different human activities viz running, walking, typing, writing, standing, writing on board, stationary, cleaning and teeth brushing which consists 900.000 lines (100.000 samples for each activity). Then data are split into parts of 200 lines (4‐second intervals) to form a pattern. Thus, each activity has 500 patterns. Features are extracted from raw accelerometer data. These features are standard deviations and average values of
4.2. Classification with statistical learning methods
Methods | Accuracy rates | F measure | ROC area | RMSE |
---|---|---|---|---|
81.33 | 0.819 | 0.974 | 0.1644 | |
91.55 | 0.916 | 0.993 | 0.1242 | |
89.68 | 0.896 | 0.971 | 0.135 | |
85.55 | 0.854 | 0.977 | 0.1507 | |
74.57 | 0.734 | 0.957 | 0.1937 |
The RMSE of a model prediction with respect to the estimated variable
where Xobs is observed values and Xmodel is modelled values at time/place
ROC (receiver operating characteristic) area is also known as area under curve (AUC) is calculated as in Eq. (3).
Here,
According to Table 2, the Bayesian network method has the best accuracy rate 91.55% and minimum RMSE value.
Classified as | A | B | C | D | E | F | G | H | I |
---|---|---|---|---|---|---|---|---|---|
249 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
1 | 185 | 0 | 14 | 13 | 3 | 0 | 1 | 39 | |
0 | 1 | 253 | 0 | 0 | 0 | 0 | 0 | 6 | |
0 | 1 | 0 | 253 | 0 | 0 | 0 | 0 | 0 | |
0 | 10 | 0 | 0 | 233 | 0 | 0 | 0 | 0 | |
0 | 9 | 2 | 0 | 0 | 207 | 0 | 0 | 29 | |
0 | 9 | 0 | 0 | 2 | 7 | 235 | 0 | 0 | |
9 | 2 | 2 | 0 | 0 | 0 | 0 | 240 | 0 | |
1 | 19 | 0 | 0 | 1 | 8 | 0 | 1 | 205 |
According to Table 3, recognition accuracy for cleaning is about 75%. This activity does not have simple characteristics and is easily confused with other activities. For example, 19 of 235 brushing teeth activity are misclassified as cleaning and 39 of 256 cleaning activity are misclassified as brushing teeth. In addition, writing board activities are confused with brushing teeth and cleaning activities are confused with running and walking. Because cleaning activity involves walking.
Developed mobile application for smartwatches collects sensor data and converts it as a pattern in 4 seconds intervals. Then it classifies the data by using trained Bayesian network model and WEKA API and shows detected activity on smartwatch screen (Figure 14b). At this step, the smartwatch application does not need the smartphone. Also it is possible to report detected activities on smartphone screen via developed application for Android smartphone (Figure 14a).
Steps of the algorithm and sample Java codes used in activity detection application are given in Figure 15.
5. Conclusion
In this chapter, human activity recognition on smartwatches by using statistical methods is studied. It is found that the Bayesian network method is the best method for the dataset used in the study. Through this work, it is possible to understand how to classify the human activities by using statistical learning methods and sensor data. Only accelerometer sensor data are used for nine different activities. To use different sensors, which smartwatches have (heart rate monitor, ambient light, GPS and gyroscope), to detect more activities by increasing the number of classes (handshake, smoking, drinking, etc.) or to separate more complex parts of activities (e.g. walking hands in pockets, walking hand in hand, etc.) can improve the studies for human activity recognition in the future.
Nowadays, smartwatches and wrist‐worn sensors are used in daily activity monitoring and healthy lifestyle applications. These devices can also help to warn the user in daily life for creating a healthy sportive habit. For example, smartwatch can send a reminder to user to warn about staying stationary for a long time. Such devices and applications can give information about people such as how much they walk, how long they sleep and how many calories they burn. In addition, this kind of work also contributes to virtual reality applications.
Acknowledgments
This study is supported by Muğla Sıtkı Koçman University Scientific Research Projects under the grant number 016‐061.
References
- 1.
John J. Guiry, Pepijn van de Ven and John Nelson. Multi‐Sensor Fusion for Enhanced Contextual Awareness of Everyday Activities with Ubiquitous Devices. Sensors. 2014; 14 (3):5687–5701. DOI: 10.3390/s140305687 - 2.
Ensar Arif Sağbaş and Serkan Ballı. Transportation Mode Detection By Using Smartphone Sensors And Machine Learning. Pamukkale University Journal of Engineering Sciences. 2016; 22 (5):376–383. DOI: 10.5505/pajes.2015.63308 - 3.
Muhammad Shoaib, Stephan Bosch, Hans Scholten, Paul J. M. Havinga and Ozlem Durmaz Incel. Towards Detection of Bad Habits by Fusing Smartphone and Smartwatch Sensors. In: Pervasive Computing and Communication Workshops; 23–27 March; St. Louis, MO. IEEE; 2015. pp. 591–596. DOI: 10.1109/PERCOMW.2015.7134104 - 4.
Bobak Jack Mortazavi, Mohammad Pourhomayoun, Gabriel Alsheikh, Nabil Alshurafa, Sunghoon Ivan Lee and Majid Sarrafzadeh. Determining the Single Best Axis for Exercise Repetition Recognition and Counting on Smartwatches. In: Wearable and Implantable Body Sensor Networks; 16–19 June; Zurich. IEEE; 2014. pp. 33–38. DOI: 10.1109/BSN.2014.21 - 5.
Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani. An Introduction to Statistical Learning. New York: Springer; 2013. 426 p. DOI: 10.1007/978‐1‐4614‐7138‐7 - 6.
He Wang, Ted Tsung‐Te Lai and Romit Roy Choudhury. MoLe: Motion Leaks through Smartwatch Sensors. In: Conference on Mobile Computing and Networking; 7–11 September; Paris, France. ACM; 2015. pp. 155–166. DOI: 10.1145/2789168.2790121 - 7.
Fernando Ginez da Silva and Elisabete Galeazzo. Accelerometer Based Intelligent System for Human Movement Recognition. In: Advances in Sensors and Interfaces; 13–14 June; Bari. IEEE; 2013. pp. 20–24. DOI: 10.1109/IWASI.2013.6576063 - 8.
Saisakul Chernbumroong, Anthony S. Atkins and Hongnian Yu. Activity Classification Using a Single Wrist‐Worn Accelerometer. In: Software, Knowledge Information, Industrial Management and Applications; 8–11 September; Benevento. IEEE; 2011. pp. 1–6. DOI: 10.1109/SKIMA.2011.6089975 - 9.
Philipp M. Scholl and Kristof van Laerhoven. A Feasibility Study of Wrist‐Worn Accelerometer Based Detection of Smoking Habits. In: Innovative Mobile and Internet Services in Ubiquitous Computing; 4–6 July; Palermo. IEEE; 2012. pp. 886–891. DOI: 10.1109/IMIS.2012.96 - 10.
Yujie Dong, Jenna Scisco, Mike Wilson, Eric Muth and Adam Hoover. Detecting Periods of Eating During Free‐Living by Tracking Wrist Motion. Journal of Biomedical and Health Informatics. 2014; 18 (4):1253–1260. DOI: 10.1109/JBHI.2013.2282471 - 11.
Raul I. Ramos‐Garcia and Adam W. Hoover. A Study of Temporal Action Sequencing During Consumption of a Meal. In: Bioinformatics, Computational Biology and Biomedical Informatics; 22–25 September; Washington DC. New York: ACM; 2013. 68 p. DOI: 10.1145/2506583.2506596 - 12.
Stewart G Trost, Yonglei Zheng and Weng‐Keen Wong. Machine Learning for Activity Recognition: Hip versus Wrist Data. Physiological Measurement. 2014; 35 (11):2183. - 13.
Adil Mehmood Khan, Ali Tufail, Asad Masood Khattak and Teemu H. Laine . Activity Recognition on Smartphones via Sensor-Fusion and KDA-Based SVMs. International Journal of Distributed Sensor Networks. 2014; 10 (5):1–14. DOI: 10.1155/2014/503291 - 14.
Farzin Dadashi, Arash Arami, Florent Crettenand, Gregoire P. Millet, John Komar, Ludovic Seifert and Kamiar Aminian. A Hidden Markov Model of the Breaststroke Swimming Temporal Phases Using Uearable Inertial Measurement Units. In: Body Sensor Networks; 6–9 May; Cambridge, MA, USA. IEEE; 2013. pp. 1–6. DOI: 10.1109/BSN.2013.6575461 - 15.
Juha Parkka, Mikka Ermes, Kari Antila, Mark Van Gils, Ari Manttari and Heikki Nieminen. Estimating Intensity of Physical Activity: A Comparison of Wearable Accelerometer and Gyro Sensors and 3 Sensor Locations. In: Engineering in Medicine and Biology Society; 22‐26 August; Lyon. IEEE; 2007. pp. 1511–1514. DOI: 10.1109/IEMBS.2007.4352588 - 16.
Sheng Shen, He Wang and Romit Roy Choudhury. I Am a Smartwatch and I Can Track My User’s Arm. In: Mobile Systems, Applications and Services; 25–30 June; Singapore. Singapore: ACM; 2016. DOI: 10.1145/2906388.2906407 - 17.
Gerald Bieber and Christian Peter. Using Physical Activity for User Behavior Analysis. In: Pervasive Technologies Related to Assistive Environments; 15–19 July; Athens, Greece. New York: ACM; 2008. DOI: 10.1145/1389586.1389692 - 18.
Ling Bao and Stephen S. Intille. Activity Recognition from User-Annotated Acceleration Data. In: Pervasive Computing; 18-23 April; Linz. Berlin: Springer-Verlag; 2004. p. 1–17. - 19.
Ki‐Hoon Kim, Mi‐Young Jeon, Ju‐Young Lee, Ji‐Hoon Jeong and Gu‐Min Jeong . A Study on the App Development Using Sensor Signals from Smartphone and Smartwatch. Advanced Science and Technology Letters. 2014; 62 :66–69. DOI: 10.14257/astl.2014.62.17 - 20.
George H. John and Pat Langley. Estimating Continuous Distributions in Bayesian Classifiers. In: Uncertainty in Artificial Intelligence; 18–20 August; Quebec. San Francisco: Morgan Kaufmann; 1995. pp. 338–345. - 21.
Ludmila I. Kuncheva. Combining Pattern Classifiers Methods and Algorithms. New Jersey: John Wiley & Sons, Inc.; 2004. 350 p. - 22.
Bjørn Kåre Alsberg, Royston Goodacre, Jem J Rowland and Douglas Kell. Classification of Pyrolysis Mass Spectra by Fuzzy Multivariate Rule Induction-Comparison with Regression, K-Nearest Neighbour, Neural and Decision-Tree Methods. Analytical Chimica Acta. 1997; 348 (1):389–407. DOI: 10.1016/S0003-2670(97)00064-0 - 23.
Chao‐Ying Joanne Peng, Kuk Lida Lee and Gary M. Ingersoll. An Introduction to Logistic Regression Analysis and Reporting. The Journal of Educational Research. 2002; 96 (1):3–14. DOI: 10.1080/00220670209598786 - 24.
Irad Ben-Gal. Bayesian Networks. In: Fabrizio Ruggeri, Ron Kenett and Frederick Faltin, editors. Encyclopedia of Statistics in Quality and Reliability. Chichester, UK: Wiley; 2007. - 25.
Rahul Ravindran, Riya Suchdev, Yash Tanna and Sridhar Swamy. Context Aware and Pattern Oriented Machine Learning Framework (CAPOMF) for Android. In: Advances in Engineering and Technology Research; 1–2 August; Unnao. IEEE; 2014. pp. 1–7. DOI: 10.1109/ICAETR.2014.7012912 - 26.
Android. Sensors Overview [Internet]. Available from: https://developer.android.com/guide/topics/sensors/sensors_overview.html [Accessed: 15.06.2016] - 27.
Motorola. Moto 360 [Internet]. Available from: http://www.motorola.com/us/products/moto‐360 [Accessed: 15.06.2016] - 28.
Stephen R. Garner. WEKA: The Waikato Environment for Knowledge Analysis. In: Computer Science Research Students Conference; April; New Zealand 1995. pp. 57–64. - 29.
Aytuğ Onan, Serdar Korukoğlu and Hasan Bulut. Ensemble of Keyword Extraction Methods and Classifiers in Text Classification. Expert Systems with Applications. 2016; 57 :232–247. - 30.
Root Mean Square Error (RMSE) [Internet]. Available from: http://www.ctec.ufal.br/professor/crfj/Graduacao/MSH/Model%20evaluation%20methods.doc [Accessed: 09.09.2016] - 31.
Calculating ROC Curves and AUC Scores [Internet]. Available from: http://www.cs.ru.nl/~tomh/onderwijs/dm/dm_files/roc_auc.pdf [Accessed: 09.09.2016]