Summary of the studies.
The aim of this study is to explore the usage of statistical learning methods on wearable devices and realize an experimental study for recognition of human activities by using smartwatch sensor data. To achieve this objective, mobile applications that run on smartwatch and smartphone were developed to gain training data and detect human activity momentarily; 500 pattern data were obtained with 4‐second intervals for each activity (walking, typing, stationary, running, standing, writing on board, brushing teeth, cleaning and writing). Created dataset was tested with five different statistical learning methods (Naive Bayes, k nearest neighbour (kNN), logistic regression, Bayesian network and multilayer perceptron) and their performances were compared.
- statistical learning
- activity recognition
- wearable devices
- Bayesian networks
The usage of wearable technology is increasing rapidly, and the effects of user healthcare are enormous. Today's smart devices have more built‐in sensors than before. Wearable sensors are small devices which are carried by people, while they are performing daily activities. These sensors such as an accelerometer, microphone, GPS and barometer record the physical condition of person such as location change, moving direction and moving speed. Latest smartphones and smartwatches have many wearable sensors as built‐in [1, 2]. Because of equipped with various on‐board sensors, smartphones and wrist‐worn devices such as smartwatches are being extensively used for activity recognition in recent studies . With the popularity of the smartwatches, wrist‐worn sensor devices will become an increasingly important tool in personal health monitoring . Statistical learning methods are generally used in activity recognition studies. Statistical learning refers to a set of tools for modelling and understanding complex datasets. It is a recently developed area in statistics and blends with parallel developments in computer science and, in particular, machine learning .
The aim of this chapter is to investigate the usage of statistical learning methods on wearable devices and carry out a case study for recognition of human activities with accelerometer data of smartwatch by using statistical learning methods. This chapter is organized as follows: related works are described in detail in Section 2. Then, overview of statistical methods is mentioned in Section 3. Next, human activity recognition with smartwatches is explained in Section 4. Finally, Section 5 concludes the chapter.
2. Related works
When examining the literature, various studies are found statistical learning methods with wearable devices. Wang et al.  imagined a user typing on a laptop keyboard while wearing a smartwatch. The accelerometer and gyroscope data, which obtained from Samsung Galaxy Live, were used as training data, and processed through a sequence of steps, including key‐press detection, hand‐motion tracking, character point cloud computation and Bayesian modelling and inference. Shoaib et al.  carried out to recognize of different living activities by using a smartphone and a smartwatch simultaneously and evaluated their effectiveness in recognizing human activities. They used J48, kNN and SVM (support vector machines) to recognize 13 various activities. da Silva and Galeazzo  presented the development of a system based on computational intelligence techniques and on an accelerometer to perform, in a comfortable and non‐intrusive manner, the recognition of basic movements of a person’s routine. Three different computational intelligence techniques were evaluated in order to search for the best performance of the recognition of the movements executed by the watch user. Chernbumroong et al.  studied classification of five human activities by using only accelerometer data and two learning algorithms: Artificial Neural Networks and Decision Tree C4.5. Scholl and van Laerhoven  presented a feasibility study with smokers wearing an accelerometer device on their wrist over the course of a week to detect their smoking habits based on detecting typical gestures carried out while smoking a cigarette. The Gaussian method was used as a classifier. Dong et al.  described a new method that uses a watch‐like configuration of sensors to continuously track wrist motion throughout the day and automatically detect periods of eating. Accelerometer and gyroscope sensor data were used in this study. Ramos‐Garcia and Hoover  developed a Hidden Markov model (HMM) and compared its recognition performance against a non‐sequential classifier (kNN), using a set of four actions (rest, utensiling, bite and drink). Trost et al.  compared the activity recognition rates of an activity classifier trained on acceleration signal collected on the wrist and hip. Features were extracted from 10 seconds windows and inputted into a regularized logistic regression model. Guiry et al.  investigated the role of smart devices including smartphones and smartwatches which can play in identifying activities of daily living. The activities examined include walking, running, cycling, standing, sitting, elevator ascents, elevator descents, stair ascents and stair descents. Data from this study were used to train and test five well‐known statistical machine learning algorithms: C4.5, CART, naïve Bayes, multilayer perceptrons and finally support vector machines. Mortazavi et al.  introduced a framework for platform creation (e.g. accelerometer only system versus accelerometer and gyroscope) and machine learning of some activities, which can be especially useful in the emerging market of smartwatches. Random forests, decision trees, Naive Bayes and SVM methods were compared. Khan et al.  implemented a smartphone‐based HAR scheme in accordance with these requirements. Time domain features were extracted from only three smartphone sensors, and a nonlinear discriminatory approach was employed to recognize 15 activities with a high accuracy. Evaluations were performed in both offline and online settings. Dadashi et al.  carried out detection of important breaststroke swimming events automatically by using Hidden Markov model (HMM) and wearable sensors. Parkka et al.  used accelerometers and gyroscopes attached to ankle, wrist and hip to estimate intensity of physical activity. Data from common everyday tasks and exercise were collected with 11 subjects. Shen et al.  tracked the 3D posture of the entire arm—both wrist and elbow—using the motion and magnetic sensors on smartwatches. Bieber and Peter  studied behaviour analysis using 3D sensor data and learning techniques and obtained sufficient results. Bao and Intille  developed an algorithm and evaluated to detect physical activities from data acquired using five small biaxial accelerometers worn simultaneously on different parts of the body. Kim et al.  developed an application by using sensor signals from smartphone and smartwatch. Summary of the literature is given in Table 1.
|||Shen et al.||2016||Arm posture||Smartwatch||Accelerometer, gyroscope, compass||Hidden Markov model|
|||Wang et al.||2015||Typing on a laptop keyboard||Samsung Gear Live||Accelerometer gyroscope||Bayessian Inference|
|||Shoaib et al.||2015||Smoking, eating, typing, writing, drinking coffee, talking, walking, jogging, biking, walking upstairs and downstairs, sitting, standing||Smartphone and smartwatch||Accelerometer gyroscope||Support vector machine, k nearest neighbour, J48 decision trees|
|||Trost et al.||2014||Lying down, sitting, standing, walking, running, basketball and dancing||ActiGraph GT3X+||Accelerometer||Logistic regression|
|||Guiry et al.||2014||Walking, running, cycling, standing, sitting, elevator ascents, elevator descents, stair ascents, stair descents||Samsung Galaxy Nexus smartphone, Motorola MotoActv smartwatch||Accelerometer, magnetometer, gyroscope, GPS, light, pressure. Smartwatch only accelerometer||C4.5, CART, Naive Bayes, multilayer perceptron and support vector machine|
|||Mortazavi et al.||2014||Bicep Curls, crunches, jumping jacks, push‐ups, shoulder lateral raises||Samsung Galaxy Gear||Accelerometer, gyroscope||Random forests, decision Trees, SVM and Naive Bayes|
|||Khan et al.||2014||16 different subject||LG Nexus 4 Smartphone||Accelerometer, pressure, microphone||Artificial neural network, Support vector machines and Gaussian mixture model|
|||Dong et al.||2014||Period of Eating||iPhone 4||Accelerometer gyroscope||Naive Bayes|
|||Ramos‐Garcia and Hoover||2013||Gesture recognition||Wrist‐worn accelerometer and gyroscope||Accelerometer gyroscope||Hidden Markov model, k Nearest neighbour|
|||Dadashi et al.||2013||Breaststroke swimming temporal phases||IMU wearable sensor||Accelerometer, gyroscope||Hidden Markov model|
|||da Silva and Galeazzo||2013||Walking, running, sitting, standing, lying, climbing stairs, coming down stairs and working on computer||Ez‐430 Choronos||Accelerometer||Multilayer perceptron, k nearest neighbour, support vector machine|
|||Scholl and van Laerhoven||2012||Cigarette smoking||Hedgehog||Accelerometer||Gaussian classifier|
|||Chernbumroong et al.||2011||Sitting, standing, lying, walking, running||Ez‐430 Choronos||Accelerometer||Artificial neural network, decision tree|
|||Bieber and Peter||2008||Walking, running, cycling, and resting||Bosch 3D‐acceleration sensor||Accelerometer||SVM, Bayesian nets and decision trees, J48|
|||Parkka et al.||2007||ironing, vacuuming, walking, running, cycling on exercise bicycle||Kionix accelerometer, XV‐3500 gyroscope||Accelerometer, gyroscope||Pearson linear correlation|
|||Bao and Intille||2004||20 different subject||ADXL210E accelerometers (On Body)||Accelerometer||Decision table, IBL, C4.5, naïve Bayes|
3. Overview of statistical learning
Statistical learning contains a large number of unsupervised and supervised tools for inferencing from data. In general terms, supervised statistical learning is employed as a statistical model to estimate or predict an output using relevant inputs in various areas such as public policy, medicine, astrophysics and business. In unsupervised statistical learning, learning of relationships and structure of data is possible without supervising the output . In this chapter, supervised statistical learning methods (Naive Bayes, logistic regression, Bayesian network, k nearest neighbour (kNN) and multilayer Perceptron) are used for activity recognition.
The Naive Bayes method is applied to learn and represent probabilistic information from data with clear and easy understanding by using supervised learning tasks in which classes are known in training phase, in which prediction of classes is realized in the test phase . Multilayer perceptron is a feedforward structure of artificial neural networks because the output of the input layer and all intermediate layers is submitted only to the higher layer. Here ‘layer’ means a layer of perceptrons. The number of hidden layers and the number of perceptrons at each hidden layer are not limited . In kNN, the whole of the calibration data set is used as a classification model. In other words, kNN does not create a different model from calibration data set due to its non‐parametric construction. In the same multidimensional hyperspace, a test set is used as the calibration set for classification. From the new test set object to the calibration objects, the K nearest neighbours are computed. The smallest length using a chosen norm is called as ‘nearest’ . Logistic regression is used to describe and test suppositions about associations between class variable and other related predictor variables by estimating probabilities using a logistic function. Logistic regression can be binomial, ordinal or multinomial . One of the probabilistic graphical models is Bayesian networks. In Bayesian networks, the knowledge about a vague subject is showed as graphical structures. In particular, variables are represented as nodes in the graph, whereas probabilistic dependencies among the variables are represented as the edges. The values of the edges in the graph can be calculated by using known computational and statistical methods . The model structure of the Bayesian Network used for the research in the case study is shown in Figure 1. Variables are standard deviations and averages of x‐, y‐ and z‐axis of accelerometer sensor.
4. Case study: activity recognition on smartwatches using statistical learning methods
In this study, activity recognition is performed by using accelerometer sensor data. Accelerometer measures the acceleration force in m/s2 that is applied to a device on all three physical axes (x, y and z) given in Figure 2, including the force of gravity [25, 26].
Figure 3 shows amplitude change of accelerometer x‐axis for nine different daily activities (typing, writing, writing on board, walking, running, cleaning, standing, brushing teeth and stationary).
Accelerometer signals of smartwatch are utilized for activity detection by using statistical learning methods. Figure 4 represents the flowchart of activity recognition which includes collecting data, feature selection, classification and development of smartwatch application steps. Information about these steps is given in the following sub‐sections.
4.1. Collecting data and feature selection
Hardware: Motorola Moto 360  smartwatch (Figure 5) is used. This device has quad core 1.2 GHz processor, 512 MB RAM and built‐in accelerometer, pedometer ambient light and optical heart rate monitor sensors. In this chapter, only accelerometer sensor is used to detect human activities. While coding the smartwatch application, SENSOR_DELAY_UI is set as sampling rate which allows the sampling rate of 50 Hz. This device is capable of tracing daily life activities about all day long with 400 mAh battery. It has Android wear operating system.
Software: For collecting dataset, two Android‐based applications are developed by using Java programming language. One of these applications runs on smartwatch (Figure 6b) and the other one runs on smartphone (Figure 6a) that connected smartwatch. Because collected sensor data are sent to smartphone to keep in internal storage.
Smartwatch application has only one push button. This button serves to begin and end collecting the sensor data. Figure 7 shows structure of storing sensor data to smartphone internal storage. The collected sensor data are transferred to the connected smartphone and stored in smartphone internal memory as CSV format with the desired label name. In order to start the data collection process, the user writes performing activity name to mobile phone application and press the ‘Begin’ button on smartwatch application. During the data collection, smartwatch must be located on the wrist.
For training statistical methods, raw sensor data are collected on nine different human activities viz running, walking, typing, writing, standing, writing on board, stationary, cleaning and teeth brushing which consists 900.000 lines (100.000 samples for each activity). Then data are split into parts of 200 lines (4‐second intervals) to form a pattern. Thus, each activity has 500 patterns. Features are extracted from raw accelerometer data. These features are standard deviations and average values of x‐, y‐ and z‐axes of accelerometer data given in Figure 1.
4.2. Classification with statistical learning methods
Experimental study: Extracted features are tested by five different statistical learning methods (Naive Bayes, kNN, logistic regression, Bayesian network and multilayer perceptron) using WEKA Toolkit  comprising these methods. Half of the data is used for training and remaining to test. For training and testing data are split randomly. Table 2 displays the comparison of evaluation metrics such as accuracy rates, F‐measure, ROC area, root mean squared error (RMSE) of statistical learning methods. F‐measure is a measure of a test’s accuracy. Formulation of F‐measure is given in Eq. (1). FN, FP, TP and TN represents the number of false negatives, the number of false positives, the number of true positives and the number of true negatives, respectively .
|Methods||Accuracy rates||F measure||ROC area||RMSE|
|kNN (k = 3)||89.68||0.896||0.971||0.135|
The RMSE of a model prediction with respect to the estimated variable Xmodel is defined as the square root of the mean squared error given in Eq. (2):
where Xobs is observed values and Xmodel is modelled values at time/place i .
ROC (receiver operating characteristic) area is also known as area under curve (AUC) is calculated as in Eq. (3).
Here, i runs over all m data points with true label 1, and j runs over all n data points with true label 0; pi and pj denote the probability score assigned by the classifier to data point i and j, respectively. 1 is the indicator function: it outputs 1 if the condition is satisfied .
According to Table 2, the Bayesian network method has the best accuracy rate 91.55% and minimum RMSE value. F‐measure is a measure of a test’s accuracy. The best values of both ROC area and F‐measure belong to the Bayesian network method. Confusion matrix for Bayesian network is given in Table 3 and ROC curves of five different methods (Bayesian network, kNN, Naïve Bayes, logistic regression and multilayer perceptron) are given in Figures 8–12.
|A = Typing||249||0||0||0||0||0||0||0||0|
|B = Cleaning||1||185||0||14||13||3||0||1||39|
|C = Writing||0||1||253||0||0||0||0||0||6|
|D = Running||0||1||0||253||0||0||0||0||0|
|E = Walking||0||10||0||0||233||0||0||0||0|
|F = Writing board||0||9||2||0||0||207||0||0||29|
|G = Standing||0||9||0||0||2||7||235||0||0|
|H = Stationary||9||2||2||0||0||0||0||240||0|
|I = Brushing teeth||1||19||0||0||1||8||0||1||205|
According to Table 3, recognition accuracy for cleaning is about 75%. This activity does not have simple characteristics and is easily confused with other activities. For example, 19 of 235 brushing teeth activity are misclassified as cleaning and 39 of 256 cleaning activity are misclassified as brushing teeth. In addition, writing board activities are confused with brushing teeth and cleaning activities are confused with running and walking. Because cleaning activity involves walking.
Development mobile application: According to the results shown in Table 2, the Bayesian network method is used in Android wear‐based classification application for recognition human activities (Figure 13).
Developed mobile application for smartwatches collects sensor data and converts it as a pattern in 4 seconds intervals. Then it classifies the data by using trained Bayesian network model and WEKA API and shows detected activity on smartwatch screen (Figure 14b). At this step, the smartwatch application does not need the smartphone. Also it is possible to report detected activities on smartphone screen via developed application for Android smartphone (Figure 14a).
Steps of the algorithm and sample Java codes used in activity detection application are given in Figure 15.
In this chapter, human activity recognition on smartwatches by using statistical methods is studied. It is found that the Bayesian network method is the best method for the dataset used in the study. Through this work, it is possible to understand how to classify the human activities by using statistical learning methods and sensor data. Only accelerometer sensor data are used for nine different activities. To use different sensors, which smartwatches have (heart rate monitor, ambient light, GPS and gyroscope), to detect more activities by increasing the number of classes (handshake, smoking, drinking, etc.) or to separate more complex parts of activities (e.g. walking hands in pockets, walking hand in hand, etc.) can improve the studies for human activity recognition in the future.
Nowadays, smartwatches and wrist‐worn sensors are used in daily activity monitoring and healthy lifestyle applications. These devices can also help to warn the user in daily life for creating a healthy sportive habit. For example, smartwatch can send a reminder to user to warn about staying stationary for a long time. Such devices and applications can give information about people such as how much they walk, how long they sleep and how many calories they burn. In addition, this kind of work also contributes to virtual reality applications.
This study is supported by Muğla Sıtkı Koçman University Scientific Research Projects under the grant number 016‐061.