Open access peer-reviewed chapter
Abstract
Myoelectric control systems as the emerging control strategies for upper limb wearable robots have shown their efficacy and applicability to effectively provide motion assistance and/or restore motor functions in people with impairment or disabilities, as well as augment physical performance in able-bodied individuals. In myoelectric control, electromyographic (EMG) signals from muscles are utilized, improving adaptability and human-robot interactions during various motion tasks. Machine learning has been widely applied in myoelectric control systems due to its advantages in detecting and classifying various human motions and motion intentions. This chapter illustrates the challenges and trends in recent machine learning algorithms implemented on myoelectric control systems designed for upper limb wearable robots, and highlights the key focus areas for future research directions. Different modalities of recent machine learning-based myoelectric control systems are described in detail, and their advantages and disadvantages are summarized. Furthermore, key design aspects and the type of experiments conducted to validate the efficacy of the proposed myoelectric controllers are explained. Finally, the challenges and limitations of current myoelectric control systems using machine learning algorithms are analyzed, from which future research directions are suggested.
Keywords
- myoelectric control
- upper limb exoskeleton
- upper limb exosuit
- pattern recognition
- machine learning
- reinforcement learning
1. Introduction
In the past few decades, the demand for upper limb exoskeletons and exosuits has grown substantially due to their promising applications across industry, medical and military sectors. The exoskeletons consists of rigid links and joints attached to the human body, whereas the exosuits use soft and flexible materials (such as fabric or soft polymer) to interact with the user’s body [1]. The applications of exoskeletons and exosuits include: (i) power augmentation to enhance physical performance or the capabilities of able-boded individuals during strenuous physical tasks [2], and (ii) assisting individuals with disabilities in performing activities of daily living (ADLs) [3].
The exoskeletons and exosuits can be controlled by many different schemes, such as the kinematics control based on the inertia measurement unit sensor (IMU) or encoder [4, 5], the force control based on load cell or torque sensor [6, 7], and the myoelectric control based on the electromyographic sensor (EMG) [8]. Among these control schemes, the myoelectric control systems have gained increasing attention over recent years [9, 10, 11]. The myoelectric control systems of the upper limb exoskeletons and exosuits use surface electromyography (EMG) signals, the electric potentials directly measured from the skeletal muscle as input of the control system for exoskeletons and exosuits (Figure 1). The surface EMG signals are generated from the motor unit activation, controlled by the human brain, and regulated by the motor neurons in the spinal cord. The mechanism for generating surface EMG signals offers surface EMG signals to detect human movement intention [12]. The critical advantage of a myoelectric control system over other control systems is its timely detection of the user’s motion intention leveraging electromechanical delay (EMD); the onset of motion can be detected about 50–100 ms earlier than the physical motion [13, 14]. Moreover, the exoskeletons and exosuits equip with myoelectric control systems have a more adaptive and intelligent interface with the users as the exoskeleton and exosuits can timely and proactively engage assistance through detecting the users’ movement intention [15].
Figure 1.
The generation of electromyography signal and the workflow of myoelectric control systems on upper limb exoskeletons and exosuits.
Myoelectric control systems for upper limb exoskeletons and exosuits initially used on-off/finite state control and proportional control, as described in Refs. [16, 17]. Although these methods are simple and easy to implement, their ability to accommodate a wide range of different movements is limited, as noted in Ref. [18]. Consequently, their primary use have been limited to a single joint function such as elbow flexion/extension or hand grip. To allow for more complex movements across multiple degrees of freedom (DOFs), machine learning (ML) and deep learning (DL) algorithms have been utilized in the myoelectric control systems. However, the myoelectric control systems with ML or DL algorithms generally require considerable computational power, which imposes practical limitations on the portability of exoskeletons and exosuits [19]. In recent years, with the advancements in more powerful and compact embedded computers, myoelectric control systems with ML or DL algorithms became feasible to implement on upper limb exoskeletons and exosuits. Compared to the early staged myoelectric control modalities, the ML or DL-based myoelectric control systems have shown superior performance and better results in complex, multi-DOF upper limb motions; yet, there still exist challenges and limitations which will be discussed in detail in the subsequent sections.
Given the growing interest in machine learning and deep learning-based myoelectric control systems for upper limb exoskeletons and exosuits, the number of publications in the relevant field has rapidly increased over the past decade. Therefore, it is imperative to understand the latest trend and challenges in machine learning and deep learning-based myoelectric control system for upper limb exoskeletons and exosutis. A systematic review that provides a comprehensive overview of the myoelectric control system for upper limb exoskeletons and exosuits [8] was published by the authors. However, the focus of that review was not specifically machine learning and deep learning-based myoelectric control system of upper limb exoskeletons and exosuits, and it does not discuss current challenges and future directions. This chapter is designed to share the extensive review of the machine learning-based myoelectric control system for upper limb exoskeletons and exosuits, particularly from scientific articles published between 2011 and 2023. The identified challenges in implementing machine learning algorithms in the myoelectric control system and future directions are suggested. In the following section, the process of machine learning-based myoelectric control system is summarized (Section 2), and the state-of-the-art implementation of machine learning algorithms in upper limb exoskeletons and exosuits is presented (Section 3). Finally, the remaining unaddressed research questions and tasks are discussed as future research directions (Section 4).
2. The procedure of machine learning-based myoelectric control system
Similar to any other types of myoelectric control systems, machine learning-based myoelectric control systems include key procedural steps: signal acquisition, pre-processing, feature extraction, and motion intention detection through the trained machine learning model (Figure 2). In this section, the process of a machine learning-based myoelectric control system will be presented in detail.
Figure 2.
The process of machine learning based myoelectric control system.
2.1 Data acquisition and signal processing
The acquisition of the EMG signal is critical to the myoelectric control system of the upper limb exoskeleton, as the accuracy of the myoelectric controller primarily depends on the quality of the EMG signal. It is, therefore, important to acquire quality and accurate EMG signals. Three essential components of EMG data acquisition systems are the electrodes for EMG, the sampling rate and signal filtering.
EMG electrodes – The electrodes of the EMG sensors include surface and intramuscular, in which the surface EMG electrode uses an insulative sticker to place the electrode on the skin overlying a muscle to detect the electrical activity of the muscle [20]. On the other hand, the intramuscular EMG electrodes utilize the needles or wires that are inserted into the muscles [21]. Compared to the surface electrode, the intramuscular EMG electrodes can minimize the crosstalk from neighboring muscle segments; however, the operation of intramuscular EMG electrodes requires trained medical expertise, and it is more difficult to use in the research environment. Thus, the surface EMGs are more commonly and widely used in myoelectric control systems of upper limb exoskeletons and exosuits.
Sampling rate – Because the acquisition of EMG signal involves converting the analog signal (voltage generated by muscular activation) to the digital signal (EMG signal used in the myoelectric control systems), which a computer can process, selecting an optimal sampling rate is critical to avoid under-sampling or oversampling. The choice of sampling rate varies between muscle segments. For example, [22] investigates the selection of sampling rate for EMG, and [23] explores the effect of sampling rate for machine learning-based myoelectric control system accuracy
Filtering and rectification – The acquisition of surface EMG measures a combination of the activation of all recruited motor units within the muscle. Therefore, the collected surface EMG signal contains the drift and artifacts which affect the accuracy of the surface EMG signal. The Butterworth filters have been widely used to remove the drift and artifact from collected surface EMG signals. Usually, filtering the raw surface EMG signal includes using a high-pass Butterworth filter to remove the drift and artifact from the raw surface EMG signal, then using a low-pass Butterworth filter to acquire an envelope indicating the magnitude of the surface EMG signal as it changes over time. However, the selection of the order and cut-off frequency of the Butterworth filter could be optimized for different muscle segments; for example, [24] presents the filter selection for surface EMG signal to remove the drift and artifacts. After high and low pass (or band pass) filters, the EMG signals need full wave rectification then a low pass filter for further processing and feature extraction.
2.2 Feature extraction
The pre-processed surface EMG signal is presented as a time sequence that includes a large number of randomness. Therefore, directly feeding the pre-processed surface EMG signal to the machine learning model is impractical. To feed the pre-processed surface EMG signal to the machine learning model, the sequence of pre-processed surface EMG signals must be mapped into a smaller dimension vector called a feature vector [25]. The process of extracting feature vectors from the pre-processed surface EMG signal is called feature extraction. In applying a myoelectric control system for upper limb exoskeletons and exosuits, feature extraction includes two types of methods: feature selection and dimensionality reduction algorithms.
Feature selection – The feature selection uses the statistic formulas to convert the pre-processed surface EMG signal to low-dimensional feature vectors. The feature selection methods are categorized as time-domain and frequency-domain features. However, according to the literature review, only the time-domain features are utilized in the myoelectric control systems for upper limb exoskeletons and exosuits because the time-domain features are computationally simple compared to the frequency-domain features. The time-domain features use the time sequence of surface EMG signal, and there are many types of time-domain features used such as integrated EMG (IEMG), mean absolute value (MAV), root mean square (RMS) and auto-regressive coefficient (AR). Among these, the root mean square and auto-regressive coefficient features are the most widely used time-domain EMG features in myoelectric control systems for upper limb exoskeletons and exosuits, according to our literature survey. Because each feature has different statistical meanings and implications, the effect of selecting different features for the machine learning model could vary. For example, [26] investigated the effect of different features on hand motion classification.
Dimensionality reduction – Similar to the statistical features, the dimensionality reduction also maps the pre-processed surface EMG signal to the low-dimensional feature vectors. However, the statistical formulas used in feature selection do not change the data, while the dimensionality reduction maps the data to the lower dimension. The dimensionality reduction uses linear or nonlinear algorithms to map the data to a low dimension, such as the Principle Component Analysis (PCA) and Linear Discriminate Analysis (LDA). In [27], the performance of dimensionality reduction in classifying the object’s weight using a machine learning model from the surface EMG signal was studied. The dimensionality reduction algorithms are less frequently used in myoelectric control systems for upper limb exoskeletons and exosuits due to their comparably heavy computational process as compared to feature selection process.
3. Taxonomy of machine learning-based myoelectric control systems
The previous section presents an overview of the process of machine learning-based myoelectric control system. This section will summarize the machine learning algorithms used in the reviewed research articles. These algorithms are categorized into three: (1) classification-based models, which are used to detect the type of movement from the surface EMG features, (2) regression-based models, which can make a continuous prediction of the human subject’s joint kinematics or torques, and (3) the reinforcement learning models which optimize the model through the interaction of human subjects and machine learning model. As shown in Figure 3, among the included research articles, the regression-based models and classification models are the most widely used modalities in the machine learning-based myoelectric control systems, while not much work has been done in the implementation and validation of reinforcement learning models.
Figure 3.
Different types of machine learning models used in the reviewed research articles.
3.1 Classification-based myoelectric control system
The classification-based Myoelectric Control System uses the classification model to detect the movement from the statistical features of the human subject’s surface EMG signal. In the classification-based myoelectric control, the labels are pre-defined by the human subjects, which includes types of upper limb movement such as elbow flexion/extension (diversified labels), and the onset of upper limb movement such as in motion or still (binary labels). To train the classification models, surface EMG data corresponding to the labeled motion must be collected from human subject. Then, the classification model can be trained by various machine learning algorithms such as the Support Vector Machine (SVM) [28], Linear Discriminant Analysis (LDA) [29], K-nearest neighbors (KNN) [30], etc. According to our literature review, the support vector machine algorithm is the most popular choice in the classification models of machine learning-based myoelectric control systems. Compared to other machine learning algorithms, the support vector machine algorithm provides better computational efficiency that makes it feasible to run on embedded computers than other types of machine learning algorithms. The support vector machine algorithm can train classification models with either diversified or binary labels. For example, [31] trained a SVM model to classify if the human subject’s finger is in motion or not. In [31], a classification-based myoelectric control system trained by the SVM algorithm was proposed for a (#DOF) finger exoskeleton using binary labels. Additionally, [32] compared the accuracy of the classification-based myoelectric control systems for a hand exoskeleton trained by different machine learning algorithms, i.e., SVM, artificial neural network with backpropagation algorithm (ANN), and K-nearest neighbors (KNN). The classification model in [32] includes five labels that correspond to five different types of hand motion, then used the classification output to trigger the predefined assistive mode in the hand exoskeleton. According to [32], the classification model trained by SVM showed the best accuracy among those compared. Moreover, Cheon et al. proposed a myoelectric interface based on the musculotendinous junctions (MTJs) of the flexor digitorum superficialis (FDS) for reliable control of a robotic glove with a single EMG sensor by identifying power grasp intentions [33] and the support vector machine (SVM) algorithm was used to optimize the classification model. Other machine learning algorithms have also been utilized to train the classification model in the machine learning-based myoelectric control systems. For example, [34] utilized the MCLPBoost – a type of decision-tree algorithm to classify the flexion and extension of elbow and wrist joints. Compared to the SVM algorithm, they showed that the MCLPBoost had better robustness against the noised training data.
Many research articles reviewed targeted to improve the performance of classification models. For instance, [16, 35] studied the impact of feature extraction on the accuracy of classification model where two types of feature extraction techniques were explored. The type 1 feature extraction technique converted the single-channel EMG signal to 14 different statistical features; the type 2 feature extraction technique converted five channel EMG signal to a single statistical feature. Both type 1 and 2 feature extraction techniques were designed for the same upper limb exoskeleton and the classification models were trained by the same machine learning algorithm. The experimental result indicated the type 1 feature extraction technique outperformed the type 2 feature extraction from which they suggested that when training the classification-based myoelectric control systems, higher dimensional training set gives better performance. Moreover, [36, 37] implemented the sensor fusion method by combining the EMG and electroencephalography (EEG) signal to improve the accuracy of the classification-based myoelectric control system trained by the artificial neural network with a backpropagation algorithm. Additionally, to prevent the misclassification caused by the unfiltered noise in EMG signals such as crosstalk and motion artifacts, [38] utilized a threshold method in which the amplitude of filtered EMG signal must be greater than a specific value to be an input to the classification model. Twardowski et al. used the machine learning algorithm to convert the motor unit firings from the sEMG signals into biomechanically informed signals that drive the actuation [39]. The resulting signal provides a smoother control scheme with less delay versus using the MAV and RMS response to modulate the actuation. The EMG signal in the study [31, 32, 34, 39] used the root mean square (RMS) as statistical features, while [40] used integrated EMG (iEMG) to train the classification model. Compared to the RMS feature, the iEMG feature requires less computational power. The classification model presented in [40] plotted the output data onto a 2D Cartesian plane that can be distinguished in real-time using a Point-in Polygon algorithm commonly used in computer graphics. This algorithm determines whether the sample in the plane belongs in or out of a given polygonal area which is the area of each given label. Among the tested classifiers, this method provided the highest classification accuracy (94%) when classifying hand grasp motions.
The abovementioned articles utilized the statistical features of EMG data as input for the classification-based myoelectric control systems. However, the raw EMG signal can also be used as input for the classification-based myoelectric control system, as demonstrated by [41], which successfully implemented a vision transformer model to classify two datasets using raw multichannel EMG data. The transformer model is commonly used in natural language processing, but the encoder-decoder network can be applied to determine the underlying characteristics of the input data without manual feature extraction or signal pre-processing. The resulting model achieved a higher classification accuracy versus a convolution neural network model and an LSTM network.
3.2 Regression-based myoelectric control system
The regression-based myoelectric control system implements regression analysis techniques. In statistics, regression analysis estimates the relationship between a dependent variable (output of regression-based myoelectric control system) and an independent variable (usually the EMG features in the regression-based myoelectric control system) by using a regression model. Compared to the classification-based myoelectric control system, the regression model can output continuous variables such as joint torque and joint angle. The regression model can be trained by various machine learning algorithms. However, there are two regression models found in our literature review, artificial neural network with backpropagation algorithm and Kalman Filters.
Among the research articles reviewed, the artificial neural network was the most widely used method to train the regression model. For example, [42] implemented a regression model to estimate the joint angle from the statistical feature of the human subject’s EMG signal. The regression model is trained by artificial neural network with a back propagation algorithm, and the results showed that the regression model could accurately estimate the joint angle of human. Additionally, the regression-based myoelectric control systems have also been widely used in the bilateral training of hand exoskeletons. Because the bilateral training focuses on using the unimpaired hand to help the impaired hand restore its motor control capability, the myoelectric control scheme must accurately estimate the joint kinematics or joint torque of the unimpaired hand which complies with the characteristics of regression-based myoelectric control systems. For example, [38, 43, 44, 45, 46, 47, 48] implemented the regression model to estimate the joint angle or joint torque from the unimpaired hand to help the impaired hand to restore its motor control capability. On the other hand, Kalman Filter is another approach used in the regression model for myoelectric control of upper limb wearable robots. Compared to the artificial neural network with backpropagation method, Kalman filter does not need much time and extensive datasets to train the model. Moreover, tuning Kalman filter requires less computational power than tuning the artificial neural network which makes it easier to run on an embedded computer. The studies [49, 50] utilized the Kalman filter to compute the joint torque based on the EMG signal whose regression models offered better accuracy when compared to the regression model trained by artificial neural network with backpropagation algorithm. Another method proposed by Kopke et al. used 6 DOF loadcells and EMG sensors to acquire the training data and the linear discriminate analysis (LDA) algorithm to train the regression model [51]. The experiment demonstrated a 92% accuracy in estimating the joint torque of human subjects’ shoulder and elbow joint.
Furthermore, some studies focused on improving the accuracy of regression models. For example, Sierotowice et al. [52] utilized a ridge regression algorithm and a feature selection algorithm called Random Fourier Features to improve the accuracy of the regression model to estimate the hand-grasping force. The regression algorithm of the controller achieved a higher classification accuracy when determining the target forces versus the random Fourier features algorithm (80% versus 73%, respectively). Moreover, the work by Meattini et al. used a soft dynamic time warping (soft-DTW) method to improve the accuracy of the neural network based regression model [53] and the result of this study shows comparable performance to the conventional neural network regression model.
3.3 Reinforcement learning based myoelectric control system
The reinforcement learning algorithm is another type of machine learning algorithm which are used as a machine learning based myoelectric control system. Different from the classification and regression models, the reinforcement learning model trains an agent to choose the optimal action under a specific state in an environment. The process of reinforcement learning can be divided into several steps; in each step, the smart agent executes an action based on a specific state and receive a reward signal as feedback. The objective of the smart agent is to find the optimal action to maximize the accumulative reward.
Compared to the other two types of machine learning myoelectric control systems, only a few included research literature implemented the reinforcement learning algorithm. Hamaya et al. [54] utilized an elbow exoskeleton and applied the Probabilistic Inference for Learning Control (PILCO) reinforcement learning algorithm. The state vector included elbow joint kinematics and EMG signals, and the reward was based on the deviation between the intended and actual trajectory. PILCO employed the Gaussian process to learn the probabilistic dynamic model of the interface between the human and the exoskeleton. The learned model was then used to assess the control policy, which was optimized using the policy gradient method [55]. This approach proved to be more efficient than other machine learning myoelectric control systems, leading to a shorter training period.
4. Discussion
This section outlines several research questions and tasks that need to be addressed in future studies, including the robustness of machine learning-based myoelectric control system, the incorporation of safety requirements in machine learning-based myoelectric control systems, and the clinical assessment of assistive and rehabilitative upper limb exoskeletons and exosuits with machine learning-based myoelectric control systems. These research questions point out crucial barriers to the effective use of machine learning-based myoelectric control systems in upper limb exoskeletons and exosuits which warrant further investigations.
4.1 Robustness of machine learning-based myoelectric control systems
The myoelectric control system’s ability to withstand disturbance from both internal and external sources within the environment, as measured by its resistance to electromyography signals [56], is referred to as its robustness. This type of disturbance is typically caused by muscle fatigue [57], electrode displacement [58], and changes in EMG patterns over time [59]. Over the past decade, there has been a significant increase in studies employing machine learning-based myoelectric control systems, which have shown promising results in preliminary or pilot testing in laboratory settings. However, none of these systems have explored methods to enhance their robustness. To bridge the gap between experimental research and commercial or clinical applications, machine learning-based myoelectric control systems should concentrate on creating a precise control scheme under well-controlled laboratory conditions while also improving robustness in real-world scenarios.
The review of research articles that utilized machine learning-based myoelectric control systems found that these systems face common issues, such as varying characteristics of sEMG signals in different physiological conditions, noise/artifacts, muscle fatigue that causes variance in sEMG signals, and electrode shift during or between sessions. However, none of the studies focused mainly on addressing these issues. Existing studies have investigated these issues in the context of myoelectric control of prosthetics, teleoperate robotic arms, and pattern recognition of sEMG signals. Potential approaches to improve the robustness of machine learning-based myoelectric control systems include using more efficient features, reducing the impact of EMG electrode shift, and improving the data collection protocol or signal processing method. However, these methods have not been studied in the included research articles. Therefore, further investigations are needed to evaluate the performance of machine learning-based myoelectric control systems with these robustness-improving methods and their performance on the upper limb exoskeleton.
In future studies, it is suggested to investigate the performance of upper limb exoskeletons with machine learning-based myoelectric control systems using different time-domain and frequency-domain features. The selection of EMG features should be expanded to account for larger time-domain and frequency-domain features, and the performance of the human-exoskeleton system with the improved myoelectric control system should be evaluated. Additionally, during laboratory research, the causes of error, such as EMG electrode shift and muscle fatigue that could affect the robustness of machine learning-based myoelectric control systems in clinical applications, should be emulated. Novel training protocols should also be investigated because using the EMG signal collected within a short period to train the machine learning-based myoelectric control system will affect its robustness. Therefore, future studies of machine learning myoelectric control systems of upper limb exoskeletons should focus on developing novel control schemes, investigating effective training protocols, and evaluating them on the upper limb exoskeletons. In the research articles reviewed, there were several common issues that were reported. These issues included differences in the characteristics of EMG signals across various physiological conditions, the presence of noise and artifacts, muscle fatigue leading to variations in EMG signals, and electrode movement during or between sessions. However, none of the included research articles specifically addressed these issues by focusing on improving the robustness of machine learning-based myoelectric control systems.
Furthermore, one of our studies explored the implementation of a variational autoencoder to improve the robustness of the classification model in using the EMG signal to recognize the motion performed by the human subject. An autoencoder is a neural network model that is trained to compress and uncompress inputted data while reducing the error between the input data and the reconstructed output data as much as possible [60]. The restrictive architecture of the autoencoder creates a model that can act as a dimensionality reduction method to perform unsupervised feature learning. Implementing autoencoder networks or more advanced encoder-decoder networks can further reduce the complexity of input myoelectric signal data or multimodal sensor data at the compressed latent layer while learning the hidden characteristics that define the system. Autoencoder networks can effectively denoise incoming EMG signal data [61], and the encoder-decoder model framework can be reused using a transfer learning-based model approach [56]. Once the model is trained offline using collected experimental data, the myoelectric control scheme can be readily implemented with little calibration time for the end user. Autoencoder models have already been implemented in research to improve the pattern recognition of myoelectric control schemes in the presence of electrode shift [62], but more research is needed to test the viability of using encoder-decoder networks in myoelectric control schemes.
4.2 Safety requirements in machine learning based myoelectric control systems
The active and powered upper limb exoskeletons and exosuits require high levels of safety to ensure that they do not pose any risks to human users for assistive or rehabilitative purposes. Previous research has primarily focused on incorporating safety measures in the mechanical design of exoskeletons by implementing mechanical stops, rotation limits, and force limits to prevent any excessive range of motion or force from being applied to the user [3]. However, these mechanisms may not always guarantee the user’s safety when there are unknown parameter variances, hardware failures, or actuator malfunctions [63]. Therefore, control strategies that can compensate for various uncertainties and external load disturbances may significantly enhance user safety when wearing the robotic exoskeleton during tasks and movements. According to state-of-the-art research articles, one potential approach to improve safety is to apply data fusion techniques to EMG signals, considering their inherent variability arising from changes in arm posture, electrode repositing, fatigue, etc. [64]. By fusing EMG data, potential errors in motion estimation can be minimized. In [64], two data-fusing algorithms, Variance Weighted Average (VWA) and Decentralized Kalman Filter, were presented as potential methods to improve safety in robotic exoskeletons.
Additionally, other works also utilize the deep reinforcement learning-controlled neuromusculoskeletal simulator (NMMS) to validate the machine learning-based myoelectric control system. The neuromusculoskeletal simulation modulates a wide range of control schemes and parameters to test the efficacy and performance of different control methods while observing model outcomes, such as muscle force, joint kinematics and power using EMG signals [65]. Compared to the conventional control schemes of the NMMS which are broadly classified as forward-type or inverse-type. The deep reinforcement learning (DRL) based NMMS controller learns the neuromusculoskeletal system dynamics by interacting with its environment without the experimental data collected from big samples with varying anthropometrics and biomechanics characteristics. For example, [66] implemented a lower limb NMMS with DRL-based locomotion controller to validate a reinforcement learning-based myoelectric control system for a lower limb orthosis. In this work, a deep reinforcement algorithm called Soft-Actor-Critic (SAC) was used to learn the dynamics of the lower-limb NMMS and served as its locomotion controller; meanwhile, a myoelectric control scheme was trained by imitation learning through interacting with the lower-limb NMMS. Compared to validating the novel myoelectric control scheme on the human body, using the RL-based NMMS can guarantee safety while maximumly emulating the feedback from real human. However, the abovementioned research only deals with the lower-limb neuromusculoskeletal simulator, while the reinforcement learning-based neuromusculoskeletal simulator for upper limb is still extensively unexplored. To address this gap, one of our recent studies utilized the MyoSuite – a Mujoco-based neuromusculoskeletal simulation kit [67] – to simulate the flexion/extension of human’s elbow joint controlled by the deep deterministic policy gradient algorithms (DDPG) – a variant of deep reinforcement learning algorithms [68]. In that work, we compared the performance between two types of action spaces – the PD-based internal model of the central neuron system, and the direct muscular activation output. The result indicated the PD-based internal model has better learning performance than the direct muscular activation output. Additionally, we also simulated the proportional myoelectric control [17] in the NMMS to validate its feasibility in validating the myoelectric control system. However, the result of muscle activation is different from the result in [17]. Therefore, further studies should focus on making the NMMS become more human-like.
4.3 Implementation of reinforcement learning algorithms
Only 3 percent of the research articles from our survey utilized the reinforcement learning algorithms for the machine learning-based myoelectric control system (Figure 3). However, as a branch of machine learning, reinforcement learning has some exclusive advantages if implemented in a control system. For example, reinforcement learning can inherently reflect how humans learn a skill in the real world, which is actively exploring the unknown environment and finding the long-term optimal solutions [69]. More importantly, the reinforcement learning algorithm can learn the optimal solution without the predefined knowledge about the dynamics of environment [70]. Due to these advantages, an increasing number of biomechanical studies implemented reinforcement learning, such as using reinforcement learning to control a lower-limb musculoskeletal model for obstacle avoidance [71], to control a functional electrical stimulation to assist movement [72], and to control an upper limb prothesis [73]. One literature implemented Probabilistic Inference for Learning Control Algorithm (PILCO) – a reinforcement learning algorithm to train a myoelectric control system – on an elbow exoskeleton and achieved a satisfactory result [54]. However, the PILCO algorithm depends on the dynamic model of the environment which increases the difficulty of computation and restricts the usage of trained myoelectric controllers in a single task. Different from [54], the reinforcement learning algorithm used in [72, 73] does not require the dynamic model of the environment, which called model-free reinforcement learning algorithms. The model-free reinforcement learning algorithms have the advantage of less computational difficulty, and wider applicability. There are many model-free reinforcement learning algorithms, for example, the Asynchronous Advantage Actor-Critic (A3C) algorithm [74] which was used to estimate the elbow joint torque from the surface EMG signal [75]. To explore the application of reinforcement learning algorithms in myoelectric control systems, further research is needed to validate these reinforcement learning algorithms (e.g., deep deterministic policy gradient (DDPG), proximal policy optimization (PPO), and asynchronous advantage actor critic (A3C)) used in the field of myoelectric control systems for upper limb wearable robots.
5. Conclusion
This chapter shares a review of the recent implementation of machine learning algorithms in the myoelectric control systems for upper limb exoskeletons and exosuits. The types of machine learning algorithms used in the myoelectric control systems include classification model, regression model and reinforcement learning model. Also, this chapter provides information on the methods, performance, and limitations of each myoelectric control modality. The machine learning algorithms in the myoelectric control systems have shown promising outcomes, including improved human-robot interactions, robot intelligence, and adaptiveness to the user, task and environment compared to traditional myoelectric control systems that did not use learning-based controls. Several challenges and limitations are identified which need to be addressed in future studies related to machine learning -based myoelectric control system for upper limb exoskeletons and exosuits, particularly in narrowing the gap between laboratory studies and clinical applications.
Acknowledgments
Article processing charges were provided in part by the UCF College of Graduate Studies Open Access Publishing Fund.
References
- 1.
Bardi E, Gandolla M, Braghin F, Resta F, Pedrocchi A, Ambrosini E. Upper limb soft robotic wearable devices: A systematic review. Journal of NeuroEngineering And Rehabilitation. 2022; 19 :87 - 2.
Fu J, Hosseini S, Simpson R, Brooks A, Huff R, Park J. A bilateral six degree of freedom cable-driven upper body exosuit. In: 2022 IEEE International Conference on Mechatronics and Automation (ICMA). Guilin, Guangxi, China: IEEE; 2022. pp. 617-623 - 3.
Gull M, Bai S, Bak T. A review on design of upper limb exoskeletons. Robotics. 2020; 9 :16 - 4.
Ding S, Ouyang X, Liu T, Li Z, Yang H. Gait event detection of a lower extremity exoskeleton robot by an intelligent IMU. IEEE Sensors Journal. 2018; 18 :9728-9735 - 5.
Wang S, Wang L, Meijneke C, Van Asseldonk E, Hoellinger T, Cheron G, et al. Design and control of the MINDWALKER exoskeleton. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2014; 23 :277-286 - 6.
Massardi S, Rodriguez-Cianca D, Pinto-Fernandez D, Moreno J, Lancini M, Torricelli D. Characterization and evaluation of human–exoskeleton interaction dynamics: A review. Sensors. 2022; 22 :3993 - 7.
Kim J, Shim M, Ahn D, Son B, Kim S, Kim D, et al. Design of a knee exoskeleton using foot pressure and knee torque sensors. International Journal of Advanced Robotic Systems. 2015; 12 :112 - 8.
Fu J, Choudhury R, Hosseini S, Simpson R, Park J. Myoelectric control Systems for Upper Limb Wearable Robotic Exoskeletons and Exosuits—A Systematic Review. Sensors. 2022; 22 :8134 - 9.
Dupont A, Morin E. A myoelectric control evaluation and trainer system. IEEE Transactions on Rehabilitation Engineering. 1994; 2 :100-107 - 10.
Englehart K, Hudgin B, Parker P. A wavelet-based continuous classification scheme for multifunction myoelectric control. IEEE Transactions on Biomedical Engineering. 2001; 48 :302-311 - 11.
Englehart K, Hudgins B. A robust, real-time control scheme for multifunction myoelectric control. IEEE Transactions on Biomedical Engineering. 2003; 50 :848-854 - 12.
Robertson D, Caldwell G, Hamill J, Kamen G, Whittlesey S. Research Methods in Biomechanics. Champaign, IL, USA: Human Kinetics; 2014 - 13.
Bi L, Feleke A, Guan C. A review on EMG-based motor intention prediction of continuous human upper limb motion for human-robot collaboration. Biomedical Signal Processing and Control. 2019; 51 :113-127 - 14.
Rechy-Ramirez E, Hu H. Bio-signal based control in assistive robots: A survey. Digital Communications and Networks. 2015; 1 :85-101 - 15.
Lotti N, Xiloyannis M, Missiroli F, Chiaradia D, Frisoli A, Sanguineti V, et al. Intention-detection strategies for upper limb exosuits: Model-based myoelectric vs dynamic-based control. In: 2020 8th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob). New York, NY, USA: IEEE; 2020. pp. 410-415 - 16.
Accogli A, Grazi L, Crea S, Panarese A, Carpaneto J, Vitiello N, et al. EMG-based detection of User’s intentions for human-machine shared control of an assistive upper-limb exoskeleton. In: Wearable Robotics: Challenges and Trends. Gewerbestrasse, Switzerland: Springer, Cham; 2016. pp. 181-185 - 17.
Lenzi T, Rossi S, Vitiello N, Carrozza M. Intention-based EMG control for powered exoskeletons. IEEE Transactions on Bio-medical Engineering. 2012; 59 :2180-2190 - 18.
Roche A, Rehbaum H, Farina D, Aszmann O. Prosthetic myoelectric control strategies: A clinical perspective. Current Surgery Reports. 2014; 2 :1-11 - 19.
Bayat N, Rastegari E, Li Q. Human Gait Recognition Using Bag of Words Feature-Representation Method. In: Ahram T, Falcão C, editors, Human Factors and Wearable Technologies. AHFE International Conference. Vol. 29. USA: AHFE Open Access, AHFE International; 2022. DOI: 10.54941/ahfe1001481 - 20.
Mesin L, Merletti R, Rainoldi A. Surface EMG: The issue of electrode location. Journal of Electromyography and Kinesiology. 2009; 19 :719-726 - 21.
Merletti R, Farina D. Analysis of intramuscular electromyogram signals. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 2008; 367 :357-368 - 22.
Li G, Li Y, Zhang Z, Geng Y, Zhou R. Selection of sampling rate for EMG pattern recognition based prosthesis control. In: 2010 Annual International Conference of the IEEE Engineering In Medicine and Biology. Buenos Aires, Argentina: IEEE; 2010. pp. 5058-5061 - 23.
Li G, Li Y, Yu L, Geng Y. Conditioning and sampling issues of EMG signals in motion recognition of multifunctional myoelectric prostheses. Annals of Biomedical Engineering. 2011; 39 :1779-1787 - 24.
De Luca C, Donald Gilmore L, Kuznetsov M, Roy S. Filtering the surface EMG signal: Movement artifact and baseline noise contamination. Journal of Biomechanics. 2010; 43 :1573-1579 - 25.
Oskoei M, Hu H. Myoelectric control systems—A survey. Biomedical Signal Processing and Control. 2007; 2 :275-294 - 26.
Phinyomark A, Phukpattaranont P, Limsakul C. Feature reduction and selection for EMG signal classification. Expert Systems with Applications. 2012; 39 :7420-7431 - 27.
Lashgari E, Maoz U. Dimensionality reduction for classification of object weight from electromyography. PLoS One. 2021; 16 :e0255926 - 28.
Oskoei M, Hu H. Support vector machine-based classification scheme for myoelectric control applied to upper limb. IEEE Transactions on Biomedical Engineering. 2008; 55 :1956-1965 - 29.
Scheme E, Hudgins B, Englehart K. Confidence-based rejection for improved pattern recognition myoelectric control. IEEE Transactions on Biomedical Engineering. 2013; 60 :1563-1570 - 30.
Scheme E, Englehart K, Hudgins B. Selective classification for improved robustness of myoelectric control under nonideal conditions. IEEE Transactions on Biomedical Engineering. 2011; 58 :1698-1705 - 31.
Jana M, Barua B, Hazarika S. Design and development of a finger exoskeleton for motor rehabilitation using electromyography signals. In: 2019 23rd International Conference on Mechatronics Technology (ICMT). Salerno, Italy: IEEE; 2019. pp. 1-6 - 32.
Arteaga M, Castiblanco J, Mondragon I, Colorado J, Alvarado-Rojas C. EMG-based adaptive trajectory generation for an exoskeleton model during hand rehabilitation exercises. In: 2020 8th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob). New York, NY, USA: IEEE; 2020. pp. 416-421 - 33.
Cheon S, Kim D, Kim S, Kang B, Lee J, Gong H, et al. Single EMG sensor-driven robotic glove control for reliable augmentation of power grasping. IEEE Transactions on Medical Robotics and Bionics. 2021; 3 :179-189 - 34.
Su H, Li Z, Li G, Yang C. EMG-based neural network control of an upper-limb power-assist exoskeleton robot. In: Advances In Neural Networks–ISNN 2013: 10th International Symposium on Neural Networks, Dalian, China, July 4–6, 2013, Proceedings, Part II 10. Dalian, China: Springer, Berlin, Heidelberg; 2013. pp. 204-211 - 35.
Trigili E, Grazi L, Crea S, Accogli A, Carpaneto J, Micera S, et al. Detection of movement onset using EMG signals for upper-limb exoskeletons in reaching tasks. Journal of Neuroengineering and Rehabilitation. 2019; 16 :45 - 36.
Kirchner E, Tabie M, Seeland A. Multimodal movement prediction – towards an individual assistance of patients. PLoS One. 2014; 9 :e85060 - 37.
Wu Q, Chen B, Wu H. Neural-network-enhanced torque estimation control of a soft wearable exoskeleton for elbow assistance. Mechatronics. 2019; 63 :102279 - 38.
Zeng H, Li K, Wei N, Song R, Tian X. A sEMG-controlled robotic hand exoskeleton for rehabilitation in post-stroke individuals. In: 2018 IEEE International Conference on Cyborg and Bionic Systems (CBS). Shenzhen, China: IEEE; 2018. pp. 652-655 - 39.
Twardowski M, Roy S, Li Z, Contessa P, De Luca G, Kline J. Motor unit drive: A neural interface for real-time upper limb prosthetic control. Journal of Neural Engineering. 2019; 16 :3-5 - 40.
Secciani N, Topini A, Ridolfi A, Meli E, Allotta B. A novel point-in-polygon-based sEMG classifier for hand exoskeleton systems. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2020; 28 :3158-3166 - 41.
Godoy R, Dwivedi A, Liarokapis M. Electromyography based decoding of dexterous, in-hand manipulation motions with temporal multichannel vision transformers. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2022; 30 :2207-2216 - 42.
Tang Z, Zhang K, Sun S, Gao Z, Zhang L, Yang Z. An upper-limb power-assist exoskeleton using proportional myoelectric control. Sensors. 2014; 14 :6677-6694 - 43.
Pang M, Guo S, Song Z. Study on the sEMG driven upper limb exoskeleton rehabilitation device in bilateral rehabilitation. Journal of Robotics and Mechatronics. 2012; 24 :585 - 44.
Loconsole C, Leonardis D, Barsotti M, Solazzi M, Frisoli A, Bergamasco M, et al. An emg-based robotic hand exoskeleton for bilateral training of grasp. In: 2013 World Haptics Conference (WHC). Daejeon, South Korea: IEEE; 2013. pp. 537-542 - 45.
Ngeo J, Tamei T, Shibata T, Orlando M, Behera L, Saxena A, et al. Control of an optimal finger exoskeleton based on continuous joint angle estimation from EMG signals. In: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Osaka, Japan: IEEE; pp. 338-341 - 46.
Leonardis D, Barsotti M, Loconsole C, Solazzi M, Troncossi M, Mazzotti C, et al. An EMG-controlled robotic hand exoskeleton for bilateral rehabilitation. IEEE Transactions on Haptics. 2015; 8 :140-151 - 47.
Burns M, Pei D, Vinjamuri R. Myoelectric control of a soft hand exoskeleton using kinematic synergies. IEEE Transactions on Biomedical Circuits and Systems. 2019; 13 :1351-1361 - 48.
Cisnal A, Pérez-Turiel J, Fraile J, Sierra D, Fuente E. RobHand: A hand exoskeleton with real-time EMG-driven embedded control. Quantifying hand gesture recognition delays for bilateral rehabilitation. IEEE Access. 2021; 9 :137809-137823 - 49.
Li Z, Wang B, Sun F, Yang C, Xie Q, Zhang W. sEMG-based joint force control for an upper-limb power-assist exoskeleton robot. IEEE Journal of Biomedical and Health Informatics. 2014; 18 :1043-1050 - 50.
Lu L, Wu Q, Chen X, Shao Z, Chen B, Wu H. Development of a sEMG-based torque estimation control strategy for a soft elbow exoskeleton. Robotics and Autonomous Systems. 2019; 111 :88-98 - 51.
Kopke J, Hargrove L, Ellis M. Applying LDA-based pattern recognition to predict isometric shoulder and elbow torque generation in individuals with chronic stroke with moderate to severe motor impairment. Journal of Neuroengineering and Rehabilitation. 2019; 16 :4-6 - 52.
Sierotowicz M, Lotti N, Nell L, Missiroli F, Alicea R, Zhang X, et al. EMG-driven machine learning control of a soft glove for grasping assistance and rehabilitation. IEEE Robotics and Automation Letters. 2022; 7 :1566-1573 - 53.
Meattini R, Bernardini A, Palli G, Melchiorri C. sEMG-based minimally supervised regression using soft-DTW neural networks for robot hand grasping control. IEEE Robotics and Automation Letters. 2022; 7 :10144-10151 - 54.
Hamaya M, Matsubara T, Noda T, Teramae T, Morimoto J. Learning assistive strategies for exoskeleton robots from user-robot physical interaction. Pattern Recognition Letters. 2017; 99 :67-76 - 55.
Deisenroth M, Rasmussen C. PILCO: A model-based and data-efficient approach to policy search. In: Proceedings of the 28th International Conference on Machine Learning (ICML-11). 2011. pp. 465-472 - 56.
Phinyomark A, Quaine F, Charbonnier S, Serviere C, Tarpin-Bernard F, Laurillau Y. EMG feature evaluation for improving myoelectric pattern recognition robustness. Expert Systems with Applications. 2013; 40 :4832-4840 - 57.
Geng Y, Samuel O, Wei Y, Li G. Improving the robustness of real-time myoelectric pattern recognition against arm position changes in Transradial amputees. BioMed Research International. 2017; 2017 :5090454 - 58.
Tkach D, Huang H, Kuiken T. Study of stability of time-domain features for electromyographic pattern recognition. Journal of NeuroEngineering and Rehabilitation. 2010; 7 :21 - 59.
Boschmann A, Kaufmann P, Platzner M, Winkler M. Towards multi-movement hand prostheses: Combining adaptive classification with high precision sockets. In: Proceedings of the 2nd European Conference Technically Assisted Rehabilitation. 2009 - 60.
Bank D, Koenigstein N, Giryes R. Autoencoders. CoRR. 2020. DOI: abs/2003.05991 - 61.
Teh Y, Hargrove L. Using latent representations of muscle activation patterns to mitigate myoelectric interface noise. In: 2021 10th International IEEE/EMBS Conference On Neural Engineering (NER). pp. 1148-1151 - 62.
Lv B, Sheng X, Zhu X. Improving myoelectric pattern recognition robustness to electrode shift by autoencoder. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). pp. 5652-5655 - 63.
Kang H, Wang J. Adaptive control of 5 DOF upper-limb exoskeleton robot with improved safety. ISA Transactions. 2013; 52 :844-852 - 64.
López N, Sciascio F, Soria C, Valentinuzzi M. Robust EMG sensing system based on data fusion for myoelectric control of a robotic arm. BioMedical Engineering OnLine. 2009; 8 :5 - 65.
Zhuang Y, Yao S, Ma C, Song R. Admittance control based on EMG-driven musculoskeletal model improves the human–robot synchronization. IEEE Transactions on Industrial Informatics. 2019; 15 :1211-1218 - 66.
Han J, Lee J, Choi H, Kim J, Choi J. Policy Design for an Ankle-Foot Orthosis Using Simulated Physical Human–Robot Interaction via deep reinforcement learning. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2022; 30 :2186-2197 - 67.
Caggiano V, Wang H, Durandau G, Sartori M, Kumar V. MyoSuite–A contact-rich simulation suite for musculoskeletal motor control. ArXiv Preprint ArXiv:2205.13600. 2022 - 68.
Fu J, Choudhury R, Park J. Deep reinforcement learning based upper limb neuromusculoskeletal simulator for modelling human motor control. IEEE. 2023, Under Review - 69.
Mnih V, Kavukcuoglu K, Silver D, Rusu A, Veness J, Bellemare M, et al. Human-level control through deep reinforcement learning. Nature. 2015; 518 :529-533 - 70.
Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, et al. Mastering the game of go without human knowledge. Nature. 2017; 550 :354-359 - 71.
Kidziński Ł, Mohanty S, Ong C, Huang Z, Zhou S, Pechenko A, et al. Learning to run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments. In: The NIPS ‘17 Competition: Building Intelligent Systems. Gewerbestrasse, Switzerland: Springer, Cham; 2018. pp. 121-153 - 72.
Wannawas N, Subramanian M, Faisal A. Neuromechanics-Based Deep Reinforcement Learning of Neurostimulation Control in FES Cycling. Italy: IEEE; 2021 - 73.
Pilarski P, Dawson M, Degris T, Fahimi F, Carey J, Sutton R. Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning. In: 2011 IEEE International Conference on Rehabilitation Robotics. 2011. pp. 1-7 - 74.
Mnih V, Badia A, Mirza M, Graves A, Lillicrap T, Harley T, et al. Asynchronous methods for deep reinforcement learning. In: International Conference on Machine Learning. New York, NY, USA: JMLR; 2016. pp. 1928-1937 - 75.
Wu W, Saul K, Huang H. Using reinforcement learning to estimate human joint moments from electromyography or joint kinematics: An alternative solution to musculoskeletal-based biomechanics. Journal of Biomechanical Engineering. 2021; 143 :2-3