Learning how to move is a challenging task. Even the most basic motor skill of walking requires years to develop and can quickly deteriorate due to aging and sedentary lifestyles. More specialized skills such as ballet and acrobatic kicks in soccer require “talent” and years of extensive practice to fully master. These practices can easily cause injuries if conducted improperly. 3D motion capture technologies are currently the best way to acquire human motor skill in biomechanical feedback training. Owing to their tremendous promise for a plethora of applications, wearable technologies have garnered great interest in biofeedback training. Using wearable technology, some physical activity parameters can be tracked in real time and a noninvasive way to indicate the physical progress of a trainee. Yet, the application of biomechanical wearables in human motor-skill learning, training, and optimization is still in its infant phase due to the absence of a reliable method. This chapter elaborates challenges faced by developing wearable biomechanical feedback devices and forecasts potential breakthroughs in this area. The overarching goal is to foster interdisciplinary studies on wearable technology to improve how we move.
- 3D motion capture technology
- body model
- real time
- feedback training
For decades, it has been known that the large and widespread anthropometrical diversity limits the effectiveness of a universal approach in human motor-skill learning and training; instead, an individualized biofeedback approach would significantly improve the learning process [1, 2, 3, 4]. Recently, wearable sensors (wearables) have garnered great interest in biofeedback training, owing to their tremendous promise for a plethora of applications [5, 6, 7, 8]. It seems that individualized biofeedback training has the potential to become an immediate reality in the motor learning realm. However, the absence of a reliable method of applying wearables in biomechanical feedback training has greatly hindered their application in human motor-skill learning and optimization .
Although wearables in sports are only a few years old, there has already been a consensus that wearable technology is leading a revolution in physical training [5, 8]. Various sensors are now fitted into sport equipment, limbs, wristbands, and/or clothes to collect crucial data in real time, sending it directly to trainers, allowing them to implement an individualized training plan for increasing athletic competence. Nevertheless, the use of real-time biomechanical feedback in training looks currently not so optimistic. A recent review paper (2019) divulges that the biomechanical development is still in its infancy . The paper reveals that while there are over 5500 published biofeedback articles in Web of Science, there are very few on real-time biomechanical feedback learning or training. Compared to the booming application of wearables in fitness as well as in health industry, the biomechanical investigations seem disproportionately low. The scarcity of biomechanical studies may due to two facts: (1) a general biomechanical body model that is suitable for wearable application in feedback learning and training is missing, and (2) a reliable method for linking biomechanical quantification and human motor learning in real time is still not available .
Clearly, the current success of wearables in sports is not yet linked to the human motor-skill learning. The overwhelming use of wearables in sports is mainly in the area of monitoring physical condition. For example, sports injuries are often caused by fatigue, overtraining, or dehydration [9, 10]. Wearables are now able to collect data related to the risk conditions from athletes’ physical conditions, muscle activities, and sweat [5, 6, 7]. The real-time biofeedback can help coaches to quickly alternate their training or competition strategies in order to decrease injury risk in training and competition [5, 6, 9]. One should note that the locomotion (e.g., distance, speed), physiological (e.g., heart rate, blood pressure), neurological (e.g., muscle activities), and biochemical feedback (e.g., electrolytes, metabolites) are only useful in analyzing the general physical condition of an athlete; however, they do not provide information related to the limbs’ control of human motor skills, and as such, the biomechanical feedback for motor control is still missing.
2. The uniqueness and challenges of developing biomechanical feedback
Why is the development of biomechanical feedback understudied? This is because of the uniqueness of biomechanical feedback. Feedbacks obtained from locomotion, physiological, biochemical, and neurological measurements deliver information of one’s general changes in speed/location, physiological and physical response, and muscle tension. The common point of these feedbacks is that they can be conserved across human motor skills, i.e., across different movement forms. Therefore, one can universally apply the feedback devices monitoring these parameters of all activities [4, 11]. On the contrary, biomechanical feedback mainly provides information related to the limb control of motor skills, which often differ from one skill to the other. To complicate matters further, skill optimization has to be adjusted depending on one’s anthropometry [4, 12, 13, 14]. In short, biomechanical feedback must be tailored to an individual activity being examined [15, 16, 17, 18].
Ergo, in order to develop a universal biomechanical feedback device, one has first to obtain a thorough understanding of a variety of motor skills in order to determine the general key parameters for monitoring [1, 4]. Further, biofeedback devices (e.g., wearables) must not interfere with the motor skill being executed. This technical limitation alone has proven to be a major hindrance to the development of biomechanical feedback in motor learning and training. Finally, a vital step in device development is to search ways/body models, which should consider the anthropometry-induced motor-control variations.
In short, there are three indispensable linchpin pieces in the development process: (1) expert knowledge obtained from extensive motion analyses of diverse human motor skills, (2) sensibleness of wearables’ application in training environment, and (3) a general method for wearable-based data analysis and interpretation.
Summarized above, there are several challenges that must be overcome during the development of the universal real-time biomechanical feedback. The obvious are:
Creating a new generalizable body model that can quantify various human motor skills
Minimizing wearable interference with the motor skill being executed
Developing wearable-based data analysis and interpretation method
Adding the anthropometrical variation into motor-control identification
3. Biomechanical steps in developing wearables for feedback training
Effective human motor-skill learning can be supported by useful and timely biomechanical feedback to learners, helping them to target at their performance defects. Previous studies have shown that regular, objective, and consistent performance monitoring and assessment through quantitative analysis of biomechanical variables can reinforce the biomechanical feedback training in practice [17, 19]. Therefore, how to increase the spatial and temporal accuracy when performing a quantification of a motor skill (i.e., the limbs coordination) would play a crucial role in developing wearables for biomechanical feedback training . Considering the uniqueness of biomechanical feedback illustrated in the previous sections, the following steps have to be undergone in developing wearables for feedback training:
Choose a motor skill.
Perform motion analysis of the skill quantitatively.
Identify dominate parameters for feedback training.
Verify the effectiveness of the selected feedback(s) in practice.
Develop a feedback device for monitoring of the critical/vital parameter(s) (e.g., coordination among certain segments or joints) for the given motor skill.
One should note that wearables developed through the current approach can only be applied to one specific motor skill. A delimitation of application in learning/training other motor skills is impossible.
Having seen the success of physiological, neurological, and biochemical wearables in practice, it would be a practitioner’s desire that the biomechanical one could also be universally applied to all motor skills for their learning and training in sports and arts. One should note that a general application means that a general methodology should exist for motor-control data collection and interpretation, i.e., a wearable system should be able to track a variety of human motor skills and to identify the motor-control patterns existing in these motor skills. Unfortunately, we are currently still far away from the goal. All existing studies are specific or isolated ones. So far, only a few studies explored the real-time biomechanical feedback application in practice [21, 22, 23].
4. A novel route in developing wearables for human motor-skill learning and biomechanical feedback training
Currently, the most reliable methodology for quantifying complicated human motor skills is the full-body biomechanical modeling based on 3D motion capture [24, 25, 26, 27]. The biomechanical body model consists of 15 segments. The 15 segments are the head, upper trunk, lower trunk, upper arms, lower arms, hands, thighs, shanks, and feet. For establishing the model, about 40 body-surface markers are needed to supply 3D coordinate inputs for mechanically determining joint kinematics in order to reveal the motor-control/limbs’ coordination. The model is widely applied in the current 3D motion capture technologies for demystifying and optimizing complex motor skills in sports and arts performance [4, 15, 17, 28, 29, 30, 31, 32, 33, 34, 35, 36]. This video-based technology uses multi-cameras to track ~40 reflective markers (their weights are negligible) attached on the body surface. Technically, the tracking can be equivalently done by using 40 wearable IMUs, a sensing technology that measures linear and angular motion with a triad of gyroscopes and triad of accelerometers [21, 37, 38, 39, 40]. As such, motion analysis could switch from labs (multi-camera environment) to the field (wearables) , i.e., quantification of limbs’ coordination would be no more restricted to labs and become an effortless daily routine for researchers and practitioners. Practically, it is not so easy.
It is wise to induce the success of the current biomechanical body model into wearable applications. Nevertheless, it is unrealistic and impractical to use ~40 IMUs for rebuilding the body and its movement. Markers used in motion capture are small (9 mm in diameter) and almost weightless [41, 42, 43], whereas the volume and weight of current IMUs are still significantly larger. The ~40 IMUs can cause unknown experimental artifacts. Obviously, we have to search a new route. Based on the current development [4, 20, 21, 44], a novel approach for a potentially successful transition to wearable applications will be introduced here.
The innovative approach will be built on previous studies on anthropometry [12, 13, 45], 3D motion analysis [25, 46, 47, 48, 49], sensing technology [21, 37, 38, 39, 40, 50], and artificial intelligence (AI) [44, 50, 51, 52, 53, 54]. This multidisciplinary approach provides a new route to develop a wearable-based method for data analysis and interpretation (motor-control depiction) as well as to distill and package new findings from various areas in realizing the real-time biomechanical feedback training in practice.
4.1 A two-chain model as a general full-body biomechanical model for realizing wearable application in human motor-skill training
In 3D motion capture, the rebuilding of an individual 15-segment model is achieved by tracking selected body-surface markers in 3D space. Alternatively, an equivalent model can be built via anthropometrical approach. Previous studies show that, using variables such as body weight (BW), body height (BH), gender, and race, one can statistically determine segmental masses and lengths to build an individual body model [12, 45]. The difference of the two approaches is that the former is “born-to-move” (video-based) and the latter has to “learn-to-move” (wearable-based). Currently, the most challenging for wearable application in sports and arts performances is practicality. Wearables attached to human body will create certain constraints for human movement and alternate the movement control in a way that may not reach the training goal. Therefore, the less wearables applied, the more practical the feedback system is. Aiming at the development of practicality, future researches/developments should focus on innovative designs of the body model that will minimize the number of wearables required yet still supply equivalent, if not better, accuracy of the current 15-segment biomechanical model. Such a novel body model is introduced below for potential breakthroughs in the future.
Based on numerous previous 3D motion studies [15, 28, 29, 30, 31, 37, 55, 56, 57, 58, 59, 60, 61, 62, 63], multitudinous complicated human movements from both sports activities and arts performances can be generally represented by using a model system with two mechanical chains: upper-body chain and lower-body chain (Figure 1). The upper-body chains consist of a base (i.e., upper trunk and head) and two sub-chains (i.e., arms) that are linked to the base. The lower-body chain has an equivalent structure, the base is the lower trunk, and the two sub-chains are the legs. With this novel design, human motor skills could be tracked by using much fewer IMUs. Theoretically, three IMUs on each chain (one on the base, one on each distal end of the chain) would likely track the movement of a chain (Figure 1a), i.e., six IMUs would be able to determine the segments’/joints’ motion and coordination as well as the orientation relationship between the two bases of the two chains. As such, researchers and practitioners could quasi-naturally track human motor skills (i.e., six wearables would minimally encumber human motor control) in learning and training. This would help the development of the real-time biomechanical feedback training. Only the six IMUs’ inputs are not sufficient to apply the class mechanics to quantitatively determine the model system. Currently, AI could alleviate this challenge due to its “learning” ability [44, 64, 65, 66].
4.2 AI for motor-control quantification
Since the inputs of the limited IMUs cannot mechanically determine the two-chain model, AI technologies are the alternative ways for the two-chain model quantification. Studies have shown that AI techniques have become a powerful tool for helping to solve many challenging problems in human motor-skill evaluations and analyses [44, 52, 53, 67].
The basic idea of AI prediction is to find a way to learn general features of existing data in order to make sense of new data [64, 65]. This description highlights the central role of data for establishing implicit knowledge. The amount of data must be sufficiently large to provide many training examples from which a large set of parameters can be extracted. In the past decades, AI techniques have experienced a resurgence following concurrent advances in computer power, large amounts of data (big data), and theoretical understanding.
Among the AI technologies, deep learning is considered as a powerful tool that percolates through to all application areas of AI, such as image identification, speech recognition, natural language processing, and, indeed, biofeedback support [68, 69, 70]. The success of deep learning networks encourages their implementation in further applications for the enhancement of human physical activities [52, 54, 67]. Recently, Nature Neuroscience has published the latest developments in the area of markerless, video-based motion tracking, indicating that the power of deep learning will enable motion tracking to human-like accuracy . This study confirms that motion capture/quantification of limbs’ coordination will move from an expensive and difficult task restricted to the laboratory to an effortless daily routine for researchers and practitioners.
From motor learning point of view, wearables would have much higher potential than video shooting in the future practice. This is not only because of the fast advance in miniature of wearables but also due to three inherited drawbacks of video-shooting approach, i.e., (1) the limited capture space, (2) the complexity of capture systems (from setup, calibration, to operation), and (3) the time-consuming nature of data processing (high cost of data processing). Reliable biomechanical feedback should be obtained from accurate quantification of human movement in field, with some requiring large space. Even with a multi-camera setting, unexpected environmental factors (e.g., interactions among athletes) will create a data gap. Further, it is true that we are already sitting on massive movement data (e.g., YouTube, Flickr) for training of deep learning models; but the video datasets are uncalibrated and have very little information on the hardware and conditions used to capture particular videos, which can bias the deep learning recognition algorithms . Currently, the availability of reliable motion capture data for developing deep learning models is significantly limited.
In summary, the combination of the two-chain full-body model with six wearable IMUs and the deep learning prediction based on IMUs’ data shows great potential in developing real-time biomechanical feedback training for an efficient human motor-skill learning and optimization. The missing piece for testing the potential is reliable massive training data.
4.3 The key for raising the reliability of wearables: creating a diversification of 3D training big data
Two factors revealed by previous studies strongly influence deep learning performance [65, 66, 72, 73, 74]. One is the massive data, and the other one is the diversity of the massive data. A systematical review article has examined 53 studies published before 2018. The scope of the review is the deep learning applications of the physiological data or signals in healthcare. The article has revealed that both the amount of data and the diversity of data would influence the prediction’s reliability . This result would signify that deep learning algorithms would perform well with large and diverse datasets.
Judging from the current knowledge, a massive and diverse 3D motion capture big data, collected from sports and arts activities, is indispensable for developing the reliable biomechanical wearables. It is well known that sports and arts skills exhibit the most diverse and complicated motor controls among all human physical activities. If such big data are available for training deep learning models, the reliability of the trained model will be raised due to the depth and specialization from training the deep learning algorithms [65, 66, 72, 73, 74]. Therefore, at present, the vital step for developing real-time biomechanical feedback tool is to apply the two-chain model in simultaneous collections of a large amount of motion data, i.e., synchronized measurements using both 3D motion capture with ~40 markers and 6 wearable IMUs . The synchronized data collection should cover large variety of sports skills and arts performances. As such, the 3D motion capture data can be served as a “supervisor” for training deep learning model to map IMU data to joints’ kinematic data. Such a deep learning model could be reliably and universally applied in motor learning and the training of sports and arts skills.
Retrospectively, the current knowledge of anthropometry, biomechanical modeling, and deep learning and the technology for miniaturizing IMUs supply an almost perfect environment for the development of the real-time biomechanical feedback tool in human motor-skill learning and training, especially in learning complicated skills in sports and arts. The missing piece is the massive and diverse motor-skill big data for deep learning.
This chapter highlights the challenges and future of wearable technology in human motor-skill learning and optimization. It introduces a novel two-chain biomechanical body model with six IMUs that are powered with deep learning technology. The framework can serve as a basis for developing real-time biomechanical feedback training in practice. In order to create a universal biomechanical feedback device for learning and training of any human motor skill, the massive and diverse big data of multifarious human motor skills have to be created first. One realistic way for obtaining the big data is through a synchronized measurement from 3D motion capture and IMUs. Evidently, gaining high-quality, full-body motion data across sports and arts performances would currently be the vital step for the real-time biomechanical feedback development.
The realization of the methodological breakthrough will allow us to transform the human motor learning paradigm from a largely subjective art into a precise scientific method. The potentials would be to (1) take scientific monitoring of motor skills from a lab-based environment into the field; (2) simplify a scientific movement quantification, transitioning from using a complicated motion capture system to easily applied wearables; and (3) transfer the vital biomechanical feedback in real time to prevent the worst/movement errors from happening while finding individual compensation/optimization. This methodology is the culmination of research programs in biomechanics, anthropometry, computer sciences, pedagogy, and equipment development. It aims to build innovative technologies for generating new knowledge as well as practical and definitive scientific methods for empowering motor learning. Fulfillment and application of the new wearable-based method in the future will benefit diverse human physical activities, including, but not limited to, motor-skill acquisition in sports, arts performances, health/fitness, and recreational activities.
The author would like to thank the Natural Sciences and Engineering Research Council of Canada (NSERC) for its decades’ financial support for his studies in human motion analysis. This chapter is a crystallization of these NSERC research programs in the author’s lab.
Conflict of interest
The author declares no conflict of interest. The founding sponsors had no role in the collection, analyses, or interpretation of data; in the writing of the manuscript; and in the decision to publish the results.
The publication of this chapter was funded by National Sciences and Engineering Research Council of Canada (NSERC), grant number RGPIN-2014-03648.