Three types of prosthetic hand are currently available: cosmetic, body-powered, and myoelectric (Laschi et al., 2000). Cosmetic prostheses are passive, and designed to look like the natural hand, with solely an aesthetic purpose. Body-powered prostheses are powered and controlled by body movements, generally of the shoulder or of the back. Myoelectric hands are electrically powered and controlled by electromyographic (EMG) signals; i.e., small electric potentials produced by contracting muscles. Myoelectric hands are typically controlled in switched or simple proportional mode, according to the amplitude of the EMG signals (Stein and Walley, 1983; Näder, 1990; Sears and Shaperman, 1991; Bergman et al., 1992; Kyberd and Chappell, 1994). The switched control is the simplest one, as it consists of only two states: on or off. Although much progress has been made in myoelectric hands, their motor functions are still not comparable with those of a natural hand, partly because they have been designed to provide only the most basic functions of a natural hand, such as grasping and holding.
Akazawa’s Lab has developed a myoelectric prosthetic hand (Osaka Hand) that simulates fundamental dynamic properties of the neuromuscular control system of the human hand, mainly the viscoelastic properties of muscles, which depend on their stiffness (Akazawa et al., 1987). This hand can be used by an amputee subject with almost the same subconscious control that he/she had prior amputation (Okuno et al., 1999).
The current design of the Osaka Hand requires the user to be fitted for and to wear a hand-made fiberglass or thermoplastic socket into which the stump is comfortably and tightly inserted. The Osaka Hand is attached to the other tip of the socket via a screw. This socket is expensive to produce and requires weeks to manufacture after measurements are taken (Sears, 1991).
An additional initial problem is that shortly after an amputation atrophy of the remnant muscles occurs, and their EMG signal becomes very weak. As that EMG signals are used to control the prosthesis, users wanting to wear the Osaka Hand (or any other myoelectric hand) must undergo a training phase in which their remnant muscles are strengthened and at the same time they re-learn how to perform fine, detailed muscles contractions, which are needed for a precise control of the prosthetic hand.
In order to solve the two problems mentioned above, we developed a graphic simulator system for the Osaka Hand that eliminates the need of a socket for attachment of that prosthetic hand to the stump and it is also used for physical training of myoelectric patients.
A number of works on prosthesis simulators have been already described, each of them fitting the specific requirements of a given prosthesis. Yamada et al. (1983) employed six different bi-dimensional (2D), fix images appearing on the screen depending on the frequency and amplitude pattern of three EMG signals in order to evaluate their proposed control method for a theoretical prosthetic hand. Daley et al. (1990) developed a simple 2D graphical simulator for operator performance comparison when using different myoelectric control strategies. Abul-Haj and Hogan (1987) performed an emulation with a combination of software and hardware for elbow-prosthesis prototypes evaluation. Perlin et al. (1989) developed a simulation program for their Utah/MIT 16-joint, four-finger Dextrous Hand.
Several works describe simulators operated by shoulder movement; Zahedi and Farahani (1995), for example, used a graphical simulator for a fuzzy EMG classifier; Durfee et al. (1991) created a 2D graphic simulator to evaluate command channels trough which control an upper limb neural prosthesis; and Zafar and Van Doren (2000) employed a video-based simulator for a shoulder-activated neuroprosthesis for spinal cord injured persons. Lin and Huang (1997) made a computer simulation of a robotic hand to test its potential use as a prosthesis.
There are already some commercially available systems such as MyoBoy –from OTTO BOCK HealthCare GmbH (Duderstadt, Germany)- that is a software tool used for the evaluation, selection, training, and documentation of myoelectric patients. MOTION CONTROL Inc. (Utah, USA) has developed an EMG tester and trainer (Myolab II) that is used to locate intact muscle activity and to help patients in strengthening and relaxation tasks.
However, all the abovementioned simulators can be used only with prosthetic hands with a switched or single proportional mode, not for those with a more complex control mode as the one of the Osaka Hand.
The goal of the present work was to develop an upper limb and hand graphic simulator system that solves the abovementioned problems, allowing amputee subjects to try virtually the myoelectric hand without needing the socket, and to perform the physical training required prior to use the real one. This simulator allows us also to easily identify the optimal electrode location for the EMG signals acquisition in each individual. In addition, the simulator allows physicians and related staff to recognize how easily the hand can be controlled and its advantages over other kinds of prosthesis.
The simulator that we have developed consists of a data acquisition system, a mathematical model that simulated the behavior of the Osaka Hand (including its model of the human neuromuscular control system dynamics), and a graphics display device.
2. Materials and methods
2.1. Structure of the Osaka Hand
A general overview of the system is shown in Figure 1. An exceptional feature of the Osaka Hand is that the user can control voluntarily the angle of its fingers and the stiffness of the grip (the resistance that the fingers oppose to change their angle) by the EMGs of flexor and extensor muscles of the wrist (see details in Akazawa et al., 1987).
To obtain such a control, the Osaka Hand mimics the properties of both muscle viscoelasticity and the gain of the stretch reflex (both varying linearly with muscle activity). The dynamics of this neuromuscular control system were determined by analyzing the tension responses of finger muscle to mechanical stretch (Akazawa et al., 1999). The dynamics are quite complex, due to the non-linearity and time delay of the stretch reflex; however, we used a simple model representing the dynamics as a first approximation. Once that model was introduced in the prosthetic hand, it was proved that a sound-limbed subject and an amputee subject were able to accurately control finger angle and stiffness of the prosthetic hand (Okuno et al., 1999).
As shown in Figure 1, for each subject a pair of surface electrodes were put on the flexor carpi radialis (wrist flexor muscle) and another pair on their extensor carpi radialis brevi (wrist extensor muscle) to measure their EMG signal. The measured signal was amplified in differential mode, full-wave rectified, and then smoothed with a low-pass filter to obtain its envelope, the amplitude of which is approximately proportional to the force exerted by the muscle (Basmajian and Deluca, 1985). Therefore, the resultant signal corresponded to the isometric contractile force (torque) of each muscle: being the torque of the flexor muscle, and the torque of the extensor muscle.
From those two calculated torques, the desired finger angle of the end effector (the target angle the user wants to achieve) was calculated as
where is the grip force exerted by the fingers of the Osaka Hand, and was measured by strain gauges (KYOWA DENGYO Co., Ltd. (Yokohama, Japan), model KFG-1N) attached to its thumb, index, and middle fingers. is the transfer function that represents the dynamics of human neuromuscular control system (Akazawa et al., 1987; Okuno et al., 1999), and is given by:
where the time constants were calculated to be and , and the gain K corresponds to the stiffness of the prosthesis fingers, which is not constant, but time-varying as:
in proportion to the contraction level of the extensor-flexor muscles pair. The user can regulate the stiffness of the hand fingers angle by varying the level of contraction of each of those muscles. The stiffness at resting state is 0.1 Nm/rad, and the coefficient is 0.98 rad-1. A software program implementing this model was introduced in the microprocessor that controls the end effector.
The position control system (see Figure 1) consists of a DC motor (MINIMOTOR SA, Croglio, Switzerland, type 2233), its servo controller (Figure 1(c2)), and a one degree-of-freedom end effector with three fingers (Figure 1(c3)). Index and middle fingers are bound between them and are endorsed with an open-close movement with respect to the thumb. This movement is produced by the DC motor, the servo controller of which works to nullify the difference between the commanded angle and the actual motor rotational angle as measured by an optical encoder.
2.2. Composition and operation of the simulator
Figure 2 shows the components of the simulator system, which can be divided into three main sub-systems: data acquisition (EMG and video), processing, and display. Ten light emitter diode (LED) markers and two pairs of surface electrodes are attached to the subject’s upper limb as shown in Figure 2(a) and 2(b). Those LEDs and electrodes provide the inputs for the processing sub-system, which is implemented in the graphic workstation (Figure 2(c)).
Figure 3 shows the block diagram of the simulator, illustrating how the processing system (Figure 3(c)) determines the position of the upper limb from the three-dimensional (3D) location of the markers on the shoulder, elbow, and wrist detected with an OPTOTRAKTM 3D camera (NORTHERN DIGITAL Inc., Ontario, Canada) (Figures 2(a) and 3(a)). The processing system determines the desired finger angle from the processed surface EMGs of both wrist flexor and extensor muscles of the subject (Figures 2(b) and 3(b)). The virtual upper limb and hand are ultimately presented on the 3D graphic workstation (display system, Figure 3(d)).
2.2.1. Data acquisition system
The OPTOTRAK 3D camera (Figure 2(a) and Figure 3(a)) detects the position of the LED markers attached to the user's shoulder, elbow, and wrist. The marker on the shoulder is attached to the point where the movement of the acromion of the scapula is smallest during the motion of the arm. The marker on the elbow is fixed in the external palate of the humeral. The arm posture is calculated from those LEDs locations.
To measure the rotational angle of the wrist during an external pronation of the arm, eight LEDs are placed on the external side of a bracelet-like device attached to the wrist (shown in the inset of Figure 2).
The EMG signals of wrist muscles are picked up with surface electrodes (Figure 2(b) and Figure 3(b)). These signals are then amplified (gain 58.8 dB, CMRR 110 dB) to the range ±5 V, full-wave rectified, smoothed with a second order low-pass filter (cut-off frequency 2.7 Hz), and then sampled at a frequency of 25 Hz, 12 bits per sample (resolution of ±2.4 mV, less than 0.01% of the maximum value) with an OPTOTRAK Data Acquisition Unit (NORTHERN DIGITAL Inc., Ontario, Canada).
2.2.2. Processing system
The location of the LEDs and the processed EMG signals are collected by a graphics workstation (GW, SILICON GRAPHICS, Inc., California, USA) that holds the processing system software (Figure 2(c) and Figure 3(c)).
The angle that the user wants to achieve with the prosthesis fingers (the target angle) is given by Eqs. (2) and (3) using the current value of user’s EMG signals and . Those equations are calculated in the real Osaka Hand by a Z-transform that gives in discrete time their solution, originally expressed in the frequency domain (see Figure 3(c1)). In the case of the processing system of the simulator, the sampling frequency is not high enough to allow using that transform. Therefore, we used the Runge-Kutta-Gill approximation method for differential equations in order to implement the transfer function (Eqs. (2) and (3)).
Dynamics of the DC motor servo system of the actual prosthetic hand were calculated in terms of the relationship between target angle and rotational angle of the motor shaft (see Figure 1). In the steady case, we assumed , with zero time delay (Figure 3(c2)).
In order to model the relationship between and final finger angle of the real Osaka Hand (see Figure 1), we performed the following measurements by attaching two LEDs to the prosthesis chassis and one on each fingertip as shown in Figure 4. The hand finger angle was defined as the angle formed between the vectors and , i.e., the angle between the fingertips with respect to the chassis. The operation range of this angle is from 0o to 110o. The relationship between and final finger angle (see Figure 1) was modeled with a piecewise approximation calculated by a least squares method. The error was always below 8% with an average of 1.7%, standard deviation (s.d.) 1.36. We roughly divided the operation range into three areas, as shown in Figure 3(c3).
2.2.3. Display system
The tasks depicted in Figure 3(c1) to (c4) were implemented in a program that used OpenGL graphical library to represent the virtual arm and prosthetic hand by the wire-frame drawing shown in Figure 3(d). The refresh rate was 25 frames/s, which was sufficient to give the impression of smooth motion. In addition, the GW displayed the processed EMG signals used as input. During the experiments, supplementary information was displayed to guide the subject to achieve the proposed goal.
2.3. Common experimental set-up
To test the performance, validity, and controllability of the simulated hand, several experiments were carried out with three male, able-bodied subjects aged 22, 24, and 32; and a 43-years-old male who had both hands amputated 18 years earlier after a traffic accident. All of them gave their informed consent.
The amputee subject uses a body-powered hook at the end of the right upper limb and a body-powered hand at the end of the left upper limb. He had worn a myoelectric prosthetic hand on the right upper-limb until four years before the experiment. Since then, he has not been actively using his forearms muscles; for this reason, he suffered from muscular atrophy (very weak muscles) in both forearms. Consequently, his EMG signals corresponding to the maximal voluntary contraction (MVC) had a value of less than 20% of the average MVC of the three non-amputee subjects (0.65V/3.51V). In addition, he exhibited a slightly higher level of involuntary co-contraction (simultaneous contraction of antagonist muscles) in his wrist flexor and extensor muscles.
All subjects performed the same protocol composed of four sessions. One session consisted of two different tasks: angle and position control. Subjects repeated these tasks from three to five times in each session.
The subject sat barefoot in a chair, with one foot on a steel sheet on the floor in front of the chair and with sleeves rolled up to expose the forearm. The steel sheet was used as reference voltage for the EMG processing unit. The subject was instructed to sit in a relaxed position in a chair, with the forearm and the arm forming an angle of about 15o. The forearm of the right-hand was cleaned with SkinPure skin abrasion gel (NIHON KOHDEN Corp., Tokyo, Japan) and ethanol. A pair of bipolar, surface electrodes (Ag-AgCl, 1 cm in diameter; NIHON KOHDEN Corp., Tokyo, Japan. Type NS-111U) was attached, with a centre-to-centre distance of about 2 cm, following the muscle fiber direction of the wrist flexor muscle (flexor carpi radialis), and another pair was positioned on the wrist extensor muscle (extensor carpi radialis brevi). Gelaid electrode paste (NIHON KOHDEN Corp., Tokyo, Japan) was placed in the contact area between the skin and the electrodes to ensure good electric conductivity between them. The subject was then given a brief explanation of how the system functions.
Before starting the experiments, the subject was instructed to exert for 1 second his maximal contraction of each target muscle from which the EMG signals were taken. The simulator calculated the MVC amplitude value for each muscle as the average around its EMG peak (the maximum detected value). The EMG signals of each subject were normalized to the range of 0-1 by their respective MVC values.
To familiarize the subject with the equipment and functioning of the simulator, the subject was firstly instructed to freely move the virtual hand contracting his forearm muscles. When he felt comfortable with the system, the different sessions of experiments were performed. In order to avoid fatigue, a rest was scheduled between tasks, and the subject was not asked to keep any of the postures for more than a few seconds (Basmajian and Deluca, 1985; Kampas, 2001).
After the experiments, a short questionnaire was given to the amputee volunteer to gather feedback on the Osaka Hand and on its simulator. Some questions were based on the surveys described by Sears and Shaperman (1998) as well as Atkins et al. (1996). This gathered information allowed us to plan the direction of our future research.
3. Experiments and results
Two types of experiments were carried out; the ones of the first block (3.1) were oriented to check whether the behavior of the simulator corresponded to the behavior of the Osaka Hand. The experiments of the second block (3.2) checked the controllability of the simulator.
3.1. Behavior of the simulator system
3.1.1. Validation of the input-output relationship
EMG signals were acquired from one subject as explained in the previous section and given as input to the simulator and, simultaneously, to the Osaka Hand. In this way, we were able to compare their respective output, which is the angle of the fingers, when both were given the same input.
Figure 5 shows the result of one of these experiments, carried out with a non-amputee subject freely moving the simulated hand. Figure 5(a) shows the inputs of the system: processed EMG signals of the wrist extensor muscle (dashed line) and those of the wrist flexor muscle (solid line). Figure 5(b) shows a comparison between the finger angle of the Osaka Hand (thick line) and the one given by the simulator (thin line). The average of the error (difference between the angle given by the simulated hand and that of the real Osaka Hand) was 0.85o (s.d. 0.39), with a maximum of 3.54o, which we consider acceptable for our purpose.
3.1.2. Variable stiffness
As one of the main features of the Osaka Hand is that the subject can control its stiffness by antagonist muscles co-contraction, we performed another experiment to corroborate that the stiffness of the virtual hand fingers can be controlled in that same way.
To simulate different levels of co-contraction, we fed the simulator with different levels of and under the condition () (see Figure 3(b) and (c1)). We sinusoidally modulated the applied grip force (see Figure 3(c4)) at a frequency of 0.2 Hz and a range between -0.08 V and 0.08 V, which corresponds to the actual output amplitude of the strain gauges. Figure 6 shows that as increased –that is, as the level of co-contraction increased-, in response to the same perturbation , the finger angle displacement decreased; that is, the stiffness increased. When there was no co-contraction (= 0V), the perturbation caused the total opening of the hand (110o is its maximal aperture), but when the level of co-contraction was maximum (= 10V), the perturbation had nearly no effect on the angle of the hand fingers. Therefore, the simulator behaves like the Osaka Hand also in stiffness.
3.1.3. Effect of force feedback
We carried out a preliminary experiment to study the simulator behavior when a subject grasped a virtual object (a sphere). The mechanical dynamics of the object were modeled in a simple fashion as a spring (Figure 3(c4)). The exerted force was calculated then as
where is the spring constant; is the finger angle when the contact with the object occurs, and is the current angle. The inputs and to the simulator were given as sinusoidal waves. Figure 7(a) compares the fingers angle when grasping the sphere: continuous line curve corresponds to the experiment carried out without feedback; and the dashed line curve when was calculated as explained above.
The first contact fingers-sphere occurred at (marked in the graph as A), when the fingers angle was around 43o. The maximal grasping force occurs when the fingers angle without was 3o. Therefore, there is a difference of approximately 40o in ΘHS providing or not PHS feedback, practically the third part of the whole simulated fingers angle range (110 o).
Figure 7(b) shows the value of the calculated . When it reaches its maximal value, approximately 25 mV (roughly one third of its maximum), the difference between the fingers target angle with and without pressure feedback is nearly 10o. Therefore, as it happens with the real prosthetic hand (Okuno et al., 1999), gives self-control to the hand over the exerted force when grasping objects, producing a smoother grasping motion.
3.2. Control experiments
3.2.1. Finger angle control
We ran a control experiment to determine how accurately the subjects could control the finger angle of the simulator hand. The effects of using the simulator were also investigated by comparing the performance of the subjects before and after two trails. In this finger angle control task, the subject was asked to achieve a series of eight different angles (from 0o to 110o) showed on the screen of the GW.
Figure 8 shows the typical results obtained, where the target angle to achieve was 55o (thick horizontal line). Figures 8(a) and 8(c) (left column) show the results of the first trial of two different subjects; (a) a sound-limbed subject and (c) the amputee subject. Both subjects needed more than 4 s to be able to keep the angle within the acceptable range, and were able to maintain it there for only less than 2 s (period between points A and B).
Figure 8(b) shows the results obtained by the sound-limbed subject after several trials for a period of about 40 min, and Figure 8(d) for the amputee subject after a similar period. In this case, both achieved the angle in just approximately 1 s (point A), and held it until they were asked to relax the muscles (point B).
To measure how accurately the subjects performed the task, we calculated the mean square error made while trying to keep a constant target angle as
where is the hand simulator finger angle in the sample n, and N is the number of samples between the points A and B. We defined ±2o as the acceptable range of error (dashed horizontal lines in Figure 8).
The average of the error was 1.19o (s.d. 0.67) for the three non-amputee subjects and 1.78o (s.d. 0.54o) for the amputee subject.
3.2.2. Grasping control
An additional control experiment was carried out to examine whether the subjects were able to grasp a virtual object using the simulator hand. In this grasping control task, six spheres with different diameters (from 2 mm to 10 cm) were depicted one at a time on a fix position on the simulator screen. The subject was instructed to grasp them with the virtual hand. To give some feedback to the subjects about the virtual force exerted over the sphere, a second index finger was drawn (with a very faint color) in the position the simulated index finger would have been if there was not a solid object in its way.
The results of this experiment were very similar to the ones shown in Figure 8. In this case, we defined the error as the distance between the index fingertip and the surface of the sphere. The average error while subjects tried to hold onto the sphere in the last two trails of each subject was 1.32 mm (s.d. 0.47). For the amputee subject, the average error was 1.77 mm (s.d. 0.63). In conclusion, subjects were able to grasp the object, and to do it in a smooth, natural way. These results prove that the simulator developed in this work is a valid tool for rehabilitation.
In this study, we have introduced a simulator of our biomimetic, myoelectric prosthetic hand (Osaka Hand), which is operated by the subject’s EMG signals, and displays in 3D a virtual arm and a prosthetic hand.
We have demonstrated that the simulator output agrees sufficiently for practical use with the finger angle of the prosthetic hand when both are given the same input.
Usefulness of the simulator has been shown in the experiments of controlling angle and stiffness of the hand. After a short period of training, subjects were able to control quite accurately the simulated hand. The precision achieved by an amputee subject was nearly as good as the precision obtained by the three non-amputee subjects, even though the amputee had not actively used his forearm muscles for four years.
This kind of powered myoelectric prostheses is not yet widely known. For example, in Japan only 350 units have been sold in the last 30 years (report of the Ministry of Health, Labour and Welfare of Japan). Our simulator could be accessible to physicians and related staff and be used to offer the opportunity to a wider group of amputees to try a myoelectrically controlled prosthesis.
The simulator can also be used for EMG signal processing and modeling. For example, when new features are added to the Osaka Hand, such as a new control program, the simulator can help in the design and testing phases, since it is easier and less expensive to make modifications in the model than in the actual prosthesis.
This simulator could be easily adapted to any myoelectric prosthesis, by performing just a few simple modifications on its software.
This work was partially funded by the Ministry of Education, Culture, Sports, Science, and Technology of Japan. G.A.G. was funded by a grant from the same Ministry (Monbusho). This work was carried out at Akazawa’s Laboratory, Graduate School of Information Science and Technology, Osaka University (Osaka, Japan).
G.A.G. thanks Professor Pedro García Teodoro (Granada University, Granada, Spain) for encouragement and scientific support during the first stages of this project.
Authors would like to thank as well Dr. Sandra Rainieri (AZTI Foundation, Bilbao, Spain) and Professor Antonio Peinado (Granada University, Granada, Spain) for useful comments and input on the original manuscript.