Channel finger relations.
In recent years, researchers have conducted many studies on the design and control of prosthesis devices that take the place of a missing limb. Functional ability of prosthesis hands that mimic biological hand functions increases depending on the number of independent finger movements possible. From this perspective, in this study, six different finger movements were given to a prosthesis hand via bioelectrical signals, and the functionality of the prosthesis hand was increased. Bioelectrical signals were recorded by surface electromyography for four muscles with the help of surface electrodes. The recorded bioelectrical signals were subjected to a series of preprocessing and feature extraction processes. In order to create meaningful patterns of motion and an effective cognitive interaction network between the human and the prosthetic hand, fuzzy logic classification algorithms were developed. A five-fingered and 15-jointed prosthetic hand was designed via SolidWorks, and a prosthetic prototype was produced by a 3D printer. In addition, prosthetic hand simulator was designed in Matlab/SimMechanics. Pattern control of both the simulator and the prototype hand in real time was achieved. Position control of motors connected to each joint of the prosthetic hand was provided by a PID controller. Thus, an effective cognitive communication network established between the user, and the real-time pattern control of the prosthesis was provided by bioelectrical signals.
- fuzzy logic classification
- multifunctional prosthesis hand
- pattern recognition
People lose limbs due to accidents and medical conditions. Robotic devices, which imitate the shape and function of a missing limb, are manufactured for use by people who lose their limb in such situations. In recent years, researchers have studied to design and control multifunctional prosthetics hand [1–7]. The complexity of the movement, that is, the number of independent movements, increases in proportion to the number of joints. There are 206 bones in the adult skeletal system. The 90 bones of the skull and face are connected to each other by non-immobilized joints, and the 33 bones of the spine are connected to each other by semi-movable joints. Movable joints are only present between the bones (except the metacarpals bones) of the arm (25) and leg (25). In light of this information, aside from the wrist joints, the human hand has 15 independent joints with three on each finger. Therefore, the biological hand movement involves the control of these joints independently. Thus, control of the hand is quite complex. Thus, of all the human parts, the hand is the most complicated in terms of kinetic analysis .
Two main factors enable the functional and visual prosthesis to be used like a biological hand:
However, no matter how perfect the design and manufacture of the prosthetic hand may be, the utility depends on the cognitive interaction, i.e., the control algorithm, being designed properly, e.g., the type of movement and coordination between fingers. If information is not transferred to the prosthetic hand rapidly enough, then the prosthesis will not assume the desired position. Cognitive interaction is the most important factor for user to use effectively. There are many studies about cognitive interaction between human and robotic devices [20–25].
All voluntary muscle movements in humans occur as a result of bioelectrical signals transmitted from the brain through the muscle nerves. Bioelectrical electromyogram (EMG) signals transmitted to the muscles carry information about the type of movement, speed, and degree of muscle contraction or relaxation. The biological hand performs the basic tasks of holding and gripping, which involve various finger movements. The wrist movements essentially constitute the axis and assist in these gripping and holding movements. The main factor that increases the functionality of the prosthetic hand is the movement of the fingers. As the number of independent movements made by the prosthetic hand increases, it can mimic the biological hand more successfully. This study realizes the design of the bioelectrical signal control algorithm and the extension of the bioelectrical signal database with the purpose of increasing the finger motion function of bioelectrical signal-controlled prosthesis hands.
Figure 1 shows bioelectrical signals in the context of the activity of the muscle movements (e.g., flexion, relaxation force), as seen from the block flow diagram. EMG can be used to detect signals from the flexor pollicis longus, flexor carpi radialis, brachioradialis, extensor carpi radialis, extensor digiti minimi, and extensor carpi ulnaris. Bioelectrical signals were recorded with the help of four surface electrodes and subjected to a series of preprocessing and classification operations to understand the relationships between EMG signals and hand and finger movements. These signals were then applied to the prosthetic hand (space and simulator) as a reference motion signal. With the designed controllers, the position of the prosthetic hand finger joints can be controlled. Thus, a cognitive interface and communication network are established between the user and the prosthetic hand. Briefly summarized, the study creates a bioelectrical database of the activities of the hand muscles and the interaction network between the human and prosthetic hand using this database and interface to design a simulator and develop a control algorithm.
2. Recording, preprocessing, and featured extractions of EMG signal
2.1. Recording of EMG signals
EMG signals were recorded from the forearm muscles (the flexor pollicis longus, flexor carpi radialis, brachioradialis, extensor carpi radialis, extensor digiti minimi, and extensor carpi ulnaris) with the help of four surface electrodes. Electrode placements are shown in Figure 2. Electrode layout was chosen according to the protocol [26–28].
The signals, which support movements of the thumb, middle, ring, index, and pinkie fingers, were recorded separately for each of the respective muscles. Channels and finger relations are shown in Table 1.
|Channel 1||Channel 2||Channel 3||Channel 4|
|Pinkie finger muscle||Ring finger muscle||Middle finger muscle||İndex finger muscle|
2.2. Preprocessing of EMG signals
The recorded EMG signals also include various noise signals. It is necessary to separate the noise signals from the EMG signals, so that the characteristics of the signal can be accurately determined. For this reason, the raw EMG signal is first preprocessed. The block diagram of the preliminary preparation stage, including the separation, rectification, and sampling of the recorded EMG signals from noise, is shown in Figure 3.
2.2.1. Numerical sampling
EMG signals are analog voltage signals. Their amplitudes change constantly over the voltage range. Analog-to-digital conversion is the process by which the amplitude of the analog signal voltage is represented by a number sequence at specific time points [29–31]. The EMG voltage signals used in this study are converted into a number sequence by sampling with a period of 0.001 s.
2.2.2. Rectification process
Rectification is the evaluation of only the positive parts of the signal. This is done either by half-wave or full-wave rectification of the signal. A full-wave rectification method was applied to preserve the energy of the signal [25, 29–34], and the expression for the method is given in Eq. (1).
2.2.3. Smoothing of signal
A bandpass filter (50–500 Hz) was designed to soften the signal by eliminating high-frequency components.
2.2.4. Separate the signal into windows
Before the attributes of the obtained EMG signals are calculated, the frame is processed by the method adjacent to the signal. Experiments in the study of Englehart [18, 19] for framing and optimal framing values (R = 256, r = 32 ms) reached with calculations were used.
2.3. Featured extractions of EMG signal
The EMG signal is a non-stationary, time-varying signal that varies in amplitude by random negative and positive values [25, 31, 32]. Bioelectrical signals have certain characteristic values, i.e., information. Features in time domain have been widely used in medical and engineering practices and researches. Time domain features are used in signal classification due to its easy and quick implementation. Furthermore, they do not need any transformation, and the features are calculated based on raw EMG time series. Moreover, much interference that is acquired through the recording because of their calculations is based on the EMG signal amplitude. However, compared to frequency domain and time-frequency domain, time domain features have been widely used because of their performances of signal classification in low noise environments and their lower computational complexity . In this study, five time domain features methods widely used in the literature have been utilized to obtain the features of the EMG signal.
2.3.1. Signal energy
Mathematically, the energy of the signal m (t) is calculated as in Eq. (2), where tj and ti denote the lower and upper bounds of the part of the signal to be integrated, respectively. The above expression represents the area below the absolute value of the signal curve at time T = ti–tj [30–35].
2.3.2. Maximum value of signal
The maximum value of the signal represents the largest of the sampled signal values in each packet divided by windows .
2.3.3. Signal average value
Mathematically, the average of the signal m (t) is calculated as Eq. (3) [30, 31], where ti and tj denote the upper and lower bounds of the part of the signal to be integrated, respectively. The above expression represents the overall average of the signal at time interval T = ti−tj.
2.3.4. Effective value of the signal
2.3.5. Variance of signal
The variance value of the signal represents the amount of deviation from the mean of the sampled signal values in each packet divided by windows . p(t) is the variance of the signal to represent the probability density function of t:
3. Pattern recognition with fuzzy logic algorithm
A classifier’s function should be able to map different patterns, match them appropriately, and, in this case, select different hand grip postures. The extracted features were then fed into the fuzzy logic (FL) classifier for the developed control system. FL developed by Lofty Zadeh [35–41] provides a simple way to arrive at a definite conclusion based solely on imprecise input information. A summary of the feature extraction process from the forearm muscles is shown in Table 2 according to motion.
|Signal||Hand closure||Hand opening||Index-thumb touch||Middle-thumb touch||Ring-thumb touch||Pinky-thumb touch|
|Maximum value||Channel 1||2,378095||1,398911||0,822295||0,61429||0,725287||2,255524|
|RMS value||Channel 4||0,474695||0,273057||0,163739||0,134428||0,148438||0,387735|
In total, there are 20 features of EMG signal for four channels. In order to make relations easier, a featured function, which occurs at RMS, AVR, MAX, VAR, and E values, is defined for each channel. Finally, the number of inputs is reduced by four. The featured function is calculated as follows in Eq. (6).
For the FL classification analysis, the triangular shape of the membership function (MF) for the inputs (Fi) and output and the centroid method for defuzzification are used. The rules are created based on information from the states of contraction. FLC rules are shown in Table 3. Recorded SEMG signals have been used to initial testing. Then real time data implemented to Prosthetic hand model.
Fi Featured functions were inputs to the FL. The limits of F were set to [0, 20]. The three linguistic variables used were Small (S), Medium (M), and Big (B). The outputs of FL were Hand closure, Hand opening, Index-thumb contact, Middle-thumb contact, Ring-thumb contact, and Pinky-thumb contact. Figure 4 shows the flow diagram of FL classification process from four SEMG signals for six hand patterns .
Performance of FL tested 200 hand motions. Classification performance value for the six motions is shown in Table 4.
|Hand pattern||Pattern number||Tested total number of motion (A + B)||Number of true classified motion (A)||Number of wrong classified motion (B)||Average percentage of success (%)|
|Hand closure||MOTION 1||84||84||0||100|
|Hand opening||MOTION 2||84||84||0||100|
|Index-thumb touch||MOTION 3||84||76||8||90.476|
|Middle-thumb touch||MOTION 4||84||66||18||78.57|
|Ring-thumb touch||MOTION 5||84||72||12||85.714|
|Pinky-thumb touch||MOTION 6||84||76||8||90.476|
In the medical decision-making process, ROC analysis method is used to determine the discrimination of the test or classification algorithm. In this study, performance of FLC algorithm for six motion class are demonstrated in Table 5 via ROC analysis.
|Classification algorithm result||Hand closure||Hand opening||Index-thumb touch||Middle-thumb touch||Ring-thumb touch||Pinky-thumb touch|
The four outcomes can be formulated in a 2 × 2 contingency table. All contingency matrixes for each motion are shown in Table 6.
4. 3D modeling and manufacturing of prosthetic hand
4.1. 3D modeling of prosthetic hand via SolidWorks
In order to develop a multifunctional prosthetic hand model, the structural characteristics of the human hand must first be determined. In other words, it is necessary to determine the number of joints, the number of links, the fingers and the length and width parameters of each finger. In order to obtain a prosthetic hand the same size as a human hand, the hand characteristics of an adult male were recorded as in Table 7 for the purposes of this study [42–44].
|First link||Second link||Third link|
|Length (mm)||Width (mm)||Length (mm)||Width (mm)||Length (mm)||Width (mm)|
4.2. Manufacturing of prosthetic hand via 3D printer
The prototype of the prosthetic hand was produced with the help of the EDISON 3D printer manufactured by 3D Design Company. The necessary adjustments for the production (e.g., resolution, amount of fullness, amount of support) were made using the Simplify 3D program, which was offered by the same company as the software program. After a hand of 16 parts was produced, it was assembled as shown in Figure 6.
5. Prosthetic hand simulator design
5.1. Mechanical design of prosthetic hand simulator via SimMechanics
SimMechanics used in the realization of simulations of mechanical systems [45, 46]. By transferring the 3D CAD model of the prosthetic hand developed in the SolidWorks program to the Matlab SimMechanics program, a chain structure containing each joint and link of the prosthetic hand was obtained as shown in Figure 7. Five fingers connected to the palm, three rotary hinges forming each finger, and three connecting links are arranged in series to form the hand SimMechanics model.
As shown in Figure 7, when SolidWorks solid model is transferred to Matlab Program, a chain structure composed of revolute and link parts is obtained.
5.2. Modeling of the DC motor
We arrive at the following open-loop transfer function by eliminating I(s) between the two equations above, where the rotation is considered the output and the armature voltage is considered the input.
Using the mathematical model of the DC servo motor, the Matlab/Simulink model is constructed as shown in Figure 9.
6. Controller design
Position of ultra-nano DC servomotors connected to joints is controlled using a PID controller. The controller’s proportional gain coefficient (Kp), integral gain coefficient (Ki), and derivative gain (Kd) values are determined by Genetic Algorithm [11, 50–52] to ensure that the system quickly reaches a steady state without overshooting as shown in Table 8. The PID controller has an input-output relationship with input e (t) and output u (t) [53–55].
|All DC motors connected the each finger joints||0.42176||0.75724||0.0048566|
7. Graphical and numerical results
Electromyography is used to measure EMG signals, which are extracted from the forearm muscles and classified with the help of four surface electrodes. The type of motion that one wishes to perform is the perceived and designed three-dimensional prosthetic hand simulator and the five-fingered and 15-jointed hand. These movements were made in real time on the prototype. Each joint of the prosthetic hand is moved with one ultra-nano servomotor, and the position control of the motors is provided by the designed PID.
The prosthetic hand was made with hand closure, hand opening, thumb-index touch, hand opening, thumb-middle touch, hand opening, thumb-ring touch, hand opening, thumb-pinkie touch, and hand opening movements. The hand opening movement is performed after the hand closing movement and touch movement.
Motion pattern was determined by motion classification algorithm.
The specified type of motion information was input to the simulator and the prototype.
According to the recognized hand pattern, the reference joint angles in Table 9 were applied as the control input signal, and the closed loop position control of the DC servomotors was performed according to feedback information from sensors connected to the simulator joints.
For all finger joints, PID performance is shown in Table 10.
|Finger||Joint no||Motion 1||Motion 2||Motion 3||Motion 4||Motion 5||Motion 6|
|Thumb finger||1||Overshoot (deg.)||2.835||0.2932||2.137||2.936||3.025||3.655|
|Steady state time (s)||9.8084||13.413||8.8084||9.988||10.8084||12.8084|
|Steady state error (deg.)||0.046||0.041||0.037||0.027||0.021||0.024|
|Steady state time (s)||4.415||2.883||1.952||0.752||1.952||1.952|
|Steady state error (deg.)||0.0052||2.6e-3||0.0001||0.0001||0.0001||0.0001|
|Steady state time (s)||4.524||1.972||0.956||0.956||0.956||0.956|
|Steady state error (deg.)||0.0053||1.5e-3||1.5e-3||1.5e-3||1.5e-3||1.5e-3|
|Index finger||1||Overshoot (deg.)||2.835||0.3299||2.835||0||0||0|
|Steady state time (s)||9.915||9.71||9.915||0||0||0|
|Steady state error (deg.)||0.045||10e-4||0.045||0||0||0|
|Steady state time (s)||9.915||4.555||7.0725||0||0||0|
|Steady state error (deg.)||0.047||0.0183||0.0219||0||0||0|
|Steady state time (s)||9.915||4.535||7.429||0||0||0|
|Steady state error (deg.)||0.047||0.0202||0.0255||0||0||0|
|Middle finger||1||Overshoot (deg.)||2.8356||0.3244||0||2.8368||0||0|
|Steady state time (s)||10.5022||10.279||0||10.52||0||0|
|Steady state error (deg.)||0.0474||1e-3||0||0.0475||0||0|
|Steady state time (s)||10.5022||10.279||0||3.437||0||0|
|Steady state error (deg.)||0.0474||1e-3||0||0.0036||0||0|
|Steady state time (s)||10.5022||10.279||0||3.9265||0||0|
|Steady state error (deg.)||0.0474||1e-3||0||0.0036||0||0|
|Ring finger||1||Overshoot (deg.)||2.8356||0.3244||0||0||2.8368||0|
|Steady state time (s)||9.922||9.907||0||0||9.914||0|
|Steady state error (deg.)||0.047||1e-3||0||0||0.047||0|
|Steady state time (s)||9.915||9.9075||0||0||3.412||0|
|Steady state error (deg.)||0.047||1e-3||0||0||0.0035||0|
|Steady state time (s)||9.9055||9.906||0||0||1.884||0|
|Steady state error (deg.)||0.047||1e-3||0||0||0.0005||0|
|Pinkie finger||1||Overshoot (deg.)||2.8356||0.3244||0||0||0||2.8368|
|Steady state time (s)||9.9094||9.9122||0||0||0||9.29|
|Steady state error (deg.)||0.0475||1e-3||0||0||0||0.0475|
|Steady state time (s)||9.9094||9.9122||0||0||0||4.8174|
|Steady state error (deg.)||0.0475||1e-3||0||0||0||0.0052|
|Steady state time (s)||9.9094||9.9122||0||0||0||1.3391|
|Steady state error (deg.)||0.0475||1e-3||0||0||0||0|
The main factor in increasing the functionality of the prosthetic hand to the extent of imitating biological hand functions is the movement of the fingers. The greater the number of movements the fingers can do independently of each other, the greater the ability of the prosthetic hand to move and the more successfully it can mimic the biological hand. Within the scope of this thesis, the function of the prosthetic hand is improved by six different finger movements. Bioelectrical signals of two separate users were recorded from the forearm muscles (the flexor pollicis longus, flexor carpi radialis, brachioradialis, extensor carpi radialis, extensor digiti minimi, and extensor carpi ulnaris) with the help of four surface electrode groups. Thus, a broad bioelectrical signal database was created. The recorded bioelectrical signals were subjected to a series of preprocessing and feature extraction processes to calculate the maximum, effective, mean, variance, and energy values of the EMG signals. An FL classification algorithm was developed to create an effective cognitive interaction network, and 90% classification success was obtained from these algorithms. The identified bioelectrical signals were applied to the designed three-dimensional prosthesis handheld simulator. The five-fingered and 15-jointed prosthetic hand prototypes produced with a 3D printer, and the positional control of the prosthetic finger joints was performed with the designed controllers. Each finger of the prosthetic hand was moved by an ultra-nano DC motor, and the position controls of the motors were provided by the designed PID. Thus, a cognitive interface and communication network were established between the person and the prosthetic hand with great success.
The subject of this chapter, which is Beyda TAŞAR’s doctoral thesis, was supported by TÜBİTAK under the Domestic Doctoral Scholarship Program for Priority Areas in 2211 C. In addition, the study was supported by Fırat University Scientific Research Projects Management Unit within the scope of PhD Thesis Project number MF-14.25.