Logical definition of needle driver operation
Recently, minimally invasive surgery such as endoscopic surgery is taking the place of laparotomy. In the field of minimally invasive surgery, a typical commercial surgical robot, such as the da Vinci system produced by Intuitive Surgical Inc., is currently in clinical use. In the robot supported surgery, master-slave system is employed. In such master-slave systems, usually motions of the master device are detected by sensors, and the slave device is controlled to follow the behavior of the master device based on the measured information by those sensors. Therefore, even the mistaken operation will be reflected.
To perform a robotic surgery, a surgeon must have considerable skill. Operation by an unskilled surgeon may result in serious malpractice. Therefore, development of a system which urges an appropriate operation to the unskilled surgeon is in demand. As described in (Tanoue et al., 2007), for training of the robotic surgery, training box or simulator has been generally used.
Recently, in order to help surgeon's dexterity, force feedback to a surgeon through the master device of a surgical robot has been studied in (Ishii et al., 2011). In order to perform safe surgery, (Ikuta et al., 2007) proposed safe operation strategies, called "Safety operation space" and "Variable compliance system" for the surgical robot. The former can prevent collision between the forceps and organs. The latter can reduce the collision force between the forceps and organs.
In addition, training systems to practice operation of surgical robot through simulation using virtual reality environment (e.g. Tokuda et al., 2009), and navigation systems which guide a surgical instrument to the targeted location during the robotic surgery (e.g. Krupa et al., 2003), have been studied.
To the best of our knowledge, however, a system that recognizes and points out any singularity in a surgical operation because of the inexpertness of an unskilled surgeon has not been established yet.
In this study, to detect any singularity in a surgical operation, surface electromyography (SEMG) is employed. Our final goal is to develop such a system that recognizes and points out any singularity in a surgical operation because of the inexpertness of the unskilled surgeon on the basis of operator's SEMG signals during the operation of the surgical robot.
To this end, a novel method for automatic identification of a surgical operation and on-line distinction of any singularity of the identified surgical operation on the basis of the SEMG measurements of an operator and movement of the forceps, is proposed.
Use of the SEMG has attracted an attention of researchers as a method of interaction between human and machines. The amplitude property of waveform and the power spectrum based on frequency analysis are typical information which can be extracted from the SEMG signal.
In (Harada et al., 2010), to control a thumb and index finger of a myoelectric prosthetic hand independently, identification of four finger motions was executed using neural networks on the basis of the SEMG measurements.
In such SEMG based interaction systems, hand gestures are identified by measuring the activities of the musculature system using the SEMG sensors. It is well known that by measuring SEMG signals, not only hand gestures but also distinction between skilled person and unskilled person, and fatigue of the muscle can be recognized (e.g. Sadoyama et al., 1981, and Kizuka et al., 2006).
In (Chen et al., 2007), recognition of 25 kinds of hand gestures consisting of various motions of wrist and fingers, was performed using only two electrodes, and the high recognition rate was successfully obtained. On the other hand, (Nakaya et al., 2010) proposed a hand gesture identification method and a distinction method of any singularity in the identified hand gesture on the basis of the SEMG measurements.
(Kita et al., 2010) proposed a self-organizing approach with level of proficiency to perform stable classification of operation. (Tada et al., 2006) proposed a distinction method of unusual manipulation of a driver when driving an automobile, using the degree of deviation on the basis of the acceleration measurements.
On the other hand, as for the surgical operation, (Hayama et al., 2009) proposed an automatic classification method of four basic surgical operations using a sensing forceps made of a forceps and strain gauges. (Kumagai et al., 2008, and Yamashita, 2009) reported that in surgical operations, a difference arises between skilled surgeon and unskilled surgeon in the following points; the magnitude and direction of the handling force of the object, the manner of having surgical instrument, and surgeon's posture. (Rosen et al., 2006) proposed an evaluation method for the state transition of the forceps operation in cholecystectomy based on comparison of skilled operator and unskilled operator.
In this chapter, a novel method for automatic identification of a surgical operation and on-line distinction of the singularity of the identified surgical operation is proposed. Suturing is divided into six operations. The features of the operation are extracted from the measurements of the movement of the forceps, and then, on the basis of the threshold criteria for the six operations, a surgical operation is identified as one of the six operations.
Next, the features of any singularity of operation are extracted from operator's surface electromyogram signals, and the identified surgical operation is classified as either normal or singular using a self-organizing map: SOM (Kohonen, 2000).
Using the built laparoscopic-surgery simulation box with two forceps, the identification of each surgical operation and the distinction of the singularity of the identified surgical operation were carried out for a specific surgical operation, namely, insertion of a needle during suturing. Each surgical operation in suturing could be identified with more than 80% accuracy, and the singularity of the surgical operation of insertion could be distinguished with approximately 80% accuracy on an average. The experimental results showed the effectiveness of the proposed method.
2. Experimental system
2.1. Simulation box
Laparoscopic-surgery simulation box is shown in Fig.1. Inside of the mannequin, a rubber sheet of 1mm thickness is installed. The image of inside of the simulation box taken by the digital video camera is projected on a central monitor. An operator performs surgical operation using the two forceps, a needle driver (right hand side) and assistant forceps (left hand side) inserted into inside of a mannequin through the trocar, by looking at the monitor. The distance between the two forceps was determined based on the spatial relationship called "triangle formation" recommended in (Hashizume et al., 2005).
In this study, an operator simulates the suturing performed in a laparoscopic surgery using the simulation box.
As shown in Fig.2, the movement of the needle driver is measured by the haptics device PHANTOM Omni and attached four strain gauges.
2.2. Measurement of surface electromyography
The SEMG signals are measured by three electrodes stuck on the forearm of the operator as shown in Fig.3. The electrode 1 was stuck on the musculus flexor carpi radialis, the electrode 2 was stuck on the musculus extensor carpi ulnaris, the electrode 3 was stuck on the musculus extensor carpi radialis longus, and the earth electrode was stuck on the wrist.
3. Distinction of singularity of surgical operation
In this study, suturing is chosen as the objective surgical operation for automatic identification, and especially “insertion of a needle” in suturing is selected as the objective surgical operation for distinction of singularity. The flow for distinction of the singularity of the surgical operation “insertion of a needle” is explained as follows.
3.1. Features of the operation
From measurements of the SEMG signals by three electrodes, the amount of distortion by four strain gauges, and the angular velocity of gimbal and stylus by haptic device, the features are defined as follows.
For identifying the surgical operation, the features of the operation are extracted from the measurements of the movement of the needle driver. Define the features as follows.
The features of any singularity of operation are extracted from operator's SEMG signals. The SEMG signals are measured by sampling frequency Fs=2 kHz, and Fast Fourier Transform (FFT) is performed to each SEMG signal for every N=512 sampled data, which is equivalent to perform FFT every 0.256 seconds.
After filtering the SEMG signals by the fourth order Butterworth type band pass filter with 10 Hz to 1 kHz range, the full wave rectification is carried out. In addition, for normalization, the measured SEMG signal of each electrode is divided by the maximum value of the pre-measured SEMG for each operation. Define the features as follows.
Average absolute value: In order to perform pattern recognition, average absolute value of each electrode is often used, which is given as follows.
Center-of-gravity: In the case where the singular operation is performed, it is expected that change of the waveform can be observed in the SEMG signal. Therefore, as a value representing change of the waveform of the SEMG signal, the value of center-of-gravity is employed, which is defined as follows.
Spectrum ratio: Also, in the case where the singular operation is performed, it is expected that change of distribution of the power spectrum can be observed in the SEMG signal. Therefore, ratio of distribution of the power spectrum of the SEMG signal is also employed.
It is well known that the SEMG signal is distributed in the frequency band between 5 Hz to 500 Hz. Therefore, to see the ratio of the spectrum, frequency band is divided into 5 to 250 Hz and 250 to 500 Hz. Thus, the value of spectrum ratio is defined as follows.
3.2. Automatic identification of surgical operation
The suturing is divided into six operations as shown in Fig.4.
Grasping: the grasping state by closing the gripper of the needle driver.
Touch: the state where the needle driver touches the objects.
Haulage: the state where the needle driver touches the object with grasping the needle disposable.
Insertion: the state where the needle disposable is inserted.
Extraction: the state where the needle disposable is extracted.
Neutral: the state where nothing is operating.
In addition, to identify the state of operation of the needle driver using a threshold value, the following new features are defined using the features (1) to (3).
For identifying the surgical operation, the following values are defined.
3.3. Distinction of singularity of surgical operation
In this study, (a)a normal operation and a (b)singular operation are defined as follows. A normal operation is a surgical operation performed in the expected manner. The singular operation is assumed to be the following surgical operations: (b-1)the surgical operation performed at a posture in which the operator’s elbow is raised, denoted as "Posture", (b-2)the surgical operation performed in the state in which the operator is straining, denoted as "Straining", and (b-3)rough surgical operation performed suddenly by the operator, denoted as "Sudden". These are illustrated in Fig.5.
The surgical operation of (4) insertion of a needle in suturing is classified as either normal or singular by using a self-organizing map: SOM. For classifying the surgical operation, the feature vector which is input to the SOM, is defined as follows using the features (4) to (6).
Where is an average of
In each state shown in Fig.5, 20 features for normal operation and 60 features for singular operation (20 features for each singular operation) were pre-measured, and total 80 feature vectors defined by (13) are used for batch learning of the SOM. The size of the SOM was determined as hexagon lattice type of 10 x 10.
In addition, k-means method was employed to divide the map into four fields, namely, (a)Normal, (b-1)Posture, (b-2)Straining and (b-3)Sudden.
A feature vector extracted from on-line surgical operation is mapped on the map of the learned SOM, and singular operation is recognized by the distribution on the map. In addition, SOM was built using SOM Toolbox.
4. Experiments and results
The one healthy 20th generation adult man was chosen as an operator, and identification of surgical operation for "suturing" and distinction of the singularity of the identified surgical operation "insertion" were performed.
4.1. Method of experiments
In the experiment, the operator repeatedly performed the suturing process (1) to (6) classified in section 3.2, under the four situations (a)Normal, (b-1)Posture, (b-2)Straining and (b-3)Sudden. The surgical operation "suturing" performed in the experiment is shown in Fig.6. Then, rate of identification of each surgical operation in suturing and rate of distinction of the singularity in the case of (4) insertion were examined.
4.2. Result for automatic identification
Recognition rate for insertion is shown in Table 2.
|A: Actual operation times||19 times|
|B: Recognition count||21 times|
|Times which was not counted although operation was performed.||0|
|Times which was counted although operation was not performed.||2|
|False recognition rate: |A-B|/A*100||10.5%|
Recognition rate for other operations is shown in Table 3.
|Operation||A: Actual operation times||B: Recognition count||Recognition rate|
|1.Grasping||8 times||9 times||87.5%|
|3.Haulage||30 times||36 times||80.0%|
|4.Insertion||19 times||21 times||89.5%|
|5.Extraction||19 times||22 times||84.2%|
|6.Neutral||6 times||6 times||100%|
The threshold value
As shown in Table 3, each surgical operation could be identified with more than 80% accuracy.
4.3. Result for singularity distinction
In order to classify the singularity of the surgical operation of (4) insertion, a SOM was used. The SOM was constructed by batch learning using the feature vectors of any singularity of operation pre-extracted from SEMG in the case of insertion. Fig.7 shows the constructed SOM and distribution of the mapping of the feature vectors extracted on-line from SEMG for each experimental operation of insertion. The domain of the SOM is roughly divided into two fields, which include the domain for the normal operation denoted as "Normal" and the domain for the singular operation denoted as "Singular." In addition, the domain for the singular operation is divided into three fields, namely, "Posture," "Straining," and "Sudden."
The number of the hexagon counted in each field on the map is shown in Table 5.
Recognition rate for singularity distinction is shown in Table 6.
As shown in Table 5, the normal and the singular operation of insertion could be distinguished with 76.5% and 81.8% accuracy, respectively. However, the accuracy of recognition of the singularity (i.e., "Posture," "Straining," or "Sudden") of the operation is approximately 30%.
As one of the reasons of this low recognition rate in the singularity distinction, the following cause is considered. In the states of "Posture", "Straining" and "Sudden", the singular operation is similar, and the difference does not appear easily in the feature vector.
In order to examine efficiency of each feature, namely average absolute value, center-of-gravity and spectrum ratio, in the feature vector defined by equation (13), singular operation was recognized by SOM using the three-dimensional feature vector which consists of each feature only. Singularity recognition rate for each feature is shown in Table 7.
From Table 7, it turns out that the average absolute value contributes to distinction of normal operation compared with the center-of-gravity and the spectrum ratio, and conversely, the center-of-gravity and the spectrum ratio contribute to the whole singularity distinction compared with the average absolute value.
Based on the above result, to raise the singularity recognition rate in each state (Posture, Straining, and Sudden), singularity distinction was performed repeatedly by combining three kinds of features in the feature vector (Average absolute value, Center-of-gravity, and Spectrum ratio) through trial and error.
As a result, the best singularity recognition rate was obtained for the following six-dimensional feature vector removing the spectrum ratio.
Then, two operators were added and the singularity distinction was performed by SOM using the feature vector defined by (14). Recognition rate for singularity distinction using the feature vector given by (14) is shown in Table 8.
|Average absolute value||Center-of-gravity||Spectrum ratio|
|Operator A||Operator B||Operator C|
From Table 8, for the operators B and C, the singularity recognition rate for "Posture" and "Straining" was improved.
In this study, a novel method for automatic identification of a surgical operation and on-line distinction of the singularity of the identified surgical operation was proposed. The surgical operation "suturing" was performed using two forceps, namely a needle driver and assistant forceps, in the built simulation box for laparoscopic-surgery. Then, the identification of the surgical operation for "suturing" and the singularity distinction of the identified surgical operation "insertion of a needle" were carried out.
As for the identification of the surgical operation, suturing was divided into six operations. The features of the operation are extracted from the measurements of the movement of the forceps, namely the amount of distortion measured by four strain gauges and the angular velocity of gimbal and stylus measured by haptic device PHANTOM Omni. Then, on the basis of the threshold criteria for the six operations, the surgical operation was identified as one of the six operations. Each surgical operation in suturing could be identified with more than 80% accuracy.
As for the singularity distinction of the identified surgical operation, when the surgical operation was identified as "insertion of a needle", general distinction of normal operation or singular operation and distinction of three kinds of the states, namely "Posture", "Straining" or "Sudden" in the singular operation, were performed by the SOM using the 6-dimensional feature vector which extracted the features from SEMG. Then, the singularity of the surgical operation of insertion could be distinguished with approximately 80% accuracy on an average. On the other hand, recognition rate of each state in the singular operation was approximately 30% to 90% accuracy depending on the individual difference. Therefore, it is difficult to distinguish three kinds of the states in the singular operation with sufficient accuracy.
However, in a complicated surgical operation such as insertion of a needle, it can be said that general distinction of normal operation or singular operation was able to be recognized with high accuracy.
6. Future directions
In this study, operator for the experiments was only three persons. In order to demonstrate the reliability of the proposed automatic identification and singularity distinction method, it is necessary to perform verification of the proposed method by many operators. However, since SEMG depends on the individuals, it is considered that learning of the SOM for singularity distinction for every operator is required.
In addition, it is also necessary to extend the proposed identification and singularity distinction method for a surgical operation performed with not only a right hand but also both hands. As for this point, we are now applying the proposed identification method to a surgical operation of ligation performed with both hands, and the singularity distinction method to a thread knotting also performed with both hands.
Furthermore, construction of the system to avoid malpractice by presenting recognition of the singular operation to the operator and to provide safe endoscopic-surgery is left as future work.
The part of this work was supported by Grant-in-Aid for Scientific Research (23650100). The author thanks Y. Nakaya for his assistance in experimental works.
Chen X. Zhang X. Zhao Z. Y. Yang J. H. 2007Multiple Hand Gesture Recognition based on Surface EMG Signal, Proceedings of International Conference on Bioinformatics and Biomedical Engineering, 506 509
Harada A. Ishii C. Nakakuki T. Hikita M. 2010Robot Finger Design for Myoelectric Prosthetic Hand and Recognition of Finger Motions via Surface EMG, Proceedings of 2010 IEEE International Conference on Automation and Logistics, 273 278
Hashizume M. Konishi K. Okazaki K. Tanoue K. 2005Fundamental Training for safe Endoscopic Surgery, Daidogakkan Press (in Japanese)
Hayama Y. Kurita Y. Kawahara T. Okajima M. Ogasawara T. 2009Automatic Measurement of Forceps Manipulation Logs for Laparoscopic Surgery, Journal of Japan Society of Computer Aided Surgery, 11 3 328 329in Japanese)
Ikuta K. Hasegawa M. Goto H. 2007Total System of Hyper Finger for Remote Minimally Invasive Surgery (The 9th Report) Proposal and Experimental Verification of Safety Operation Strategies, Proceedings of the 16th Annual Meeting of The Japan Society of Computer Aided Surgery, 43 44
Ishii C. Mikami H. Nakakuki T. Hashimoto H. 2011Bilateral Control for Remote Controlled Robotic Forceps System with Time Varying Delay, Proceedings of IEEE International Conference on Human System Interaction, 330 335
Kita K. Kato R. Yokoi H. 2010EMG-to-Motion Classification for Prosthetic Applications ? A Self-Organizing Approach with Level of Proficiency-, Journal of the Robotics Society of Japan, 28 7 783 791in Japanese)
Kizuka T. Masuda T. Kiryu T. Sadoyama T. 2006Biomechanism Library Practical Usage of Surface Electromyogram, Tokyo Denki University Press (in Japanese)
Kohonen T. 2000Self-Organizing Maps, Springer
Krupa A. Gangloff J. Doignon C. de Mathelin M. F. Morel G. Leroy J. Soler L. Marescaux J. 2003Autonomous 3-D Positioning of Surgical Instruments in Robotized Laparoscopic Surgery Using Visual Servoing, IEEE Transactions on Robotics and Automation, 19 5 842 853
Kumagai T. Yamashita J. Morikawa O. Yokoyama K. Fujimaki S. Konishi T. Ishimasa H. Murata H. Tomoda K. 2008Distance Education System for Teaching Manual Skills in Endoscopic Paranasal Sinus Surgery Using ”HyperMirror” Telecommunication Interface, Proceedings of IEEE Virtual Reality 08, 233 236
Nakaya Y. Ishii C. Nakakuki T. Hikita M. 2010A Practical Approach for Recognition of Hand Gesture and Distinction of Its Singularity, Proceedings of 2010 IEEE International Conference on Automation and Logistics, 474 479
Rosen J. Brown J. D. Chang L. Sinanan M. N. Hannaford B. 2006Generalized Approach for Modeling Minimally Invasive Surgery as a Stochastic Process Using a Discrete Markov Model, IEEE Transactions on Biomedical Engineering, 53 3 399 413
Sadoyama T. Miyano H. 1981Frequency analysis of surface EMG to evaluation of muscle fatigue, European Journal of Applied Physiology and Occupational Physiology, 47 3 239 246
Tada M. Omura R. Naya F. Noma H. Toriyama T. Kogure K. 2006Analysis of Steering Control Behavior Using 3D-Accelerometers, IPSJ SIG Technical Report, SIG-CVIM-93, 233 240in Japanese)
Tanoue K. Hashizume M. 2007Advanced medicine and innovative technology, Fukuoka Acta Medica, 98 4 100 105in Japanese)
Tokuda J. Fischer G. S. Papademetris X. Yaniv Z. Ibanez L. Cheng P. Liu H. Blevins J. Arata J. Golby A. J. Kapur T. Pieper S. Burdette E. C. Fichtinger G. Tempany C. M. Hata N. 2009OpenIGTLink: an open network protocol for image-guided therapy environment, The International Journal of Medical Robotics and Computer Assisted Surgery, 5 4 423 434
Yamashita J. 2009A distance- and Self-Education System to Learn Experts’ Postures for Technical Skills Training in Endoscopic Sinus Surgery, Journal of Japan Society of Computer Aided Surgery, 11 3 170 171in Japanese)