Telerobotic 3D Articulated Arm-Assisted Surgery Tools with Augmented Reality for Surgery Training

Abstract

In this research, human body will be marked and tracked using depth camera. The arm motion from the trainer will be sent through network and then mapped into 3D robotic arm in the destination server. The robotic arm will move according to the trainer. In the meantime, trainee will follow the movement and they can learn how to do particular tasks according to the trainer. The telerobotic-assisted surgery tools will give guidance how to slice or do simple surgery in several steps through the 3D medical images which are displayed in the human body. User will do training and selects some of the body parts and then analyzes it. The system provide specific task to be completed during training and measure how many tasks the user can accomplish during the surgical time. The telerobotic-assisted virtual surgery tools using augmented reality (AR) is expected to be used widely in medical education as an alternative system with low-cost solution.

1. Introduction

The study of robot controlled to follow certain paths by considering the collision with surrounding object has been studied extensively for the past years. The robot is usually manipulated through a particular device. In order to adjust the robot movement, user should reprogram the microcontroller that is attached to the robot to do certain task.

Robotic arm is one of the robotic parts, which is widely used for several fields including in the medical rehabilitation to assist a disable people. Implementing the real 3D arm is quite costly; the 3D simulated articulated robotic arm is offering new ways of simulation. In addition, the gesture tracking also offers the natural ways of interaction by providing the real-time synchronization between human arm and 3D arm.

Kinect was originally invented as a device for game; however, it also can be used for other purposes such as helping the recovery process of patient who got stroke. The doctor can monitor the improvement of patient and trail their nerve movement. By investigating the nerve of a person’s skeletal connection, healing specialists will have competency to find zones of body parts that requires intense training. The response that person acquired throughout or subsequently through a rehabilitation period is still possible to be perfected to solve precise badly behaved zones for the patient’s movement. The Kinect has a possibility as rehabilitation tools in house. Rehabilitation in the house provides suppleness for the patient to do regular reiteration of therapy.

Furthermore, to motivate neuron recurrence inside human mind that manipulates body change, therapy should be repetitive for a period of time. Idea behind the interface is that new position commands for the robot will be derived from the depth map captured by the Kinect. The interface offers methods to start and stop the Kinect, translate points from the Kinect’s position to robot’s position, and retrieve the latest position calculated by the Kinect. The control algorithm which is able to find the next position of the robot and, so that he/she can get exactly the wanted functionality. The Kinect uses a few seconds to start and to calibrate itself to give accurate depth measurements, so it is recommended that the Kinect interface is initialized at the same time as the robot system is turned on.

Most of the devices that used in hospital are really costly, and the procedure can be so complicated. They require very strict permission with authorized personnel to run the device and the queue can be quite long. Furthermore, some researchers focus on studying the communication between patient and therapist by providing the remote access through network to do home-based therapy [1]. In addition, the training system for elderly is also being studied to provide better design for attractive training system, since it can be conducted in home personally [2].

Augmented reality is also widely used for helping people on finding the path/route for pilgrim by combining AR technology and GPS trackers [3, 4, 5, 6]. In addition, the telecommunication in medical field such as telemedicine also has strong correlation with brain computer interface and haptic technology that increases the realism of telemedicine technology. The virtual character that assists the user will do their best to imitate human behavior as well as their interaction among themselves that will strive collision response in virtual environment [7, 8, 9, 10, 11, 12, 13, 14, 15]. With advanced collision handling, it will also reflect the realism of interaction between user and virtual agent [15]. Besides, researcher also studied the hand and finger tracking to assist doctor on viewing and manipulating the 3D model of human body [16, 17]. While another research try to improve augmented or virtual reality technology by producing realistic facial expression in the conventional teaching system [18].

2. Research method

2.1. Research methodology

The process is initiated by placing the Kinect camera in front of the user and then adjusting the optimum distance for acquiring finest depth image. Subsequently, it will capture the human body, which is then transferred into depth image stream. The second phase is random decision forest algorithm application by choosing a set of threshold for segmentation. The threshold and attribute which have high-density information are chosen and then the process is repeated. The final phase is mapping between joint skeleton and robotic arms and then starting the simulation by controlling remotely.

The methodology of the project consists of three main processes as shown in Figure 1.

Figure 1.

Project methodology.

Figure 1 describes that the telerobotic arm was initiated by placing the depth camera in front of the user to capture the human body and produce a stream of depth image. The algorithm used here is random decision forest that can determine the arm position of human body in real time. The process will be repeated until the desired accuracy has been satisfied. There are two vectors that built joint direction. Three axes, which define the joint frame, are the local or joint axis, the normal axis, and the binormal axis as shown in Figure 2.

Figure 2.

Join frame diagram.

2.2. Result of software requirement

The proposed application provides some features for user to interact with the system that is classified as functional and nonfunctional requirement as listed below:

2.2.1. Functional requirement

2.2.1.1. Gesture tracking

Gesture recognition can be divided into four main phases: hand movement detection, classifying the gesture from a collection of images, pull out the characteristic and then distinguish the gesture. The hand motion is determined through color of the skin and movement analysis. The speed of hand motion is computed to interpret the gesture localization from repeated images. Therefore, by collecting the depth images and analyzing the images, it can stimulate meaningful gestures that contain a particular command.

2.2.1.2. Real-time feedback

The real-time feedback is provided with depth image stream that is send out by Kinect and then analyzed using random forest algorithm. The algorithm will produce a skeleton joint that will be mapped into 3D robotic arm remotely.

2.2.2. Nonfunctional requirement

2.2.2.1. Finger tracking

The finger tracking is not covered in this research due to its complexity and it requires very short distance during tracking process while gesture requires a longer distance.

2.2.2.2. Full body synchronization

The robotic arm does not need full body tracking and synchronization because the system will focus on the arm of human.

2.3. Use case diagram

Figure 3 shows the use case diagram of the system. There are three main use cases: Kinect sensor, 3D articulated arm, and 3D arm control.

Figure 3.

Use case diagram.

2.3.1. Actor description

The actor of the above system represents the patient or other users who interact with the arm robot system

2.3.2. Use case description

The Kinect sensor is a motion-sensing controller that is able to sense user arm in motion mode. The Kinect needs sensor calibration before it is used, and the Kinect is able to detect depth by using an IR camera. The 3D articulated arm communicates with Microsoft Robotic Studio, while 3D arm control with gesture tracking.

2.4. Analysis phase

This section describes the sequence of project phases such as sequence diagram, activity diagram, and architecture of the system.

2.4.1. Sequence diagram

Figure 4 shows the sequence process of the system that starts with simulation and calibration with Kinect sensor and then it is continued by reading the depth data of user arm. Microsoft Robotic Studio will render the 3D arm and is synchronized with interpreted command of 3D arm and then performs 3D arm control.

Figure 4.

Sequence diagram.

2.4.2. Activity diagram

Figures 5 and 6 show two activity diagrams: diagram (A) is started with simulation, adjusting distance, tracking, capturing human depth image, and then classifying gesture.

Figure 5.

Activity diagram (A).

Figure 6.

Activity diagram (B).

While the activity diagram in Figure 6 starts with gesture classification, joint classification, than joint synchronization between human joint with 3D arm joint, and provide real time interaction between 3D articulated arm.

2.5. Architecture design phase

Figure 7 depicts the architecture design phase that has Kinect sensor component, and the other component is Microsoft Robotic Simulator that has capability to connect with Kinect driver and Kinect software development kit (SDK). Microsoft Robotic Studio consists of visual programming language and 3D environment model.

Figure 7.

Architecture design phase diagram.

3. Result and discussion

The first testing shows how to manage the tracker for hand control; it is started by initiating the Kinect camera to capture the human joint and render 3D hand that will follow the mouse movement.

3.1. Hand tracking control

The hand tracking control is used for motion detection of human hand as shown in Figure 8. The system is to track and differentiate right hand or left hand of human. Figure 9 is the result of robotic arm rotation using Kinect gesture.

Figure 8.

Kinect depth camera tracking human hand.

Figure 9.

Robotics arm which is connected to Kinect in steady position with domino blocks.

Figures 10 and 11 also demonstrate the movement of joint of robotic arm using Kinect gesture. The robotic arm will move according to the joint data that are sent through network by following the trainer movement remotely.

Figure 10.

User controlling the robotic arm to hit the blocks of domino.

Figure 11.

Robotics arm in wireframe mode.

3.2. Performance testing

The performance of rendering is satisfying when the frame per second (FPS) is very high and the highest score reaches 60 while the lowest is 54, as shown in Figure 12.

Figure 12.

The statistical measurement of frame rate per second.

Figure 13 and Table 1 also portray the statistical measurement of the rendering process of 3D articulated robotic arm during the testing.

Figure 13.

The statistical measurement of frame over time.

Frames	Time (ms)	Min	Max	Avg
982	16,411	55	61	59.838
1935	33,119	5	61	58.426
99	1638	59	61	60.44

Table 1.

Frame, time, max, and average.

4. Conclusions

Telerobotic is one of the essential research topics, which is widely applied into medical rehabilitation and even manufacturing process in industry. This paper aims to provide telerobotic arm with six joints that represent human arm. This arm can be rotated and can act like human arm. The idea of the project is to provide training or exercise for poststroke patient to move their hand by controlling the 3D articulated arm. The human hand is tracked down by depth camera, and the behavior of real hand and 3D Arm is synchronized in real time. In this project, we provide five joints of 3D robotic arms that can be rotated according to its angle. The 3D arms will be simulated with domino effect when they collide with each other. The user will control the arm by performing a gesture in front of Kinect camera, and the synchronization of joints between human arm and 3D arm is performed in real time. The control process through Kinect by imitating mouse cursor movement also runs smoothly during the testing process. This finding is believed to bring potential benefits to rehabilitation for certain patients such as poststroke rehabilitation. The result is very convincing, while interaction is conducted naturally just by waving their hands or rotating the joints of our hand, 3D arm will do rotation as well. For future recommendation, the result of project can be improved further by conducting clinical test to real patient in medical rehabilitation and it can be used as a simulator in manufacturing process in industries.

Acknowledgments

This work was supported by the deanship of scientific research (DSR), King Abdulaziz University, and Jeddah Saudi Arabia. The authors, therefore, gratefully acknowledge the DSR for technical and financial support.