Auto-tracking camera for dry-box laparoscopic training

Abstract

While laparoscopic surgery is less invasive than open surgery and is now common in various medical fields, laparoscopic surgery often requires more time for the operator to achieve mastery. Dry box training is one of the most important methods for developing laparoscopic skill. However, the camera is usually fixed to a particular point, which is different from practical surgery, during which the operational field is constantly adjusted by an assistant. Therefore, we introduced a camera for dry box training that can be moved by surgeons as desired by using computer vision. By detecting the ArUco marker, the camera attached onto the servomotor successfully tracked the forceps automatically. This system could easily be modified and become operable by a foot switch or voice, and collaborations between surgeons and medical engineers are expected.

1. Introduction

Laparoscopic surgery is one of the surgical approaches, and the proportion is increasingly compared to conventional open surgery [1]. In laparoscopic surgery, surgeons use a few and small incisions (5–10 mm in general) to enter into the abdomen. Through the incision, surgeons insert specialized instruments and camera and perform surgery inside the abdomen. While laparoscopic surgery is less invasive than conventional open surgery and is now common in various medical fields, laparoscopic surgery often requires more time for the operator to achieve mastery [2]. This is because the field of laparoscopic surgery is two-dimensional through a camera, and it takes a lot of time for surgeons who have often performed conventional open surgery to get accustomed to the camera view [2]. Thus, surgeons must practice extensively before performing practical laparoscopic surgery. Dry box training is one of the most important methods for developing laparoscopic skill [3, 4]. Dry box training consists of real instruments (such as forceps, scissors, needles, etc.) inserted into a box with a camera [5]. In general, organ models made from rubber or silicon are placed inside the box, and the surgeons practice how to use forceps, suturing, and other basic skills for laparoscopic surgery. The advantage of dry box training is the low cost. On the other hand, the camera is usually fixed to a particular point, which is different from practical surgery, during which the operational field is constantly adjusted by an assistant [5, 6]. Thus, surgeons need to manually change the camera angle to focus on where they are manipulating during training. This requirement leads to inefficiency because surgeons must pause the procedure every time they move the camera. Additionally, this inefficiency may prevent surgeons from practicing with the dry box training because some are annoyed by the interruptions. Some researchers elegantly investigated controlling camera by computer vision, but it would be expensive to introduce and not be suitable for the dry box training [7].

Therefore, we introduced a reasonable camera for dry box training that can be moved by surgeons as desired by using computer vision. In recent years, deep learning has been widely used for visual tracking in the field of computer vision analysis [8]. On the other hand, operation is now exclusively performed by single surgeon, which is called solo surgery [9]. Thus, the system which enables surgeons to be free from manipulating various instruments and to focus on surgical maneuvering will be more and more expected. In that context, artificial intelligence or deep learning has a great possibility to realize it.

This chapter is organized as follows. In Section 2, we introduce our applied methodology of autotracking camera. Section 3 deals with the results we obtained from the instrument. Section 4 covers the possibilities of computer vision in the field of surgery.

2. Methods and materials

2.1. ArUco marker

We used the OpenCV2 library for ArUco marker detection (https://opencv.org) [10, 11]. Briefly, OpenCV (Open Source Computer Vision Library) was designed for computational efficiency with a strong focus on real-time applications. Providers say that “An ArUco marker is a synthetic square marker composed by a wide black border and an inner binary matrix which determines its identifier (id). The black border facilitates its fast detection in the image and the binary codification allows its identification and the application of error detection and correction techniques. The marker size determines the size of the internal matrix. For instance, a marker size of 4 × 4 is composed by 16 bits” (https://docs.opencv.org/3.1.0/d5/dae/tutorial_aruco_detection.html). We referred to the code in the official tutorial. We printed an ArUco marker and attached it to the needle holder. We used the ArUco marker because of its robustness. As we recently demonstrated, the marker can be detected at a size as small as the forceps [12]. This means that their applications in the dry box training may be extended to clinical practice.

2.2. Servomotor

A servomotor is required to move the camera. In this study, we used the Pan-Tilt HAT (Pimoroni, Sheffield, United Kingdom) as the servomotor, as shown in Figure 1 (https://shop.pimoroni.com/products/pan-tilt-hat). We rotated the servomotor to the degree at which the ArUco marker was detected. The brief rationale is as follows: first, Raspberry Pi (a small computer) obtained consecutive images through the camera and then calculated the distance from the center of the current frame image to the center of the ArUco marker when it was recognized. The distance was expressed similar to a vector: x-axis (horizontal pixel) or y-axis (vertical pixel). Then, the angles were calculated, and the servomotor was rotated according to those calculations (x-axis for a pan rotation and y-axis for a tilt rotation) [13].

Figure 1.

Raspberry Pi and Pan-Tilt HAT. A small computer, Raspberry Pi, and a servomotor, Pan-Tilt HAT, were connected, and the ordinal web camera was attached on top.

We referred to the code of FaceTracker in the official tutorial (https://github.com/pimoroni/PanTiltFacetracker). Our modified code is shown in SI Text 1.

2.3. Development environment

The development platform used in this study was Raspberry Pi 3, model B (https://www.raspberrypi.org) [12]. Raspberry Pi and Pan-Tilt HAT are shown in Figure 1.

3. Results

3.1. ArUco marker detection and autotracking

This investigation consists of two parts as described in Section 2. The first objective was to detect the ArUco marker, and the other objective was to track the marker by using the servomotor. As shown in Figure 2 and Video 1, we succeeded in detecting and tracking the marker.

Figure 2.

Autotracking model. The detected ArUco marker was automatically tracked by using the servomotor.

The ArUco marker sometimes failed to be detected, but this occurred when the marker was too far from the camera. In addition, we thought that it was not problematic as long as surgeons used the forceps for training, as the marker would typically be within the detectable range.

As shown in Video 1, the speed and smoothness of the rotation from the Pan-Tilt HAT were satisfactory. Interestingly, we felt that the Pan-Tilt HAT might be better at performing a pan rotation than a tilt rotation. This might be due to using a non-expensive machine; however, we still consider our system to be suitable for dry box training because of its low cost and simplicity to create.

4. Discussion

Herein, we introduced a camera that can be moved by surgeons as they wish. The idea of cooperative human-machine coexistence has been the most researcher’s focus [14]. And, in that context, we are now interested in investigating cooperative system for surgery.

While dry box training is important for laparoscopy training, it differs from practical surgery in that the camera is usually fixed and does not move [3, 4]. Thus, surgeons need to manually change the camera angle every time they would like to focus on where they are suturing, thus producing unwanted inefficiency in training. Therefore, we introduced a camera that can automatically track the forceps to which an ArUco marker has been attached.

We propose two future directions. One research direction is the method for marker detection. We detected the ArUco marker in this study; however, this marker may not always be necessary if the energy device or forceps can be detected by using computer vision [6]. Indeed, recent studies have investigated using an autotracking camera to track the device [15, 16, 17, 18]. We consider these studies to be very important and may be essential for future robotic surgery. The advantage of using the ArUco marker is its robustness. As we previously described, we succeeded in detecting the ArUco marker at a size as small as the forceps [12]. Thus, we imagine that the marker may be used by embedding or depicting it directly on the forceps. Another advantage of using the ArUco marker is that we can use various markers at a time. The ArUco marker can be recognized at the same time as many as 1024 markers. This means that the marker can be assigned to all the instruments during laparoscopic surgery. This might be applied to the handling and cooperation of instruments by robots or artificial intelligence in the future. The other research direction is where and how to use the camera system. Although we showed an autotracking model in the present study, we have also investigated a tracking model that reacts only when the surgeon presses a certain key (e.g., the “s” key; SI Text 2). This means that the system can be modified and become operable by a foot switch or voice [19, 20]. Indeed, a robotic arm that can be operated and moved by a foot switch or voice is now commercially available [19, 21, 22, 23]. In addition, we believe that more sophisticated products can be developed by combining these ideas and systems.

5. Conclusions

We introduced a camera that can be moved by surgeons as they wish. We conclude that this investigation could easily be applied to a practical camera and robotic arm. In addition, collaborations between surgeons and medical engineers are expected.

Acknowledgments

This chapter has not been published elsewhere and is not being considered for publication in other journals.

Conflicts of interest

Authors declare that there are no conflicts of interest.

Supporting information

SI Text 1. ArUco marker tracker.

The code consists of two parts. The first part was used to detect the ArUco marker, and the other part was used to track the marker by using the servomotor.

SI Text 2 Modified ArUco marker tracker.

The model shown in SI was modified so that the camera reacted only when the surgeon pressed a certain key (e.g., the “s” key).

Video 1. ArUco marker detection and autotracking (https://www.youtube.com/watch?v=PIZod4SALsY%27%27).

The detected ArUco marker was automatically tracked by using the servomotor.