Application and Prospect of Telesurgery: The Role of Artificial Intelligence

Abstract

Remote surgery refers to a new surgical mode in which doctors operate on patients with the help of surgical robots, network technology, and virtual reality technology. These robots are located far away from patients. The remote surgical robot system integrates key technologies such as robot, communication technology, remote control technology, space mapping algorithm, and fault tolerance analysis. Apply a variety of emerging networking modes such as 5G, optical fiber private network, fusion network technology, and deterministic network to realize the motion of the subordinate surgical robot and the vision of the main knife, and ensure stable signal transmission and safe remote operation. The development and application of remote surgical robots has become a new trend, which helps to break the barriers of unbalanced regional medical resource allocation, promote the rational allocation of high-quality medical resources, and solve the telemedicine problems in special areas and special circumstances. The development prospect is broad. In the future, relying on the 5G network technology with high speed, low power consumption, and low latency, remote surgery can operate more efficiently and stably, and the surgical robot will also develop toward a more portable and flexible direction, so as to better serve patients.

1. Introduction

Telesurgery is an emerging model in which the physician and the patient are located in geographically distant locations and the physician performs surgery on the patient with the help of surgical robotics, network technology, and virtual reality. The idea of telesurgery was first proposed during wartime, with the aim of providing fast, high-quality surgical treatment to forward hospital trauma patients. However, the progress of related research was slow due to the limited level of robotics, network, and other technologies at that time. The uneven distribution of modern medical resources and the limited distribution of resources in special areas have led to many patients losing the best surgical opportunities, and the demand for telesurgery has increased in modern society. With the development of telecommunication technology and surgical robotics, the idea of telesurgery has gradually become a reality and has already benefited some patients. Artificial intelligence is a new technical science that delves into the development of theories, methods, core technologies, application software, and control systems for simulating the extension and expansion of the intelligence of the human brain. Its areas include robotics, image recognition, and expert systems. AI has important applications in many medical disciplines.

1.1 Method

We searched for literature on PubMed and the Internet using keywords such as “remote surgery,” “artificial intelligence,” “digital twin technology,” “quantum communication,” and “5G network,” and summarized the retrieved literature.

2. History and current status of telesurgery development

2.1 History of the development of telesurgery abroad

As a country with advanced medical care, the United States has conducted early research in the field of telemedicine, and basic research such as teleconsultation in multiple hospitals and televideo medical education has been conducted before entering into formal research on telesurgery. Since the 1990s, telemedicine for surgical procedures has developed rapidly, and a large number of research results have been reported. The first real-time teleconsultation for telesurgery was reported in 1992 in which a standard telephone network was used to transfer pathology slides between surgical procedures and to give real-time pathology diagnosis by a remote pathologist, but only 37% of the 35 cases received diagnostic help due to the limited network technology and medical level at that time [1]. In the same year, Satava et al. used the SRI remote operating system to directly control the movement of the mechanical needle tip to perform part of the operation and developed the famous da Vinci robot based on this operating system [2], which was the beginning of telesurgical robotics and the turning point of telesurgery.

In the 1990s, based on the continuous exploration of telesurgery, some foreign countries have mastered the key of technology in telesurgery, from the initial remote simple operation to the basic formation of telesurgery system, and began to try the real meaning of independent telesurgery. In 2001, the first real telesurgery was completed, namely the famous “Charles Lindbergh surgery” [3]. The patient was a 68-year-old woman with gallbladder stones in Strasbourg, France, and the surgeon was operating 7000 km away in New York, USA. A special dedicated network was applied to transfer signals between the surgeries, and the data transmission was stable during the surgery, with smooth transmission of operational and imaging signals and low network latency maintained. This is a milestone in telesurgery, and it validates the feasibility of telesurgery technology.

2.2 History of telesurgery development in China

Medical resources in China are unevenly distributed. Highly qualified surgeons and advanced medical equipment are basically distributed in large- and medium-sized cities, while rural and remote areas are significantly lagging behind, and there are obvious geographical differences. Many patients in need of surgery are unable to receive timely and high-quality surgical treatment, which seriously threatens their lives and health safety. Therefore, there is a need for the development of telesurgery in China. Although the research on telesurgery in China started late, it is developing rapidly.

In the early twenty first century, the Naval General Hospital used a remote surgical robot system to perform stereotactic biopsy surgery for brain tumor patients, completing the first off-site brain surgery in China. Beijing Jishuitan Hospital applied the master-slave robotic surgery system to perform remote orthopedic assisted surgery operations. Although the two surgeries were completed successfully, the telesurgery system in the study only played the role of auxiliary positioning and image transmission.

The application of 4G network communication has promoted the development of telesurgery in China, and its network stability is better than that of satellite communication. In 2015, the domestic “Myriad S” surgical robot completed remote wireless animal experiments at an interval of about 170 km with the help of 10 Mb/s bandwidth commercial network. Although the whole experiment was relatively successful, the narrow bandwidth and high latency of the 4G network still limit the extension of the clinical application of telesurgery.

2.3 The status of telesurgery abroad

In 2003, Anvari M’s team set up a telesurgery system between a teaching hospital and a rural hospital 400 km apart in Hamilton. This study led to the completion of 21 telesurgeries in 2005, successfully establishing the world’s first tele-robotic system to serve rural communities.

In 2014, a study by Xu et al. showed that a time delay below 200 ms is ideal for remote surgery, but up to 300 ms does not affect the successful completion of the procedure. A higher network latency would affect the safety and accuracy of the procedure, or even make it impossible for the operator to perform the operation. This conclusion has also been used as a criterion for network selection in many domestic and international remote studies [4].

Since the beginning of 2019, many foreign studies have started to use 5G networks for remote surgery. And satisfactory surgical results have been obtained. Lacy’s team applies 5G network to remote surgical coaching of young physicians in off-site locations [5]. In February 2019, a Spanish medical team used 5G network to remotely perform an intestinal tumor resection. This is the world’s first human remote surgery done using 5G network.

2.4 The current status of telesurgery in China

The commercialization of 5G network is a sign that China’s telesurgery has entered modernization, and its low latency, high bandwidth, and high mobility meet the demand for real time, high efficiency and stability of remote surgery, advancing the research boom of domestic telesurgery. In December 2018, the People’s Liberation Army General Hospital applied a domestic self-developed surgical robot to successfully complete 5G telesurgery animal experiments in Fuzhou. The physician remotely controlled the robotic arm and the lens arm to remove part of the pig liver, and the intraoperative high-definition 3D image and sound transmission were in real time and stable. The robotic arm operated flexibly with good master-slave consistency, and the one-way average time delay between the two ends was less than 150 ms. In September 2019, the National Institute of Hepatobiliary Surgery completed the world’s first multi-point collaborative 5G remote multidisciplinary animal experiment with stable network latency, smooth surgical operation, and stable intraoperative animal vital signs. The surgical experiment allowed two physicians located in Beijing and Suzhou to perform gastrointestinal resection and liver resection on the experimental animals through remote control of the robotic arm. This experiment broke the traditional mode of single-point consultation and surgery between doctors and patients, and provided patients with multidisciplinary remote consultation and treatment options, realizing multidisciplinary cooperation in telesurgery.

Chinese telesurgery is developing rapidly with the help of 5G networks. And it has transformed from a single-center, few-sample exploration model to a multi-center, large-sample clinical research model. Since September 2020, Prof. Niu Haitao’s team at the Affiliated Hospital of Qingdao University has conducted a large sample, multicenter remote robot-assisted urological surgery study based on animal experiments and simulated time-delay experiments. More than 50 cases of telesurgery have been completed, further confirming the safety and feasibility of telesurgery [6].

In February 2023, Prof. Niu Haitao’s team completed the first case of quantum telesurgery in China. The implementation of telesurgery with the help of quantum communication technology means that the development of telesurgery has entered a brand new stage, which greatly expands the spatial scope for physicians to perform complex surgeries and creates more convenient conditions for people in remote areas to enjoy high-quality medical services.

In order to ensure the high requirements of network for telesurgery, more and more new network technologies are applied to the network line. The multi-link aggregation transmission technology is a more mature and widely used network technology in telesurgery. Multi-link aggregation technology ensures data transmission capability for remote surgery. It adds a virtual layer on top of the traditional link layer, which implements the distribution of data frames that are distributed to each link through a rotation algorithm, successfully aggregating the transmission bandwidth of multiple physical links, thus achieving the effect of high-speed transmission with bandwidth overlay on the same terminal. Multi-link transmission and single-link transmission can coexist, and either multi-link transmission or single-link transmission can be selected according to the actual needs of the application. As an auxiliary technology of the networking scheme, it can guarantee the future development of telesurgery after the 5G network mode is popularized and has great development potential.

3. Application of digital twin technology based on artificial intelligence in remote surgery

The application of artificial intelligence in remote surgery is a new emerging technology, also known as remote intelligent surgery or robot-assisted remote surgery. Its core is to combine robotic surgery systems with Internet technology, allowing surgeons to perform surgery remotely via the network. This technology can help patients in areas with limited medical resources to receive better medical services.

In this technology, artificial intelligence can identify human tissues and organs during surgery and provide accurate location and size information through deep learning and image recognition technology. In addition, based on patient medical data and surgical history, artificial intelligence can provide personalized surgical plans and treatment recommendations to surgeons. Moreover, artificial intelligence can also perform automatic control and adjustment during surgery to ensure accuracy and safety. Combining digital twin technology derived from artificial intelligence with remote surgery greatly enhances the safety of the surgical process.

Digital twin is a digital representation of real-world entities or systems, creating virtual models of physical entities in a digital way that can describe the process of “symbiosis” between physical objects and their dynamic processes throughout their lifecycle. By simulating the behavior of physical entities in a real environment with interactive feedback between virtual and real worlds, data fusion analysis, and iterative decision optimization, and digital twin can add or expand new capabilities to physical entities and serve as a bridge and link between the physical and digital worlds. Currently, the concept and technology of robotic digital twin are also entering the medical and health fields from the industrial sector. The potential applications of digital twin include patient health monitoring, personalized medication, medical equipment, hospital operation management, etc. These characteristics are expected to play an important role in overcoming the limitations of remote surgery technology and expanding its application scenarios.

3.1 Establishment of artificial intelligence digital twin models in remote robot-assisted surgery interactive processes

In remote robot-assisted surgery, stereoscopic image processing and remote transmission are the main factors causing delays, making it difficult for doctors to obtain real-time information about the current movement of instruments within the body cavity. Therefore, using artificial intelligence technology, real-time monitoring of the surgical process can be achieved through 3D images and laser scanning, helping doctors to more accurately locate and handle problems during surgery. Artificial intelligence can also apply digital twin technology to the surgical process and perform twinning of the expected movements of instruments in real time at the doctor’s end. By establishing a digital representation of the entity model’s geometric dimensions, physical relationships, and motion behaviors in multiple dimensions and expressing the entity’s characteristics using mesh simplification and entity rendering methods, a twinning model of instrument movement can be constructed. This twinning surgical instrument can interact with the main operating hand operated by the doctor in real time to reflect the ideal posture of the instrument under the current instructions of the doctor.

3.2 Dynamic fusion of digital twin model and real surgical scene

The interactive process observed by doctors during remote surgery is a delayed scene, while the digital twin environment can accurately reflect the real-time position and posture of surgical instruments being operated by the doctor. By integrating these two, comprehensive remote operation information can be provided to ensure operational safety and avoid damage to intra-abdominal organs. Since both are time-varying scenes, a virtual scene perspective coordinate system is first constructed based on the endoscopic camera coordinate system. Then, artificial intelligence techniques such as deep learning and image morphology are used to accurately segment the real instrument pole, determine the control points and the scaling relationship between the twin and real scenes, and reverse correct the coordinate system deviation introduced by the camera and disparity. Based on the control point registration, virtual and real image fusion is achieved to accurately present the twin instrument model under the real endoscopic image.

3.3 Representation of motion correlation between real surgical instruments and their digital twin models

The motion of the digital twin surgical instrument and the real surgical instrument is homologous but not synchronous, and the motion deviation of the two needs to be extracted to ensure remote operation safety. In the abovementioned fused environment of real and twin scenes, intelligent algorithms are used to establish the expression of the twin and real instruments in the image coordinate system based on the principle of projective geometry, and the spatial deviation between them is described by pixel difference. Based on this, the transparency of the twin instrument is controlled by the deviation, with smaller deviations resulting in greater transparency and larger deviations resulting in greater visibility. Without interfering with the doctor, the delay of the remote instrument is visually displayed to the doctor.

3.4 Virtual fixture and safety force feedback strategy for virtual-real fusion

A virtual fixture is constructed by introducing a gravitational potential field, with the reference point of the real instrument’s end point in the image as the zero point of the potential energy, and the distance between this reference point and the twin instrument’s end point used to represent the potential energy size. Dead zone and extreme value processing are performed on this potential energy to ensure model stability. Based on this, the potential energy generated by the virtual-real fusion is fed back to the primary operator as impedance force within the extreme value range. When the twin model’s potential energy reaches the safety threshold, the system automatically generates feedback force to resist the driving force of the primary hand and waits for delayed motion to follow. If the feedback exceeds the limit value, the master-slave operation pause command is executed, and the remote robot motion is stopped within one network transmission and robot execution cycle (less than 40 ms).

Remote surgical operation delay is currently a critical factor affecting the success of remote surgeries and patient safety. To address the core challenge of operation delay in robot-assisted remote surgeries, remote surgeons combine artificial intelligence and digital twin technology. The AI system analyzes and deeply learns the basic principles of expressing delay based on the remote surgeon’s ideal operation behavior in virtual space and the observed real operation behavior. It achieves quantitative expression in both visual and tactile dimensions, making delay a visible and tangible physical quantity. Based on this, the surgeon’s operation process is intervened to provide safe and reliable guarantees for remote operations.

4. Remote surgical robot equipment and technology

4.1 Remote surgical robot system architecture

Remote surgical robot is a complex of multidisciplinary and high-tech means. It is composed of two parts: the doctor’s operation end and the operation end. It has two modes of traditional local surgical function and remote surgical function. The emergence of remote surgical robot is of great significance. It is completely different from the traditional surgical concept. Through the remote surgical robot system, surgeons can carry out surgical treatment for patients in different geographical locations [7, 8]. This is another milestone in the history of surgical development.

The remote surgical robot control system consists of five parts: the doctor console, the master communication control box, the patient console, the slave communication control box, and the three-dimensional endoscope camera system.

4.1.1 Doctor console

The doctor console is the control center of the surgical robot system and the interactive platform of the system at the doctor’s end. The surgeon controls the surgical instruments and three-dimensional laparoscopy by operating the two mechanical arms on the doctor’s console. The motion scaling function is added to the remote surgery robot system, which maps the motion of the doctor’s mechanical arm to the motion of the patient’s mechanical arm after reducing it to a certain proportion, minimizing the unconscious motion of the doctor’s hand, and improving the operation quality of the remote surgery.

In addition, the doctor console in the remote surgical robot system also includes the main communication control box, which is composed of industrial computer, display, controller, image processor, keyboard, etc. At the main operation end of remote surgery, the communication control box at the main end is connected with the doctor’s console. At the same time, all signals are collected and transmitted to the remote patient end, and the transmission signals and three-dimensional images from the remote patient end are received to the doctor’s console.

4.1.2 Patient console

The patient console is the executive part of the remote surgical robot system to implement minimally invasive surgery. Its main structure is to provide support for two patient manipulators and one image arm. In the process of remote surgery, the surgical assistant is also required to work next to the patient operating table in the sterile area, responsible for replacing the surgical instruments and three-dimensional laparoscopy, and assisting the chief surgeon to complete the operation. The operation of the remote surgical robot system must always be under the absolute control of the chief surgeon or surgical assistant and meet a certain priority relationship, that is, the surgical assistant next to the patient’s console has the highest priority, and they can adjust the robot’s motion at any time according to the actual situation of the operation.

Similarly, the patient console also includes the slave communication control box, which is composed of industrial computer, display, image processor, keyboard, etc. At the slave operation end of remote surgery, the slave communication control box is connected with the patient console for use, receives all signals from the remote master operation end, and simultaneously sends all signals and three-dimensional images to the remote master operation end.

4.1.3 3D endoscope camera system

The three-dimensional endoscope camera system collects the three-dimensional images in the body cavity area and then presents the image data on the three-dimensional display. During the operation, it is located outside the sterile area and can be operated by itinerant nurses, and various auxiliary surgical equipment can be placed. Three-dimensional laparoscopy is a high-resolution optical three-dimensional lens, which can magnify the surgical field by more than 10 times, and can obtain three-dimensional high-definition images of the surgical field, so that the surgeon can get a clearer understanding of the structure, reduce visual fatigue, and improve the accuracy of surgery.

4.1.4 Other equipment

The doctor’s console, patient’s console, and 3D endoscope camera system all need separate power supply. There is a backup battery on the patient console. In order to prevent emergencies, always connect the power supply to ensure that the backup battery is in full charge.

In the process of remote surgery, the patient’s slave operation terminal and the doctor’s console at the master operation terminal are connected through 5G network. The rest of the equipment is connected through the control signal line. Each time the remote surgery is performed, the labels at both ends of each cable need to be confirmed to ensure that the connection is correct.

4.2 Core technology and security processing mechanism of remote surgical robot

4.2.1 System architecture

The key to implement remote surgery is the development of remote surgery robot system. Traditional commercial surgical robots are mainly used in the same physical space. To build a remote surgical robot system, the key is to add a reliable remote communication system. Among them, the remote robot system includes the patient console, the doctor console, the attached endoscope system, and the surgical instrument unit. The remote communication system mainly provides a network channel for the transmission of multimodal signals at the master and slave ends of the robot.

4.2.2 Remote signal transmission mechanism

In order to ensure the smooth operation of remote surgery, doctors not only need to control the robotic arm at the patient end to perform surgery, but also need to constantly confirm the feedback information at the patient end. Therefore, the transmission of remote signals needs to have a high real-time two-way transmission mechanism. At present, the transmission hardware of the remote communication system is mainly based on the upper computer, and most of the transmission mechanisms adopt the robot control information transmission protocol based on UDP [9]. However, due to the low transmission reliability of UDP transmission protocol, developing a real-time and reliable transmission mechanism is one of the important research directions for the future development of remote surgical robots.

4.2.3 Video compression processing mechanism

The doctors of remote robot surgery can obtain a wider and clearer operation field through 3D laparoscopy, and how to obtain a high-resolution 3D laparoscopy image is one of the key technologies of robot remote surgery. Because the laparoscopic image is a 3D high-definition image, which requires high bandwidth and network real-time, an external high-speed data acquisition card needs to be used at the patient end for image acquisition and 3D compression processing, which can effectively reduce the image transmission time and delay. In order to ensure the continuity of the intraoperative image and add the breakpoint continuation function, the image can be continued from the breakpoint when the image is disconnected. With the development of the fifth generation mobile communication technology, 3D high-definition laparoscopic image transmission is expected to be further improved. The emergence of 5G network technology is particularly important for the development of 3D laparoscopic high-definition images in remote surgery.

4.2.4 Remote master-slave security processing mechanism

In the robot remote surgery, doctors and patients are in different physical environments. The security processing mechanism of the doctor’s console and the patient’s console of the remote surgery robot is mainly controlled by the communication transmission. In case of an unexpected communication situation, such as continuous packet loss on the network, the remote control boxes at both ends will immediately stop two-way signal transmission. The holding brake at all joints at both ends of the doctor’s console and the patient’s console of the surgical robot will be activated immediately to stop all movements of the robot. At the same time, an alarm signal will be sent, and the robot will enter the standby state, thus ensuring the safety of remote surgery. It is particularly important to ensure the safety and operability of remote surgery by ensuring the master-slave security processing mechanism between “doctors and patients.”

5. Basic principle and configuration conditions of the master and slave end of the remote surgical robot

The remote surgical robot system integrates key technologies such as robot technology, communication technology, remote control technology, space mapping algorithm, and fault tolerance analysis. The intraoperative endoscope image is compressed by the image encoder and then transmitted to the decoder at the main hand end through the network for decoding. Then, the doctor can observe the transmitted surgical image through the display, so as to operate the main hand; the signals of each joint sensor in the master hand are collected and processed in real time and then output. The data packets are encapsulated by the master communication controller and sent to the slave hand via the dedicated Internet. The received data packets are verified and filtered by the slave controller at the hand end and sent to the robot motion controller. The motion controller performs motion calculation, and finally inputs the data information to the drivers of each motor, then controls the manipulator to complete the operation of the main end physician.

5.1 Master and slave control communication method for remote surgery

The Internet is the basis of communication. It has not only complex physical circuits, but also complex protocol families, verification mechanisms and network security mechanisms. The network delay mainly depends on the transmission distance and the physical link through which data transmission passes, including the number of routers and the processing time of routers. The transmission routes and routing routes of fixed transmission nodes are usually fixed. However, due to the sharing and competition of the network, the processing time and processing tasks of the router are changing, and the processing time of data packets on the router at different times is also changing. Therefore, data packet disorder, delay, and other problems will occur. Therefore, in order to meet the stringent requirements of surgical operation, it is necessary to use a dedicated network and solve the problem of data transmission fluctuations through delay compensation and filtering.

5.1.1 Network control model

The network control system is generally divided into two structures, direct control and indirect control. The main difference between them is the signal transmission mode. Both the direct control signal and the sensor signal are transmitted through the network, and there is no restriction on the transmission network. The network that transmits the two signal flows can be independent of each other. The remote end of the indirect control structure is an independent closed-loop control system. The actuator signal collected by the sensor is directly fed back to the control system at the remote end and no longer fed back to the main controller to reduce the impact of the network on signal transmission.

5.1.2 Construction and implementation of control system

There are many ways of data communication, including wireless or wired Internet, optical fiber, 5G communication, etc. Each communication method contains a variety of different communication protocols. The remote control system uses socket to complete the communication protocol. The transport layer is implemented based on TCP protocol. When packet loss occurs in network congestion, we directly receive the next group of data packets to ensure the reliability of remote control.

5.1.3 Quantitative analysis of control model

In order to meet the requirements of remote surgery, it is necessary to test and verify the proposed method. In order to quantitatively analyze the practicability of predictive filtering algorithm, we build a remote operation simulation platform and randomly introduce 10-30 ms delay into the master-slave tracking system. The master-slave tracking test is carried out in combination with the motion frequency of the human hand, and good prediction results are finally obtained [10].

5.2 Remote surgery stereo image transmission method

5.2.1 Video encoding method

Unlike local surgical robots, remote surgery requires the transmission of endoscopic high-definition images through the Internet. Under certain network bandwidth conditions, in order to ensure the real time of image transmission, image compression means are needed to reduce the amount of data transmitted, and image compression and decompression processing will introduce new delays. Common video coding modes include H.264/MPEG-AVC coding, H.265/MPEG-HEVC coding, etc [11].

Compared with H.264, in order to improve the compression and coding efficiency of high-definition video, H.265 adopts the ultra-large quadtree coding architecture, and uses three basic units, namely, coding unit (CU), prediction unit (PU), and transformation unit (TU), to implement the whole coding process, which improves the coding efficiency and effectively reduces the decoding time [12].

5.2.2 Stereo image transmission mode

Three-dimensional stereo images are composed of two cameras taking pictures of objects from different angles of view, and then interleaving the images with odd and even lines. The 3D stereo image synthesis process adopted by the remote surgery robot, At any time, the 3D stereo endoscope camera outputs two high-definition images (a) and (b) with a resolution of 1920 * 1080, scales the two images to an image (c) with a resolution of 1920 * 540, and then splices the two images to form a 1920 * 1080 Top-Bottom format high-definition image, which is then input to the image encoder for compression and remote network transmission. After receiving the image at the main hand end, through parameter adjustment, the two images spliced up and down are displayed alternately in odd and even rows to form a three-dimensional stereo image (d), and finally, the stereo image under the endoscope field of vision is displayed on the display at the main hand operation end.

5.2.3 Delay and optimization of remote surgery

Low transparency and large network delay of remote minimally invasive surgical robot will prolong the response time of surgeons. According to the experiment, when the delay of remote surgery exceeds 500 ms, the operation risk will be significantly increased [13]; According to the statistics of the transatlantic remote “Lindbergh operation,” the delay doctors can tolerate is 330 ms. For the developer of robot equipment, a detailed quantitative description of the system delay will help to find deficiencies and continuously optimize, so the delay test of surgical robot is very meaningful.

The delay of the remote robot system is mainly composed of two parts: ① the sample-communication-execution delay between the master and slave hands; ② capture-transmission-display delay between the endoscope and the display. Therefore, it is necessary to measure the delay of these two parts separately. After continuous measurement and optimization of the test results, the final test results completely test the system delay of the remote surgical robot, and theoretically ensure that its reliability meets the use requirements.

5.2.4 Master and slave configuration of remote surgical robot

Based on the above test results, combined with the architecture of the minimally invasive surgical robot and the requirements for signal and video transmission, we designed a remote communication control system based on 5G/Internet dedicated line, which integrates the remote surgical robot.

The main terminal communication control box is the receiving and sending and control module of all kinds of information at the doctor’s operating terminal under network conditions. It consists of a box, an image encoding and decoding unit, a power supply unit, a network communication unit, a motion control and signal processing unit, a status display unit, an interaction unit, an interface unit, etc. The functions of each unit are as follows:

Box: integrated with each component unit to facilitate overall transportation.

Image encoding and decoding unit: composed of image encoder, used for encoding and decoding stereo endoscope dual-channel images and transmitting them at both ends of remote surgery.

Power supply unit: It is a switch power supply conforming to medical specifications, which is used to supply power to all units inside the box.

Network communication unit: It is a special industrial computer used to transmit control signals at both ends of remote surgery and monitor the network status.

Motion control and signal processing unit: interacts with the network communication unit, can collect the motion information of the master hand, and can actively control the motion of the master hand.

Status display unit: used to display the working status of network communication unit and motion control and signal processing unit.

Interaction unit: human-computer interaction interface, which is used to set network connection, start/stop data transmission, etc., and can feedback the robot operation status to the operator through prompt tone, etc.

Interface unit: including power supply interface, network interface, video output interface, foot switch signal acquisition interface, robot main end operation data output interface, etc.

The main communication control workflow is to connect the interface of the main end of the robot itself and the main end medical monitor with the main end communication control box, and then connect the main end communication control box to the Internet through the RJ45 interface. At the same time, supply 220 V AC power through the interface unit, and start-up.

The slave communication control box is the receiving and sending and control module of all kinds of information at the slave end of the robot under network conditions. It is composed of box, image encoding and decoding unit, power supply unit, network communication unit, energy instrument control unit, status display unit, interaction unit, interface unit, etc. The functions of each unit are as follows.

Box, power supply unit, network communication unit and status display unit: the same as the main control box.

Image encoding and decoding unit: composed of image encoder, used for encoding and decoding stereo endoscope dual-channel images and transmitting them at both ends of remote surgery.

Energy instrument control unit: It is composed of PLC modules, which is used to simulate the control signal output by the main machine of the excitation energy tool.

Interaction unit: human-computer interaction interface, which is used to set network connection, start/stop data transmission, etc., and can feedback the robot operation status to the slave assistant through prompt tone.

Interface unit: including power supply interface, network interface, video input interface, robot slave operation data output interface, etc.

The work flow of the slave communication control box is as follows: connect the communication port of the slave robot to the slave communication box through the network cable, and then connect the slave communication control box to the Internet through the RJ45 interface. At the same time, supply 220 V AC power through the interface unit and start-up.

5.2.5 Use of remote surgical robot

After testing, in order to ensure the smoothness and security of remote operation, the average network delay should not exceed 30 ms, and two dedicated networks with a bandwidth of not less than 50 Mb should be provided to ensure network stability.

The remote surgical operation doctors should not only receive the operation training of local surgical robots, but also receive the operation training in the remote environment. The operating physician also needs to be familiar with the operation mode and operation specification of the robot, be able to know the meaning of various operation status prompts fed back by the robot in a timely and accurate manner, and intervene with the robot to ensure the safety of remote surgery.

The remote surgical operation assistant shall also receive sufficient local and remote surgical robot operation training of the system. The assistant also needs to be familiar with the operation mode and operation specification of the robot, be able to know the meaning of various operation status prompts fed back by the robot in time, and be able to accurately intervene the robot according to the surgical requirements to ensure the safety of remote surgery.

The assistant of the remote surgical robot should be familiar with the connection, setting, and testing processes of the robot, and the connection between the robot and each module should be accurate and reliable. A comprehensive test should be carried out 2 hours before the robot implements the remote operation to complete the initialization and operation test of the robot.

6. Network solution for remote surgery

The construction and networking scheme for remote surgery support is a critical technology that ensures the smooth development of remote surgery. Since the inception of the concept of remote surgery, the choice of network communication mode has been crucial to ensure stable, speedy, and efficient signal data transmission. Additionally, minimizing operational delays caused by remote communication and surgical failure due to remote signal interruption has been a top priority for designers and users of remote surgery systems. From the first proposal of remote surgery to the realization of the first remote surgery and the current boom in the field, the selection of an appropriate network communication mode remains a vital consideration for the advancement of remote surgery technology.

6.1 Traditional networking schemes

6.1.1 Dedicated fiber optic cable network solution

In 2001, Professor Jacques Marescaux in France completed the world’s first remote surgery, Lindbergh surgery, through the Zeus robot system, using a dedicated submarine fiber optic cable transmission network. This network directly connects the master operator and the slave operator through a dedicated fiber optic cable. It has several advantages, including wide bandwidth, large capacity, good signal quality, and high reliability. However, the drawback of this network is the limitation of the point-to-point physical connection, which requires special erection and maintenance of the fiber optic cable dedicated line and is extremely expensive, making it not widely promoted.

6.1.2 Satellite communication network solution

The satellite communication network uses artificial earth satellites as relay stations to relay radio waves, allowing interconnection between two or more earth stations. This network has many advantages, including wide coverage, large communication capacity, good transmission quality, less geographical restrictions, convenient and rapid networking, and easy global seamless connection. However, the main disadvantages of satellite transmission are a delay of about 0.6 seconds for audio and video, high cost of satellite signal transceiver equipment and channel usage at the user end, and high professional requirements for maintenance personnel. Currently, the satellite communication networking method can provide more than a few megabytes of communication rate, mainly used for mobile medical emergency rescue equipment to participate in telemedicine.

6.1.3 ADSL Internet access networking program

This networking scheme refers to client network or equipment access to the Internet through asymmetric digital subscriber line (ADSL) and the use of the Internet to form an interconnected network. This type of networking can provide up to 3.5 Mbps uplink and up to 24 Mbps downlink, and the cost of access is usually between several hundred and several thousand dollars per year in various provinces and cities. Although the cost of this solution is lower than that of private network and fiber optic Internet access methods, the actual available bandwidth drops when public network resources are insufficient, and upstream and downstream bandwidths are inconsistent because the data streams pass through the public network and share the public network bandwidth with the other user data streams. Therefore, the bandwidth stability is poor with this networking method, which may adversely affect two-way audio and video interaction applications with high bandwidth stability requirements, such as non-smooth video. Nevertheless, the networking scheme still has the advantages of easy networking and low cost and is suitable for building a telemedicine system using software video among hospitals below the county level. Some of the hospitals’ self-built teleconsultation systems using software video partially adopt this networking method, which is one of the most common Internet access methods for small institutions and individual users.

6.1.4 3G/4G communication networking scheme

This networking scheme means that both the user side and the data center use 3G/4G communication to connect to the Internet and achieve interconnection between them. When a business relationship is established between multiple users, the information flow is delivered to the destination device through the Internet. However, since the data stream must pass through both 3G/4G and public network bottlenecks while sharing the public network bandwidth, the actual available bandwidth is reduced when the 3G/4G signal is weak or public network resources are insufficient. The upstream and downstream bandwidth may also be asymmetric. Therefore, the bandwidth stability using this networking method is poor, which can negatively impact applications that require two-way audio/video interaction.

6.2 Emerging networking schemes for remote surgery: the optional 5G communication networking scheme

6.2.1 5G communication networking scheme

The latest generation of cellular mobile communication technology is 5G, which inherits the advantages of previous systems such as 4G (LTE-A, WiMax), 3G (UMTS, LTE), and 2G (GSM), and also adds new features. Compared to previous technologies, 5G technology offers high-speed data transmission, low-latency rates, high capacity, large-scale device connectivity, low cost, and low energy consumption. The development of 5G technology as a bearer network for new technologies has revolutionized the development of areas such as telesurgery.

The International Telecommunication Union (ITU) has identified three main application scenarios for 5G: enhanced mobile broadband, ultra-high reliability low-latency communication, and massive machine-like communication. The key performance indicators of 5G include high speed, low latency, and large connectivity. Enhanced mobile broadband (eMBB) provides a better application experience for mobile Internet users, while ultra-high reliability low-latency communication (uRLLC) is used for telemedicine, industrial control, autonomous driving, and other applications with high requirements for low latency. The most prominent features of 5G are high speed, low latency, and large connectivity, with user experience rates of up to 1Gbps, latency down to milliseconds, and user connectivity up to 1 million connections per square kilometer.

6.2.2 5G network architecture for remote surgery

The 5G remote surgery network communication system requires two-way network communication. One is for remotely controlling the robotic arm, and the other is for transmitting live surgical video feedback. To ensure the smooth progress of surgery, we adopt a dual 5G (or dual gigabit dedicated line) multi-guaranteed network, which ensures the stability, reliability, and low latency required for remote medical operations. To ensure the stability and reliability of the 5G access network, we use high-performance indoor distributed systems (Pico RRU) for 5G. We use dedicated 5G core network equipment to ensure low latency on both ends of the network, guaranteeing system independence and security. We use a new type of distributed Pico Site to provide indoor coverage and configure uninterruptible power supply (UPS) backup power to ensure reliable power supply. These measures ensure the stability and reliability of the 5G remote surgery network communication system, meeting the requirements for remote medical care.

The use of network slicing technology in the 5G network can greatly improve the speed and security of remote surgery. With the development of 5G technology, the slicing packet network (SPN) has emerged, supporting the next-generation transport network architecture, bandwidth, traffic patterns, slicing, latency, and time synchronization. The core advantage of the SPN network is its flexibility. By binding elastic ethernet or FlexE technology with SPN, a larger physical link can be divided into multiple smaller physical channels, ensuring quality of service and isolation between transport layers. The SPN technology is a fiber-optic network transmission technology architecture independently developed in China, which has been successfully applied in the transmission of China’s 5G network, achieving the organic integration of TDM transmission technology and packet transmission technology, fully meeting the requirements of lossless and efficient 5G transport. The SPN transmission technology has the advantages of large bandwidth, ultra-low latency, ultra-high precision synchronization, flexible control, and network slicing, which are essential in 5G remote surgery communication. By adopting SPN technology, efficient, secure, and fast network transmission can be achieved, ensuring the stability and precision of remote surgery.

Communication plan during surgery: If possible, a video conferencing system or a 5G smart bedside car can be utilized to ensure voice and video communication between the surgical control and the controlled end. In the absence of this equipment, mobile phones can be used for voice communication between both parties. However, wireless or wired headphones should be provided for doctors to ensure convenience, real-time communication and to avoid external interference.

Quality of 5G remote surgery network communication and monitoring method for surgical equipment: During remote surgery, the network should be subjected to a PING test, and the network delay should be monitored in real time. Both test routes should be tested. To reduce the impact on the network, the size of the PING packet should be set to the smallest possible size. During the surgery, three safeguard plans should be implemented in the following order of priority: The first plan is to use 5G for both control and video transmission, the second plan is to use two dedicated 5G lines for transmission, and the third plan is to use two dedicated lines for transmission. The first plan should be adopted initially. If there are network quality problems resulting in increased delay or difficulty in controlling the robot, the second and third plans should be used.

Standards for diagnosing network and surgical equipment failures in 5G remote surgery: The ideal network delay for 5G during surgery should be within 30 milliseconds, and the delay for dedicated lines should be within 10 milliseconds. If the average delay of 5G or dedicated lines exceeds 50 milliseconds within 3 minutes, or if there is unstable jitter in the instantaneous delay, the fallback plan should be initiated. The switch between 5G and dedicated lines should be completed within 3 minutes to ensure the smooth completion of the surgery.

The fusion of aggregate network technology and quantum communication encryption technology ensures the security of surgical networks. By adopting heterogeneous multi-link aggregation transmission technology, data is split and transmitted across different networks at the transmission layer of the network, endowing the network with features such as multi-link parallel transmission, link weight adjustment, forward error correction encoding technology, network self-adaptation, and real-time determination of the total network bandwidth. This achieves lower latency, higher stability, and higher efficiency, fully exploiting the adaptability of the public network for data transmission. The implementation of this technology can greatly reduce the cost of remote surgery and promote the normalization process of remote surgery. In the process of applying quantum encryption communication technology, the project team combines quantum encryption communication technology with remote robotic surgery to achieve the theoretically “unconditionally secure” communication mode for remote laparoscopic surgery. By using quantum superposition states and entanglement effects, combined with quantum random number generators (QRNG), quantum key distribution devices (QKD), and other equipment, key resources are generated, distributed, and received for quantum key production, distribution, and reception, providing encrypted transmission for remote surgery.

6.2.3 Fiber optic private network configuration

A fiber optic private network configuration is a star-shaped network that connects user LANs or devices through a dedicated line at a single point. Multiple private networks can also form a tree-shaped network by cascading. Fiber optic private networks are usually constructed using synchronous digital hierarchy (SDH) technology and are physically isolated from public networks. Therefore, from an application perspective, they offer high security, stable bandwidth, and high standardization of terminal equipment interfaces. However, the disadvantage of this network configuration is its higher cost compared to Internet-based networking. This networking approach can provide fully optical transparent channels ranging from 2 Mbps to 10 Gbps and offers data, image, and audio transmission services for point-to-point and point-to-multipoint connections. It is suitable for networking remote medical systems between county-level hospitals or above in provinces and cities that require high-quality audio-video interaction, frequent usage, and large image data volume.

During the specific implementation process of remote surgery, a gigabit dedicated line with dual router access is used, relying on clear networking architecture of SDH, packet transport network (PTN), and optical transport network (OTN) equipment, which provides high-risk security operation guarantees. In terms of maintenance, it has a 7×24-hour full-time scheduling and maintenance capability, enabling quick fault repair and restoration. The above network guarantees should be deployed at least 1 day before the operation, and network debugging should be completed. After the network debugging is completed, it is necessary to perform joint debugging with the surgical robot to ensure that both 5G and the dedicated line can support the surgical tasks. The networking adopts a disaster recovery mechanism. The 5G equipment and the dedicated line are protected by main and standby transmission equipment, and the reliability of transmission is ensured using dual router and dual-loop methods. All equipment is protected by UPS for power supply, and in case the power supply cannot be guaranteed, dual power supply or oil machine protection should be considered.

6.2.4 Aggregation network technology

Aggregation network technology is composed of 5G fused communication terminals and cloud servers. The 2-channel 1920×1080P60 video signals of the endoscope are collected, and 3D signals are synthesized and encoded, and then transmitted through multiple 5G links via the aggregation transmission. The cloud server deploys the 5G fused communication system software, which provides the service-side function of heterogeneous multi-link aggregation transmission. In terms of aggregation network technology, the transmission of the surgical endoscope video signal and control signal uses heterogeneous multi-link network aggregation transmission technology to improve transmission efficiency and stability. The control command signal is issued by the main hand, connected to the 5G fused communication gateway next to the main hand through the aggregation link, and the gateway transmits the control signal through the network port to the 5G network for transmission. The cloud server deploys the 5G fused communication system software, which implements the service-side function of heterogeneous multi-link aggregation transmission in the kernel layer, supporting both uplink and downlink aggregations. Therefore, aggregation network technology has the advantages of signal stability, fast transmission speed, environmental independence, and strong universality, and has good application potential in future remote surgery.

6.2.5 Deterministic network

“Deterministic network” is a new technology that provides end-to-end network service quality assurance for different users and businesses, and can provide differentiated business services for remote surgery. Its determinism is reflected in three aspects: Firstly, security isolation determinism, achieved through logical or physical segmentation of the network using slicing technology, as well as measures such as user access authorization, data storage filtering, and transmission security checks to achieve security isolation. Secondly, latency and jitter determinism. In the 5G era, many network applications such as remote robotic surgery, autonomous driving, and VR games require end-to-end latency to be controlled within a few milliseconds, and jitter to be controlled within the range of seconds. Thirdly, bandwidth determinism. In the era of traffic, there are higher requirements for upstream and downstream bandwidth. Remote surgery has extremely strict requirements for network latency, jitter, packet loss, redundancy protection, and fast switching. Deterministic network is the key means to achieve these standards. It can cooperate with network slicing and edge computing to sink AI and other technologies to the grassroots level, promote the integration of data and 5G’s “cloud-edge-end” functions, fully leverage the advantages and characteristics of 5G independent networking, adjust the network architecture, and meet the overall requirements of remote surgery. Therefore, deterministic network also has good application potential in future remote surgery.

7. Personnel and cost analysis of remote surgery

Generally, the cost of hospital medical services includes labor costs, fixed asset costs, material costs, administrative expenses, business expenses, and other expenses. Remote surgery involves special cost expenditures (such as remote networks) in addition to the general medical service costs.

7.1 Standard configuration for remote surgery resources

Remote Regional Medical Center

Equipment and software: remote surgical robot and its operating system, communication equipment, and data transmission system;

Personnel: one surgeon, one surgical assistant, one equipment maintenance personnel, and one communication transmission maintenance personnel.

Remote Primary Healthcare Institution

Equipment and software: remote surgical robot and its operating system, communication equipment, and data transmission system;

Personnel: two collaborating surgeons, one anesthesiologist, two nursing staff, one equipment maintenance personnel, and one communication transmission maintenance personnel.

7.2 Composition of remote surgery costs

Depreciation and maintenance costs of equipment (including regional medical center and primary healthcare institution equipment): equipment depreciation and maintenance costs of the remote surgical robot system;

Usage costs of specialized instruments and consumables: costs of instruments and disposable materials used multiple times during remote surgical procedures;

Personnel costs: costs of personnel such as the surgeon, surgical assistant, remote collaboration surgeon, nursing staff, anesthesiologist, equipment maintenance personnel, and communication transmission maintenance personnel;

Other costs: depreciation costs of the operating room and auxiliary equipment, and other related consumables.

7.3 Cost-benefit analysis of remote surgery for patients

The hospitalization costs of remote surgery patients include two types of costs: direct costs and indirect costs. Direct costs include medical and non-medical expenses. Medical expenses include various treatment-related expenses, the cost depreciation of the surgical robot system, and personnel costs of remote surgical physicians. Non-medical expenses include living, transportation, and accommodation expenses during hospitalization. Indirect costs include family members’ lost income, patients’ lost income, and other costs. For remote surgery patients, the structure of these costs is the same, but the amount of cost consumption will differ. Conducting remote surgery treatment will reduce the patient’s indirect cost expenditure.

7.4 Economic analysis at the regional medical center level

Hospitals are the main body for introducing and using new medical technologies, and they are also the most commonly used perspective for economic analysis. For hospitals, the cost of remote surgery includes resource costs used during surgery (instrumentation and surgical supplies), drugs, food and lodging, and nursing, as well as indirect costs such as hospital management and operating expenses. Remote surgery is a medical activity that connects experts with patients and medical workers using computer communication technology and medical technology to achieve long-distance data, text, voice, and image data transmission. Considering the characteristics of remote medical services, the cost of remote medical services should include hardware costs, software costs, housing costs, labor costs, and operating costs.

1. Hardware costs refer to the relevant hardware equipment purchased for the remote medical service project at the regional medical center and the patient’s primary healthcare institution.

2. Software costs refer to the development and purchase of software used for remote medical services at the regional medical center and the primary healthcare institution.

3. Housing costs refer to the cost of housing for the remote medical service project at the regional medical center and the primary healthcare institution.

4. Labor costs refer to the cost of human resources required for the remote medical service project at the regional medical center and the primary healthcare institution.

5. Operating costs refer to the costs incurred during the operation and maintenance of the remote medical service project at the regional medical center and the primary healthcare institution.

8. The future direction of telesurgery

The development and application of telesurgery robots has become a new trend worldwide, which helps solve the problem of telemedicine in special geographical areas and special situations, and has great prospects in the future [14].

8.1 The application in regions with scarce medical resources

As an important part of telemedicine, telesurgery can effectively improve the biased distribution of medical resources, coordinate medical resources, and increase the rate of patient treatment and medical resource utilization.

8.2 The application in the field of special environment

In military medicine, in 2009, the U.S. Army developed a complete surgical robot system based on the da Vinci system to cope with the wartime environment to achieve unmanned processing [15], including surgical robot system, management and display system, control and supervision system, monitoring system, equipment replacement system, equipment delivery system, and drug supply system. Although the system is not used in the clinic, the study of the system suggests that telemedicine will enter the era of fully remote surgery in an unmanned mode.

8.3 Driving the development of other areas of technology

Faced with complex cases requiring multidisciplinary surgery to resolve the disease, telesurgery can also be performed many-to-one, with two or even more control systems controlling the same patient. Combined multidisciplinary surgery solves surgical problems of neighboring organs with the same surgical orifice or combines different hospitals to perform remote surgery [16]. In addition, the development of telesurgery can simultaneously lead to the development of other telemedicine disciplines, such as telecare and telerehabilitation after telesurgery [17]. The Telecare Medical Information System (TMIS) utilizes wireless communication technology and smart devices, enabling patients to receive remote medical treatment from doctors via the internet without the need to visit the hospital, thus providing convenience for postoperative rehabilitation care following remote surgeries [18]. It combines healthcare and information technology to achieve electronic medical information management and remote collaboration. Doctors and nurses can record patients’ medical information, and the system supports remote collaboration and consultations. Doctors can remotely access patients’ imaging data, provide remote guidance and diagnostic opinions. TMIS also collects and analyzes medical data, generates reports for medical quality assessment and decision support, thereby improving the service quality and efficiency of healthcare institutions.

It can also promote the development of imaging medicine. Data conversion in telesurgery cannot be achieved without remote proximity systems in the field of imaging, which can present information about the surgical field of view and the surgical environment to the operator in an image-audio format to create a sense of presence [18]. A typical robotic telepresence system includes a light source, a digital image and audio acquisition and processing system, and an intelligent decision and control execution system. The remote presence system has evolved from a simple image-audio acquisition and processing system to an integrated system that incorporates surgical field of view, surgical environment, and other image-audio information with some learning and adaptive capabilities. The way forward now is to combine intraoperative images with patient-specific 3D models and to combine them with virtual/augmented reality imaging.

With limited medical resources in the deep sea and high altitudes, conventional medical resources may not be able to solve problems in a timely and effective manner in case of sudden surgical emergencies. The potential of telesurgery for applications in maritime aviation and space stations is enormous.

Telemedicine can help eliminate distance barriers and provide medical expertise to remote areas. Due to the relative shortage of surgeons and the need to explore new approaches to surgical education, surgical tele-mentoring may be a solution to enhance and improve surgical education models. Although remote robotic surgical teaching may not replace local surgical instructors, studies have demonstrated that it is a valuable tool for remote instruction in minimally invasive surgery.

Telesurgery can serve as a tele-education function. By remotely interrogating multiple surgical specialists and remotely training hands-on surgeons, the professionalism of new hand surgeons around the world can be more effectively enhanced. This can revolutionize surgical education by creating an interactive, scalable and accessible education system with support and guidance from experts around the world [19, 20].

9. Prospect

As an emerging breakthrough technology in the twenty-first century, remote robotic surgery technology has been classified as a major research project by many countries by virtue of its advanced real-time transmission technology and robotic surgery system. Artificial intelligence has great potential for development as a strategic development plan in China, and the development of autonomous surgical capabilities has received a great deal of attention from researchers as one of the development directions of surgical robots. Surgical robots have replaced surgeons into numerous dangerous environments to independently complete remote rescue and treatment work, and have played a significant role in national defense and military, major disasters, future battlefield, and aerospace fields. It is believed that with the increasing volume of remote surgery and further development of artificial intelligence, robotic systems capable of autonomously completing remote surgery will be further developed.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (52122501), the Major Scientific and Technological Innovation Project of Shandong Province (2019JZZY021002), Qingdao People’s Life Science and Technology Project (18-6-1-64-nsh), and the Taishan Scholar Program of Shandong Province (tsqn20161077). The sponsors played a role in the design and conduct of the study and collection of the data.