Wearable Technology for Assessment and Surgical Assistance in Minimally Invasive Surgery

Abstract

Wearable technology is an emerging field that has the potential to revolutionize healthcare. Advances in sensors, augmented reality devices, the internet of things, and artificial intelligence offer clinically relevant and promising functionalities in the field of surgery. Apart from its well-known benefits for the patient, minimally invasive surgery (MIS) is a technically demanding surgical discipline for the surgeon. In this regard, wearable technology has been used in various fields of application in MIS such as the assessment of the surgeon’s ergonomic conditions, interaction with the patient or the quality of surgical performance, as well as in providing tools for surgical planning and assistance during surgery. The aim of this chapter is to provide an overview based on the scientific literature and our experience regarding the use of wearable technology in MIS, both in experimental and clinical settings.

1. Introduction

There is a wide variety of wearable devices, such as smartwatches, wearable mobile sensors, mobile hub medical devices, among others. This technology makes it possible to collect data on the user’s health status. A widespread example of the use of this technology is the detection systems of blood glucose levels in diabetic patients [1]. However, wearable technology has also been used in other fields of clinical application, such as minimally invasive surgery (MIS). In this case, this type of technology could make it possible to evaluate the surgeon’s ergonomic conditions, the interaction with the patient or the quality of the surgical performance, as well as to provide tools for medical training and surgical assistance.

Apart from the numerous advantages of MIS for the patient, these surgical techniques present several limitations for the surgeon. Some of these challenges include the loss of depth perception due to two-dimensional vision, awkward postures during surgery due to restricted movements, and decreased tactile sensation. All this leads to an increased mental and physical burden on the surgeon during surgery, as well as the possible onset of musculoskeletal disorders. The constant evolution of wearable technology allows for a comprehensive analysis of these parameters in order to improve the surgeon’s ergonomics during surgery, and therefore the patient’s surgical outcomes [2, 3].

Medical education is a long and demanding process which involves learning complex theoretical and practical aspects. During its early stages, training methods are often based on static and unrealistic learning content. Currently, these methods are being replaced thanks to advances in information and communication technologies. In this regard, new technologies, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), present the potential to provide medical students with interactive and realistic training contents using head mounted displays (HMD) [4].

During surgical training, the assessment of the technical surgical skills to be acquired by novice surgeons has traditionally been performed by expert surgeons, being a subjective assessment that may be biased. Recent advances in the Internet of Things (IoT), the ability to embed sensors in objects and environments to collect large amounts of data, and advances in machine learning allow for a more objective and automated assessment of MIS skills [5].

On the other hand, advances in preoperative imaging systems have made it a fundamental element in surgical planning [6, 7]. In particular, when facing complex surgeries, surgical planning provides valuable information to predict and reduce any potential risks during surgery, thus improving its safety levels. The application of wearable technology (mainly HMD) in this area provides the surgical team with access to this information in situ and without compromising the asepsis of the surgical procedure [8]. The use of three-dimensional (3D) representations of this data in immersive environments provides new ways to explore the patient information and further enhance the tools available to medical professionals in several areas, such as medical training, surgical planning and intraoperative guidance.

In this chapter we will review some of the aforementioned technologies and medical applications, both described in the scientific literature and those developed by our research group.

2. Evaluation of surgeon’s ergonomics

2.1 Physiological parameters

Physiological sensing remains one of the most challenging topics in minimally invasive biomedical signal acquisition today. The clinical gold standard protocols needed to measure biochemical and physiological parameters often require invasive and time-consuming techniques, thus lacking the real-time and comfort factor requirements present in wearable technology.

To date, the most commonly used physiological parameters in the context of wearable technology are still heart rate variability (HRV) [9], as a function of heart rate, and galvanic skin response [10]. Other alternatives are biomarker analysis measured from eccrine sweat [11] and saliva [12], as well as the well-known electroencephalogram (EEG) [13] and surface electromyography (sEMG) [14] techniques.

Smartwatches in particular present a suitable solution as they offer the possibility of acquiring heart rate and motion data simultaneously from the user. Although heart rate sensors included in smartwatches are not adequate for medical use or diagnosis, they offer a quick and affordable approach to measure parameters like HRV.

In MIS field, we have made use of smartwatches (Figure 1) to monitor hand motion and HRV data of surgeons during the performance of laparoscopic training tasks. Apart from evaluating the surgeon’s physiological parameters and performance, we sought to build a machine learning model that could predict stressful situations and quality of surgical performance during laparoscopic practice [2, 15]. Although more data is required in order to build a reliable model, smartwatches are proving to be a promising alternative to more invasive and expensive solutions that are often used in these types of studies.

Figure 1.

Smartwatch being used during laparoscopy to acquire surgeon’s hand motion and HR.

2.1.1 Muscular activity

The level of effort involved in performing work tasks in a given job is a risk factor for musculoskeletal pathologies. In this sense, different instrumental techniques and wearable systems have been used to evaluate this factor, such as sEMG. Surface electromyography makes it possible to characterize the intensity of muscular intervention of a particular muscle and to identify the onset of muscle fatigue [3]. In the surgical setting, in a study by Wong et al. [16] it was observed that 83% of surgeons showed musculoskeletal symptoms during microlaryngeal surgery, especially in the neck and upper back. However, the assessment of fatigue in microsurgical techniques is especially difficult due to the minimal range of body movement and the low amount of muscle activation required for the performance of the surgical tasks, making them challenging to analyze.

We have studied the effect of experimental microsurgery training on surgeon ergonomics. For this purpose, a total of ten surgeons of different levels of surgical experience were evaluated during the performance of eleven anastomoses in a simulator. A wireless sEMG system (Delsys Trigno; Natick, MA, USA) was used to evaluate the muscle activation of the analyzed muscles (paravertebral, upper trapezius, lower lumbar, and supinator longus muscles) (Figure 2). All surgeons showed improvement in their surgical performance. The results revealed that novice surgeons showed, on average, a higher level of muscle activation than intermediate and expert surgeons. In addition, novice surgeons showed greater activation in the muscles on their dominant side, whereas expert surgeons had similar levels of activation on both sides.

Figure 2.

Use of the Delsys Trigno™ EMG wireless system as a tool for recording sEMG and motion data to assess the surgeon’s posture and workload in microsurgery.

2.2 Kinematic parameters

The adoption of forced postures for prolonged periods of time is one of the well-known limitations of laparoscopic surgery. This is why the analysis and evaluation of the working conditions in the operating room (OR) is essential to improve the its ergonomics during MIS. To perform these ergonomic studies, which have traditionally used techniques including photogrammetry and cumbersome EMG systems, techniques based on wearable technologies such as body tracking systems, data gloves, electrogoniometers, smart devices or pressure sensors are now being used [3, 17]. The ultimate goal is to provide practical objective and reliable ergonomic criteria during surgical activity.

Kinematic parameters are associated with the motion of body parts. The technology related to kinematics has been widely studied and implemented in multiple fields as diverse as aeronautics, navigation, video games, or health, among others. Several approaches can be found depending on the needs of the clinical application. The most popular include the use of acceleration sensors, followed by optical tracking solutions. These solutions range from a simple and affordable smartwatch that can measure a subject’s hand accelerations from accelerometer sensors [2], to more complex and expensive systems such as the Xsens™ (Xsens Technologies B.V.) that uses inertial sensors [17].

2.2.1 Full body posture

The application of wearable technology for the analysis of full body posture is carried out using markers or sensors placed on the subject’s body to quantify the movements of the different body segments. In the case of optical tracking techniques, use is made of a set of retroreflective markers identified by a system of cameras and infrared light [17]. The main limitation of these solutions is the occlusions they may have in the working environment, which is worsened in crowded settings such as an OR.

In recent years, the introduction of inertial sensors has facilitated the application of kinematic analysis techniques in more complex environments as the OR. These inertial measurement systems are not affected by visual occlusions, making them suitable for clinical settings (Figure 3). This is a more efficient instrumental technique for postural characterization in MIS than conventional techniques such as 3D photogrammetry [3]. In this sense, a study was carried out in which the body posture of 8 laparoscopic surgeons (4 novices and 4 experts) was analyzed by means of the Xsens™ system during the performance of laparoscopic suturing tasks. The kinematic results showed a high variability of the surgeons’ body posture, with a coefficient of variation greater than 70% in all the joints, especially in flexion-extension of the wrist and its radial and ulnar deviation [3].

Figure 3.

Motion tracking system based on inertial sensors placed on the surgeon’s body (left). A biomechanical model of the subject is created in real time (right). (Source: [17]).

2.2.2 Hand posture

Surgical tasks and instruments directly affect the position of the surgeon’s wrist and hand. Data gloves allow comprehensive analysis of the subject’s hand movements (Figure 4A). Typically, such systems employ electromechanical technology or conductive sensors.

Figure 4.

Use of the CyberGlove™ device (A) and an electrogoniometer (B) for recording the surgeon’s hand posture during laparoscopic practice. (Source: [17]).

In the clinical setting, the CyberGlove™ system (CyberGlove Systems, San José, CA, USA) has been used to analyze the surgeon’s hand movements while using different surgical instruments during laparoscopic practice [18]. The data were analyzed according to an adaptation of the Rapid Upper Limb Assessment (RULA) ergonomic evaluation method. In this study, the hand posture of experienced surgeons in laparoscopic surgery was analyzed during the handling of various handle designs for laparoscopic instruments: axial, axial with rings, and pistol. For all handle designs analyzed, most of the surgeons showed unfavorable flexion-extension angles, with the pistol handle being the most ergonomically suitable design. In another study, instruments for laparoendoscopic single-site surgery (LESS) were found to be ergonomically more suitable than conventional laparoscopic instruments [19].

Wrist flexion-extension and radioulnar deviation during surgical practice can also be recorded using electrogoniometers (Figure 4B). These are devices whose measurement signals (usually electrical voltages) are directly related to flexion-extension or rotation between body segments. They must be precalibrated to relate the measured voltage to the angles described by the analyzed joint. As with data gloves, the information provided by electrogoniometers would allow us to measure the degree of risk of the surgeon’s posture. However, in both cases these devices are difficult to integrate into the surgical environment, and are relegated mainly to experimental studies. This technology has been used to objectively study the surgeon’s wrist posture in the use of new handle designs for laparoscopic instruments [20].

2.2.3 Human-instrument interaction

Another relevant risk factor in the surgical context is the localized contact pressure during the use of laparoscopic instruments. Excessive pressure, repeatedly or for long periods of time, can result in nerve damage to the surgeon’s fingers. In order to analyze this factor during laparoscopic practice and thus potentially improve the ergonomic conditions of the surgeon and the design of the surgical instruments, wearable systems such as the FingerTPS™ (Pressure Profile Systems, Inc., Hawthorne, CA, USA) can be used. This system makes it possible to measure and map the contact pressures exerted on or by the fingers and the palm of the hand when using the laparoscopic tools (Figure 5). This technology makes it possible to assess whether pressure levels are detrimental to the surgeon and to draw conclusions about the most appropriate design for the laparoscopic instruments.

Figure 5.

Use of the FingerTPS™ system during the handling of laparoscopic instruments with a ring handle. (Source: [3]).

In a study, we evaluated the pressure exerted by the surgeon’s distal phalanges of the thumb and the index, middle and ring fingers, as well as the palm of the hand during the use of new handle designs for laparoscopic instruments and with different sizes [20]. The participating surgeons performed three laparoscopic basic tasks to analyze the influence of handle size. The results showed that there was a significant increase in palmar pressure with the incorrectly sized instrument handle.

On the other hand, in another study we compared the grip pressure in the use of a robotic laparoscopic instrument (Dex Device™; Dex Surgical, Verrières-le-Buisson, France) with respect to conventional laparoscopic tools during urethrovesical anastomosis in experimental model [3]. The results showed that the pressure exerted by the thumb was significantly higher during the use of the robotic instrument. This was due to the interaction with the controls installed on the instrument handle. Additionally, the results showed that the force exerted by the distal phalanx of the index finger was significantly higher with the conventional handle. The palm of the hand was the area that received the greatest pressure while using both instruments.

3. Telementoring and surgical assistance

3.1 Telementoring

Telementoring, defined as mentoring by means of telecommunications and computer networks, allows an experienced physician to directly provide information and share knowledge at a distance with a less experienced practitioner. This is a safe modality for delivering intraoperative surgical education and provides some equivalence to on-site mentoring with regard to clinical and educational outcomes [21].

Meijer et al. demonstrated the feasibility of using wearable technology in combination with the TedCube device (TedCas Medical Systems; Pamplona, Spain) for hands-free interaction with the computer during the course of surgery and for telementoring purposes. The combination of wearable sensors, an integrating device and internet-based remote desktop sharing software proved a feasible set-up for telementoring in situations when asepsis for both the mentor and the mentee is necessary, and distance needs to be overcome. A successful connection without any downtime was established between the Academic Medical Center in Amsterdam, The Netherlands, and Jesús Usón Minimally Invasive Surgery Centre (JUMISC) in Cáceres, Spain [22].

3.2 Surgical assistance

In MIS, preoperative imaging studies are fundamental to facilitate diagnosis and surgical planning. These personalized patient studies allow analysis of anatomical details prior to surgery, improving the course of the procedure. However, surgical environments require very strict aseptic conditions. This makes it difficult to use traditional interaction devices such as keyboard and mouse to consult preoperative patient information during surgery. In addition, some equipment requires the surgeon to leave the OR to access preoperative images, which hinders the surgical process.

The use of wearable systems could offer suitable solutions to these limitations and help maintain surgeon asepsis while interacting with the patient’s preoperative data during surgery. One possible device is the MYO armband (Thalmic Labs Inc., Kitchener, ON, Canada), which is worn on the arm (placed just below the elbow). The MYO is equipped with 8 EMG sensors that allow it to recognize the user’s hand gestures and arm movements. In a study during the development of several laparoscopic procedures, we demonstrated the feasibility of using a gestural control system by means of the MYO device in conjunction with voice commands to interact with the patient’s preoperative imaging studies while maintaining aseptic conditions [23].

4. Head-mounted displays

Head-mounted displays (HMD) are, of all the devices that allow viewing digital content, the ones that achieve greater user immersion in digital content. Technologies such as stereoscopy (projecting a different image of the same environment in each eye, thus achieving a sense of depth) allow user immersion in a virtual environment. The possibility of using hand gestures, voice commands or eye-tracking devices that allow interaction with the environment represents a paradigm shift since, without these elements, HMDs would be passive devices, being able only to display content but not to interact with it.

These wearable devices allow the visualization of content through different types of technologies [24]. At one extreme is Virtual Reality (VR) where all the content displayed is completely digital and the real environment is ignored. At the other extreme is Augmented Reality (AR), in which the environment is visualized and layers of digital information are added based on what is seen. Mixed Reality (MR) is in the middle ground, as it displays computer-generated content that is aware of the environment, allowing users to interact with the content but taking into account the environment around them. VR can be a useful tool for various simulators or educational applications, although it has limitations in terms of assistance during surgical applications (Figure 6).

Figure 6.

VR application oriented to anatomical education.

Until 2016, Google Glasses (Google Inc.; Mountain View, California, USA) [25] was the most widely used HMD for viewing digital content. However, several publications have mentioned the limitations of this device for its application in the medical field. Some of them are its restricted functionalities, low computational capacity, poor display resolution, information projected on a single eye (which negates stereoscopy), and usability limitations due to its incompatibility with the use of glasses or privacy aspects [26]. Over time, MR devices have become a standard, as they allow the inclusion of holographic images or 3D objects, which are displayed in the user’s field of view and integrated with the real environment, improving the immersive experience during work.

4.1 Surgical training

In the educational field, most VR applications dedicated to surgeon training have a fully immersive nature, due to the inherent characteristics of this type of technology. Simulation using devices such as Oculus Rift (Oculus VR; Menlo Park, CA, USA) offers complete immersion and VR controls mimic the surgeon’s surgical tools [27]. Other applications are able to offer a much more detailed visualization of the human anatomy using HMD devices such as HTC Vive (HTC Corporation; Xindian, Taiwan) [28]. In most cases, this methodology offers possibilities for training outside the OR, collecting surgical performance metrics from the user as if it were a serious game [29]. These solutions allow procedures to be repeated without any restriction, reducing training costs [30], and obtaining also progression feedback for each user, thus enabling the improvement of less developed surgical skills.

In the case of Google Glass, these have been used mainly as a means of sharing with students the surgeon’s view during his/her performance, along with real-time comments, which allows the generation of comprehensive content for learning [31]. Finally, it is worth mentioning the use of this device together with an ultrasound probe for the visualization of the 3D ultrasound images generated, facilitating the learning of human anatomy [32].

Regarding AR/MR devices, most training applications are based on the visualization of anatomical elements [33]. As the main potential of these technologies lies in knowing the environment and coupling spatially augmented information to it, numerous applications opt for using visual markers and superimposing digital content on them [34]. On the other hand, it is worth highlighting the wide possibilities offered by these devices in telemedicine and remote medical training [35]. This technology offers a significant improvement in 3D perception compared to two-dimensional illustrations or more traditional training content (Figure 7A). MR devices, such as HoloLens (Microsoft; Redmond, Washington, USA), make it possible to scan the working environment (Figure 7B) and provide 3D anatomical models, as well as organize them into different systems (nervous, muscle, bone, or vascular) (Figure 7C). The user can interact (move, rotate or scale) with the holograms by means of gestures or voice commands (Figure 7D). These applications have been tested in various contexts by expert surgeons at the JUMISC, concluding that the visualization of 3D anatomical models using this technology is useful and facilitates the transfer of knowledge to real clinical practice [36].

Figure 7.

MR application for anatomical training: (A) traditional simulator with medical illustrations; (B) spatial detection of the simulator using the HoloLens depth camera; (C) selection of the anatomical element to be displayed; (D) example of gestural interaction with the 3D hologram.

4.2 Surgical assistance

In relation to surgical assistance, Google Glass emerged as a means of broadcasting the surgeon’s vision for telementoring applications and requesting expert opinion outside the OR [37]. On the other hand, this device has also been used to provide the endoscopic image in place of traditional laparoscopic monitors. Another surgical assistance application is the display of checklists to proceed in a more orderly and safe way with the surgical procedure [38]. Various sensors of this device (camera, microphone) have also been exploited in combination with other devices such as Fitbit™ (Fitbit, San Francisco, California, USA) to collect data and facilitate surgical evaluation and performance [39]. Of note is the feasibility, safety and usability study by Borgmann et al. in which preoperative studies were shown during ten types of urological surgery procedures [40].

Since 2016, the use of Microsoft’s HoloLens glasses has been gaining presence in the field of AR applied to surgery to the detriment of Google Glass. This device marked a milestone in AR HMD devices, even defining a new term (mixed reality -MR-) since, due to the built-in depth cameras, the device is aware of both the surrounding spatial environment (Figure 8A and D) and the gestures that the user performs with the hands even when holding the surgical tools (Figure 8B and C), thus resulting in a more complete integration of the information displayed on the glasses with the OR. The nature of this device makes it an exceptional choice for use in surgical assisting applications. Recently, applications have been presented on the use of this type of device in MIS, in kidney and prostate surgical procedures [41], lung, and uterus [42], among others.

Figure 8.

MR application for surgical assistance: spatial detection of the surgical environment (A and D); detection of the surgeon’s hand gestures (B and C).

The Remote Assist application (Microsoft) for HoloLens has recently been presented, which could facilitate medical training and telementoring tasks. This application is capable of video calls sharing the surgeon’s point of view, allowing both communicating surgical knowledge with trainees and consulting with clinical experts in real time. The application accesses the depth camera to spatially locate 3D elements in the OR environment, allowing to point out elements such as surgical material or equipment or to explain the operation of the equipment in an intuitive way (Figure 9). In addition, the user using the device is be able to see the expert/apprentice in the form of a hologram (Figure 9D). Due to the recent need for distance learning/mentoring solutions, these tools are emerging as useful alternatives.

Figure 9.

Use of the Remote Assist (Microsoft) application: spatial markers (A–C); video call sharing the surgeon’s point of view (D).

MR technology can also be used for surgical planning. It could facilitate visualization and analysis of the patient’s preoperative imaging studies for a better and safer approach to surgery. MR devices, in combination with new emerging medical imaging techniques, have been successfully applied as a planning tool in different surgical disciplines such as urology [43], thoracic surgery [44], neurosurgery, colorectal surgery, and bariatric surgery [45], among others.

Surgeons can interact with the 3D models generated from preoperative studies of the patient for a better understanding of the anatomy to be operated on (Figure 10B). Recent studies have managed to visualize in real time the preoperative studies stored in DICOM format, allowing a more complete and interactive view in the form of a hologram. These applications make it possible to visualize the preoperative imaging study based on the density of each point, as well as to perform filters on the point cloud (Figure 10A, C and D). We have used this surgical planning application during a laparoscopic renal tumorectomy in experimental model [46] and during a laparoscopic lobectomy at the University Hospital of Cáceres (Spain) [47]. Surgeons highlighted the usefulness of the application for planning laparoscopic procedures, although they also reported some ergonomic limitation of the MR device.

Figure 10.

MR application used to visualize preoperative studies: Visualization of the volumetric point cloud (A, C, D); visualization of the 3D model (B).

Another application of this technology for surgical assistance is its use as an additional monitor during surgery. Ten surgeons evaluated this function during the performance of a simple laparoscopic training task (object transfer task) (Figure 11A, C and D). For this purpose, the participants used the HoloLens v2 glasses as a holographic monitor, instead of a conventional laparoscopic monitor. Most surgeons concluded that they could perform more complex tasks using the holographic monitor. The application achieved positive results in terms of latency, image quality and user experience. The surgeons showed a slight learning curve when it came to spatially positioning the monitor (Figure 11B), mainly due to the fact that in most cases it was their first experience with this type of device.

Figure 11.

Use of MR application as a holographic monitor during laparoscopic practice: setting of the study (A); positioning and scaling of the holographic monitor (B); start and end of the transfer task (C and D).

5. Conclusions

The constant evolution of wearable technology has boosted its application in the surgical field, especially in minimally invasive surgery. This technology allows an exhaustive analysis of the surgeon’s physiological and ergonomic conditions during the surgical practice, improving the surgeon health and surgical equipment design, and therefore surgical outcomes for the patient. On the other hand, this technology has led to a paradigm shift in medical training, taking the student to the same operating room in which an intervention is being performed or offering holographic and interactive 3D anatomical models close to reality. In addition, head-mounted displays offer surgeons advanced tools for surgical planning, providing access to the patient’s preoperative information in the operating room, while maintaining aseptic conditions. Undoubtedly, developments in sensors, data analysis techniques, artificial intelligence and mixed reality will continue to offer new and innovative solutions to clinical needs.

Acknowledgments

This study has been partially funded by the MISTI Global Seed Funds, “la Caixa” Foundation, Junta de Extremadura (Spain), European Social Fund, European Regional Development Fund (ERDF) “A way to make Europe”, and the Spanish Ministry of Science, Innovation and Universities through ERDF funds of the Intelligent Growth Operational Program (LCF/PR/MIT18/11830006, TA18023, PD18077, GR18199, CPI-2019-2033-1-TRE −14).

Conflict of interest

The authors declare no conflict of interest.