VR and AR simulator examples with respect to each educational field.
Virtual reality (VR) and augmented reality (AR) are two contemporary simulation models that are currently upgrading medical education. VR provides a 3D and dynamic view of structures and the ability of the user to interact with them. The recent technological advances in haptics, display systems, and motion detection allow the user to have a realistic and interactive experience, enabling VR to be ideal for training in hands-on procedures. Consequently, surgical and other interventional procedures are the main fields of application of VR. AR provides the ability of projecting virtual information and structures over physical objects, thus enhancing or altering the real environment. The integration of AR applications in the understanding of anatomical structures and physiological mechanisms seems to be beneficial. Studies have tried to demonstrate the validity and educational effect of many VR and AR applications, in many different areas, employed via various hardware platforms. Some of them even propose a curriculum that integrates these methods. This chapter provides a brief history of VR and AR in medicine, as well as the principles and standards of their function. Finally, the studies that show the effect of the implementation of these methods in different fields of medical training are summarized and presented.
- virtual reality
- augmented reality
- medical education
- advanced learning
Virtual reality (VR) and augmented reality (AR) are the current trends in medical education. VR is the virtual construction of an artificial world. The key element of VR is the high level of user’s immersion over the virtual environment, namely, the high level of structures’ fidelity, as well as the interaction of the user with them in a realistic manner. This has become recently available, as it requires high standards of certain technologies, including advanced haptic devices with force-feedback capabilities (bidirectional stimuli), high-resolution audio-visual effects, motion detection technology, and high-performance processing power to transmit, and processes all this information with near-zero latency.
The first virtual system in medicine was introduced in 1965 by Robert Mann, in order to facilitate a new training environment for orthopedic. In the late 1980s, the head-mounted display (HMD) was introduced as a wearable device for VR visualizations in medicine . The first pioneering applications in medical education were some on hands-on procedures that appeared over a decade later [2, 3]. In the 1990s, many adverse effects from the use of VR were reported. Nausea, dizziness, temporarily impaired vision, and lack in the sense of presence, even after 20 min of use, were some of them. The adverse effects were attributed to technical defects, such as the lag time and the inability of the human eye to fixate in-depth to “artificially distant” 3D objects [4, 5]. Although these effects are minor and subside quickly nowadays, the risk of learning inappropriate handling moves, when low-fidelity or badly simulated models are used and particularly under unsupervised teaching, still remains .
Although AR and VR share many technical aspects, AR differs from VR as its target is not to construct a fully artificial environment but to overlay computer-generated images onto images of the real world . Therefore, it uses machines that allow physical view of the surrounding environment to be visible but enhanced with virtual images. Tablets, mobile phones, AR glasses, and other optimized devices can be employed as hardware for running AR applications. Historically, the development of AR starts in the 1960s, but the term “augmented reality” was established in 1990. The increase in the use of AR came along with the technological advances that made it available and useful. Nowadays, AR is used widely in the clinical setting, providing extra information for the clinician during interventional procedures (CT/MRI guidance, visualization of paths), but it is also applied in the educational world. Anatomy, with the 3D visualization of hardly comprehensible structures, and physiology, with the representation of mechanisms in 4D (in space and time dimensions), are the main areas of interest .
2. Technical aspects
VR and AR come with certain hardware requirements in order to retain high standards of simulation. The computerized 3D pictures and audio have to be realistic and simulate both real and abstract structures. The motion of the user should be detected with high precision so that the visual field (size, shape, angle of objects) and auditory stimuli (volume, sound balance) can change accordingly. The user must be able to affect the virtual environment but also to be affected via haptic feedback stimuli. Haptic devices, such as joysticks, gloves, and other specialized tools, serve these haptic interaction needs. High-performance computer power is needed to process the huge amount of information produced, with low, unnoticeable latency. All these contribute to the key element of VR and AR: immersion of the user over the environment. Moreover, fast and reliable Internet connection is needed to support the latest trend in VR field, namely, the establishment of VR fora where users meet and interact with each other online, in a common virtual environment [1, 3, 9].
Many VR settings involve HMD. HMD is a wearable device that offers visualization of the constructed virtual reality in wide angle and does not permit the disturbance of the virtual experience by external visual stimuli. Although the first HMDs had a restricted resolution of 800 × 600 pixels and a narrow vision angle of 30°, the current HMDs display fully high-definition images, with angles that can reach 360°. Some of them are wireless and may incorporate position and motion detection systems for the eye and head movements. Although HMDs play a significant role in many VR settings, they are not mandatory for all medical applications. The simulation of many procedures, such as laparoscopic or endoscopic, requires the display of the virtual images on a computer screen . Sound quality is also important for a complete VR experience. Volume and balance adjustments according to the moves of the users are significant elements of the VR environment .
Input devices consist of all the devices that transmit the user’s stimuli to the VR system. Gloves, joysticks, or other specialized tools (laparoscopic or endoscopic instruments) are usually employed. A special subcategory of necessary input devices for VR is sensors and trackers. Sensors and trackers identify the position and the direction of the user in space as an input stimulus, allowing interaction. They employ various technologies such as lasers, infrared radiation, and mechanical detectors and can detect the properties of the whole body, the head, or even the subtle gestures and moves of the hands, ability that is extremely useful in medical procedures’ simulation. The recent technological improvement of these devices, which made the VR setting suitable for medical training, is the tactile-feedback ability. Adjusted force is exerted via the handling instruments, depending on the hand moves. Thus, the operator has a more realistic, bidirectional haptic experience when performing a virtual surgical process or other interventional processes .
Most AR hardware use glasses that project virtual 3D images onto the real environment. While
3. Quality control and educational impact
VR and AR simulators must possess quality standards so as to be suitable for medical education purposes. However, even if these standards are met, the educational benefit is still not guaranteed a priori. There are many published studies that try to test VR and AR simulators in specific educational settings (concerning different medical fields and stages of training) in order to examine their validity, the transferability of the taught skills to the real world, the acceleration of the learning curve, and the retention period of the skills. Following the determination of these parameters, establishing a curriculum that integrates this evidence in a productive and cost-efficient manner is an essential consequence.
High-resolution images, quality in sound, haptic input and feedback devices, and high processing power are the parameters that are required to create realistic VR/AR environment. Moreover, the structures of the organs and tissues must be of high fidelity, and the change in their shape, size, and angle of view must correspond completely to the user’s moves and handlings. Otherwise, there is a risk that the moves and skills learnt will be inappropriate in real situations . ISO criteria have also been established. They include initial assessment of the educational needs and establishment of a technical solution. This requires the in-depth knowledge of the procedures or mechanisms that will be simulated. After the concept generation, the setting must be validated, in order to examine if it meets the educational expectations. Moreover, technical experts, medical experts, representatives from end-user groups, human factor experts, and designers should collaborate and form an interdisciplinary team that will design, evaluate, and upgrade the VR/AR products [11, 12].
Furthermore, VR and AR systems meeting these standards have to undergo post-market validation studies, in order to examine if they provide the educational effect they are expected to. In terms of medical education, the studies examine the following types of validity: (1) face validity—usually the end users make subjective comments as feedback for the quality and realism of the setting. (2) Content validity: once again, it is subjectively defined by the users’ comments but with emphasis to the content of the simulation (procedures, fidelity of organs, etc.). (3) Construct validity: this employs objective statistical methods that aim to correlate the actual skill level of the participants with their performance in the VR setting. Usually, there are two or more groups that differ only to the level of skills. Their performance in the VR/AR setting is evaluated, either by external human evaluators or, if it is feasible, by the VR system itself. If there is a correlation, that means that the VR setting can effectively discriminate between novices and experienced users. This is extremely useful when using VR/AR settings as assessment tools for the progress of trainees, as they provide an objective evaluation that may not require the need of human evaluators. (4) Concurrent validity: another objective validity control that aims to correlate the performance in the VR/AR setting with other existing evaluation tools. This practically checks whether the VR/AR evaluation results point to the same direction with other evaluation tools that are considered as gold standards. (5) Predictive validity: the last but probably most useful objective validity control. Two homogenous cohorts of trainees are formed. After that, only one receives a specific VR/AR training, with predefined parameters (such as the simulator model, the procedure/topic learnt, the time, the frequency and the difficulty level of the procedure, and the presence of a supervisor). On the contrary, the other cohort receives training with differently optimized parameters (perhaps another VR/AR simulator or even a completely different simulation method) or does not receive training at all and is used as a control group. Subsequently, the skills of both cohorts are evaluated in real patients or other simulation settings that are considered as gold standards with proven discriminating ability. If a correlation exists, the VR setting can have an educational effect that can be indeed transferred to real patients. Some studies evaluate the effect of the training after a long time period to examine the retention period of the skills [8, 13].
If there is confirmed predictive validity of a VR/AR simulator for acquiring a specific skill set, then the next step is to wonder: “How much does the VR/AR training contribute to the acceleration of the acquired skills?” This is a more complex question that can be approached with learning curves. A learning curve is a graph that correlates the level of acquired skills with the amount of effort exerted (measured in time, number of trials or procedures performed). In general, it is desired that the learning curve has a high plateau, meaning that the final level of skills acquired with this method will be high, but also is steep, meaning that there is an early high acceleration of skills with little effort. If a trainee that has completed the training in the VR setting (namely, he/she has reached the plateau of the VR/AR learning curve) and then is able to receive training in other settings (e.g., other simulation settings or real patients) from a higher than the basic level, then we assume that the VR/AR training offers transferable skills and has “accelerated” the learning curve. The common characteristics of the learning curves on most VR/AR simulators are the low plateau, the transferability of the skills, and the steep shape. Low plateau means that the total level of skills that can be acquired is lower than the one with other education methods; thus, the VR/AR is usually appropriate for novices, but is not of any significant benefit for more experienced users. Fortunately, it usually has a steep shape and transferability properties, meaning that these skills are acquired quickly and then can be transferred to other education settings. In other words, VR/AR training compensates for some training period in the initial steps of other educational settings.
After these VR/AR setting-specific properties are evaluated, the next step is to define a curriculum that incorporates VR/AR methods in education. This curriculum must also take into account the cost of both VR/AR and alternative ways of training, as well as the patient risk for both settings. For example, if there is a high risk in training with real patients or high cost with animal-simulation models, a VR/AR simulator (with low plateau and steep learning curve) can compensate quickly for the initial steps, offering a safe, relatively inexpensive, and reproducible alternative choice that may provide transferrable skills.
As there are many validation and evaluating studies in literature, an overall profile of VR/AR applications can be formed. In general, it seems that VR offers an appropriate environment for training in hands-on procedures. As Table 1 shows, laparoscopic surgery, neurosurgery, catheterization techniques, and endoscopic procedures seem to be the main fields of VR applications. Novices are the most benefited trainees, while there is not a significant learning effect in experienced users. VR offers great diversity with only one setting. The educational setting can be reproducible with a variety of different optimization details, not restricted by the physical world . Although VR simulators are expensive to purchase, they can be used recurrently with low cost, require less staff than other methods (animal models or real patients), and guarantee no risk for the patient [8, 12]. However, there are certain disadvantages in their use. In the common case of crashing, the virtual experience is destroyed. From a technical point of view, they are still not ready to present a virtual environment with absolute fidelity, as if it was real. Last but not least, wrong handling can be learnt, particularly under unsupervised learning. The VR simulators can reconstruct a whole virtual surgery, specific procedural tasks, or even more abstract procedures .
|Fields of educational training||VR and AR simulator examples|
|Physiology and anatomy||Visible Human Project, Visible Korean Human, The Virtual Body, The Virtual Human Embryo, The Visible Human Server, AR glasses, mobile phone and tablet based AR applications|
|Open surgery||Virtual Reality Educational Surgical Tools|
|Laparoscopic surgery||MIST-VR, LaparoscopyVR™, LapMentor™, LapSim™, SINERGIA, Xitact LS500®, ProMIS®|
|Robotic surgery||RoSS™, DV-Trainer®, SEP Robot, da Vinci Skills Simulator (dVSS) ™|
|Oesophagogastroduodenoscopy, colonoscopy, ERCP||GI Mentor™, EndoVR™, Olympus Endo TS-1|
|Neurosurgery||NeuroVR (NeuroTouch, NeuroTouch Cranio)™, ImmersiveTouch®, RoboSim, Vascular Intervention Simulation Trainer®, EasyGuide Neuro, ANGIO Mentor™, VIVENDI, Dextroscope®, Anatomical Simulator for Pediatric Neurosurgery|
|Interventional cardiology and cardiothoracic surgery||ANGIO Mentor™, Vascular Intervention Simulation Trainer (VIST)®, Vimedix (equipped with Hololens) ™, Nakao Cardiac Model, Minimally Invasive Cardiac Surgery Simulator, dVSS™, EchoCom|
|Urology||URO Mentor™, University of Washington TURP Trainer, UROSim™, PelvicVisionTURP simulator, GreenLight laser simulator, Kansai HoLEP, ProMIS®|
|Orthopedics||ImmersiveTouch®, Phantom haptics interface®, Gaumard HAL S2001® and S3000® Mannequins, Novint Falcon®, Medtronic model, Arthro-VR®, Arthro MENTOR™, ArthroSIM, ArthroS™|
|Endovascular surgery||ANGIO Mentor™, Vascular Intervention Simulation Trainer (VIST)®, Cardio CT, SimSuite, Compass 2™|
|Obstetrics and gynecology||HystSim™, EssureSim™, AccuTouch (and newly version from CAE Healthcare), MIST-VR, LapSim™|
|ENT||OtoSim™, VOXEL-MAN (supports Phantom haptics), Ohio State University surgical simulator, Stanford Surgical Simulator, Mediseus, ImmersiveTouch, Endoscopic Sinus Surgery Simulator (supports Phantom haptics), Dextroscope®, dVSS ™|
|Ophthalmology||EyeSi®, MicrovisTouch™, PhacoVision®|
|Intubation and bronchoscopy||EndoVR Simulator™, BRONCH Mentor™, ORSIM®|
Moreover, many of the simulators have incorporated assessment tools of the user’s performance. They evaluate parameters like path distance, operation time, moves, and errors and then deliver quantified, timely, and objective metrics. If these simulators meet construct validity criteria, they can be used to objectively assess the trainee’s performance. However, if this ability is not supported, human experts can also play the role of the evaluator, and studies have also shown acceptable metrics .
Although AR finds many clinical applications in medicine, its implementation in medical education is not so extensive (see Table 1). However, tablets and mobile phones have AR applications that enrich traditional reading with pop-up videos, links, and interactive material . Moreover, in medical training there are AR applications that project traumas and other lesions onto healthy humans, aiming at training students and residents in the management of these situations. However, interventional procedures are not performed, and the AR applications are restricted in teaching the theoretical context of these subjects in an increased reality setting .
4. Applications in educational fields
4.1. Preclinical teaching
Anatomy and physiology are the classic paradigms that have the majority of VR/AR applications . A checkpoint in the development of VR/AR applications that study human structures and mechanisms is the construction of massive online databases that contain human images and information, mostly obtained with CT/MRI scanning. The first relevant project, called the
VR and AR find most of their applications in surgery and particularly in laparoscopic surgery training. Endoscopic procedures and neurosurgery are also popular. Fidelity and realism are extremely important factors in these fields, not only for the comfort of the user but also for guaranteeing that no improper handling will be learnt and that the actual stressful conditions will be recreated. VR/AR simulators demonstrate certain advantages, such the minimal cost per use, the absence of ethical issues, and the safety compared to training on actual patients. Moreover, greater diversity and complexity in the procedures can be achieved. However, the cost of purchasing and maintaining is high, and also despite the rapid advances in VR/AR technologies, the realism still fails in representing the operating room settings with high fidelity .
4.2.1. Laparoscopic surgery
The current VR laparoscopic simulators are characterized as “hybrid” as they have real instruments and a virtual operating field, projected on a screen or other display devices. They provide haptic feedback to users according to their handlings, and most of them have assessment systems that measure parameters such as the time to complete a task, the errors made during surgery, and the surgeon’s economy of movements. They can have both “task trainer” modules, for training in more simple and abstract handlings, and also “complete operation” modules that simulate the whole operating procedure . They seem to benefit the novices, especially when intensive feedback by the instructor is applied . The following simulators are among the most popular:
The sustainability of the skills obtained with VR simulators is supported by few studies with small samples and a short follow-up period . Other VR/AR models are
The main application of AR in laparoscopic surgical training is telementoring: the supervisor can teach the trainee by indicating the proper surgical moves, paths, and handlings on the AR screen. These pieces of information are displayed to the trainee and guide them . However, research has shown that this might also distract the user. A study proposes displaying this information on an additional sub-monitor, rather than on a single head-up display .
4.2.2. Robotic surgery
As the current robotic surgery systems do not rely on tactile feedback, their simulators are also freed from this need. However, for educational purposes an alteration in color in the surgical field exists, depending on the force that the users apply. Training in robotic surgery with VR simulators seems to provide better and transferable surgical skills, as shown by some studies [28, 29]. Novices are found to be particularly benefited from training in robotic simulators .
4.2.3. Open surgery
There is a limited variety in open surgery VR simulators and a lack of validation studies, probably because the complexity and the hands-on nature (without the involvement of machine systems) of open surgery operations do not provide an easily simulated environment.
4.3. Endoscopic procedures
4.3.1. Oesophagogastroduodenoscopy (OGD)
More VR simulators are available for colonoscopy training. The evidence that supports the use of VR trainers in the early stages of training is well established. The repetitions required to reach the plateau of VR training are 7–10, depending on the study. After this checkpoint there must be a switch to traditional endoscopy training. The combination of these two methods has been found more beneficial than either of the two alone. The skills acquired with VR training were maintained for at least 9 months .
4.3.3. Endoscopic retrograde cholangiopancreatography (ERCP)
Neurosurgery training is a field where VR and AR simulators are usually found. Such simulators must be comfortable enough to wear for many hours; extremely realistic, so that they represent effectively the detailed neurosurgery structures; and affordable . They demonstrate several advantages when compared to physical models (animal or bench models): they have a lower cost in use, there is no limit in the repetitions, and they offer a great sense of variety and diversity in the simulated cases. However, the resolution and realism of the constructed world are still issues to be resolved .
Various other VR/AR simulators exist.
4.5. Interventional cardiology and cardiothoracic surgery
Other TURP simulators using VR are the
A module employing
Three VR models are validated and commercially available for training in arthroscopy. Both
4.8. Endovascular surgery
Usually, endovascular VR trainers have metric tools that evaluate the procedural time, the volume of the contrast used, and the time for fluoroscopy. They provide plenty of modules including angioplasty and stenting of coronary, renal, iliac, and femoral vessels. They also seem to benefit the inexperienced users, and the skills acquired seem to be transferable .
4.9. Obstetrics and gynecology
Cataract and vitreoretinal surgery is the main field of VR training in ophthalmology.
4.12. Other fields
Pediatrics and emergency medicine are promising fields for integration of VR training. Rigorous work is being done to create realistic virtual humans and virtual pediatric models. Moreover, highly realistic virtual environments that recreate massive destruction situations are being developed for use in emergency medicine [70, 71].
AR has been used in intubation training. More modern systems, such as
Ultrasonography simulators have been used for assessing trainees.
Some AR applications that indicate the landmarks for central venous catheterization have been developed. An ultrasound probe locates the vein and a microprojector projects it onto the skin. A study has found positive learning effect, when used in landmark finding training .
5. Conclusion and future perspectives
Most of the educational applications of VR and AR seem to have construct and predictive validity, with the acquired skills to be transferable to real situations. However, the credibility of several of these studied might be questionable. First of all, many of them are not randomized studies, with cohorts of different characteristics and inadequate number of participants (less than 30 in most studies). Moreover, there are few studies for each simulator, and there are no similar standards in their design, so that they could be summarized and directly compared. In addition, many of the studies referring to specific simulators become quickly outdated as they do not take into account the simulators’ continuous upgrades. More randomized control trials, comparing the effect of VR training against no training, other simulation-based training, or different VR training systems, are needed. The samples must be larger to strengthen the results and the designs of different studies similar [5, 7, 8, 21]. Moreover, the extent of the decay of the skills over time must be elucidated. When these properties, along with the cost factors, are clarified, then we can examine the way that VR and AR can be officially incorporated in medical education curricula.
However, the published literature suggests a positive educational impact. VR/AR training displays certain advantages toward other simulation techniques. Although expensive to buy, VR/AR simulators provide a relatively costless opportunity for reproducible training under various environments and difficulty levels. Moreover, they do not raise ethical issues, compared with other animal and living tissue simulation models. They provide immersion for the user and the ability to perform complete procedures, in contrast with partial task trainers. Multiple studies have shown a favorable impact of VR/AR trainers on inexperienced trainees, and we can intuitively assume that they are technically evolving in high pace as the technology progresses. Future improvements could include the integration of olfactory stimuli. Odors can be used as diagnostic tools or even to recreate stressful conditions (e.g., in a combat or in the operating room) with greater realism [1, 8]. Medical informatics is also an evolving field. Medical data will be visualized more clearly and impressively with VR/AR technology. Users will be able to dive into statistical plots and reports, watch them in 4D (in 3D space and time), manipulate them, and “wander” around them. Although significant progress has been made, there is still a need for more processing power, higher resolution, better design of the scenarios, and more advanced haptic devices in order to achieve highly realistic environments [31, 76].