Medicine » "Virtual Reality in Psychological, Medical and Pedagogical Applications", book edited by Christiane Eichenberg, ISBN 978-953-51-0732-3, Published: September 12, 2012 under CC BY 3.0 license

Chapter 10

Applications of Virtual Reality Technology in Brain Imaging Studies

By Ying-hui Chou, Carol P. Weingarten, David J. Madden, Allen W. Song and Nan-kuei Chen
DOI: 10.5772/48445

Article top

Applications of Virtual Reality Technology in Brain Imaging Studies

Ying-hui Chou1, David J. Madden1, 2, 3, Allen W. Song1, Nan-kuei Chen1 and Carol P. Weingarten2

1. Introduction

Virtual reality is an evolving technology that allows for the possibility of interactive environments with stereoscopic, three-dimensional (3D) visual displays, auditory input, haptic feedback, and immersive interaction from a first person perspective [1]. Thus virtual reality technology makes it possible to simulate an environment with better ecological validity and control than previously possible for brain research and clinical applications. Virtual reality is compatible with many brain imaging methods and this has allowed researchers to evaluate typical and atypical brain function when users are immersed in a virtual reality environment. Virtual reality is also being developed as a therapeutic tool for a wide range of clinical populations, and because the brain is a primary mediator of effects of virtual reality treatments brain imaging is an important method for assessing some types of treatment effects. In a very different type of therapeutic application, virtual reality is also being developed to augment clinical use of brain imaging results for presurgical planning. This chapter provides an overview of these kinds of studies that employ both virtual reality and brain imaging technologies. In the first part of the chapter we will describe brain correlates of a few examples of tasks that are very difficult, if not impossible, to employ inside a brain imaging scanner without virtual reality technology. These tasks are spatial navigation, car driving, and social interactions. In the second part of this chapter we will survey examples of the application of virtual reality and brain imaging methods to clinical populations. These applications are in early stages of development and will require further studies to assess the value of using a virtual reality versus conventional task and to provide specific evidence of a therapeutic benefit from a virtual reality treatment. Nonetheless these examples will indicate some innovative ways being explored to apply virtual reality and brain imaging in the clinical sphere. We will conclude with a prospective view on using real-time magnetic resonance imaging (MRI) technology combined with virtual reality environments for research or therapeutic applications.

Many imaging modalities have been employed in virtual reality studies reviewed in this chapter. These imaging modalities include structural MRI [2] and diffusion tensor imaging (DTI) [3] that are used to measure brain anatomical structure, and positron emission tomography (PET) [4], functional MRI (fMRI) [5], electroencephalography (EEG) [6], and magnetoencephalography (MEG) [7] that are used to assess brain functional activity.

2. Measuring brain function in virtual reality environments

Brain function can be measured using imaging methods in a straightforward fashion for many cognitive and sensorimotor performances, such as attention, memory, facial recognition, finger tapping, or language. However, during these measurements of brain function participants usually must be imaged individually and must remain immobile (especially the head). Therefore it has been very difficult to measure brain function associated with tasks that require participants to move through the environment or to have face-to-face social interactions. Virtual reality technology makes it possible to “walk” or “drive” or to perform social interactions in simulated environments during measurements of brain function. Here we will review selected topics in spatial navigation, car driving, and social cognition for which attempts have been made to relate brain function to these behaviors using virtual reality and brain imaging techniques in healthy adults. We will also describe some studies that probed if the brain responds differently to tasks with two-dimensional (2D) versus 3D stimuli or real objects, as steps towards a more complex understanding of the neural correlates of virtual reality tasks.

2.1. Spatial navigation

“Where am I?”

“Where are other places with respect to me?”

“How do I get to other places from here?”

These are the main questions associated with spatial navigation [8, p.305]. Spatial navigation is the process of determining and maintaining a course or trajectory to a goal location accurately and efficiently [9] and is a requirement of daily life. Behavioral studies of spatial navigation indicate two important aspects for navigation involving allocentric or egocentric representations [10, 11]. Allocentric representations (e.g., South or North) are linked to a reference frame based on the external environment and independent of one’s current location in it, while egocentric representations (e.g., left or right) reference spatial locations in the external world with respect to individual body space. Different cognitive strategies have also been observed during spatial navigation [12]. One is called spatial strategy, which involves a more Euclidean representation of space allowing a target location to be reached in a direct path from any given location. The other is a non-spatial strategy, which is related to using environmental information, such as turning left or right at certain points, without knowing the relationships between the start and the target locations.

In addition, gender- and age-related differences in spatial navigation have been widely reported from behavioral studies [13, 14]. It has been suggested that the gender-related differences may result from disparate cognitive strategies [12, 15-17]. For example, in a self-report study [12], men are more likely to report using a spatial strategy whereas women are more likely to report using a non-spatial strategy. Men perform best when using instructions indicating the directions (e.g., North or West) and metric distances (e.g., 100m), whereas women performed best when using instructions indicating the salient landmarks (e.g., the purple doors) and egocentric (e.g., left or right) turn directions [17].

Age-related deficits in spatial navigation have also been studied extensively in animals and humans. For example, older animals show decreased performance in a water maze task [18-20]. In this task, rats are placed in a large circular pool of opaque water. Younger rats quickly learn to escape by finding and climbing onto a small platform hidden beneath the water surface if this platform remains in a fixed location over a series of trials. However, older rats take longer to find the hidden platform, travel a longer distance in locating the platform, and may require more trials before reaching a designated criterion performance. Moffat and Resnick [21] developed a virtual water maze task for human application and found that, compared to younger adults, older adults traversed a longer linear distance to locate a hidden platform. Older adults also showed evidence of impairment in cognitive mapping as revealed by their poorer map constructions of the previously visited virtual environment and their impaired ability to locate the platform on the experimenter-provided 2D maps of the virtual environment.

With the development of 3D virtual reality environments it has now become possible to probe human brain function involved in spatial navigation using brain imaging methods. Virtual environments, rendered in a first-person view, have been created for spatial navigation brain imaging research. Participants could navigate through the virtual environment with the use of keyboard, control pad, joystick, or mouse. The most frequently used virtual environment is the virtual maze, in which participants were instructed to learn and recall the topographical information to locate a target object or to find their way out of the maze [22-27]. Another frequently used virtual environment is a complex virtual town, in which participants were asked to freely navigate the environment first and later they were required to either head directly toward the goal location or follow a trail of arrows [28-31]. In addition, a water maze task used in animal studies to examine the brain correlates of spatial navigation [32] has also been employed in many virtual reality studies [33-35]. In this task participants are required to search for a hidden target in a large pool of water.

Based on one PET [29] and five fMRI [24-26, 31, 36] virtual reality studies, the hippocampus (especially in the right hemisphere) is one of the most consistently identified substrates associated with spatial navigation in these virtual reality studies. Involvement of the hippocampus has usually been implicated in allocentric representations of space that allow the computation of the direction from any start location to any goal location [29]. The parietal cortex is another brain region that has been frequently identified as being involved in spatial navigation [24, 26, 29, 31, 36] and has been implicated in providing complementary egocentric representations of locations [37]. Various brain correlates were attributed to different cognitive strategies during spatial navigation. The hippocampus was specifically more involved in spatial strategy, and the caudate nucleus more associated with a non-spatial strategy [23, 25].

In addition to the hippocampus many other brain regions, including parahippocampal gyrus, parietal cortex, caudate nucleus, frontal cortex, posterior cingulate cortex, cerebellum, putamen, thalamus, and retrosplenial cortex, have also been identified as possible elements of a navigation system from numerous PET [29, 38] and fMRI virtual reality findings [22, 24, 26, 31, 36]. For example, the parahippocampal gyrus has been implicated in allocentric representations with a different role than the hippocampus. When objects can be used as specific landmarks for navigation there is significant activity of the parahippocampal gyrus, but no activity of the hippocampus, suggesting a role for the parahippocampal gyrus in object-location associations [36, 38]. In addition to its role in non-spatial strategy, activation of the caudate nucleus is also associated with getting to the target locations quickly [29]. Activations in the frontal areas (especially the inferior frontal gyrus and frontal eye fields) may be involved in planning, decision making, and attention during spatial navigation [29].

Virtual reality studies using fMRI to explore the brain mechanisms underlying gender-related differences in cognitive strategies during spatial navigation have been reported [24, 27, 34]. In a study by Gron et al. [24] participants were scanned as they searched for the way out of a complex, 3D, virtual reality maze. Men were significantly faster than women at finding the way out of the maze. Women coped with the task by engaging a right parietal and a right prefrontal area, whereas men recruited the left hippocampal region. Two later fMRI studies matched spatial navigation performance between men and women and consistently found that women showed significantly increased activation of parahippocampus in comparison with men [27, 34]. In addition, in one of the studies [34] women also showed increased activation in the hippocampus and cingulate cortex while in the other study [27] men showed increased activation of posterior cingulate cortex and retrosplenial cortex relative to women. These findings suggest that even when men and woman are well-matched on navigation performance they appear to use different brain mechanisms to achieve the same behavioral end point, and the distinct functional correlates of spatial navigation in women versus men may be related to the differential use of cognitive strategies during spatial navigation.

Virtual reality studies using fMRI to study age-related differences in spatial navigation have found that, compared to younger adults, older adults showed reduced activation in the hippocampus, parahippocampal gyrus, and parietal cortex [26, 39, 40]. For both age groups, level of activation in hippocampus and parahippocampal gyrus was positively correlated with navigation accuracy [26]. In addition, age-related attenuation in functional activation in prefrontal and parahippocampal areas was accompanied by brain volume reduction [39]. Findings from these studies suggest that hippocampus, parahippocampal gyrus, parietal cortex, prefrontal cortex, and caudate nucleus play a critical role in age-related decline in spatial navigation.

Collectively, these findings are consistent with previous animal [32, 41] and human lesion [42, 43] studies on spatial navigation. When virtual reality technology was not incorporated into functional brain imaging paradigm, many brain imaging studies could only measure brain correlates of “mental navigation” by asking participants to imagine and make decisions about the routes and landmarks from their mental representations of the environment [44, 45]. Although the feasibility of virtual reality technology in understanding the neuronal underpinnings of spatial navigation in human beings has been demonstrated, future studies investigating similarity and disparity in brain correlates of spatial navigation between viewing virtual reality and conventional 2D visual display will be needed.

2.2. Car driving

Car driving is a complex behavior involving interrelated cognitive elements, which may include attention, perception, visuomotor integration, working memory, and decision-making. Simulated driving environments using virtual reality have been used to understand the cognitive elements involved in driving behavior and to study how driving performance is affected by factors such as alcohol intoxication [46] and distraction [47, 48]. Understanding the neuronal underpinnings of these cognitive elements and factors degrading driving performance is crucial for safe driving. The combination of virtual reality and brain imaging techniques is a useful approach for studying the brain during driving.

Virtual environments created for simulated driving are relatively homogeneous across studies. Participants are presented with an ‘in-car’ view of a road and a readout of speed. Most studies included both driving and observing conditions. During driving conditions, participants were instructed to drive within a predetermined speed range [49-57], to maintain a constant distance from a preceding car traveling at varying speeds [58], or to drive without deviating from the road [59]. Different input modalities, such as a brake pedal plus accelerator and a steering wheel, joystick or a controller with buttons, were used to brake, and to adjust speed and driving direction. During observing conditions, participants passively viewed a simulated driving scene. Some studies added traffic signs (e.g., speed limit, stop signs, and yield for pedestrians) and/or a stream of oncoming traffic and asked participants to abide by all conventional traffic rules [49, 53, 54, 56, 57].

Using these virtual environments, fMRI studies of healthy adults showed that a network of brain regions, including cerebellum and fronto-parietal areas, was more active during simulated driving than passive viewing conditions [51, 55-59]. The increased activity in these brain regions was common to all prepared movement executions (such as starting, stopping and turning), with the cerebellum more specifically linked to fine-control during movement execution and the fronto-parietal region related to visual attention [57, 58]. Involvement of other brain regions, including occipital cortex, basal ganglia, thalamus, and amygdala, has also been reported [51, 55, 56, 58, 59], although without clear consensus regarding what cognitive elements were associated with these other brain regions.

FMRI studies have also examined the alcohol dose effects on brain networks during simulated driving in healthy adults [52-54]. Participants received single-blind individualized doses of beverage alcohol designed to produce moderate, high or placebo (i.e., sober baseline) blood alcohol content. Compared to the placebo condition, many brain networks, including cerebellum and fronto-temporal-basal ganglia circuits, were significantly affected by the high levels of alcohol, which was associated with unstable motor vehicle steering, increased driving speed, and opposite white line crossings during simulated driving [52-54].

Distraction is another factor that impacts driving performance reported in behavioral studies [47, 48]. However, there is scant evidence showing how distraction during driving modulates brain function, mainly due to difficulties in applying ecologically valid driving hardware to the investigation. Recently, an fMRI study examined distraction effects on brain correlates of simulated driving in a virtual reality environment [56]. In this study, healthy participants were instructed to respond to audio tasks consisting of simple true or false questions, such as “a triangle has four sides”, by pressing a left (true) or right (false) buttons on the steering wheel during simulated driving. Driving with this type of distraction, increased activations in auditory cortex, precuneus, frontal cortex, and cerebellum and slightly decreased activations in the visual cortex were observed suggesting that distraction might draw resources away from the primary task of driving. This compromise in resources also indicates the potential risk for danger when distracted. This finding may have implications for screening drivers with a history of head injuries or pre-existing cognitive impairment.

Together, these findings have demonstrated that virtual reality technology allows simulations of driving tasks that would be impractical, dangerous, unethical, or even impossible in real contexts and enables the measurement of brain function during simulated driving [60].

2.3. Social cognition

Social cognition is a key topic in brain imaging studies. Almost all brain imaging studies in social cognition have been performed on single subjects who are isolated from a real social presence or interaction relevant to the study. Thus studies of theory of mind or social emotions may employ stories or pictures of others, attachment studies may use photographs or texts about loved ones, etc. This isolation is due in part to limitations imposed by brain imaging technologies, such as MRI and PET scanners that do not allow for most types of direct social contact or interactions during imaging. Although an individual can engage alone in important types of social cognitive processing, such as social memories, imagination, prospection, etc., by its very nature social cognition depends on the dyad or group of individuals [61, 62]. Virtual reality is a means by which some degree of simulation of social interactions within the constraints of many brain imaging methods is possible. A few examples of how virtual reality has been employed in MRI studies of social cognition are described.

Eye gaze is an important social act. Virtual reality methods have been used in several innovative ways to study brain responses when an individual observes the gaze of another’s eyes. Pelphrey et al. [63] used fMRI to assess brain activations while adult participants used virtual reality goggles to observe an animated man who shifted the gaze of his eyes either toward or away from the participant. Activations were observed in the superior temporal sulcus, known for its role in social cognition, and the fusiform gyrus. A similar study was then conducted on seven to ten year old children, which also showed the importance of the superior temporal sulcus and several other regions including the fusiform gyrus, middle temporal gyrus, and inferior parietal lobule [64]. Another study has examined brain activations when participants observed virtual characters for eye gaze and facial expressions and pointed to the importance of the medial prefrontal cortex [65]. All of these studies were based on passive observance of a virtual character. However, in naturalistic settings social acts such as eye gaze are socially responsive and interactive acts. Thus a more technically advanced methodology for interactive study of eye gaze during MR imaging has been developed [66]. The method employs an eyetracking system with real-time data transmission that can modulate a visual stimulus, such as the gaze of a virtual character, “allowing the participant to engage in ‘on-line’ interaction with this virtual other in real-time” [66, p.98]. Using this approach a virtual “joint attention task” was examined. Results included increased activations in the medial prefrontal cortex, posterior cingulate cortex, and anterior temporal poles. Note that some of these regions overlap the default mode network [66]. Regions of the default mode network typically decrease during cognitively demanding tasks [67]. However, regions in the medial prefrontal cortex can also be involved in outcome monitoring, understanding intentionality, and triadic relationships (joint attention toward a third object) [66, p.105].

Joint action is a key type of behavioral aspect of social interactions. Because joint actions are difficult to study during MRI studies, virtual actions have been employed to probe the neural correlates of joint action. In one approach, the participant performed a task of lifting a virtual bar while balancing a ball on the bar during MR imaging and did so either alone or with the assistance of another person (who was not being imaged) [68]. Joint action versus solo action showed increased activations in regions of the mirror neuron system of the inferior frontal gyrus and inferior parietal lobule.

Finally, there is an important emerging brain imaging technique of hyperscanning for which the use of virtual reality environments has also been proposed [69]. Hyperscanning employs simultaneous brain imaging of more than one participant interacting with a task [70]. Simultaneous scanning of two (or more) interacting participants allows for observation of the complex interplay of brain responses that emerge in real-time during an interpersonal interaction. Among these responses some of the most interesting new observations made possible by hyperscanning are inter-brain synchronization or coherence [71, 72]. For example, interbrain synchronization of EEG or near infrared spectroscopy (NIRS) brain signals has been observed when two participants imitated hand movements [72] or played a computer game together [71] respectively. FMRI hyperscanning has also been conducted [70]. In one study fMRI hyperscanning of two participants was conducted simultaneously while the participants worked together or alone to “drive” through a maze displayed on a video screen [73]. One result was that highest activation in the caudate, putamen, and orbitofrontal cortex – regions in the reward system – was observed when the two participants worked together to successfully complete the task, indicating the importance of cooperation in activating the reward system. Note that in all of the above hyperscanning studies, whether fMRI, EEG, or NIRS, the participants being simultaneously scanned did not watch the other participant but, instead, watched a video screen of a maze, televised hand movements, or computer game respectively. Thus virtual reality paradigms could be useful for hyperscanning. Preliminary reports of hyperscanning employing virtual reality include a study in which participants competed to increase regional brain activations [74].

2.4. Measuring brain function using 3D, 2D, or real stimuli

Many types of virtual reality tasks in studies cited in this chapter could have conventional counterparts. “Driving” could be performed using a virtual environment [49-59] or a nonvirtual environment [73]. Pain distractors could comprise virtual reality protocols [75-77] or 2D equivalents. Would the use of virtual reality 3D versus conventional 2D versions of task stimuli lead to observable differences in brain responses? Such questions will be important not only for understanding the neural substrates of virtual reality tasks but can also contribute to testing and understanding the efficacy of virtual reality versus conventional therapies. From another perspective it is also the case that, although 3D virtual reality appears more consistent with the 3D nature of reality than 2D representations, virtual reality is not identical to reality. Would differences be observed for brain responses to stimuli comprising 2D or 3D representations versus real objects, such as response to a 2D televised display of someone’s hands moving [72] versus direct observation of the person’s hands; or to 2D versus 3D renditions of televised displays of someone’s hands?

At this time these types of questions are only beginning to be addressed. Virtual reality protocols are still relatively infrequent in brain imaging studies and our most productive brain imaging methods for neurocognitive studies, namely MRI and PET, have usually precluded the use of real objects, environments, and interpersonal interactions for comparisons. Studies that have been conducted so far, however, suggest that there can be significant differences in how the brain responds to 2D, 3D, and real stimuli. A recent fMRI study on the neural substrates of visual motion processing included observations of brain regions involved in 2D versus 3D flow stimuli [78]. The cingulate sulcus visual area, a region in the dorsal posterior cingulate cortex, showed stronger activation for 2D stimuli, while regions in the occipital and temporal lobes (V5/middle temporal and middle superior temporal areas) were more involved in 3D stimuli. There has also been an innovative fMRI study of brain responses to repeated observation of 2D pictures of objects versus the real objects [79]. Repeated observation of 2D pictures of objects showed well-known repetition effects in activations in two regions of the lateral occipital complex, a posterior dorsal portion and anterior ventral portion in the posterior fusiform sulcus. The repetition effect is an attenuation of signal referred to as “fMRI adaptation” or “repetition suppression” [79, p.1]. Repeated observation of the real objects, however, showed a surprisingly weak repetition effect. Overall, results from these two studies indicate that neural processing of 2D, 3D, and real stimuli may differ in important ways. Although this topic will require many further studies these results [78, 79] suggest that there can be significant differences from use of 2D and 3D representations and real objects as stimuli in brain imaging studies. Understanding these differences will contribute to an understanding of the neural correlates of virtual reality and can facilitate the development of more ecologically valid virtual simulations for research and therapeutic applications.

3. Applications to studies of clinical disorders

Virtual reality has been integrated into non-invasive brain imaging in hopes of improving some kinds of clinical assessments and treatments. Specifically, virtual reality tasks have been employed as cognitive paradigms in brain imaging assessments of several neurological and psychiatric disorders. This approach is being developed to understand pathophysiological mechanisms underlying the diseases or to evaluate treatment effects. Virtual reality treatments have also begun to be developed for some clinical populations and brain imaging is being explored to assess effects of treatment. Finally, use of virtual reality to augment imaging procedures for presurgical planning is also being developed. In this section we will describe some of these applications for different clinical populations. All of these applications are in early stages of development and will require further studies to determine if and how these applications of virtual reality add clinical benefits and to explicate the neural mechanisms of action. Nonetheless the applications are interesting examples of approaches being explored for assessment and treatment of clinical disorders.

3.1. Addiction

Successful recovery from additions is often hampered by craving or strong desire for the substance of addiction. Craving can be initiated or enhanced by cues of the substance and associated factors. One approach to treatment for nicotine addiction and cigarette smoking is Cue Exposure Treatment (CET), which attempts to extinguish cue-induced craving by use of repeated exposure to cues [80]. CET, however, has limits on the nature of environmental cues that can be simulated, the degree of sense of “immersion” and reality, and has also shown decreased efficacy over time. Thus a virtual reality variation of cue exposure treatment, the Virtual Environment- CET (VR-CET), has been developed as an alternative to CET [80, 81]. To explore VR-CET treatment effects fMRI was conducted on eight adolescents who were cigarette smokers before and after they received VR-CET [81]. (No comparison treatment was assessed.) The fMRI task was to observe photographs of smoking cues versus neutral images during the scan. Participants did not report significantly decreased craving for cigarettes after treatment with VR-CET. However, fMRI showed decreased activation in the inferior frontal and superior frontal gyri in response to smoking cues after treatment with VR-CET. In a previous study, the inferior frontal gyrus region was one of several regions that showed increased activation from smoking cues in comparison with neutral cues, as well as increased activation in smokers versus nonsmokers for smoking minus neutral cues [82]. The results suggested that there can be a decreased brain response to smoking cues after VR-CET. Further studies will be required to examine effects of VR-CET on brain function, subjective craving, and smoking behaviors, and whether VR-CET can become an effective and specific treatment for smoking addiction.

3.2. Cerebral palsy

Cerebral palsy is a disorder of brain development that includes dysfunction of movement, posture, perception, communication, and other cognitive functions. Virtual reality therapies are being developed to treat cerebral palsy [83]. The use of fMRI to assess effects of virtual reality therapy has been described for a single case of a child with hemiparetic cerebral palsy [84]. The child had a right hemiparesis with encephalomalacia in the left temperoparietal cortex and corona radiata. Virtual reality therapy was given to improve function in the right shoulder and elbow. FMRI of right elbow flexion-extension movement was conducted before and after therapy. Results included observation of bilateral activation of the primary sensorimotor cortices before virtual reality therapy but only contralateral activation after treatment, indicating that the brain was capable of neuroplastic changes in cerebral palsy. The investigators noted that further studies would be required to compare virtual reality to other forms of rehabilitation to determine whether effects were specific or not to use of a virtual reality therapy.

3.3. Depression

Depression is one of the most common medical disorders and an increasing global health problem. It has been associated with many complex changes in brain structure and function. Virtual reality tasks have contributed to understanding the brain in depression. For example, MEG has been used to compare brain function of depressed versus healthy individuals during a virtual Morris water navigation task [85]. Depressed individuals showed decreased navigational performance of virtual Morris water tasks in comparison with healthy controls. In comparison with healthy individuals depressed individuals also showed decreased regional theta (4-8 Hz) activity in multiple regions, including the hippocampus and parahippocampal regions and lateral prefrontal cortices, and increased theta activity in posterior cortical regions and cerebellum. Results suggested that “dysfunction of right anterior hippocampus and parahippocampal cortices may underlie this deficit (impaired spatial navigation)” [85, p.836] in depressed individuals.

Given the abnormal findings associated with depression the question has arisen whether abnormalities are normalized after treatment. Hviid et al. [86] used virtual reality and MRI to assess long-term hippocampal structure and function in individuals in remission from depression. The participants comprised individuals in remission from depression and healthy controls, all of whom had been originally recruited for a PET/Depression project and then followed for 8 years. Baseline PET studies had shown that depressed participants had increased regional cerebral blood flow (rCBF) in the right hippocampus during rest [87]. The depressed participants had received several kinds of pharmacologic treatments for depression. Hippocampal volume at 8 year follow-up was assessed with structural MRI. Hippocampal function was assessed with a virtual reality navigation task in a shopping mall, the Counter-Strike Citymall Task (CSCT). A performance measure of CSCT, i.e. precision of navigation, had been shown to correlate with PET regional cerebral blood flow in the right hippocampus and, therefore, this task was used as a “right hippocampus-dependent task” [86, p.179]. Results showed that there were no significant differences between individuals in remission from depression and healthy controls for performance of the virtual reality precision of navigation task or MRI hippocampal volumes. Results suggested that individuals with depression in remission can show results similar to healthy individuals for some measures of hippocampal volume and function.

3.4. Epilepsy

Individuals with epilepsy may have seizures in unpredictable circumstances and, therefore, can face limitations in their activities of daily life. An important daily activity for many people is driving. Because of the potential hazards of a seizure during driving many people with epilepsy face the prospect of being restricted from driving. To better understand how driving may be impaired during a seizure Yang et al. [88] used EEG to monitor seizure activity in individuals with epilepsy while they played a virtual reality driving simulation game. Seizure activity and “driving” impairment were able to be observed and showed the potential value of virtual reality driving simulation for patient assessments.

Some cases of temporal lobe epilepsy are treated with surgical resection of the anterior temporal lobe when seizures cannot be managed medically. However, surgery comes with high risk for cognitive dysfunction. For example, the temporal lobe supports memory and this may become impaired after surgery. Thus it is helpful to have assessments that could indicate the relative risks of memory loss versus seizure relief. One presurgical measure to assess postsurgical cognitive risk is the degree of lateralization of memory function. FMRI of a memory task for object location in a virtual environment has been used to assess medial temporal lobe function prior to surgery [89]. Increased lateralization of presurgical hippocampal activation during the virtual reality task towards the ipsilesional side correlated with decline in verbal memory after surgery.

3.5. Movement disorders

3.5.1. Huntington’s disease

Huntington’s disease is a genetic neurological disorder that shows progressive impairment of movement, cognition, and mood and other psychological functions. In early stages of the disorder it manifests with relatively selective impairment of the caudate. The basal ganglia (caudate) and hippocampus are important regions that have been related to two systems for navigational memory, with evidence for both competitive and non-competitive or compensatory interactions between the caudate and the hippocampus. To further assess possible interactions between the caudate and hippocampus Voermans and colleagues [90] selected early stage Huntington’s disease as a model of caudate dysfunction and predicted that individuals with early stage Huntington’s disease and caudate dysfunction would show compensatory hippocampal function and near normal performance of a navigational task. FMRI was used to assess brain function in individuals with early Huntington’s disease versus healthy controls during a navigational memory task employing a virtual home. A key measure was route recognition performance. Results included that right caudate activity associated with route recognition correlated negatively with disease severity while bilateral hippocampal activity correlated positively with disease severity. Further, healthy controls showed greater activity in the right caudate associated with route recognition than observed in individuals with Huntington’s disease, while individuals with Huntington’s disease showed greater activity in the right hippocampus and parahippocampus associated with route recognition than observed in healthy controls. Results indicated that the hippocampus could compensate for impaired caudate function in individuals with early Huntington’s disease. This compensatory function could explain observations of relatively normal navigational performance in these individuals.

3.5.2. Parkinson’s disease

Parkinson’s disease is a movement disorder with many symptoms including tremor, bradykinesia, rigidity, and cognitive and mood changes. About 50% of individuals with Parkinson’s disease exhibit spontaneous “freezing” while trying to move. A single case fMRI study identified the neural mechanisms involved in freezing while a patient with Parkinson’s disease used foot pedals to “walk” in a virtual reality environment, both on and off dopaminergic medication [91]. Different brain activation and deactivation patterns were observed for different walking tasks and freezing episodes. For example, freezing episodes occurring when the individual was off medications showed increased activations in the “pSMA (pre supplementary motor area), motor cortices, DLPFC, VLPFC, (dorsolateral and ventrolateral prefrontal cortices) and posterior parietal regions… deactivation within the frontopolar cortices and precuneus” [91, p.809]. The study demonstrated the ability to obtain knowledge of functional brain activation patterns related to walking and freezing by employing a virtual reality protocol.

3.6. Pain

Pain can sometimes be unremitting and treatment resistant even to maximal doses of opioid medications. Thus alternative or adjunctive treatments are required. The use of virtual reality as an adjunct treatment for control of pain has been proposed. One hypothesis is that virtual reality will engage attention and therefore distract or decrease attention to pain [75]. Virtual reality approaches to treatment of pain have been assessed with fMRI using normal volunteers. In one study normal individuals subjected to thermal pain showed activations in the anterior cingulate, insula, thalamus, and primary and secondary somatosensory cortex regions that decreased during immersion in a virtual reality protocol (throwing snowballs in SnowWorld) [75, 76]. In another study fMRI was conducted to assess effects of no treatment, virtual reality, opioid, and combined virtual reality and opioid treatments of thermal pain in normal individuals [77]. Both virtual reality and opioid treatments given separately decreased brain activation in the insula and thalamus, and virtual reality treatment also decreased activation in the primary somatosensory cortex. Combined virtual reality and opioid treatment led to further decreases in brain activations in the above regions as well as decreased activation in the secondary somatosensory cortex. Additional studies, such as brain imaging comparisons employing other kinds of distractors and treatment combinations, are required to better understand the neural mechanisms specific to use of virtual reality distractors to augment analgesia.

A protocol for an fMRI study of a pain syndrome – fibromyalgia – has also been published that would examine effects of a virtual reality exposure therapy [92]. Therapy would comprise virtual reality exposure to exercises that could induce pain catastrophizing, with fMRI conducted before and after treatment.

3.7. Post-Traumatic Stress Disorder (PTSD)

Post-traumatic stress disorder (often combined with traumatic brain injury) affects many service members returning from deployment in conflict and war zones. Although symptoms of PTSD in early stages may be a normal response to trauma, chronic PTSD is diagnosed when recovery fails to continue. Cognitive behavioral therapy that includes imagined reliving of the traumatic experience, such as Prolonged Exposure (PE) imaginal therapy, has been developed and shown benefits as a treatment for PTSD [93]. However, virtual reality exposure therapy (VRET) has been developed as an alternative treatment that may have advantages over PE imaginal treatments, including the possibility for “shared experience” of the battlefield with the therapist, preference as a treatment by some individuals because it uses computers, etc. [93, p.130]. Roy et al. [94] have begun to employ VRET as a treatment for PTSD in service members and assess its effects in a small number of participants using fMRI of an Affective Stroop test. Participants were randomly assigned to receive VRET or PE treatments. FMRI results for VRET and PE participants were pooled because of the small number (eight) of total fMRI participants. Results before treatment showed increased activations in the amygdala and lateral prefrontal cortex, as observed in other studies of PTSD and consistent with hypervigilance and negative arousal, and in the subcallosal gyrus as observed for depression. There were also deactivations in the anterior cingulate region. FMRI after treatment indicated normalized responses in these regions.

3.8. Schizophrenia

Schizophrenia can be a highly debilitating psychiatric disorder with many complex psychological, cognitive, and behavioral symptoms. To advance understanding of the neurocognitive underpinnings of schizophrenia investigators have drawn from cognitive behavioral paradigms employed in animal studies. One such paradigm is the Morris water task described above. This is a task that has often been used to study spatial navigation, learning, and memory in nonhuman species, for which virtual variants exist that allow MRI studies of human brain function. Several brain regions involved in performance of MWT have also been implicated in schizophrenia. Thus Folley et al. [95] examined individuals with schizophrenia using structural MRI and fMRI during performance of a novel virtual Morris water task to assess brain morphometry, neural circuits derived from independent component analysis, and regional brain activations. In comparison with normal controls, individuals with schizophrenia revealed differences in several distributed brain networks engaged during the tasks. They also demonstrated a lack of association between hippocampal activation and gray matter concentration that was observed in normal controls. The study demonstrated the value of virtual reality paradigms and MRI for translational research from animal studies.

3.9. Traumatic Brain Injury (TBI)

A concussion is a mild traumatic brain injury (MTBI) with mild symptoms of brain dysfunction. It can be one of the sequelae of sports injuries. Although symptoms of brain dysfunction after MTBI have usually been temporary there has been increasing interest in the possibility of residual brain impairment in otherwise asymptomatic persons. Evidence of brain pathology has, however, been difficult to demonstrate. Recently, fMRI of a virtual reality task was used to probe for impaired brain function in asymptomatic individuals with a recent history (within 30 days) of MTBI [96]. The virtual reality task was designed to probe spatial memory functioning. It comprised moving through a virtual corridor to find a virtual room. Behavioral measures showed that individuals with MTBI were able to perform the virtual reality navigation task similar to normal individuals. However, fMRI revealed that individuals with MTBI showed increased cluster size of activations in the parietal cortex, dorsolateral prefrontal cortex, and hippocampus indicating increased activation in these regions. These results provided fMRI evidence for regional brain impairment in individuals who appeared normal based on behavioral measures. The possibility that “efficient performance of navigation task … may give rise to hyperactivation of focal clusters and to recruitment of additional cerebral resources” [96, p.352] was one explanation given for the increased activations. It was also noted that further studies would be required to assess whether the findings were transient early phase changes or would be observed long-term.

More serious traumatic brain injuries can lead to a variety of cognitive and social dysfunctions. A virtual reality interpersonal paradigm has been employed to assess and social functioning in individuals with a history of TBI in conjunction with structural MRI studies [97]. The main task consisted of observing virtual reality presentations of people in conflict. Structural MRI was conducted for measurements of cortical thickness. Participants with TBI showed impaired social performance, with more impulsive and self-centered results. Cortical thickness of several brain regions correlated with task performance, including the orbitofrontal cortex with its role in motivation and reward.

3.10. Tumors

Surgery to remove tumors of the brain is accompanied by high risk for impaired function that depends on location of the tumor. Qiu et al. [98] have described the use of a virtual reality environment to display MRI results to the surgeon for presurgical planning for resection of gliomas located near cortical and subcortical motor pathways. Structural 3D MRI was conducted to image the tumor and primary motor cortex. DTI was conducted for tractography of the pyramidal tracts (white matter tracts of myelinated axons of pyramidal cells that go from the primary motor cortex to the spinal cord and brain stem). The structural MRI and DTI results were then used for a stereoscopic 3D virtual reality display of the tumor, motor cortex, and pyramidal tracts for pre-surgical planning and surgery simulations. The virtual reality system allowed the surgeon to be able to visualize both the brain tumor and pyramidal tracts from multiple directions and planes. A segmentation tool was used to virtually rehearse tumor resection and compare different surgical approaches. This virtual reality 3D stereoscopic visualization of the brain provided increased information for presurgical planning and showed promise for improving surgical outcomes.

3.11. Vascular diseases

Study of cognition, emotion, and behavior in individuals with brain lesions has been an important method to elucidate neuroanatomical substrates of normal brain function. This approach was employed by Weniger et al. [99] to help identify the neuroanatomical substrates of egocentric navigation as introduced above. The investigators examined individuals with unilateral parietal lobe infarctions or hemorrhagic lesions using structural MRI. They assessed performance of two virtual reality navigation tasks in both normal individuals and those with parietal lobe lesions. The tasks were navigation in a virtual maze to assess egocentric navigation or navigation in a virtual park to assess allocentric navigation. Results showed that individuals with parietal cortical lesions had impaired navigation on the virtual maze task but normal performance on the virtual park task. Performance on the virtual maze task in individuals with parietal lesions also correlated with increasing size of the right precuneus. Results provided evidence for a role of the parietal cortex in egocentric navigation.

4. Brain-computer interfaces, real-time neuroimaging, and virtual reality

Virtual reality technology also plays an important role in enhancing brain-computer interfaces (BCI) that are uniquely enabled by real-time neuroimaging, specifically EEG and fMRI. The integration of virtual reality, BCI and real-time imaging has shown promise for training healthy adults and treating neurological and psychiatric disorders. Combined virtual reality and brain imaging may see its most sophisticated technical development in future applications of real-time fMRI biofeedback employing virtual reality environments. Overall, although combined use of virtual reality and brain imaging technologies for application to clinical disorders and enhancement of normal levels of functioning is still in early stages of development, many innovative approaches have appeared that herald the possibility of important future contributions to understanding and treatment of many clinical disorders.

4.1. EEG-based approach

EEG monitoring during virtual reality protocols has been used for training purposes in normal persons. These studies may have implications for future development of cognitive, behavioral, and psychological treatments of clinical disorders. As an example of training in normal persons, training of actors has been enhanced by using an EEG based sensory motor rhythm (SMR) neurofeedback method with eyes-open training in a virtual reality auditorium performance environment [100]. Assessments of training outcome included ratings of Hamlet performances on the Globe Theatre stage, London. Results showed higher ratings for acting performance using SMR with the virtual reality environment versus a 2D computer screen.

EEG monitoring can take place in very fast time scales (milliseconds) with relatively good mobility and comfort and has become the most important modality in brain-computer interfaces. A novel study of a single individual with tetraplegia and muscular dystrophy was conducted in which BCI with EEG monitoring of the sensorimotor cortex was developed to allow the individual to use his motor intention to control an avatar in an internet virtual environment [101]. The individual “successfully walked and chatted with other virtual users while using the BCI at home” [101, p.3]. EEG results also showed changes in event-related synchronizations and desynchronizations over the 5 month period of the study that suggested cortical plasticity. The possibility of BCI in conjunction with virtual reality as a therapeutic regimen has also been discussed for walking rehabilitation and treatment of autism [102, 103].

4.2. fMRI-based approach

Functional MRI studies may be performed with a real-time and interactive manner, enabling participants to adaptively modify their cognitive and emotional processing strategies according to dynamic brain activities measured with fMRI. This real-time fMRI based biofeedback scheme makes it possible to enhance subjects’ ability to control and modulate their central nervous system, and is expected to be valuable for learning and various therapeutic applications such as cognitive rehabilitation. The dynamic and vivid information provided by virtual reality technologies, not surprisingly, can directly improve the performance of real-time fMRI based biofeedback, in comparison with 2D visual feedback used in most conventional fMRI experiments, as summarized in this section.

Real-time biofeedback of brain activity information has been previously demonstrated with EEG, which measure changes of the electric fields associated with neuronal activation at high temporal resolution (on the order of milliseconds). The measured information can be fed back to subjects for behavioral changes [104-106]. A limitation of EEG and EEG-based biofeedback is that the spatial-resolution and spatial accuracy of functional mapping are limited. Furthermore, it is difficult to reliably measure neuronal activities in brain regions away from the skull using EEG.

As compared with EEG, fMRI provides imaging data at a much higher spatial resolution, and can image the whole brain at approximately equal sensitivity and accuracy. Because of these advantages, real-time fMRI based biofeedback is becoming an important research topic [107, 108]. Note that the data acquired with fMRI have lower temporal resolution (on the order of seconds) than EEG. Nevertheless, the biofeedback with information updated every few seconds is adequate for most behavioral paradigms.

The real-time fMRI based biofeedback is schematically illustrated in Figure, using an experimental design of fear control as an example. A subject is lying in the MRI scanner, watching a video showing a roller coaster scene while brain signals are dynamically acquired with an MRI system (panel A). The MRI signals are then transferred to a workstation (panel B) for real-time reconstruction of functional activation maps (panel C). Functional activation in pre-selected regions of interest, such as the amygdala (i.e., a region associated with fear: yellow circle in panel C), is quantified [109]. The level of amygdala activation is converted to a scale bar (panel D), and displayed in the computer screen (panel E) so that the subject is aware of his own level of fear quantified by functional MRI. The subject is then asked to suppress the fear based on the information provided visually and dynamically.

media/image2.jpeg

Figure 1.

Schematic diagram of real-time fMRI biofeedback.

This real-time fMRI based biofeedback study can be improved by virtual reality technologies in several ways. First, if the roller coaster scene is displayed with 3D virtual reality through goggles instead of 2D viewing through a computer screen then the fear induced brain signals is expected to be significantly higher. Second, instead of showing a 2D thermometer-like scale bar in the computer screen, the biofeedback can be made more intense by employing 3D flames of different sizes to show the level of functional signals (panel F) stimulated by the virtual reality task [110].

In addition to the amygdala signal modulation that has been shown in previous real-time fMRI studies [109], other brain regions that have been investigated with real-time fMRI include somatomotor cortex [111-113], parahippocampal gyrus [114], the auditory cortex [115], the insular cortex [116], and the anterior cingulate cortex [112, 117].

Recent studies further showed that, in contrast to relying on fMRI signals in pre-selected regions of interest, the brain state derived from analyzing the overall activation patterns can be used in real-time fMRI based biofeedback without needing prior knowledge of activation regions [118]. It is expected that the brain state based biofeedback, without needing prior assumption on brain regions involved, should prove valuable for various types of virtual reality-based training in which multiple brain regions or even multiple neuronal connectivity networks are involved.

As summarized by de Charms [110], real-time fMRI biofeedback has several promising therapeutic applications, including treatment of neurological diseases and enhanced psychotherapy. For example, through training patients to modulate their central nervous system activities, the level of chronic pain can be reduced [117]. It appears that what a participant does internally while learning highly specific cognitive strategies for controlling pain using real-time fMRI is closely related to what the same participant might do while using other methods for learning cognitive control over pain, such as in cognitive behavioral therapy [110, p.725]. It is hoped that virtual reality will improve the efficacy of therapeutic procedures based on real-time fMRI biofeedback.

5. Conclusion

Many studies have now employed virtual reality paradigms in brain imaging studies performed with healthy individuals or those with clinical diseases. One of the most significant contributions of these studies has been to initiate brain imaging studies of some human behaviors that would have been difficult or impossible to assess with imaging methods such as MRI. Most often virtual reality paradigms have appeared as cognitive behavioral stimuli employed as tasks in brain imaging studies or as components of treatment protocols for which brain imaging is an important measure of treatment effects. There are also emerging possibilities for therapies with brain computer interfaces and real-time biofeedback at the neurological level that could employ complex virtual reality environments for which real-world counterparts would be impractical or impossible to use. We believe that with continued improvements in virtual reality and brain imaging technology this application holds considerable promise, both for theoretical frameworks of brain science and for translational applications.

Acknowledgement

This chapter was support by research grant R01-NS074045 from National Institute of Neurological Disorders and Stroke (NKC), and research grant R01 AG039684 (DJM) and training grant T32 AG000029 (YHC) from the National Institute on Aging.

References