Virtual Reality (VR) has been used in an increasingly wide range of applications, and technological advances have made it accessible to the everyday consumer. Virtual reality games have become the norm in today’s consumer marketplace. This makes it possible to consider such devices for more productive use of leisure time, such as in exercise and health maintenance. Gaming systems such as the Nintendo® wii-fit™ have already introduced the concept of VR guided balance and fitness training.
Studies have shown that virtual reality has potential for assisting recovery from illnesses such as stroke and vestibular disorders and improvements in cerebral palsy. Custom designed VR systems in the laboratory have been successful with balance training . Active video game systems, like the Sony PlayStation® 2 with the Sony EyeToy®, a camera accessory, have also been used in studies of rehabilitation. Flynn et al. used this system in a case study of a stroke patient and documented the types of training for each game . The Nintendo® wii™ system was designed for in-home training as well as gaming. This system was tested on a 13-year-old patient with cerebral palsy who showed improvements in visual perceptual processing, postural control and functional mobility . Therefore, these gaming systems show promise in rehabilitation, and have the added benefit of entertainment.
Other custom-designed games and virtual environments have also been developed to address specific disabilities and user requirements. Why do these systems work? In order to answer this question, the basics of human postural balance control have to be considered. The balance control system is a complex integration of multiple sensory inputs along with feedback from the motor component. The sensory inputs include visual, vestibular, auditory, tactile, and proprioceptive signals. The motor aspect is the “plant”, driven by neuromuscular control. Figure 1 demonstrates the complex interaction between the various sensory systems in the central processing of incoming signals. Different inputs are assigned weights based on motor learning and experience, i.e. the reliability of the incoming cues.
Visual inputs are generally dominant among human sensory inputs for balance control as well as a general source of information, and virtual reality systems have evolved with an emphasis on visual displays. Balance improvement has been demonstrated through both static and dynamic visual stimuli in VR systems.  This results in a natural fit for visual-modality systems for both cognitive and neuromuscular control training along with entertainment value. However, other channels of information cannot be discounted, including the vestibular system, which is located in the inner ear and senses head acceleration.
The visual and vestibular systems are very closely linked, for example eye movements respond to vestibular inputs. Since the vestibular system senses head movements, it can be hypothesized that head movements together with visual stimuli should be used in training the balance control system.  Cohen et al. showed that purposeful head movement exercises improved balance in vestibular patients.  Therefore active games that track player head and body movements could demonstrate potential for balance improvement.
Other inputs include auditory and tactile modalities. Auditory signals provide orientation cues based on sound localization. Tactile cues provide pressure feedback and external inputs regarding contact with surfaces. Proprioceptive inputs are also important, providing us cues about our body position in space, through joint angle feedback signals. These signals are directly relevant to the control of ankles, knees, hips and upper body to maintain balance. Furthermore, the arms can be used for stabilization control and fall prevention.
In this chapter, we will investigate both commercial and custom-designed virtual reality interfaces for balance training and rehabilitation, considering the various inputs that are important for balance control. We can then begin to evaluate the important elements of virtual reality games, so that they can be developed more effectively for rehabilitation applications.
2. Arm movement importance and potential training system
Arm movements have been shown to be very important in balance maintenance and recovery . Both unilateral and bilateral arm movements generate anticipatory and reactive responses in postural leg muscles  and hip and trunk muscles . Furthermore, reaching and grasping actions are important in balance recovery and prevention of falls .
In a recent study we showed the importance of free arm movements in a series of classical balance and mobility tests in which three out of four tests showed significant improvement with the free use of arm movements :
Maximal step length test (length and timing of a single maximally long step)
Step test (number of steps up and down onto an elevated surface in one minute)
Timed get-up-and-go test (time to get up from a chair, walk to a target, and return to sitting)
Walk along an elliptical line
Only test (4) did not show signficant improvement, possibly because there is less change of support motion of the body relative to the task of walking, compared to launching a single large or elevated stepping motion, and certainly compared to rising from a chair. These results show the importance of arm movements in controlling body balance.
Arm movements are affected by stroke and progressive disease, such as Parkinson’s disease (PD). Given the importance of arm movements for balance, such an effect could result in decreased balancing ability or steadiness of stance in these patients. Arm movement training is thus an important part of rehabilitation. We have developed a virtual reality arm movement training system , in which different games can be programmed to suit user abilities and interests. This system uses the Microsoft Kinect™ interface, along with pressure sensors attached to the index finger and thumb as an object selection tool. This system shows the actual image of the user on the screen, and we have developed games, which appear overlaid on the user’s image. This allows the user to reach out to the target letters and “select” them by making contact between the thumb and index finger. Initial testing on healthy subjects provided a baseline scores, so that we can now proceed to evaluate patients’ performance.
3. Balance training: Sony PlayStation® 2 with the Sony EyeToy® and the Harmonix AntiGrav™ game 
Based on the importance of head and arm movements along with visual inputs in balance training, the Harmonix AntiGrav™ game was selected for evaluation. The purpose of this study was to to characterize balance-related motor learning effects through game performance and to evaluate the effectiveness and specificity of the video game for balance training in healthy subjects. 
3.1. Game description
Sony's EyeToy® used video capture technology, in which it imaged the user with a small video camera mounted above the display and used face-recognition and motion-tracking technologies (no body markers were required). [13, 14] The system used two-dimensional user motion in the frontal plane as inputs. Each user was individually calibrated before each session, to account for height differences and range of head motion. During this procedure, the subject’s own image was shown on the screen overlaid with a calibration figure, instructions and controls. The subject had to move to a position such that his/her face was centered on the figure, and then was instructed to wave his/her hand over the “lock” control. In the subsequent screen, the subject was required to bend to the left and right such that the head was aligned with target positions. If this procedure was not properly executed, it became very apparent in the early stages of game-play that the game was not responsive (the avatar movements did not respond to user movements), and the calibration was repeated. A diagram of the system and screenshots is shown in Figure 2.
The user was portrayed in the game as an avatar and the user’s image was not shown after the calibration procedure. The avatar raced on a hoverboard along roadways and rails, over jumps, and when airborne, through rings. The user, standing in front of the display, used head movements to guide the avatar, and was required to jump or duck to avoid obstacles. The user also had to reach with either arm or both to strike targets as the avatar passed them. The arm motions were directed diagonally up or down or to the side, requiring additional head/upper body motions to contact the targets. The game was fast-paced and challenging, and included sound effects and "life or death" consequences. It presented unpredictable situations requiring maintenance of balance and decision making at the same time. As the user gained experience in the game, anticipation of upcoming movements became easier. Upon achieving a pre-set score, the user was advanced to playing both difficulty Level 1 and Level 2 in each training session.
3.2. Experimental protocol
The control (no training, n=7) and experimental (video game training, n=7) groups completed the balance tests (as described below) at the beginning and at the end of a three-week period. Both the control group and the experimental group were free to conduct their daily activities as normal, and thus, the control group controlled for the effect of the specific experimental video game training. The experimental group trained 9 times during the three-week period, and the control group received no training. For details, see .
Game performance was measured by the number of tokens or targets struck with arm movements and the speed and accuracy in navigating the avatar through the race course using head movements. Additional points could be achieved by doing tricks, executed by arm movements while airborne. The scores were provided after each game, and points were displayed when awarded, which reinforced the immediate feedback provided during game play.
Three balance tests (Balance Board (BB), Tandem Romberg (TR), and One-leg Standing (OL)) were selected to evaluate transfer of the video game training to the real world. These tests evaluated transfer of learning to the real world because they were conducted without the virtual environment and thus tested for changes in balance outside of the game. Many balance-related measures were found in the literature ranging from the Dynamic Gait Index to the Sensory Organization Test to various functional measures and clinical tests. Our tests were selected based on common usage and relevance to overall balance [15-17] rather than factors such as sensory contribution or functional outcomes such as locomotion.
The average scores from two trials for each test were used. To minimize a “ceiling effect” (which occurs if most subjects pass a balance test given for a specified time, e.g. 30 seconds), the time was recorded when balance was lost. This gave a finer measure to detect and quantify improvement.
3.3.1. Game scores
As expected, the game scores in both levels of difficulty improved over the 9 training sessions. The performance in the game was based on the game scores in both levels of difficulty. All subjects except one achieved the target score for Level 1 in the first session, and were moved to training at both Levels 1 and 2 by the second session. Overall, there was a gradual increase in the total scores for both Levels over the period of the experiment.
3.3.2. Model of motor learning through game scores
The improvement in game scores was consistent with our common understanding of improved performance with practice and with studies of motor learning.  The game scores were averaged across subjects and fit to an exponential given in Equations 1 and 2 for Level 1 and Level 2 games respectively :
where n is the session number and y is the curve fit value. All three variables were initially used to obtain curve fits. It was found that constraining the DC term to be the same for both curves did not affect the quality of the fit, and it represented the average performance plateau after a large number of sessions. These curves are shown together with the averaged data in Figure 3. The curve fits reflected the increase and plateau of motor learning as a function of the session number for the first level (Level 1), and suggested that the subjects had not yet mastered Level 2 because of the absence of the plateau.
3.3.3. Balance tests
Performance on balance tests improved over the training period for the training subjects as compared to controls. During pilot trials, it was found that some volunteers were balancing for longer than three minutes on the balance board and two minutes on the Tandem Romberg tests. The BB and TR tests were discontinued after three and two minutes respectively because of the potential for muscle fatigue, which would interfere with the measure of balancing ability.  Of the 7 experimental subjects in each group (training and control), three control subjects reached the maximum time in the BB pre-test and one control subject in the TR pre-test.
Figure 4 shows the balancing times in seconds averaged for training and control subjects for each test. Trained subjects showed significant improvement in the BB test, moderate improvement in the TR test, and no significant difference in the OL test, while control subjects did not show significant difference on any of the tests.
3.3.4. Simulator sickness
Virtual Reality games have the potential for simulator sickness, which is related to motion sickness, with many of the same symptoms. Kennedy et al. have established a Simulator Sickness Questionnaire (SSQ) based on work with the US Navy, which is often used in evaluating simulator sickness.  It contains 16 items on which the subjects rated their responses (ranging on a 4 point scale; 0 (none), 1 (slight), 2 (moderate), 3 (severe)).
In this study, simulator sickness symptoms were recorded in every training session. Figure 5 shows the incidence of all symptoms across the 7 subjects and 9 training sessions. Most subjects who experienced symptoms reported slight sweating. No severe symptoms were present, and the symptoms subsided over the course of the training sessions. The presence of the symptoms suggested potential for vestibular rehabilitation, and their mildness demonstrated the safety of this system for the general public.
The results showed that game scores improved over the sessions as expected. The average game scores were modeled, and showed an almost linearly increasing component followed by a plateau. Motor learning analysis showed that Level 1 was mastered by the participants while Level 2 was still showing learning. We developed a game difficulty parameter, which could be used in rehabilitation game design to provide appropriate increases in difficulty for continued learning and training.
Pre- and post-test results showed that subjects who played the game had an improvement in balance board and tandem Romberg tests over the 9 sessions. One-legged standing tests did not show improvement, which could be the result of training specificity, since the training was not performed on one leg.
The simulator sickness questionnaire showed the presence of several symptoms, especially in the earliest sessions. These were mild and decreased over the training period. The symptoms indicated that the game had a physiological effect on the participants, and the decrease in symptoms paralleled the motor learning.
4. Auditory inputs
The auditory system also provides important cues to an individual’s self-orientation in space due to sound-localization capability. This ability is based on the difference in arrival time of a sound to each ear. We have used this concept in a prototype balance training aid, in which both visual and auditory feedback were provided to four healthy subjects. The system was tested with the subjects standing on a balance board. The auditory feedback was provided through stereo headphones worn by the participant. Both the Inter-aural Intensity Difference and the Inter-aural Timing Difference were controlled based on accelerometer signals from the tilt of the balance board. The combined audio-visual feedback resulted in significant improvement in the subjects’ balancing ability . Future studies will test for the relative contribution of each of the two feedback modalities.
5. Tactile inputs
Tactile inputs can be utilized in VR systems with the increasing availability of haptic feedback. Multimodal feedback increases the sense of immersion in the virtual environment, and different sensory inputs can be harnessed for sensory substitution, additional information channels and most importantly, redundancy. Redundancy is a means of bringing important signals to the attention of the user, and alerting the user in emergencies. For this reason, we considered it important to include tactile feedback in the design of an alerting system to inform subjects about potential loss of balance.
We developed a Wireless Vibrotactile Rehabilitation System (WVRS), which provides vibration feedback during body tilt.  The system incorporates three vibrating disks on each arm, which are activated when an accelerometer, mounted on the mid-back detects body tilt within certain ranges. Figure 6 shows the design of the system, which includes a wireless link to a computer. The system can record and display the accelerometer signals. The three vibrating disks on each arm are activated as a function of the degree of body tilt and tilt direction.
The arms were selected for the vibration transducers to naturally produce a reaction in the appropriate arm, and by eliminating unnecessary processing steps, the arm movement could be rapid enough to help stabilize posture. An example result from one subject is shown in Figure 7.
Such a system can be used as a VR rehabilitation tool, utilizing multimodal feedback with the visual display and providing progress tracking with the data recording capability. It is also anticipated that in the next design phase, the recording could also be housed in the main module, making the system portable. Such a wearable device could be worn as an everyday balance maintenance aid. This is an example of taking the virtual reality into the real world, or augmented reality.
6. Ankle training
In addition to head/body and arm movements, balance also depends on the quality of control in the joints of the lower limbs. In a balance training program, these joints can be involved in general training, or targetted specifically. When a joint injury occurs, for example in the knee or ankle, then a phased and progressive rehabilitation program is essential.
We have demonstrated ankle VR rehabilitation systems in which ankle movements in up to two degrees of freedom can be used to control a custom-designed VR game. The program was motivated by a sports injury, which can often take a long time to heal without properly designed rest and training. The games tracked the progress of the training and could be used as an inexpensive home-based training tool that motivates exercise adherence. The system had the following objectives :
the training has a range of difficulty levels to suit different patient abilities, including isometric contraction when no joint rotation is desired
the system has the ability to constrain movement in both magnitude and direction so that specific muscle groups can be trained without the risk of re-injury, and
the game provides visual feedback through a game, which provides scores of the ankle control ability related to game performance.
These objectives have been satisfied with the incorporation of either a balance board simulating a 2 degree-of-freedom foot pedal interface , or a boot with a frame and locking pins that either prevent motion, or constrain motion to the plantarflexion/dorsiflexion plane . Force sensors and accelerometers measured the muscle activity and ankle motion and provided inputs to the game. Visual interfaces were developed for calibration, as shown in Figure 8. Game scores were tracked and recorded across the training sessions, so that the patient and also the caregiver can monitor progress.
The important side-benefit of such a visual gaming interface is the motivation to improve scores, and the entertainment value of the game. Sayenko et al. have shown that a custom-designed VR game provides useful feedback and motivation for a spinal cord-injured patient to prevent muscle atrophy through functional electrical stimulation training.  The game encouraged adherence to the exercise program. It is important to conduct further studies on patients to develop optimal training protocols.
7. Occupational health and safety: Training for correct lifting posture
Very specific aspects of postural balance can also be isolated and trained through VR. In the field of occupational safety, VR can be a very important training tool and real-time aid in injury prevention. We have developed a lifting technique trainer, in which an avatar shows the correct motion on the screen, based on published positioning guidelines.  The trainee lifts a training object wearing a harness instrumented with accelerometers as shown in Figure 9.
The user can view the image of the avatar reflecting actual measured values from his/her upper body during the lifting motion. The discrepancy between the ideal and actual lifting motions is displayed, which guides the trainee to improve his/her lifting technique. Such a VR training aid provides instant visual feedback to allow the user to learn the body proprioceptive feedback signals associated with the correct postural positioning.
Different virtual reality training environments were evaluated or developed for improving postural balance and neuromuscular control. Balance training was shown to transfer to the real world from commercial video-game based training, and because balance is related to overall fitness, these training interfaces represent an important means of combining leisure and health maintenance. The results of such training could reduce the likelihood of falls and injury in everyday activities.
The benefits of VR training include the ability to isolate specific sensory inputs and motor outputs. Clearly, VR also provides immediate feedback to the user, which enhances learning. Further, VR provides an automated way to guide and track the progress of the training. Adaptive systems can sense the user’s performance and adjust parameters to provide the appropriate level of difficulty in each training session. Finally, VR provides a relatively safe environment for training compared to many real-world environments.
I would like to thank Dr. Ken Norwich, University of Toronto for his insight and wisdom and endless encouragement. I am also grateful to all of my students who have helped me immensely in this work, especially Matija Milosevic and Sumandeep Virk. I also thank the Natural Sciences and Engineering Research Council of Canada for funding the research.