A Virtual Harp for Therapy in an Augmented Reality Environment

1. Introduction

In the United States, between thirty-four and forty-three million people have some type of disability, with physical, sensory, mental, and self-care effects ranging from mild to severe. For people over age sixty-five, nearly 13.9 million people, or thirty-five percent, have some level of disability (Statistics of Disability, 2007). According to the U.S. Department of Commerce, a person is considered to have a disability when that person has difficulty with any of the following tasks (Kraus et al., 1996):

1. Normal body functions, such as seeing, hearing, talking, or walking

2. Activities of daily living, such as bathing or dressing

3. Instrumental activities of daily living, such as shopping or doing laundry

4. Certain expected roles, such as doing housework, school work, or working at a job

5. Performing usual activities, such as driving or taking a bus

In the U.S., an estimated four percent of the non-institutionalized population age five and over, that is 9.2 million individuals, need personal assistance with one or more activities. People who need assistance with instrumental activities of daily living, also considered independent living activities, number over 5.8 million, while 3.4 million people need assistance in activities of daily living, which are personal functioning activities (Kraus et al., 1996). Several different factors can contribute to developing some level of disability, for example, strokes, arthritis, or injuries, or individuals may be born with disabilities such as cerebral palsy.

There are many different treatment methods to reduce or correct disabilities. The most common treatment method is to enrolling the disabled individual in some type of therapy program, such as physical, occupational, recreational, musical, or speech therapy.

Many traditional therapeutic interventions used in rehabilitation programs have demonstrated that intensive and repeated practice may be necessary to modify neural organization, effect recovery of functional motor skills, and promote functional recovery (Jack et al., 2000). With traditional therapy, patients usually visit a therapist once or twice a day for half hour sessions. Their visit frequency drops to once or twice a week when they become outpatients. Finally, when they reach a predetermined level of function, their access to therapy is terminated even if residual deficits remain. It has been reported that while in sessions, common problems during therapy, such as boredom, fatigue, lack of motivation, or cooperation, result in low interaction function between the patient and the environment (Sveistrup, 2004).

Due to decreasing equipment costs and increasing processor speed, computer simulations have become more common over the last decade. Such technology is currently being explored for potential benefits in therapeutic intervention for retraining coordinated movement patterns (Jack et al., 2000). Virtual reality (VR) can create the most appropriate individualized treatment needs and training protocols by providing an environment in which the intensity of feedback and training can be systematically manipulated and enhanced (Jack et al., 2000; Sveistrup, 2004; Kuhlen & Dohle, 1995). In addition, VR-based rehabilitation systems can provide a unique experience that is engaging, functional, and purposeful, in a manner similar to computer games, which is important for stimulating patient motivation. Additionally, VR systems can provide precise real-time feedback, which further increases the motivating effect for patients (Kuhlen & Dohle, 1995).

As an example, Todorov et al. (1997) used a VR system to create a computer animation of a table tennis task for training a difficult multi-joint movement. The system had a virtual ball and virtual paddles which provided the teacher and the trainee the movement variables most relevant for successful performance of the task by the trainee. The system also used augmented feedback. Their results showed that virtual environment training provided better training for the task compared to a comparable amount of training in a real environment.

In a different virtual environment training study, two patients with hemiplegia were trained on an upper-extremity reaching task (Holden, 1999). The two patients were evaluated pre- and post-VR training using motor recovery and functional ability tests and a real-world test. During the training, subjects practiced a virtual task similar to the real task, trying to imitate a virtual teacher's performance. Post-training, reaching errors during real-world performance were reduced by 50%. Both subjects improved in the trained task, indicating transfer of skills from the virtual environment to real world performance. On the other hand, motor recovery and functional scores showed little to no change. However, one subject acquired the ability to perform several functional tasks. The results suggest that virtual environment training has a positive impact in stroke rehabilitation.

Reiss and Weghorst (1995) used augmented reality (AR) to help patients with akinesia Parkinson's disease overcome difficulties stepping over objects in their paths when initiating and sustaining walking. They found that a stable cue appearing about six inches in front of the toes was required to initiate the first step, while cues scrolling toward the feet were needed to sustain walking. In addition, the effectiveness of the visual cue was dependent on the degree and type of akinesia, showing that cues that are more realistic are needed as the severity of akinesia increases.

For individuals with spinal cord injuries, a VR-enhanced orthopedic appliance was developed linking a gait-inducing exoskeleton to a HUD. The exoskeleton used a semi-rigid sling to support the bust and lower limbs of the user and move the lower extremities to support and induce human gait. In preliminary study results, a 26-year old with complete paraplegia showed improvements in self-confidence, increased relaxation, and activity scores, and higher levels of optimism and motivation (Riva, 1998).

VR assessment and intervention tools have also been developed. A hand system with force feedback was developed in a virtual environment to complete various tasks such as a virtual PegBoard and reach-to-grasp exercises. The system was useful for augmenting rehabilitation chronic-phase patients following strokes. Skills also transferred to a functional clinical outcome measure as well as improvement on a variety of movement parameters (Adamovich, 2003).

Luo et al. (2005) conducted a pilot experiment on finger extensions for three male stoke survivors participating in 30-minute training sessions held three times per week for six weeks. Users were placed in an AR environment trying to grasp 15 virtual objects followed by 15 real objects, one using assistance from a body-powered orthosis (BPO), one using assistance from a pneumatic-powered device (PPD) and one using only AR. Their results show an encouraging trend of improvement for finger extension capability in the impaired hand after 6 weeks of training in functional tests scores, peak angular extension speed, maximal displacement, and BPO assistive force.

Environments in VR are being used in therapy treatment for anorexia, psychological ailments, post-traumatic stress disorder, and pain. Using real-world scenarios in the treatment for post-traumatic stress disorder and phobias patients would have enough exposure for habituation to occur and overcome their ailment to help live their lives. In order to help individuals with pain such as burn patients VR is being used for distraction. When the patients explored the VR winter environment, their pain tolerance lowered allowing their painful wounds to be debrided. Patients with anorexia are placed in a VR house with many doors to meet various person-sized animations. These animations include their judged ideal size, this help the patient form more accurate mental pictures of themselves (Westerhoff, 2007).

For individuals with spinal cord injuries, a video-capture virtual reality system was used for balance training. The study had 13 participants experience three virtual environments, birds and balls, soccer, and snowboard, compared to a group of 12 non-disabled participants. The participants expressed interest in having additional sessions suggesting this enhances motivation and enjoyment during a therapy-like session. The amount of effort exerted from the participants performing the three environments at different levels of difficulties without having any frustrations or discouragement shows that the ability to adjust the level of difficulty of environments and the related proper motor responses corresponds to the key rehabilitation principle (Kizony, 2005).

VR in rehabilitation has unique attributes for achieving many rehabilitation goals including the encouragement of experiential, active learning; the provision of challenging but safe and ecologically valid environments; the power to motivate patients to perform to their utmost ability; and the capacity to record objective measures of performance (Weiss & Katz, 2004). In this paper, a realistic virtual harp was created to assist individuals with disabilities for their rehabilitation therapy using augmented reality technology and a haptic device. In our prior study, we created a prototype computer-generated harp for interactive musical experiences (Taylor, et al., 2007). A simple harp was created and force feedback was provided using a Sensable Phantom Omni haptic device.

2. Augmented reality

Head Mounted Device (HMD). Such a device presents an image acquired by a camera mounted on the HMD, producing the effect of seeing through the video display while allowing virtual objects to be displayed. When users interact with a special marker in the real world, this marker is tracked by the system allowing the overlay of computer graphics with respect to the marker. The user can thus move virtual objects in and with respect to the real world by simple manipulation of the marker. The movements of the camera or the movement of the marker will move the virtual harp. This allows the user to manually move the harp to "zoom" in or out, or to select its exact position in their view.

A popular open source package for tracking such markers and handling the see-through video display is the ARToolkit (Lamb, 2007). The system is cross platform and transparently handles the video capture, video display and marker extraction, and calculates the marker transform relative the camera. It is possible to use many different markers, which are then individually identified and tracked, for the simultaneous tracking of multiple marked objects in the real world.

ARToolKit uses computer vision algorithms to track the users' viewpoint, using video tracking libraries to calculate, in real time, the real camera position and orientation relative to physical markers. ARToolKit tracking works by capturing camera video of the real world allowing the computer to search through each video frame for any square shape. Then, it calculates the position of the camera relative to a black square, which draws a computer graphics model from that position to render graphics overlaid on the real world (Kato & Billinghurst, 1999).

The challenge is to integrate the video capturing and marker tracking of ARToolkit with the computer graphics and haptic feedback. The software package H3D API is chosen for graphical rendering and haptics. H3D API is an X3D based open source system for handling multi modal rendering and simulation with support for a wide range of display technologies and haptic devices. It is available for the three major operating systems and allows for programming at three different levels: X3D for scene graph definition and modeling, Python for interactive behavior, animation, and dynamics, and C++ for the extension and implementation of base functionality.

A bridge called HART handles the connection between ARToolkit and H3D API. The bridge is implemented in C++ and provides a set of X3D scene graph nodes for the convenient representation of basic ARToolkit functionality inside of H3D API. With HART an augmented reality applications can be easily built which take advantage of all the power provided by the H3D API with the ease of using ARToolkit.

The bridge connects the H3D API functionality with ARToolkit through a singleton object, which is represented by a "Bindable" node in the H3D scene graph. One and only one such node is required for the system to work. As this node is created and initialized the required initialization of ARToolkit is performed and all the properties specified for the node through X3D are used to configure the video capturing and loading marker for the ARToolkit. Any other node representing a functionality of ARToolkit make calls to the singleton node to perform their respective action.

There are two general types of nodes in the bridge: a see-through background node (SeeThroughBackground) for displaying the camera image as a background, and various nodes for using marker transforms. The presented prototype uses a see-through viewpoint node (SeeThroughViewpoint) that updates the viewpoint of the virtual environment to match the viewpoint of the camera in the real world relative the coordinate system defined by a specified marker. This way the actual coordinate system for the virtual environment will be stationary regardless of marker movements. This is done by querying ARToolkit for the viewpoint transform that matches the selected marker and applying that to the viewpoint in H3D. Using the SeeThroughViewpoint node together with the SeeThroughBackground node produces an augmented reality environment, as shown in Figure 1.

Since the coordinate system for the virtual environment does not change as the HMD moves relative the marker, the harp object will not move relative the haptics device. This is essential for the stability of the haptic rendering of the harp since the positional precision required for haptic rendering is several magnitudes higher than the precision, which can be provided by optical tracking today.

Figure 1.

Harp in augmented environment

3. Harp string node and vibration model

Originally, a Python code was used to produce the string vibration animations (Taylor et al., 2007). However, the first limitation was the code needed to have each string coded with all the necessary properties to do the animations and routed to the X3D file. The next limitation was caused by the fact that each string and their properties had to be written in the Python code; any change to the strings properties in the X3D file or movement to the harp’s placement on the screen would cause the haptic device to be unable to pluck the harp strings. This code did not have any of the vibration model equations implemented, and the code was limited in which mathematical functions it had to use. The final harp design required the use of thirty-six strings and realistic real-time vibrations. These limitations in Python would have made the code very long and confusing for using thirty-six strings, resulting in the loss of the ability to allow the users to move the harp in any position without providing the real-time vibrations. Therefore, a H3D node for the string animation was written in C++ to eliminate the limitations found in using the Python code.

The string is modeled as a mass-spring system where having N number of masses would produce N number of modes of transverse oscillations. For this computer-generated harp, the first six modes of oscillation are considered for the string vibration animation interaction to minimize the computation time. Each mass-spring section will be considered as a mass point along the string and can only move in one plane. On a harp, the strings are fixed at both ends of the instrument and since the string is modeled with many mass points along the line, the vibration motion will be analyzed using a Fourier analysis for each point along the line.

Since the strings on a realistic harp are not positioned on a starting base location, it was needed to take into account a starting position and an ending position in order to draw the strings in the proper location as a realistic harp. Using the y-axis starting point position (P ₂) and the y-axis ending point position (P ₁) the string length L will be P ₁-P ₂. As shown in Figure 2, this simulates the realistic pulling motion limit variables in the string to provide the calculation of the pulling limit along the string.

Figure 2.

String pulling limit

The string pulling limit now needs to take into account the y-axis starting point position and the y-axis ending point position in order to use this equation for any string in any position on the harp model. The maximum pulling limit (H) is based on half of the distance between two strings before hitting the next string along the x-axis. The string pulling limit equation requires the use of two equations for the lower half and the upper half as in Eq. (1).

String pulling limit:

$$h(y)=\left(\begin{array}{l}\left(\raisebox{1ex}{$2H$}\!\left/ \!\raisebox{-1ex}{$L$}\right.\right)\left(y-P2\right),P2\le y\le \raisebox{1ex}{$L$}\!\left/ \!\raisebox{-1ex}{$2$}\right.\\ \left(\raisebox{1ex}{$-2H$}\!\left/ \!\raisebox{-1ex}{$L$}\right.\right)\left(y-P1\right),\raisebox{1ex}{$L$}\!\left/ \!\raisebox{-1ex}{$2$}\right.\le y\le P1\end{array}\right)$$ E1

The periodic function of the string was defined based on the pluck location with the fundamental frequency having a wavelength of 2L giving the limit on the y-axis of –L and L as shown in Figure 3. This diagram leads to three sets of equations based on the pluck location in the periodic function used to do the analysis of the Fourier coefficient for the first six modes of oscillation, as shown in Eq. (2).

Fourier coefficient:

$$\begin{array}{l}{C}_n(y)=\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2L$}\right.{\displaystyle {\int }_{-L}^{L}y(z)\sin (n{k}_{n}z)dz}=\\ \left(\begin{array}{l}\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2L$}\right.{\left[\left(\raisebox{1ex}{$h(y)$}\!\left/ \!\raisebox{-1ex}{$y$}\right.\left(\raisebox{1ex}{$\sin (n{k}_{1}z)$}\!\left/ \!\raisebox{-1ex}{$n^2{k}_1^2$}\right.-\raisebox{1ex}{$z*\cos (n{k}_{1}z)$}\!\left/ \!\raisebox{-1ex}{$n{k}_1$}\right.\right)\right)\right]}_{-L}^L,-yzy\\ \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2L$}\right.{\left[\left(\raisebox{1ex}{$h(y)$}\!\left/ \!\raisebox{-1ex}{$\left(L-y\right)$}\right.\left(\raisebox{1ex}{$\left(z-L\right)*\cos (n{k}_{1}z)$}\!\left/ \!\raisebox{-1ex}{$n{k}_1$}\right.-\raisebox{1ex}{$\sin (n{k}_{1}z)$}\!\left/ \!\raisebox{-1ex}{$n^2{k}_1^2$}\right.\right)\right)\right]}_{-L}^L,yzL\\ \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2L$}\right.{\left[\left(\raisebox{1ex}{$-h(y)$}\!\left/ \!\raisebox{-1ex}{$\left(L-y\right)$}\right.\left(\raisebox{1ex}{$\sin (n{k}_{1}z)$}\!\left/ \!\raisebox{-1ex}{$n^2{k}_1^2$}\right.-\raisebox{1ex}{$\left(L+z\right)*\cos (n{k}_{1}z)$}\!\left/ \!\raisebox{-1ex}{$n{k}_1$}\right.\right)\right)\right]}_{-L}^L,-Lz-y\end{array}\right)\end{array}$$ E2

Figure 3.

Periodic motion diagram

To simulate the vibration motion for each time frame for each node along the string, the Fourier coefficient values are calculated in the Fourier analysis by Eq. (3). The string properties, wavenumber by Eq. (4), and eigenfrequency by Eq. (5), also known as harmonics would be defined as functions calls to be used during the Fourier analysis calculation. The damping term (Γ) value was defined on a trial and error basis by matching the vibration decay to the sound decay, and the frequency (f) value in the eigenfrequency equation was defined using the MIDI note frequency definition for each string. The string’s note frequency is placed in a list and selected based on the user’s string number input.

Fourier analysis:

$$x(y,t)={\displaystyle \sum _{n=1}^N{C}_n(y)\sin (k_{n}z)\cos (}2\pi \ast v_{n}t)e^{-\Gamma t}$$ E3

Wavenumber:

$$k_n=\raisebox{1ex}{$n\pi $}\!\left/ \!\raisebox{-1ex}{$L$}\right.$$ E4

Eigenfrequency:

$$v_n=n*f$$ E5

A flow chart for describing the process is shown in Figure 4. The program can be divided into three sections with section one handling the line drawing for the string called Harpline, section two handling the forces for the string called Harpforce, and section three handling the sound for the string called Harpsound, all being place in the group node called Harp.

When the program starts Harpline, an IndexedLineSet node initially draws the string with a vertical line using two hundred points equally spaced from the line length equation being divided by the number of points in the direction of the y-axis. To manipulate the line’s points, the IndexedLineSetIsTouched feature is routed in the X3D code and is initially set to false, and the haptic device is informed if the line is touched which starts the animation for pulling the string. One example is shown in Figure 5.

Figure 4.

Program flow chart

Figure 5.

Plucking harp string

During the pulling of the string, Harpforce an H3DForceEffect node is activated. It creates a spring effect by using the distance difference from the initial position to the current position multiplied by the spring constant to give the users the feeling of string tension while the line is pulled within the pulling limit using the Phantom Omni’s haptic device. Once the user reaches the string’s calculated pulling limit, the string would start the vibration animation removing the spring effect and producing the appropriate MIDI harp note sound.

The vibrate function in Harpline starts a real-time timer loop for drawing each of the two hundred points along the string in the correct position from the Fourier analysis equations, for which the timer is reset every time the string is touched. The vibration damping term value was found by matching the MIDI sound decay to the visual string vibration decay.

4. Realistic harp design and sound

To model the realistic harp, SolidWorks 2001 was used, and it was able to convert the model file into an X3D file that H3D uses. Using a model guide, the harp frame was divided into four parts consisting of the column, base, soundboard, and string frame. This SolidWorks assembly file was then converted into an X3D file for which now the model is a series of IndexFaceSets points with a wood texture applied on the face. The strings were colored a light gray color to resemble the color of metal strings.

The Harpsound node created in C++ used MIDIShare for the sound output. The harp’s sound range is from C2 to C7 on the MIDI sound range scale. When the program starts Harpsound, an X3DChildNode node initially opens the MIDIShare system and changes the sound output to produce harp type MIDI sounds. When the string’s calculated pulling limit is reached, the MIDI pitch value of the string was sent to the playnote function in Harpsound. Once the vibration animation starts, there is a switch to activate the sound.

5. Virtual therapist

In most therapy sessions, the therapist would prompt the individual in therapy to perform an exercise and observe the individual during therapy exercises for errors and improvements. Once the individual leaves the therapy sessions, the therapist wants the individual in therapy to continue performing the exercises on their own. Therapy exercises are repetitive, causing the individual performing the therapy exercises to become bored and unmotivated to continue performing the exercises on their own. Therefore to allow the individual to engage in self-motivating therapy exercises outside the therapy sessions and have the therapist observe the individual during therapy exercises without musical knowledge, in this study, a virtual therapist feature was developed.

The virtual therapist feature includes a sequence of colored strings. The patients can play a song by plucking the harp by following the colored strings. Here, Python is used to create the virtual therapist guiding feature because Python allows the users to modify the string sequence easily using a text editor and the program does not need to be compiled again.

The virtual scene uses a 3D sphere as a 3D button to active the virtual therapist feature. As shown in Figure 6, touching this 3D sphere allows individuals to follow a sequence of notes by following the colored strings to help the user identify the notes, similar to playing the Simon game. In this study, the virtual therapist will highlight the strings and guide the users to play the song “Joy to the World”.

5.1. User study

This study consisted of 20 participants, requirements of our participants included having upper limb limitations or any disability affecting their arm or hand. The age group ranged from ages 18-60. The total age distributaries are 55% of participants in the age range of 18 and 30, 10% in the range of 31 and 40, 15% in the range of 41 and 50, and 20% in the rage of 51 and 60. Of these participants, 80% have been in traditional therapy where Figure 6 shows the distribution of years.

Figure 6.

Years in therapy

In addition, 15 participants (75%) had have some type of musical experience, 17 participants (85%) have not used any type of computer-aided music software or equipment, and the remaining 3 participants (15%) had experience with computer software such as sheet music writing, keyboarding, and Guitar Hero.

There were 12 females and 8 males that participated in the study. The user study testing procedure consisted of the four following sections:

1. Greeting and background questionnaire

2. Orientation

3. Performance test

4. Participant debriefing

The participant were greeted and made to feel comfortable while the study was explained. Each participant was required to sign a consent form before the study began. During this time, the participants were informed on the issues of confidentiality and allowed an opportunity to ask any questions they might have concerning the test. Each participant was asked to fill out a short questionnaire gathering basic background information before using the system. Participants were given an explanation of the virtual harp therapy system equipment so that they would become familiar with the technology and understand how it is being used. This was followed by an informed discussion about how the system will work and an explanation of the interface with the equipment. During the observed testing, the participants were positioned in front of the virtual harp setup. In order to help adjust to the unique instrument, participants were asked to familiarize themselves with the harp and its interface by freely playing the virtual harp.

After the familiarization period, the participants were asked to follow a simple sequence of notes using the virtual therapist feature, as shown in Figure 7. After the testing tasks were complete, the participant were asked to fill out a brief questionnaire pertaining to the usability of the virtual harp, development of skills, improvement in range of motion, and entertainment value.

Figure 7.

User study participants playing the virtual harp

5.2. Study results

Based on the participants’ type of disability they were formed into two groups. There were 9 participants classified with long-term disability (cerebral palsy, arthritis) and 11 participants classified with short-term disability (pinched nerves, injuries). Statistical analysis was done using one sample t-test for the whole study group (20) and to compare the long-term disability and short-term disability. For users with either long-term disability or short-term disability the hypotheses are the following:

1. The virtual harp system would be easy and comfortable from using to the users

2. The users would improve their range of motion and skills

3. Users would be motivated to continue playing the system for therapy

The following questions are used as a composite result related to the ease and comfort from using for testing for the first hypothesis.

1. How would you describe the level of difficulty in using the virtual harp?

2. (1=very easy, 5= very difficult)

3. Did you find that the virtual harp acted in a predictable manner?

   (1=strongly agree, 5= strongly disagree)

4. How satisfying was the virtual harp to use?

   (1=very satisfying, very unsatisfying)

5. Was the virtual harp comfortable to use physically?

   (1=strongly agree, 5= strongly disagree)

6. Were you able to play notes or individual strings with little or no error?

   (1=strongly agree, 5= strongly disagree)

The following questions are used as a composite result related to the improvement in range of motion and skills for testing for the second hypothesis.

1. 6. Did you experience any improvement in your range of motion?

   (1=much improvement, 4= no improvement)

2. 7. Did you experience any improvement in your skill level?

   (1=yes, 3=no)

The following questions are used as a composite result related to the user motivation from playing for testing for the third hypothesis.

1. 8. I felt motivated to continue playing this harp over traditional therapy exercises.

   (1=strongly agree, 5= strongly disagree)

2. 9. I would recommend this along with traditional therapy.

   (1=strongly agree, 5= strongly disagree)

3. 10. I would recommend this augmented reality format for other therapy.

   (1=strongly agree, 5= strongly disagree)

4. 11. Would you like to play the virtual harp or another musical instrument presented in augmented reality again in the future?

   (1=yes, 3=no)

The composite result for questions 6-7 related to the improvement in range of motion and skills was calculated by Eq. (6). The composite result for questions 8-11 related to the user motivation from playing was calculated by Eq. (7). These two equations change the composite result questions scale from 0-1 in order to find the average mean since the questionnaire scale was different between the questions.

Composite result for questions 6-7:

$$x_i=\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.\left(\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$4$}\right.x_6+\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$3$}\right.x_7\right)$$ E6

Composite result for questions 8-11:

$$x_t=\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.\left(\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$5$}\right.\left(\raisebox{1ex}{$\left(x_8+x_9+x_{10}\right)$}\!\left/ \!\raisebox{-1ex}{$3$}\right.\right)+\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$3$}\right.x_{11}\right)$$ E7

Table 1 shows the results of the one sample t-test for the whole study group, the long-term disability, and short-term disability on the three hypotheses.

Users with either long-term disability or short-term disability felt the virtual harp system was neither easy nor difficult when using, so the first hypothesis is rejected. Short-term disability users gained improvement in their range of motion and skills. Long-term disability users did not show improvement. However, overall, as a group, users did show improvement, so the second hypothesis is accepted. Users with either long-term disability or short-term disability were motivated to continue playing the system for therapy, so the third hypothesis is accepted.

ɑ = :05	Whole Group	Long Term	Short Term
Composite results for problems 1-5 Ho: µ=3 Ha: µ<3	µ=3.21 p =.9 not sig.	µ=3.13 p =.7 not sig.	µ=3.27 p =.88 not sig.
Composite results for problems 6-7 Ho: µ=.875 Ha: µ<.875	µ=.68 p =.0004 sig.	µ=.73 p =.036 not sig.	µ=.65 p =.0034 sig.
Composite results for problems 8-11 Ho: µ=.8 Ha: µ<.8	µ=0.443 p =.0001 sig.	µ=.44 p =.0001 sig.	µ=.44 p =.0001 sig.

Table 1.

One-sample t-test results

6. Discussion

The use of the virtual harp therapy system had an impact on the participants in the program, most were very excited when they first arrived. While almost all of the participants have received some form of traditional therapy training, most have never experienced virtual or augmented reality before and were therefore, able to look at the therapy training that they have seen before in a different way. Part of the problem with traditional therapy training exercises is that patients get bored or fatigued with the repetitive activity and lose motivation to continue with the training exercises.

This augmented reality based therapy system allows participants not only to experience a new type of technology with a hands-on therapy exercise, but also to be motivated to continue exercising through playing the virtual harp. During the observational testing, users playing performance did increase as they continued plucking the harp strings. They would start the test searching for the strings with long periods of pauses between notes gradually being able to shorten the period of pauses or strumming the harp. Participants were able to follow the virtual therapist feature with little error from selecting the wrong string. There were social interactions from the users such as asking questions, emotional expressions, and non-verbal communication while playing the harp.

Most of the participants stated that they enjoyed the experience; however, there were a few complaints that seemed to come up often. The primary two of these were the haptic device cursor sinking through the harp and the flashing background making it hard to see our harp. The model we used was very detailed and producing realistic vibrations with the haptics in the augmented scene is a memory intensive operation so at times the program can be slow. Haptics was used over optical finger tracking to reduce the challenge of trying to find the correct string in the air; since haptics provides force feedback and could have the benefit of muscle resistance exercises. Other glove haptic devices could be used to provide a more realistic plucking action but this device was used to minimize cost and system complexity in order to be used at various therapy facilities.

Another common complaint was the headset was too heavy and big, making it hard to keep in place. Based on the results for the level of difficulty in using the virtual harp, the users felt it was difficult using the system. Some of the users’ comments mentioned that this will take awhile to get used to, but there could be good skills learned and that it was more interesting than traditional therapy because playing the harp gives an immediate reward for trying. One reason may have been the equipment was new to them and since the users had to hold up the head mounted display on their head the accuracy of tracking the marker did have an affect on the playing difficulty. Another comment mentioned with improvement the system would be versatile for use in therapy like hand/eye coordination.

Measuring improvement and learned skills is one of the greatest challenges of a training program. In the survey the participants were asked to rate how much they improved and learned during the experience. Even though the study lasted about thirty minutes for one session, users expressed they experienced some improvement in their range of motion but really noticed improvement in their skills. While this is not a concrete measure of skills and improvement acquired since repetitive sessions are needed, it suggests that participants at least felt that they had gained a great deal of information utilizing our program. Despite these issues, most participants indicated that the VR training as their favorite method of therapy training. Most participants thought the system was entertaining and desired to continue playing the harp after the testing session. In addition, users would recommend this along with traditional therapy and like to play the virtual harp or another musical instrument presented in augmented reality again in the future.

7. Conclusion and future work

The virtual harp system was an attempt to provide interactive musical experiences and development of skills during therapy for individuals with disabilities in a motivating and fun method. This system provides individuals a means to engage in a self-motivating therapy exercises outside the therapy room and therapists without musical knowledge to observe the individual during therapy exercises. Virtual reality technology has the potential benefit as a therapeutic intervention for retraining coordinated movement patterns because it has the unique capability to be engaging, functional, and purposeful for patient motivation, the virtual harp system results associate with the statement. The system is coupled with music to increase the therapeutic goals from music therapy and to help stimulate and motivate the use of therapy exercises. The users from the study did have some frustrations using the system but felt that they could improve with practice over time. They felt this system was more enjoyable, interesting, and a fun challenge over traditional therapy. Although measuring improvement skills in therapy can be difficult, individuals cannot forget the experience and rewards they received using this system.

The virtual harp therapy system would benefit being used at a healthcare facility such as the hospital or nursing home along with an individuals’ traditional therapy program. The virtual harp therapy system would benefit from an assessment of how the video camera image is conducted for the ARToolKit background image in H3D to keep the background image from flashing. While the current method worked better on the computer used for the user testing, users still had difficulties and frustrations with using the system. In addition, the head mounted display should be replaced with a lighter version to provide users with more comfort. The virtual therapist system could also be extended to represent different difficulty levels of pieces of music by adding more three-dimensional buttons that provide different difficulty levels in therapy. In addition, this system could be expanded to include different types of instruments or different types of therapy scenarios such as lifting different weighted objects. Overall, the virtual harp system would be a beneficial extension to a therapy program.

8. Nomenclature

H Defined pulling limit value (m)

Ho Null hypothesis

Ha Alternative hypothesis

L Length of string (m)

N Number of mass points on string

h(y) Pulling limit based from position (m)

n Number of mode

p P-value for T-test

t Time (s)

x Position (m)

y(z) Node position along y and z-axis (m)

z Node position along z-axis (m)

α Significance level

Γ Damping term

μ Mean