Music is widely believed as one of the most effective media forms that effect on human psycho-physiologically. Many people expect the effects of music and uses music pieces in various situations. For example, relaxation effect of music is used in change in personal mind in daily life, creation of sedative atmosphere in a home, and therapeutic purposes. Oppositely, parts of music pieces excite people in disco and party. These various effects of music have been investigated in many previous studies. Especially, relaxation effects of music were investigated from various view points with psycho-physiological experiments. Although many previous studies investigated the effects, the effects have not been clarified at all. One of the reasons of that is existence of many music factors; melody, tempo, harmony, rhythm, etc. Anyway, investigations of music and its effects will contribute to theoretical use of music for therapy and so on.
This chapter aims to investigate the temporal change in heartbeat intervals in a transition between different sound stimuli. Heartbeat is one of the most important physiological indices and is often used as physiological index to investigate the effect of music and sound stimuli, because the heartbeat reflects autonomic nervous activity [Pappano, 2008] of a listener and is easy to measure. Furthermore, a device measuring electrocardiogram is generally cheaper than devices measuring other physiological indices such as electroencephalogram. Although many previous studies have investigated the effect of music and sound with heartbeat interval as physiological index, very few previous studies have investigated the change in heartbeat in the transition of different stimulus; most of the previous studies have observed average of heartbeat intervals a certain range in pre- and post-listening sound stimulus. Observing temporal change in heartbeat is important and contributes to improvement of exposure method of music and sound to the listeners. For example, time intervals eliciting effect of music and sound and decreasing the effect are important information to determine the time length to exposure them to the listeners.
The experimental method in the present study is set by referring to our previous study [Fukumoto et al., 2009]. As further investigations, we newly add No-sound stimulus to relaxation music piece (Air by Bach), white noise, which are music and sound stimuli employed as sound stimuli in the previous study. In the listening experiment of our previous study, two min relaxing music piece and white noise were employed as the different sound stimuli, and these sound stimuli were played twice alternately after five min rest. The alternate exposure of different sound stimuli is also employed in the present study. By employing No-sound stimulus as sound stimulus, we can observe change in heartbeat in start and end of noise and music stimuli. It means observing the change in heartbeat caused by listening sound stimuli.
Objectives of this study are fundamental investigations of temporal change in heartbeat in a transition between sound stimuli; music, noise, and No-sound. Figure 1 illustrates the objectives and explains that listening sounds elicit deceleration or acceleration of heartbeat. Time cost is also investigated. The results of this study will contribute to develop the studies in music therapy and the studies on musical system reflecting a user's KANSEI information automatically.
As mentioned above, we generally believe relaxation and excitation effects of sounds and utilize the effect in music therapy, reducing patient’s anxiety during a surgical operation and personal use to quickly change our mind and so on. Especially for the effects of music, many researchers have investigated the effect with psycho-physiological indices [Dainow, 1977], however, the effects have not been clarified completely yet.
Heartbeat has been often used to investigate the effects of music, because heartbeat is non-invasive physiological index being measured with relatively convenient and low-cost device. Moreover, as mentioned above, heartbeat reflects autonomic nervous activity, and the decrease of the heart rate is caused by a combination of two factors; increase in parasympathetic activity and decrease in sympathetic activity [Pappano, 2008]. Most of the previous studies using heartbeat as physiological index have tried to investigate the effect of music by comparing heartbeats before and after listening to the musical piece. However, heartbeat information has not been used well, because we do not know how long time the sound stimuli would cost to elicit the change of heartbeat. According to a previous study, impression for the listening to music piece on the listener is determined with 1 s [Bigand et al., 2005]. Referring to this finding, response of heartbeat needs longer than 1 s, because general physiological changes come after psychological changes.
Some previous studies have investigated the temporal cost that sound elicits heartbeat approximately with different sound stimuli and different conditions. Etzel et al. have investigated the physiological effects of musical pieces inducing different moods on listener [Etzel et al., 2005]. They have not mentioned about the concrete time cost, however, the results of heart rate development seem to elicit the change of heart rate from 30 s to 40 s. Gomez et al. have investigated the physiological effects of 30 s noise and music [Gomez & Danuser, 2004]. The results of their study showed that part of noises and musical pieces elicited physiological change including heart rate within 30 s. Moreover, Hazama et al. have associated listener’s preference (with 2-point scale) and heartbeat interval for 40 s musical pieces obtained from their evolutionary computation method composing musical piece [Hazama & Fukumoto, 2009]. Their result showed different distribution of heartbeat intervals for preferred and not preferred musical pieces. These previous studies give us useful and interesting information about the time cost, however, precise time that sound stimulus need to elicit the change of heartbeat has not been clarified.
In our previous study [Fukumoto et al., 2009], average of heartbeat intervals in music section was larger than that in noise section significantly (P < 0.05). Furthermore, after the transition of sound stimuli, it took about 30 s to change heartbeat interval from previous section. In the investigation of the change in heartbeat interval, 20 s sliding window was employed. Additionally, a questionnaire asking relaxation feeling was used as psychological index. After the exposure of the sound stimuli, a questionnaire asks the subjects relaxation feeling for noise and music sections, respectively. Psychological result showed that music section induced the higher relaxation feeling than noise section significantly (P < 0.001). The significant difference in subjective relaxation feeling meant that the relaxing music piece and the noise were greatly different. These results in our previous study support the findings by other previous studies that investigated the effects of sound stimuli on heartbeat.
What kind areas does this study contribute for? First, as described above, theoretical use of music is first candidate of the application of this study. For example, in Guided Imagery Method, patients listen to several music pieces for a long time. If a therapist know a time length that music pieces need to elicit change in heartbeat, the knowledge must be useful to make a selection of music pieces.
From engineering point of view, with a mind to develop a musical system that reflects user’s KANSEI and psychological condition of listeners automatically, several previous studies have investigated the psycho-physiological response for sound stimulus with various physiological indices [Aoto & Ookura, 2007; Chung & Vercoe, 2006; Hazama & Fukumoto, 2009; Healy et al., 1998; Kim & André, 2004; Sugimoto et al., 2008; Yoshida et al., 2006]. Heartbeat is included the indices, and revealing time cost eliciting the change of heartbeat by sounds will contribute to musical information techniques such as automatic musical composition based on physiological index [Fukumoto & Imai, 2008].
This section has explained the background, the previous studies, and the applications as introductions of this study. The remains of this chapter are constructed as below. The section 2 describes experimental method and sound materials, and the section 3 shows results of the experiment. The section 4 discusses the effects of the sound stimuli on temporal change in heartbeat based on the experimental results. Finally, the section 5 concludes this chapter.
2. Procedure and materials
This section describes experimental method used in this study. Basically, the experimental method used in this study is referring to our previous study [Fukumoto et al., 2009]: different sound stimuli are included in one experimental set, and electrocardiogram is measured in listening to the sound stimuli. Music and noise were used in our previous study. To investigate the effects of various transitions of sound stimuli, we add mute sound as a sound stimulus. Therefore, three kinds of transitions and their inverted sequences are used in the listening experiment.
Time length of one experimental set was thirteen min. The set of the listening experiment was basically constructed from two parts; five min rest and eight min sound stimuli. The part of eight min sound stimuli was composed of two min different sound stimuli. These sounds were played two times respectively with change places their sequence. Figure 2 shows example of one experimental set. Especially, the change in heartbeat intervals in second presentations of Sound A and B were used in analysis, because the subjects already realized the sound stimuli in second presentations; remove of surprise for first time listening to the sound stimulus. Based on this experimental set, three experiments were performed, and each of the experiments included two different sound stimuli; Noise and Music, Noise and No-sound, Music and No-sound.
Sixteen males and females (mean age: 21.9±0.7 years) participated in the listening experiments as subjects. None of the subjects had professional or college-level music experience. Beforehand for the experiments, the subjects were instructed not to eat, drink and smoke anything from 30 min before the listening experiment. As described in the next section, each of three experiments included two conditions, and all of the subjects participated in the one condition in all of three experiments. Therefore, there were eight kind of combination of experimental conditions: Each of the subjects participated in three experimental sets. Sixteen subjects were randomly and counter-balanced assigned to the eight combinatins.
2.2. Sound stimuli
As sound stimuli, a music piece, white noise, and no sound were employed. In the listening experiment, Air in G composed by Bach was used as relaxing musical piece in music sections. This music piece is well known as its relaxing mood and was used as sedative musical piece in a previous study [Yamada et al., 2000]. White noise was used in noise sections. Format of these sound stimuli was WAVE format, and these sound stimuli were played by notebook. Both of the music piece and white noise were stereo recorded sound. In the no-sound section, sound was not played. The subjects listened to these sound stimuli through a headphone.
With three experiments, the change in heart beat in the transition between different sounds was investigated. Each of the three experiments was mainly composed of two different sound stimuli. Figures from 3 to 5 show outline of waveforms of sound stimuli including first 5 min rest. Horizontal axis means time, and vertical axis means sound amplitude. As shown in the outline of waveforms, to prevent to induce strange feeling to the subjects, music stimuli and noise were introduced after 10 s of fade-in and finished with 10 s of fade-out. This control of volume enabled us to construct the experiment composed of continuous different sound stimuli. In our previous study [Fukumoto et al., 2009], volume of sound stimuli was adjusted initially by the subjects themselves by listening to white noise before the listening experiment. In the present study, to adjust the experimental condition between the subjects further, the volume of the sound stimuli were fixed. The volume of noise and musical piece was arround 66.0 to 70.0 dB(A).
2.3. Psychological index
In the questionnaire after listening all of the sound stimuli, semantic differential method [Osgood et al., 1957] was used to ask the subjects relaxation feeling. The subjects estimated their relaxation feelings with 7-point scale of “relaxed - stressful” for the afforded two sound stimuli, respectively. Additionally, in the expeiment 1, the questionnaire also asked the subjects that the subject had an experience listening to the musical piece played in music sections. These questions were written on a paper in Japanese and were explained by an experimenter, and the subjects answered with writing on the paper by themselves. Additionally, in statistical analysis for relaxation feeling, sign test was used.
2.4. Physiological index
In the physiological analysis, R-waves were detected from the subjects’ electrocardiogram. R-wave approximately represents time of heartbeat, and R-R intervals represent temporal development of heartbeat intervals. Development of heartbeat interval is represented from (tn, tn+1-tn), where tn means time of n-th R-wave. In the analyses, 2 min and 20 s windows were used to observe the change in heartbeat intervals. First, all heartbeat intervals were detected from electrocardiogram. Then, average of heartbeat intervals included in each window (tn is in the temporal range of each window) was calculated.
Two kinds of windows were used for different observation. 2 min window was same as its temporal length of each section and was used for broad and rough observation for change in heartbeat intervals in each section. On the other hand, 20 s was used for detail observation, and temporal length of 20 s window was determined referring to a method of a previous study [Yoshida et al., 2006]. Heartbeat intervals in general condition contain two kinds of period of heartbeat fluctuations reflecting autonomic nervous activity and respiration, and the temporal lengths are 4 s and 10 s (in 60 heartbeats per 1-min). Length of 20 s window was available for omitting these heartbeat fluctuations. In a part of the analyses, the 20 s window slid as queue processing, and time of the window was defined as central time obtained from average of first and last time of the window. The analyses with 20 s window were mainly applied for latter two listening sections.
3. Experimental results
This section shows the psychological and physiological results of the listening experiment. In the statistical analyses, pairwise comparison was used, because individual variation of subjective evaluation and physiological indices were considered large.
3.1. Results of psychological index
First, result of questionnaire only for the experiment 1 showed that all of the subjects knew the music piece played in music sections.
Figure 6 shows subjective relaxation feeling in the experiment 1. Higher point means higher subjective relaxation feeling. As shown in Fig. 6, large difference between relaxation feelings of Noise and Music conditions was observed. Statistical analysis for this result showed that there was significant difference between Noise and Music conditions (P < 0.001).
Figure 7 shows subjective relaxation feelings of Noise and No-music Stimuli in the experiment 2. Average point of No-sound was larger than that of Noise a little bit, however, there was no significant difference.
Figure 8 shows subjective relaxation feelings of Music and No-sound in the experiment 3. Relaxation point of Music was also larger than that of No-sound (P < 0.001).
As summary of the results of relaxation feelings, Music stimulus elicited the largest relaxation feeling. Noise elicited the smallest. No-sound was intermediate, however, Noise and No-sound were almost same level.
3.2. Results of physiological index
This subsection shows the results of physiological index. First, rough change in heartbeat intervals in each section is investigated. Then, detail change in heartbeat is observed with 20 s window.
3.2.1. Change in heartbeat in each 2 min section
Figures from 9 to 11 show whole changes in heartbeat in each three experiment, respectively. In the analysis, average heartbeat intervals in 2 min sections were used. For rest section prior to listening to sound stimuli, last 2 min in 5 min rest was utilized as analyzed section. Each result was composed of average and standard deviation between subjects and was obtained after analysis of each subject.
A common tendency between the results shown in Figures from 9 to 11 was gradual extend of heartbeat intervals in correspondence with time development. It means that lowest average heartbeat was observed in the rest section except Music to No-sound condition in the experiment 3. From these results, large differences between different sound stimuli were not observed.
3.2.2. Detail change in heartbeat in a transition of sound stimuli
Figures 12 and 13 show the results of the experiment 1. Figure 12 shows change in heartbeat in latter 2 sections from 540 s to 780 s with 20 s window. The sound changed its content at 660 s. We can observe gradual change in heartbeat and tendencies that shorten of heartbeat in noise sections and extension of heartbeat in music sections.To investigate time cost eliciting change in heartbeat by listening to each sound stimulus, 20 s sliding window was used. As previous analysis section, 650 s window (640 s to 660 s) was used. From 650 s and latter windows are target sections and compared with 650 s statistically. Figure 13 shows
the results of analysis with sliding window. Gradual change with sliding window was observed, and plotted line means P-value obtained from statistical analysis. In the result of Noise to Music condition, heartbeat interval tended to be extended from previous section in 761 s window. It means that 111 s was needed to change heartbeat interval by listening to music from listening to noise. In the result of Music to Noise, heartbeat interval was shortened in 678 s window significantly: Time cost to elicit the change of heartbeat from Music to Noise was 28 s.
Figure 14 shows change in heartbeat in latter 2 sections in the experiment 2. Gradual change in heartbeat was also observed in this result.
Figure 15 shows the results of analysis with sliding window. In the result of Noise to No-sound condition, heartbeat interval tended to be extended from previous section in 671 s window: Smallest P-value was observed in the section (P = 0.0547). After that, heartbeat was gradually extended. 21 s was needed to change heartbeat interval from listening to noise. In the result of No-sound to Noise, heartbeat interval was shortened around 690 s window, furthermore, it was shortened in 756 s window significantly.
Figure 16 shows change in heartbeat in latter 2 sections in the experiment 3. Gradual and rapid changes in heartbeat were observed in this result.
Figure 17 shows the results of analysis with sliding window. In the result of Music to No-sound condition, heartbeat interval tended to be shortened from previous section in 686 s window. After that, heartbeat interval kept same level till the end of the listening experiment. In the result of No-sound to Music, heartbeat interval was extended around 752 s window.
Result of relaxation feelings of sound stimuli showed that music elicited highest relaxation and noise elicited lowest one. These results support our previous study with more concrete investigations. No-sound was intermediate. The sequence of them is reasonable, however, difference between Noise and No-sound was little (no significant). Generally, noise is believed as sound everyone dislikes. One of the reasons for that is the subjects were in the listening experiment without any task. If they have tasks to do, noise deny doing the tasks, therefore, they might feel the noise as more negative. In addition, adjustment of volume of sound stimuli between the subjects might reduce the volume for some of the subjects (The adjustment was not applied for the sound stimuli in our previous study). The reduction of volume might elicit little negative impression for noise.
Entire change in 2 min average heartbeat interval did not show large difference between sections, while a significant difference was observed between music and noise conditions in our previous study [Fukumoto et al., 2009]. Adjustment of volume might effect on change in heartbeat interval as small change. Same tendency between the present and the previous study as gradual extend of heartbeat interval was observed. The tendency is considered as caused of sitting on a chair for a long time.
Detail observation in a transition between different stimuli showed obvious change in heartbeat intervals. In the all experiments, tendency and significant change in heartbeat were observed. Time cost eliciting the change and direction of the change (extension or shorten) was quite different between experimental conditions: The direction of change obeyed subjective relaxation feelings. In our previous study that employed music and noise as sound stimuli, about 30 s was needed to observe obvious change in heartbeat interval from previous condition to post transition. In the present study, time cost of noise to elicit the change from music piece was 28 s. While the time cost of noise was almost same time length as our previous study, the time cost of music was longer than our previous study: 111 s was needed. As mentioned above, the difference was considered as caused of adjustment of sounds’ volume.
For Noise condition, the time costs were 28 s from Music. From No-sound to Noise, to observe significant change, it costs 106 s. However, around 690 s window, shorten of heartbeat was observed. With these results, Noise elicited the change in heartbeat than Music. The reason why noise affects on heartbeat earlier than music is considered that noise is a continuous sound content: from start to end, noise sound contents unpleasant sound. On the other hand, music effects on psycho-physiologically by its development. The different of sound contents might be cause of difference of time cost. Additionally, physiological change is considered faster for unpleasant and dangerous stimuli, because the human have to defend or run away from dangerous things quickly.
No-sound was a new additional condition from our previous study, and it played a role of release from music and noise and obvious start of the stimuli from no-sound. From noise to no-sound, it was needed almost 20 s to elicit the extension heartbeat intervals: With the analysis used in the present study, shorter reaction was not measured. The time cost was the fastest among the experimental conditions. Furthermore, from Music to No-sound, 36 s was needed to shorten the heartbeat intervals. The time cost was shorter than that of music. Generally, impressions and feelings to sound stimuli are believed to remain for a long time, however, the results with no-sound condition suggest that change in heartbeat interval occurred by sound stimuli disappear from 20 s to 36 s. Strength of the impressions and feelings would relate to the time cost.
In some results, although temporal changes in heartbeat intervals just after the transition were observed, higher P-values around the end of post section were also observed. This tendency was also shown in temporal development of heartbeat intervals. Homeostasis is an important function assisting the body in maintaining a constant internal environment [Rubinson & Lang, 2008], and this function keeps heartbeat interval in certain range. In the physiological evaluation processes, it should be noted that heartbeat interval has its limits to change.
Furthermore, the change in heartbeat was mainly caused of psychological change as discussed above, however, some previous studies indicated the possibility that tempo of the sound stimulus entrain listener’s heartbeat. Previous studies have investigated the physiological effect of tempo of sound stimuli on heartbeat with simple tone [Bason & Celler, 1976] and musical piece [Kusunoki et al., 1972] and have observed the entrainment and synchronization of heartbeat by the sound stimulus. According to the results of these previous studies, the change of heartbeat interval might be partly affected physiologically from tempo of sound stimulus. On the other hand, there was no possibility that heartbeat interval in noise section was entrained by white noise because white noise does not have any tempo and cycle. The difference of physical property of sound stimuli used in the listening experiment might affect the change of heartbeat intervals in each section. To clarify the effect of the tempo of sound stimuli, further investigation comparing tempo of musical piece and heartbeat interval is needed.
In this study, as fundamental investigation of temporal development of heartbeat interval in listening to sound stimuli, we investigated the effects of relaxing musical piece and white noise, and no-sound on heartbeat through three listening experiments. These sound stimuli induced the subjects’ different relaxation feelings between the sound stimuli. Averages of heartbeat intervals were almost same level. As precise observation with statistical analysis, detail temporal development of heartbeat interval in transitions between two diffrent sound stimuli were observed using 20-s sliding window. Some of the results, especiallyl for noise, supports previous studies. However, time cost of change in heartbeat by listening to music were longer than the result in our previous study.
Some previous studies have investigated the relationship user’s KANSEI and physiological response in listening sound and have tried to apply the relation between them for developing musical system; selection, arrangement and creation of musical piece. These approach aims to reflect user’s KANSEI to the system and to absorb individual difference. The results of this study will contribute to these trials, especially if the system uses heartbeat interval as physiological index.
As future study, we will investigate the psycho-physiological effects of the sound stimuli inducing different relaxation feeling with spectral analysis [Akselrod et al., 1981]. With the analysis, we can separately evaluates autonomic nervous activities based on heartbeat intervals; sympathetic and parasympathetic nervous activity. The results with the analysis would show us more precise physiological change in the change in the sound stimuli, and it will contribute to more effective applications based on user’s physiological information.
This work was supported partly by Grant from Computer Science Laboratory, Fukuoka Institute of Technology.