Audiovisual Music from the Audience’s Viewpoint

Abstract

The music of everyday life crosses areas whose borders are blurred, takes place on screens and forms part of the trinomial that connects it with the visual and narrative realms. This space of high emotional tension is created with characters, settings, stories and roles, and is of a spectacular nature. All this allows it to provide meaning, which confirms the need for this study. This work looks into audiovisual music and its connections with the narrative, image and story. To perform this approach, we will study the leitmotif, music as a figure and background, diegetic/extra-diegetic music and emotions from the audience’s viewpoint. Furthermore, to explore intermediality we will use a mixed method, with tools that include bibliographic reviews, content analyses and maps. With them, we will carry out relational and specific studies, while analysing the connections between music, image and narrative. This study explores the audiovisual experience and its perceptions through music.

1. Introduction

1.1 Music

1.1.1 Music and mediums

Today, references to ‘music’ encompass a wide range of expressions and descriptions that have always been subjected to the conditioning factors of each time and place. In turn, these factors were closely linked to the possibilities of expression and action offered by their individual context. Thus, both making music and listening to it as an audience, or its performance with voices, instruments and devices, has been a constant factor since mythological tales. The most noteworthy figure is Orpheus, who was able to rule the world with his music: mountains, trees and animals were all swayed by his voice and lyre. He even descended into the underworld to rescue his wife, Eurydice. Even though he was unable to do so, he continued singing [1]. Today, music can be heard on numerous sonic devices based on playing it back, and very frequently connected to screens. The invention of electricity represented a turning point that is only comparable to the appearance of Ars Nova and polyphony [2]. Music has evolved constantly, but especially since the late nineteenth century. It then took off after the second half of the twentieth century with the ability to play music, the emergence of popular music with rock and roll, the technological development of sound and the industry of information and leisure. In the history of western music, after Graeco-Latin culture (of which there are few remains) come the Early Middle Ages, with the gestural medium of Gregorian chants and hand movements. This was followed by writing neumes above and below a line, until the establishment of the stave, with five lines that made it possible to place the notes at different heights and express their duration. This method finally became established in Europe in the eighteenth century. The perpetuation of sound was another landmark, thanks to the appearance of the press and printed musical scores making music permanent, so it could be performed. Then, in the twentieth century, there was another major leap thanks to mechanical reproduction [3] and large multimodal spaces that combine narration with sound and images that are almost always in motion and often synchronised with the music. Then came the digital transformation, bringing together numerous means of expression, communication and art, as well as recordings, stages, screens or online stages. It encompassed the technique, the audience, live music, recorded music and noise while being a prominent setting for popular and traditional culture. Experience with music is another sign of this era, a time where silence seems to have no place or, as expressed by Benjamin, Aguirre, de Alba [3]. This evolution has made music one of the main (and almost mandatory) constructors of multimedia experiences.

1.1.2 Approach

Music is a coherent organisation of sounds and silences in its essential parameters: melody, harmony and rhythm. The way of defining its elements varies among cultures, and also among schools of thought. In general, we can say that the melody is a set of sounds and silences—created within a specific sonic environment—that are played in succession, and are perceived as having an identity and a sense of unity; the basic unit of harmony is the chord, and it regulates the concordance between sounds played simultaneously and their connection to neighbouring sounds; the metre refers to the repeated pattern, at regular (and sometimes irregular) intervals, of loud sounds, weak sounds and silences. In a composition, the rhythm is the end result of these elements. It could generally be defined as the ability to create contrasts in music, and is caused by different dynamics, tones, textures and sounds; the tone is the characteristic of sound that is caused by a specific sonic agent. In addition, there are accents, which alter the intensity (dynamic accents) or speed (agogic accent), and the musical structure or form. Lastly, it is worth mentioning that musical elements are directly linked to the characteristics of sound. The rhythm is linked to the duration, the melody and harmony to the pitch and the accents to the intensity and duration. Current artistic trends differentiate between music and sonic art, with the main difference between them being that music uses a language where the musical sound or tone is framed within scales, counterpoints, harmony and rhythm, unlike sonic art, which uses sound as a generating element, releases it and then groups it together again. This is why music must not depend on audio technology. However, Reyes (2006) believes that sonic art emerged from the connection between music, art and technology, where music and sound are located in a new expressive or gestural interface. In this interface, the sonic elements have meaning depending on the other elements and, far from being independent, they act as the sum of their parts, as was proposed by the artistic vanguard. Familiar concepts in mass media linked to sound (e.g. recorded sound, loops, sound simultaneity and manipulation, editing, tempo changes and phonetic language expressed through items) and others (e.g. stretch, compress, switch, manipulate, see the sound or perceive the visual rhythm) have become part of vanguard art in the twentieth century, and of the general uses made by the youth, as a trend, until today [4].

1.2 Language and communication

Several authors have studied audiovisual productions as constructors of reality. More specifically, their music has been studied by [5] Morley [6] Porta and Sloboda [5, 6, 7]. Furthermore, it was a subject of interest in cinema for authors such as Adorno, Leppert and Chion [8, 9, 10].

1.2.1 Narrative

The story told by music has been studied from several viewpoints. Meir Sternberg [11] talks about the differences in fiction/non-fiction (historical narrative, autobiography, reports, etc.) regarding the action of the individual who remembers, narrates and uses deictic markers that place the actions in specific space/time contexts. From the viewpoint of cinema, the narrative has been studied by Michel Chion, especially in his work of reference in the field, Audiovision [9]. Others have specifically studied the narrative possibilities of music in this field, such as Adorno, [8], among others, and its connection to images. The intervening elements from the viewpoint of major media companies are all the elements of the story, the text, the image, the narrative and the staging, with key roles being held by conflict, catharsis and the figure of the hero. From the viewpoint of semiotics, Bathes [12] studied the significant connections through narratives that express shared cultural values. Dewey [13], taking the viewpoint of the Theory of Art, made a key contribution to this field with Art as Experience. In the field of art, proposed research is based on art, in an attempt to connect what is said with how it is said. Meanwhile, narrative art analysed the form and material mentioned by Vygotsky [14] (‘form’ in terms of the plot or story, and ‘material’ in terms of the content or tale). In its connection to music, we can see growing trends towards cognitive aspects, with increased interest in the depiction Gómez-Ariza, Porta [6, 15]. The music and soundtrack are part of the audiovisual production, along with the narration and image. In some scenes, their role can be critical for the cinematographic narrative, as well as a major driver of emotions [6, 16].

1.2.2 Semiotics

Audiovisual music can be seen as a semiotic system that is structured and configured to create expressive meanings by staging a communicative chain, and whose meaning can be approached from two viewpoints: the person who speaks and the person who listens [6].

Its key points are summarised in the topic at issue. The reader is encouraged to read Porta [16] to flesh out the information. Semiotics is the science that studies the communication systems of human societies. It says that a sign is an item or event that takes the place of another, which is absent, by virtue of a certain code. In these lines, we will go over four of its main proponents. Ferdinand Saussure was the first to talk about semiology, establishing the bases for modern communication by studying everything from fashion to shows, customs, culture, verbal and non-verbal language and multimodal speeches [16]. Peirce [17] believes the sign, item and interpretation are the three main components, providing feedback to one another, creating a spiral process. Roland Barthes [12] studied popular culture and the significant connections through narratives that express shared cultural values, whereas Umberto Eco, in The Absent Structure [18], talked about the meaning as a cultural unit [4]. Davidson and Peirce summarised the contributions, agreeing on 1) defending a justification, broad and social vision of the meaning, 2) the external reality being the main source of our beliefs and 3) interdependence between thought, language and action. Lastly, we have the contribution of Vygotsky: ‘All that is internal in its higher forms must have been external. In other words, it must have been for others what it is now for one’s self […]. Any higher psychological function must have gone through an external phase in its development, as it is initially a social function’ [19].

1.2.3 Cognitive aspects

The works reviewed on music show a transition from psychoacoustic to more cognitive viewpoints, with greater interest in aspects such as learning, memory or the performance; Gómez-Ariza, Porta [6, 15]. Specifically, some authors studied the cognitive and significant effects of the media, highlighting the singularity of mental processes, the development of specific skills and its systems of symbols. There is also the research of Tal on comprehension, which took into account the restructuring theories of cognitive psychology. These theories are of a constructivist nature, and use molar units due to their meaningful nature [20]. This contribution is valuable because it shifts the importance towards the context, the habitat and its elements of significance, which, as happens in reality, are never separate [21]. The third contribution we take into account is López-Varela [22], who connected the psychological-cognitive development to the affective relationships created in the family environment [22].

1.3 Emotions

Psychology commonly defines emotion as a feeling or perception of the elements and relationships of reality or the imagination that are expressed physically, and which include behavioural reactions. Ekman, Levenson and Friesen [23] proposed six basic emotions, with the most complete definition being in Kleinginna and Kleinginna [24], who suggested several categories (e.g. affective and cognitive categories) based on external, physiological, emotional/expressive, disruptive, adaptive, multifactorial, restrictive or motivational stimulation. Furthermore, Vygotsky [25] talked about emotions as primitive feelings that cannot remain indifferent or ineffective on behaviour, and which have signed as mediators. These concepts were used in Porta and Porta, Herrera [4, 21, 26] to create experiences referenced in this study on children’s experience when listening to audiovisual productions.

1.3.1 Emotions and experience

According to Vygotsky [27], emotions are one of the hardest research topics to address. He believed that they should be approached through empirical studies on experience, its indications and connections to thought and internal language. He considered emotions as the internal organiser of our behaviour, which add tension, excite and stimulate us, causing three groups of reactions: reflexive, motor, somatic and secretory; bodily reactions; and a secondary perception of the proprioceptive field, divided into positive and negative feelings [25]. Emotions are involved in the meanings we create from our life experiences [28].

1.3.2 Music and emotions

The effects of music on emotions are a field of rising interest, especially in psychology, health sciences and social studies, with a broad bibliography, that we will go over in the bibliographic review. From the experiential side on the effects of music on filmic and audiovisual tales and narratives, Amparo Porta and Porta, Herrera [16, 26] performed audiovisual experiences to study their effects on children from a Vygotskian perspective, finding connections between music, emotions and some narrative elements, the first of which is the hero. The perception of the story, the plot and the personality of the hero cause conflicting emotions and are a key process in the tragedy, which requires specific features [29]. The second is the catharsis. From the Vygotskian viewpoint, emotions, in cinema, create an alternating balancing system of opposite feelings. For the viewer, there is a catharsis when negative emotions are offset or neutralised by positive ones in the filmic process [16].

1.4 Cognition, perception and comprehension

1.4.1 Perception

Luria, from the viewpoint of cognitive psychology, says that perception is the first chain in human cognition [30]. In the case of auditive perception, it is generated by the sense of ‘hearing’, which he differentiates from listening as the cognitive generative element of speech, sound and music.

1.4.2 Trends

To approach cognition and comprehension, we will begin with the post-structuralist viewpoints that address the construction of conscience [19]. The theory of art focuses on the representation of the world [31]. Music requires a specific approach [15] that helps understand listening as a social creation that has meaning. The musical approach is necessary because music speaks its own language [6]. The literature review revealed that psychology highlights the contributions of cognitive psychology on the processing of information and restructuring theories, with noteworthy works and authors such as Piaget, Vygotsky or Gestalt psychology [17, 19, 32]. One of the significant changes in significance lies in the analysis units, which propose the use of molar units. Of these, we highlight Vygotsky’s socio-historical theory, which was subjected to a process of reinterpretation and possible application to audiovisual comprehension, according, and whose principles and some practical implementations have been performed in works by Porta [4]. The Russian author shows how mental processes are explained by the instruments and signs, which act as mediators and have meaning. For several authors, this is the most original and important contribution of Vygotsky, who leans on the study of the communicative nature of the signs [4].

1.4.3 Understanding music and the listening process

Music cognition has been studied by several authors, such as Meyer or Sloboda [7, 33]. We are interested in the act of listening to the music and the soundtrack, in a significant way and with meaning. Taken as such, audiovisual music represents a polyhedron with numerous sides: musical, narrative, persuasive, patrimonial, temporal, cultural and cognitive, in addition to its connections with identity, communication and society.

1.4.4 Method

The scientific method frees thought of all doubt, and makes it possible to obtain increasingly reliable and more stable patterns of action, as it sets beliefs based on ‘something permanently external’ (i.e. a reality detached from our opinions) [34]. In a complementary way in multimodal discourse, modes are resources that allow the simultaneous realisation of discourses and types of (inter) action. This is why the first premise when studying this topic is to locate the different communicative practices and their procedures.

Regarding the positions of the disciplines and their methods, Vygotsky highlights as an issue the connection of emotions with adjacent fields of knowledge, such as the psychology of language, linguistics and the psychology of education. Intermediality suggests dialogue between disciplines interested in expanding fields and the value of experience.

1.5 Intermediality

According to Irina Rawjesky [35], there are two different approaches to studying intermediality. 1) The approach resulting from literary studies and narrations, and 2) the approach of communication studies.

In this work, we include both approaches, based on audiovisual experiences with music studied as a discourse and content, understanding intermediality as a meeting point between material, semiotic and cultural aspects. We are interested in ‘Where are they?’ and ‘What do they say?’ from the viewpoint of music.

1.5.1 Where are they?

When we talk about audiovisual music in this live and changing environment, music and the media form an inseparable pairing [21]. The audio is the space for the soundtrack and the place where its music lies together with the image, shaping a story (the tale). We approach all of this, taking into account its form and content, as well as its texts and discourse [36]. However, empirical research requires the use of instruments. Music is produced over time, which is why its analysis must have a time-based anchor that must be doubled, in order to answer what is being listened to, and where the material being listened to is located. In this case, the first issue to address is its limitations to being observed and analysed objectively [6, 21].

1.5.2 What do they say?

Audiovisual productions tell stories with music and moving images. Therefore, they shape a space of interaction that we want to explore as a narrative and also as a discourse. The meaning of audiovisual music can be approached from two viewpoints: from the person who speaks, and the person who listens [6]. From the viewpoint of the person who speaks, linked to poiesis it has been studied by musical language, the theory of music and musicology. From the viewpoint of the person who listens, it has been studied by ethnomusicology, sociology and cultural studies, among others, focusing on social discourses. Communication sees audiovisual music as a discourse, an instrument of signs [37], whereas education takes part in both viewpoints due to its scope and history [16]. All this involves the context, the background and the listening skill as a dynamic and unfinished process [16]. From the viewer’s perspective, listening and the meaning take place through the summation of languages [6, 21]. This has been studied by different schools, including the School of Frankfurt, with authors such as Adorno, Leppert, W. Benjamin and E. Gombrich [3, 8, 31]. Sometimes, music has a spectacular nature and becomes a ritual. This happens in rock concerts, where mass expressions of popular urban music take place [6, 38]. Considering all the above, music reaches everyone through platforms of mass dissemination, which is received anonymously and produces effects that have hardly been studied [4].

1.6 The importance of experience

We provide three ideas to shape this section. The first is the concept of habitus. Pierre Bourdieu [39] says that habitus is the generation of actions limited by the social circumstances that withstand them. They are imprinted in the human body and mind and shape subjective social structures.

The second idea is the possibility of describing the world in an objective (consensual) way, which depends on our intersubjective communication skills [22]. The third is to find a place or medium that can make it possible, and, according to López-Varela, the common factor among these platforms is the inclusion of one or several mediums in another form of communication. The methodological result of all this entails not only different physical ways of coding the human experience but an amalgamation of cultural relationships (processes) that include several channels for processing the information obtained from those experiences (visual, aural, etc.) with changes in the communicative contexts [22].

In this study, we want to show music together with the image and narrative: the synchronisation, the creation of worlds and realities from sound, their ability to tell stories, the place and position of the narrator, their evocative power, the task of informing of and describing elements without listening (such as space/time and other external elements), their ability to hold diegetic and extra-diegetic positions, cause reactions that affect a different sense through sound [26] and create dialogues between emotional, cognitive and narrative aspects in an intermedial way.

2. Method

This study preferentially used a qualitative methodology, analysing content by the importance of the context [40] and its intermediality, as well as quantitative elements, to achieve, through triangulation, the best results possible. All from the global viewpoint of the project that we have been conducting since 2017, with more recent results in Porta [16], and which allows us to continue expanding our position, contents and coverage. We used the grounded theory [41] and the constant comparative method of Glaser and Strauss. This study has a descriptive and interpretive nature, with the noteworthy presence of a qualitative analysis [36, 42] that is systematic and objective.

Objective: This study focuses on audiovisual music and its relationship with the story, the narrative and the image. Its objectives are as follows:

1. To identify the prominent topics in active audiovisual experiences conducted by 150 participants;

2. To analyse its presence in Google, Google Scholar and Web of Science (WOS).

3. To understand the relationships, specificities and connections between the music, image and narrative using MxQda.

To conduct it, we performed the following actions: a) Starting point: Selecting experiences performed by 150 students of bachelor’s degrees and postgraduate studies in music-related courses, b) Bibliographic review, c) Content analysis, d) Creation of maps using MxQda and e) With them, we will perform relational and specificity studies, as well as examining the connections between music, image and narrative.

2.1 Design of the investigation, materials and procedure

This work is part of a financed project on audiovisual listening. This paper addresses one of its objectives. Selecting the methodological system allowed us to approach our objectives, learn the background and implement the necessary tools to conduct it. In all, the design and research were conducted in four phases, with three tools and two content analyses (Table 1).

Phase	Purpose	Tools	Evidence
PHASE 1	Selection of participants’ activities for the initial selection of contents on prominent topics	Excel for quantitative measurements and MxQda	Revision of activities, results and analysis
PHASE 2	Bibliographic review of the field	Bibliographic selections: Databases, WoS, Google Scholar.
PHASE 3	Selection of concepts and topics	Analysis: MXQDA	Content analysis
PHASE 4	Two content analyses	MxQda and the word cloud for the qualitative analysis	QA1; QA2

Table 1.

Research design, materials and procedure.

2.2 Starting materials, participants and experiences

We performed an initial exploration of experiences with audiovisual music, preferably cinema and video games, performed by students of higher studies in musical education [42]. We selected the activities whose topics and approaches had been chosen freely by the students themselves. With the results obtained, we performed an initial content analysis, which determined the topics that were going to be the centrepiece of this chapter.

2.3 Participants

A total of 88 bachelor’s degree dissertations (TFG, in Spanish) and master’s degree dissertations (TFM, in Spanish) in Music Didactics were selected for reasons of continuity with our current projects.

2.4 Tools used

We used the Excel program for the quantitative measurements and the bibliometric analysis, and the MxQda program for the qualitative analysis. For the content analysis, we used the procedure that had already been implemented [16, 42].

3. Results

3.1 Results objective 1. Content analysis 1

Our objective was to identify the most prominent topics in audiovisual experiences performed by 150 students with bachelor’s degrees and postgraduate degrees related to musical education. To do so, we chose the most prominent topics from the dissertations of students’ projects from 2020, 2021 and 2022. The audiovisual topic had to have been chosen freely. When searching for them with content analysis 1, using the program MxQda, we found 88 documents. Of these, the most commonly selected topics were: emotion, feeling, diegetic, extra-diegetic, leitmotif and background music. This was the first step to study the presence of this group’s interests in academic and research texts. We searched and filtered results in Google, Google Scholar and the Web of Science, thus completing a journey on the topics chosen by the students from the informative sphere to the scientific realm [42].

3.2 Results objective 2. Bibliometric study

The second objective proposed was to search for the topics on Google, Google Scholar and the Web of Science (from less restrictive to most restrictive), to analyse their presence and communicative interests.

3.2.1 The audiovisual music experience on Google

We initially searched for the words ‘music’, ‘audiovisual’ and ‘experience’ to limit the scope of the search. Then, the search targeted the topics chosen by the students, obtaining the following results, ordered by the number of results found: Audiovisual music experience: approximately 10,600,000 results; Audiovisual music emotions: approximately 3,740,000 results; Audiovisual music background: approximately 3,330,000 results; Diegetic and/or extra-diegetic audiovisual music: 46,300 results for diegetic music, and 18,900 for extra-diegetic music, for a total of 65,200 results; Audiovisual music leitmotiv: approximately 63,200 results.

3.2.2 Searches in Google scholar

The results already show a drastic decrease in results. For example, in the case of audiovisual music experience, the number of results went from 10,600,000 to 237. The bibliometric study was performed with the latter amount, using the entries obtained in the report on publications, citations, fields and main concepts. It provided the following results and analysis. The information obtained in publications and citations reveals a slow start until 2007. We obtained the following results, ordered by their appearance (as a percentage): Fields: Behavioural Sciences (74, 35.71%); Psychology (70, 33.17%); Music (66, 31.28% - third place); Neurosciences Neurology (47, 22.27%); Computer Science (43, 20.379%); Arts Humanities Other Topics (39, 18.483%); General Internal Medicine (29, 13.744%); Communication (28, 13.270%); Film Radio Television (22, 10.427%); Education Educational Research (21, tenth place, last). Regarding the main concepts, the first two spots were held by ‘behavioural sciences’ and ‘psychology’, whereas ‘music’ is in third place, ‘method’ in sixth and ‘educational research’ last.

3.2.3 The research response

We then searched for the students’ most prominent topics, in addition to studying the results of their perceptions and research methods used. All in order to learn the research activity present on the Web of Science. We obtained the following results:

3.2.3.1 Featured topics. Music

In connection to the topics explored in the experiences performed by 150 students, we found 68 results, connected to emotion OR feeling OR diegetic OR extra-diegetic OR leitmotif OR background music. Of these, we analysed the report of citations and publications, fields and main concepts. The analysis yielded the following results: Publications and citations. Publications start being significant in 2007, and citations start rising in 2019. Fields: 1 and 2 psychology, 3 music, 4 neurology and 5 computer science. Main concepts: 1 Coordination, 2 nervous system 3 behaviour, 4 method and 5,6,7 medicine. Country with the most results is the USA.

3.2.3.2 Perceptions

We conducted a specific study of the perceptions, analysing the relationships between the experience of music and cognitive construction from the standpoint of people’s perceptions. A total of 80 results were obtained, of which we analysed the fields of research that had this topic as a focus of interest. The report of citations, fields of research and main concepts were used, with the following results and analyses: Information on publications and citations: Publications begin in 2002, creating constantly rising double and triple peaks. The highest number of citations was registered in 2016, with 240. Regarding fields, the first place was held by ‘neural coordination’. Second was ‘sensory reception’, ‘method’ was eighth and ‘music’ and ‘education’ do not appear. As regards concepts, the most prominent was ‘methods and Ttchniques’ (8). Most noteworthy country is the USA.

3.2.3.3 Audiovisual music experience. Method

Lastly, from the results obtained from the audiovisual experience, we explored the research methods. A total of 84 results were obtained, which we analysed through a report of citations and publications, fields and main concepts. Report on publications and citations: Publications were uneven, but follow an upward trend. The highest number of citations was 160. There were five main concepts, which did not include ‘method’, ‘music’ or ‘education’. Countries: First, the USA.

3.2.4 The elements of analysis in WOS

Through reports on citations and publications, we obtained the following results:

3.2.4.1 Audiovisual music experience

Specific results of interest: Experience, perception and method. The highest specificity of music was obtained in ‘experience-perception’. The most prominent field of interest was ‘experience’, with 23 articles and 360 citations, both obtained between 2002 and 2022. It was followed by ‘perceptions’, which also began in 2002 but started increasing in 2007 (2), reaching its peak in 2016 (8), and registering 250 citations in 2021. Lastly, ‘methods’ began in 1976 (1) and increased in 2007 (2), growing unevenly until reaching its peak in 2019 (10 articles and 150 citations).

Audiovisual music experience publications: A very slow start in 1976, appearing later, in 1987 and 1988 (1 publication). Since 2007, it had a continued rise, reaching a peak of 21 in 2019. Citations: Significant since 2007, upward trend. Maximum of 360 in 2021.

Perceptions and publications: Started in 2002, maximum of 250 in 2016. Citations: Significant since the maximum of 250 in 2021.

Audiovisual music experience. Method and publications: Beginning in 1976. Highest of 10 (2019). Citations: Significant since the maximum of 160 (2022).

3.2.4.2 Topics and the study of emotions

Of the prominent topics of interest highlighted in the experiences conducted by students, the Web of Science produced 57 results on emotions and feelings. We conducted a detailed analysis through a report of citations and publications, fields and main concepts.

Report of publications and citations: The publications began around 2008, with uneven peaks and an upward trend. The most prominent years for citations were 2019 and 2022.

Fields: ‘Psychology’ is first, ‘neurology’ second and ‘music’ fifth.

Regarding the main concepts, the first place was taken by ‘methods and techniques’.

Countries: First, the USA.

Reports of publications and citations:

The four favourite topics were leitmotiv, background music, diegetic/extra-diegetic and emotions. The answer by the Web of Science: The students’ four favourite topics start appearing on the WoS around 1976, when there is just one publication, until 2001. The number of publications rose in 2007, and in 2015 it reached its peak, with eight.

Publications: Beginning in 1976 (1 publication), with its peak in 2015 (eight publications). Citations: Significant, with 240 citations in seven publications in 2021.

Only ‘emotions’ and ‘feelings’. Publications: Beginning in 2002, with a peak of 7 in 2019 and 2022. Citations: Significant since 2011, with a maximum of 180.

3.2.4.3 By fields of knowledge

The presence of music in an analysis by fields: Audiovisual music experience: ‘music’ in third place, and ‘education educational research’ tenth; perceptions: ‘music’ in third place; audiovisual music experience method ‘music’ in third place and ‘education educational’ tenth.

3.2.4.4 Concepts

The presence of music in the main concepts: Audiovisual music experience ‘methods and techniques’ sixth place; perceptions: ‘methods and techniques’ eighth place; audiovisual music experience method ‘methods and techniques’ fifth place.

3.3 Results objective 3. Content analysis 3

The third objective was to understand the relationships, specificities and connections between music, image and narrative. To do so, we used the program MxQda (for qualitative analyses) to perform a second content analysis. We selected the topics and their presence from the exploration performed in the Web of Science, summarised.

This section has responded to typologies and domains, for which the themes and their records have been used. We present the three tables that comprise it with codes: documents, segments, codes and superior codes (Table 2).

Table 2.

Content analysis 2. Documents, topics, codes, segments and records.

3.3.1 Typologies

Documents with codes.

On music and sound: The word ‘music’ in WoS (72, 85.71%), ‘tone’ in WoS (28, 33.33%) and characteristics of music (9, 10.71%).

On Perception: The perception of emotions (4, 4.76%), the perception of movement (3, 3.57%) and the perception of time (5, 5.95%).

On the Perception of Health: Perception of pain (1, 1.19%) and perception of anxiety (1, 1.19%).

Specifically, on the perception of music and sound, in WoS, we found ‘perception’ in ninth place, with 47 results (55.95%). ‘Aural and musical perception’ had six results (7.14%), and ‘the perception of audiovisual music’ had two (2.38%).

Segments with codes

Here are the segments obtained and the percentage for three groupings: music and sound, perception and health.

Music and sound: The words ‘music’ (368, 60.63%), ‘perception’ (103, 16.97%).

and ‘tone’ in WoS (61, 10.05%). Characteristics of music (27, 4.45%), the perception of music (9, 1.48%), the perception of audiovisual music (3, 0.49%) and aural perception (10, 1.65%).

Perception: The perception of time (13, 2.14%), the perception of emotions (5, 0.82%) and the perception of movement (4, 0.66%).

Health: The perception of pain (3, 0.49%) and the perception of anxiety (1, 0.16%).

3.3.2 Domains

Total of 88 documents containing music and perceptions on the Web of Science were analysed, showing the results obtained by large categories (Figure 1) and by codes (Figure 2). It has been made by Matrix of Codes of a selection of documents and marked codes, with indication of the quantities added by adjustment of the figure. The domains observed by large categories correspond to musical characteristics, perception, music, perception in RIS.

Figure 1.

Domains observed in the major thematic categories.

Figure 2.

Domains observed by codes.

The word ‘music’: (368), ‘perception’: (151), all music: (88).

In order, the results were as follows: Timbre (61), perception of time (13) and music perception (12) (Figure 2).

3.3.3 Relationships between the codes

Typologies: The nine categories (Figure 3).

Figure 3.

Relationships between the codes.

MxQda 2018. View of the relationships between close codes, with a distance of one paragraph.

Highest number: ‘Timbre’ (166) and ‘musical perception’ (123).

Lowest number: ‘Anxiety’ (11) and ‘pain’ (6) (health-related topics).

‘Timbre’ is connected to ‘musical perception’ (21), ‘aural perception’ (9), ‘emotions’ (9), ‘the perception of movement’ (6), ‘audiovisual music’ (5) and ‘anxiety’ (2).

‘Aural perception’ is connected to ‘tone’ (9), ‘the perception of audiovisual music’ (3) and ‘musical perception’ (8).

‘The perception of emotions’ is connected to ‘plantilla’ (12) and ‘tone’ (9).

‘The perception of audiovisual music’ is connected to ‘tone’ (5) and ‘aural perception’ (3).

‘Musical perception’ is connected to ‘tone’ (21) and ‘aural perception’ (8).

The word ‘music’ in WoS is connected to ‘tone’ (185), ‘aural perception’ (48), ‘musical perception’ (53), ‘the perception of audiovisual music’ (27), ‘the perception of time’ (26) and ‘emotions’ (19).

3.3.4 Method by comparison of documents

5 MxQda 2018 graph of the comparison of documents (Figure 4).

Figure 4.

Graph of the comparison of documents. BLUE the characteristics of music. PINK perception. BROWN method.

3.3.5 Method WoS the model of a code

Regarding the method, represented in the map obtained by a code model (Figure 5), the results of the investigations (WOS) are shown separated by a line. Above the line, we show the results of the perceptions of music in a broad sense and, below the line, we show the most outstanding non-musical perceptions. In this second group, we highlight the highest results and their relationship with music. They are the perception of time, obtained by the sum of temporal perception and audiovisual time perception (13), followed by the perception of emotions (5).

Figure 5.

Method WoS the model of a code.

4. Conclusions

Music is a form of expression that has accompanied society since its origins. Boulez defined music as sound in time. Its forms have changed, but it has been present in all stages of life, accompanying people, institutions, societies and cultures. This chapter has looked at music as an expressive, communicative and aesthetic element that affects people and takes shape through voices, instruments and devices. One of the most common spaces is screens, where real and fictional spaces are created. It is also through them that stories are told, words are spoken and the viewers understand the message. From the viewpoint of intermediality, Irina Rawjesky suggests two ways of approaching its study and what to focus on: one derived from literary studies and the narrative, and another one derived from communication studies. In this paper we looked at both, understanding intermediality as a meeting point between expressive, material, semiotic and cultural aspects. We strived to answer ‘Where are they?’ and ‘What do they say?’ regarding music in audiovisual experiences. On screens, music can be found in the soundtrack, which, together with the image, tells us stories. The second question is no less important: What does music say? Music, together with the narrative and the image, creates a synchronisation and creates worlds. It sometimes takes the position of the narrator and describes aspects that are unnoticeable through sound, such as the temperature, the focal point and others. It settles into invisible angles and achieves a significant symbiosis between emotional, cognitive and narrative aspects in an intermedial way. We have approached the topic using a scientific analysis.

This study focuses on audiovisual music and its relationship with the narrative, the image and the story. Its objectives have been 1. To identify the prominent topics in active audiovisual experiences developed by 150 participants, 2. To analyse their presence on Google, Google Scholar and the Web of Science (WoS) and 3. To understand the relationships, specificities and connections between the music, image and narrative.

For this approach, we used a qualitative methodology by applying a content analysis on the importance of the context and intermediality, and a quantitative methodology to achieve, through triangulation, the best results possible. We used the grounded theory, which has a descriptive and interpretative nature, and whose most noteworthy feature is the qualitative analysis, which is systematic and objective. To do so, we started with a selection of experiences of bachelor’s degree and master’s degree students, performing the following actions: a bibliographic revision, content analyses, creating maps with MxQda and relational and specific studies.

Selecting the methodological system allowed us to approach our objectives, learn the background and implement the necessary tools to conduct the study. The research was conducted in four phases, with three tools and two content analyses, as well as using expressive and analytical activities. The four phases were the following: P1: Identifying the participants’ activities for the initial selection of prominent topics in Excel and MxQda, P2: Bibliographic revision on the Google Scholar and WoS databases, P3: Selection of concepts and topics and analysing them using MxQda and P4: Performing two content analyses, CA1 and CA2, with MxQda.

We performed an initial exploration of experiences with audiovisual music, preferably cinema, cartoons and videogames, performed by students with the Bachelor’s Degree in Musical Education. We selected the activities whose topics and approaches had been chosen freely by the students. With the results obtained, we performed an initial content analysis to determine the topics that were going to be the focal point of this chapter.

Participants: A total of 88 bachelor’s degree dissertations (TFG, in Spanish) and master’s degree dissertations (TFM, in Spanish) in Music Didactics were selected.

Tools used: We used the Excel program for the quantitative measurements and the bibliometric analysis, and the MxQda program for the qualitative analysis. For the content analysis, we used the procedure implemented in Porta [16, 21].

4.1 Results and their analysis

4.1.1 Objective 1 results

We selected the most prominent topics from the projects of students from school years 2020 to 2022 with topics related to audiovisual music. Through content analysis 1 with MxQda, we obtained 88 documents. Of these, the most prominent topics were: Emotion, feeling, diegetic, extra-diegetic, leitmotif and background music.

Then, we explored these initial results in academic and research documents by using progressive filtering on Google, Google Scholar and the Web of Science to make a journey from the communicative or initial environment (of students) to scientific databases, regarding the topics chosen.

The second objective was to perform thematic searches. First on Google, for being the least restrictive, and then on Google Scholar and the WoS to make a journey from the communicative or initial environment (of students) to scientific databases, regarding the topics chosen. The search revealed a drastic decrease in results, with ‘audiovisual music experience’ going from 10.6 million results to 237 in the different search engines. The exploration of the main publications, citations, fields and concepts provided elements of interest.

4.1.2 Presence of the prominent topics, perceptions and methods in the WoS

The students’ most prominent topics fit into two main categories: ‘Music’ (diegetic/extra-diegetic, leitmotiv, and music as a figure and background) and ‘emotions’. Their publications began in 2007, with ‘music’ being the third most common keyword, and ‘method’ the fourth. The most prolific country is the USA in all the analyses performed.

4.1.3 The topics of the Web of Science

The case of emotions. Of the 57 results found, publications began in 2008, with peaks in 2019 and 2022, and an upward trend.

4.1.4 By fields

The first two spots were held by ‘psychology’, and ‘music’ was fifth, with ‘methods’ and ‘techniques’ being the most prominent concepts.

4.1.5 Perceptions

We specifically conducted a study on perceptions, analysing the connections between the experience of music and cognitive construction from the viewer’s perception. We obtained 80 results and analysed their fields of research. The publications began in 2002, with different peaks. The highest number of citations (240) was registered in 2016. By fields, the two most common ones were ‘neuronal coordination’ and ‘sensory reception’, while ‘method’ was eighth. Meanwhile, ‘music’ and ‘education’ do not make an appearance. By concepts, the most noteworthy is ‘methods and techniques’.

4.1.6 Audiovisual music experience. Method

There is an uneven but upward trend of publications. Regarding citations, the highest number was 160, and there were five prominent concepts, which do not include ‘method’, ‘music’ or ‘education’.

4.2 From the study of the web of Science and its elements of analysis

In the report of citations and publications, we obtained the following results: 1) Audiovisual music experience and the most prominent topics of students (leitmotiv, figure/background, diegetic/extra-diegetic, emotions). The results in the WoS of the students’ favourite topics start yielding results in 1976 when there was just one publication. There were no more until 2001. The number of publications rose in 2007, reaching its highest peak in 2015. 2) Specific analysis of emotions and feelings: Publications began in 2002, with peaks in 2019 and 2022. Citations began in 2011, with a peak of 180. 3) By fields of knowledge: There is a prominent presence of ‘music’ (third place) and ‘educational research’ (tenth place) in the ‘audiovisual music experience’ category. ‘Music’ is also in third place in the ‘perceptions’ and ‘audiovisual music experience method’ categories. Lastly, ‘educational research’ is in seventh place in the ‘audiovisual music experience method’ category.

4.2.1 Objective 3 results: Content analysis 2

Performed with MxQda to understand the relationships, specificities and connections between music and the other elements and categories, selecting the topics and how often they appeared. Regarding ‘typologies’, the WoS had ‘perception’ in 55.95% of the results, ‘aural perception’ and ‘musical perception’ in 7.14% and, more specifically, ‘the perception of audiovisual music’ in 2.38%. Regarding ‘domains’, we used a matrix of codes to obtain the main categories, which were ‘music’ and ‘perceptions’, in 88 documents.

4.2.2 The connections between the chosen codes

They were obtained using a single-code model. By typologies and number of connections, the highest categories were ‘tone’ with 166 results and ‘musical perception’ with 77. The lowest positions were held by ‘anxiety’, ‘pain’ and ‘health’. Specifically, from highest to lowest: ‘tone’ is connected to ‘musical perception’, ‘emotions’, ‘the perception of movement’ and ‘audiovisual music’; ‘aural perception’ is connected to ‘tone’, ‘the perception of audiovisual music’ and ‘musical perception’; ‘emotions’ is connected to ‘musical perception’ and ‘tone’; ‘the perception of audiovisual music’ is connected to ‘tone’ and ‘aural perception’.

4.2.3 Method

The connections were obtained by comparing documents with a single-code model. This allowed us to obtain a map and analyse ‘music’, ‘tone’, ‘perception’ and ‘method’.

4.3 In conclusion

In this way, we have been able to verify the great difference in interests that exist in audiovisual themes related to music, among students, and the interests of scientific society. We have verified that scientific studies are interested in experience, followed by perceptions and finally by research methods. The analysis of the fields of knowledge indicates that the most prominent are occupied by neuronal and psychological aspects. ‘Method’ ranked 8th (being the most prominent), and, very importantly, ‘music’ and ‘education’ do not appear. Regarding concepts, five appear as the main ones, which did not include ‘method’, ‘music’ or ‘education’. About their domains and relationships. The domains observed by large categories correspond to musical characteristics, perception and music. Regarding the rest of the categories, music maintains relationships in order of importance with emotions, the perception of time and elements such as the perception of movement, time, anxiety and pain. The second content analysis has allowed us to observe through various lenses simultaneously and successively; the presence, domains and relationships of the audiovisual from the music side, covering a territory made up of 88 documents, synchronously analysing its cognitive, emotional, narrative and musical elements from the perspective and sum of each of its components and experiences (Figure 6).

Figure 6.

Document portrait of one of the experiences used in conducting the study.

Acknowledgments

Project “Television as a sound habitat. Study of the effects of the soundtrack and its audiovisual narratives in childhood” of the R + D + i call of the Ministry of Economy and Competitiveness of Spain (MINECO) (EDU2012-36404).

Conflict of interest

The authors declare no conflict of interest.