This chapter sketches recent evolutions of semiotics as applied to music. Rather than providing merely a historical overview, it focuses mainly on the pragmatic turn in semiotics and the role of sensory experience in the process of musical sense-making. In order to elaborate on this experience, it delves into theoretical groundings of second-order cybernetics, biosemiotics, and ecological psychology, which are then applied to the field of music. Much effort is made to provide a broader framework to illustrate the transition from a disembodied to an embodied approach to musical semiotics. Special emphasis is laid on the concept of affordance and the role of interactions with the sounds.
- musical sense-making
- musical experience
- pragmatic turn
- embodied cognition
- enactive cognition
1. Introduction: semiotics as a discipline
Semiotics has a long tradition as the science of signs. There is, however, no agreement as to a general definition of the field. Three major descriptions have been proposed: semiotics as the science of signs and communication systems, semiotics as a description that leans upon linguistic methodology, and semiotics simply as scientific description . The linguistic strand has received most attention in the past, with a strong impetus from French structural linguists as de Saussure and Greimas. This structural approach has been valuable in providing basic conceptual tools, e.g., “signifier” (the material sign that signifies) and “signified” (that which is referred to) . This structural approach has been challenged, however, by its blindness to the role of the “sign user” in the process of sense-making. Semiotics thus had to broaden its field by encompassing also the interpreting mind with a transition from a dyadic to triadic approach as emphasized already within analytical philosophy, action theory, general systems theory, and the semiotic tradition of Morris and Peirce. Basic in this approach is the dynamic relationship between three levels of semiotic reference: the material sign vehicle, the object it refers to, and the final decoding by the interpreter.
This has been most notable in the pragmatic approach to sense-making, with an initial opposition between two traditions in semiotics which were greatly independent from each other: the “Anglo-Saxon tradition” which was oriented mostly to the theoretical framework of Peirce and the “continental tradition” (mostly Italian, French, and Slavic) with a principal orientation to the schools of de Saussure and Hjelmslev. Both positions, however, have started to come nearer to each other as a result of the pragmatic turn in philosophy [3–7]. This has been the case also for the domain of music and performing arts in which music has a primordial role [8–11]. There is, in fact, a growing influence of the experiential dimension in the study of music in all its aspects. This embraces theories of externalization (embodiment, corporality, individual biographies of composers/performers/listeners) as well as other disciplines such as theories of performance, neurosciences, and cognitive sciences and other methodological approaches which are based on the subjectivity of expression (presence, effects of presence, transmediality, and relation between body and machine). There is, so to say, a broadening of the field of study which investigates music from the points of view of cultural esthetic practice, the performing process, and historical-cultural forms. Music, in this view, can be considered as a spectacular phenomenon with multiple dimensions which can be studied in its intermedial dimension . As such, concepts as transdisciplinarity and transmediality have been substituted for interdisciplinarity and intermediality in the sense that the prefix “inter” implies that the different disciplines can at least be distinguished from each other, whereas the prefix “trans” calls forth an interpenetration and abolition of possible differences. This is obvious, for example, in the context of an opera or an installation, in which case it is very difficult to distinguish between what is merely musical and the physical presence of the musician and the poetic language that is used. Many efforts have been done and are being done, therefore, to describe this complexity in a more systematic manner [10, 11].
2. Semiotics: an operational approach
Semiotics, as the “science of signs,” has been criticized for the lack of intersubjective validity of its conceptual framework. This holds true for the French tradition but also for the numerous taxonomies of signs that were proposed by Peirce. There is, however, a more operational approach to semiotics which was introduced by Morris [13, 14] who distinguished between three dimensions of the semiotic process with respect to the relation of signs to signs, objects, and interpreters and which he coined, respectively, as syntactics, semantics, and pragmatics. The latter, especially, has broadened the field to include the reactions of sign users.
Pragmatics, in fact, has been fruitful in introducing the observer as part of the semiotic process. It was Peirce, in particular, who put the sign in a triadic relation. Going beyond the scholastic conception of reference—aliquid stat pro aliquo—with the sign process breaking up in something that signifies and something that is signified, he broadened the Saussurian distinction between signifier and signified to include the consciousness of the interpreter in the sign process: A sign, or representamen, is something which stands to somebody for something in some respect or capacity. It addresses somebody, that is, creates in the mind of that person an equivalent sign, or perhaps a more developed sign. That sign which it creates I call the interpretant of the first sign. The sign stands for something, its object.  (p. 135).
A sign, or representamen, is something which stands to somebody for something in some respect or capacity. It addresses somebody, that is, creates in the mind of that person an equivalent sign, or perhaps a more developed sign. That sign which it creates I call the interpretant of the first sign. The sign stands for something, its object.  (p. 135).
This distinction between sign, object, and interpretant has been a major contribution to the operational approach of the sign process. It has even received renewed impetus with theoretical elaborations in the field of “second-order cybernetics” and the field of “biosemiotics.”
Second-order cybernetics presents a paradigm change in scientific discourse which conceives of the observer as a participant and as part of the observed system, with a major focus on the role of interaction, emphasizing the role of the knower and observer rather than the known things or events [16–23]. It stresses the role of subjectivity and its influence on our reactions to the environmental outer world. As such, it must be considered through the first-person perspective and with active verbs.
Biosemiotics, on the other hand, can be described as that area of knowledge which describes the biological bases of the interaction between an organism and its environment [24–26]. It typically studies those signification processes which are typical for living organisms in general and which are rooted in their biology (for an overview, see  and [28, 29] for an application to music) and can be considered as an interdisciplinary field of theoretical and empirical research of communication and signification in living systems, with a focus on the study of the behavior of living systems in their interaction with the environment.
As such, a full description of perceiving cannot be given by analyzing only either the organism or its environment (organism-environment dualism). What is needed, on the contrary, is an approach which is not “animal/organism neutral” but which treats the environment as perceived.
Central in this approach is the role of circularity between action and perception. It is an idea which has been retaken in current research, with a culmination in the recent boost of perception-action studies [30–36]. They all stress the role of the observer in establishing new semiotic links with his or her environment as the result of previous interactions with the outer world. Starting from seminal contributions by von Uexküll and the ecological framework by Gibson, revolving around the concept of affordance and active search for information, a whole research program has been set up, which is likely to provide a substantial body of empirical grounding for semiotics as the science of sense-making.
Von Uexküll’s work (see [29, 37, 38] for musical applications) has been seminal to the field. His key concept of functional cycle (Funktionskreis) is a very useful contribution to the study of interactions between a human/animal organism and the objects of its surrounding world: Figuratively speaking, every animal grasps its object with two arms of a forceps, receptor and effector. With the one it invests the object with a receptor cue or perceptual meaning, with the other, an effector cue or operational meaning. But since all of the traits of an object are structurally interconnected, the traits given operational meaning must affect those bearing perceptual meaning through the object, and so change the object itself.  (p. 10)
Figuratively speaking, every animal grasps its object with two arms of a forceps, receptor and effector. With the one it invests the object with a receptor cue or perceptual meaning, with the other, an effector cue or operational meaning. But since all of the traits of an object are structurally interconnected, the traits given operational meaning must affect those bearing perceptual meaning through the object, and so change the object itself.  (p. 10)
The basic mechanism of the functional cycle is a simple, recursive loop between action and perception. It stresses the role of the organism as the subject of interaction in terms of sensorimotor integration, with behaviors consisting of perception and action which are organized in a meaningful way. The concept has proven to be fruitful. It has its origins in the concept of the reflex arc, but the linearity of the stimulus-reaction chain is replaced by the concept of circularity. Every stimulus, in this view, presupposes a readiness to react, allowing the organism or animal to select as a stimulus a phenomenon of the environment which has been neutral up to that point. Rather than thinking in terms of reactivity to an external environment, we should conceive of the construction of an internal model of the world. The external environment is objectively there, but it can be assessed only as part of the subjectively perceived environment or Umwelt, as von Uexküll coined the term. Such a phenomenal world calls forth a set of “mapping relations” between an organism and the external reality, a semiotic world of subjective meanings imprinted on all objects as a private subset of the world at large . Functional cycles, then, encompass all the meaningful aspects of the world for a particular organism—they make up their respective Umwelts—and are the actual root of intentionality, bringing together the world of sensing and acting through processes of signification which invest the objects with perceptual and effector tones.
The critical element in this approach is the sensitivity to the functional characteristics of the environment. This has also been the basic claim of Gibson’s ecological psychology, which revolves around the central concept of affordance ([40, 41], see also [42, 43]), which stated that animals perceive their environment in terms of what it affords to the consummation of their behavior, rather than in terms of their objective and perceptual qualities: The affordances of an environment are what if offers the animal, what it provides or furnishes, either for good or ill.  (p. 127)
The affordances of an environment are what if offers the animal, what it provides or furnishes, either for good or ill.  (p. 127)
Affordances refer to the environmental supports for an organism’s intentional activities by claiming that animals—and by extension also human beings—are sensitive to the functional characteristics of their environment. As such, they can be considered as subjective qualities that render the environment apt for specific activities, such as supporting locomotion, concealment, manipulation, nutrition, and social interaction for the animal. It is a conception that points to an important quality of the world, namely, that its features are meaningful for an active perceiver who perceives this world in terms of functional significance of an object, event, or place.
Affordances, moreover, are interesting conceptual tools. They rely on objective environmental features of the world but also on perceiver-specific qualities, which are variable and subjective to a great extent. As such, they go beyond an objective/subjective dichotomy by claiming that there is no outside standing over against an inside, but only ways to classify experiences .
The concept of functional significance is really important here. It stresses the importance of sense-making as an act of deliberate attention and epistemic autonomy and brings together ecological, pragmatic, and biosemiotic claims. Listeners, in fact, build up relations with their sonic world by selecting some elements to give them special meanings. In doing so, they construct their own sonic Umwelt, as a collection of subjective meanings that are assigned to a specific subset of the sounding environment.
3. Semiotics and music
There is actually a huge body of semiotic studies as related to music. Most of the earlier studies are related to structural, phenomenological, or hermeneutical approaches. Though valuable, these approaches do not yet fully embrace empirical facts that validate the grounding theories. An interesting attempt to broaden the field has been initiated by the reliance on Morris’ division of semiotics in three dimensions (syntactics, semantics, and pragmatics) and the tripartition proposed by Molino  and Nattiez  with a distinction between an “esthesic,” “poietic,” and “neutral” level of description, referring, respectively, to the process of creation (poietic), the process of reception (esthesic), and the form and content (neutral) of the music. This tripartition, which has been contested also to some extent, has enabled semioticians to free themselves from certain constraints that were imposed by mere structural analyses which conceive of music as a closed system.
In the domain of music, the traditional analytical approach has been directed mostly to the syntactic level, leaning heavily on the contributions from linguistics. Scholars such as Molino , Nattiez , and Ruwet  have been exponents of taxonomic-empirical research. Starting from a neutral level of description, they have made major attempts to classify the sound (i.e., the empirical data) in an objective and scientific way, using a kind of taxonomy in order to select and identify the classes of objects that can be arranged in terms of similarity and difference. Central in this approach are procedures of division and extraction of structural elements, which offer decoding strategies that work “from text to code” with structural units that are describable in a formal way. To quote Nattiez: …it is no longer a question of knowing whether one of the fragments … is a motif or a cellule: it becomes an a, or A, or x, no matter which, possessing certain characteristics, which are defined by a group of features (melodic, rhythmic) which make it possible to compare it and classify it, that is to place it in hierarchy in relation to all the other segments of the piece. At the level of the metalanguage of the analysis one can guess what the immediate tasks of musicology will be: to develop fully a formal, artificial, explicit language which can take into account all the units one can find in music and their combinations.  (p. 63)
…it is no longer a question of knowing whether one of the fragments … is a motif or a cellule: it becomes an a, or A, or x, no matter which, possessing certain characteristics, which are defined by a group of features (melodic, rhythmic) which make it possible to compare it and classify it, that is to place it in hierarchy in relation to all the other segments of the piece. At the level of the metalanguage of the analysis one can guess what the immediate tasks of musicology will be: to develop fully a formal, artificial, explicit language which can take into account all the units one can find in music and their combinations.  (p. 63)
Such an analytical methodology operates at the neutral level of description. It reduces structural units to a purely formal level, stressing the more essential parts and eliminating nonessential aspects as being unimportant. The way of doing this is to use signs and symbols instead of real things. Signs, however, represent objects at a reduced level of cues, which means that the sign will not call forth all the responses that the object itself could do. This is the price we pay for the transposability of the sign system that we use instead of the less transposable original. The advantages, on the other hand, are numerous. They are, however, not sufficient to explain the richness and fullness of a real-time listening experience.
There have been contributions to the level of musical semantics as well [49–52], with a distinction between musical meaning being defined as referring to something outside of the music (“external” or “real” semantics) and as referring merely to itself (“internal” or “self-referential” semantics) (see  for definitions and  for empirical grounding). The level of self-referential semantics, however, is somewhat ill defined, as it conflates somewhat with the syntactic level. It calls forth the syntactization of semantics as advocated already in the 1930s—the logical semantics of Carnap and the model-theoretic semantics of Tarski—with an approach that is accomplished by completely encoding the world so that the elements (mostly formal symbols) are seen in relation to completely logical-symbolic structures without the need of specifying any set of observables and without the need of verifying their truth values with respect to an outer world. In Saussurian terms, this should mean that signifier and signified blend together and that musical signifieds are internal to the musical system, without any reference to something outside of the system. The signifieds, in this view, are not denotative or lexical but self-reflective  which means that they refer mainly to themselves. What matters merely is the identification of sonic events and their interrelations, without any relation to the external world. Music, then, is a carrier of immanent meaning, with sounding elements as recognizable entities that can be assigned some meaning or semantic weight. Unlike language where attention is directed away from the text in order to grasp the meaning outside of the written text—the centrifugal tendency of linguistic meaning—music is characterized by a centripetal tendency with a focus on the auditory material .
The distinction between internal and external semantics, however, is not so radical as it may seem. Music, as a sounding phenomenon, relies on both of them in the sense that elements that are referring to themselves may trigger processes of sense-making that refer to the external (the sounding environments) or the internal world of the listener (bodily resonance). To the extent that a listener experiences a particular sound as a real sounding thing that originates in the external environment, there is an aspect of external reference and of external semantics. As soon, however, as the listener starts doing mental computations on this sound, there is a shift from presentational immediacy to cognitive mediation. The listener, then, does no longer conceive of the sound in its experiential qualities but at a symbolic level of representation, with processes of recognition and identification that replace the fullness and richness of an actual real-time experience. Music, in that case, is conceived “in absentia” and not “in praesentia,” to use de Saussure’s terms . The reference to the internal environment of the listener, on the other hand, has received considerable impetus from the hard sciences, in particular from cognitive neuroscience and the neurobiological research with a special focus on the inductive power of music and its effects on the body and the brain. It means that stimuli do not necessarily originate from the outer environmental world. They can have their origin in our proper body with all kinds of sensory or motor reactions to the sounds. The issue is somewhat related to the distinction between distal and proximal stimuli in perception. Distal stimuli correspond to what is considered an actual object or event in the environment; the proximal stimulus is more narrowly defined as the pattern of energy impinging on the observer’s sensory system. The energy is associated with the distal stimuli, but the observer depends most directly on proximal stimuli for perceiving the world. For certain perceptions, however, there is little distinction between the two. Touch is an example, as the distal stimulus that is responsible for the sensation is created when the object that serves as distal stimulus is in physical contact with the observer . The distinction, however, needs further elaboration as proximal stimuli are situated mostly at the boundary (mostly the skin and special sense organs) between the inside and the outside of the body. Yet, there is also the visceral part of our body, together with our bones, muscles, and connective tissues which all are able to trigger reactions to the sounds to the extent that are resonating to these sounds. This is, in fact, the province of vibro-acoustic medicine [59, 60] which investigates the bodily and visceral reactions together with that kind of information processing that is tuned at monitoring the internal environment of our body. It seems, in fact, that sound vibrations may be organized and targeted to arouse certain bodily functions to induce particular physiological responses. Musical sense-making, in this view, cannot be reduced to a detached and disembodied nature of cognition . It calls forth, on the contrary, an embodied and enactive approach that conceives of music users as organisms that are endowed with a sensory and motor apparatus that enables them to carry out interactions with their environment.
This brings us to the third dimension of musical sense-making, the pragmatic level, which investigates the relations between sign vehicles and their users and the processes involved in the interpretation of signs. Meaning, in this view, is not to be defined in terms of ontological categories but in terms of dispositions to react to external stimuli. It includes the listener—or more in general, the music user—as a principal participant in the semiotic process, both at the level of reception, action, and mental processing and computation. As such, it calls forth dimensions that go beyond a mere object-centered, esthesic, or poietic approach. The configuration of our body and our cognitive faculties, in fact, determines not only our ways of listening but also the execution and creation of the music, which make it possible to understand and to live a musical experience.
As a discipline, musical pragmatics is still in continuous development (see [10, 11]). Starting, to some extent, from the conceptual framework by Peirce and Morris, it has made considerable efforts to describe the music in a richer and more complex way. This is even more the case nowadays with multiple contributions that are borrowing avidly from other disciplines such as the cognitive sciences, psychology, neurosciences, and even philosophy and neuropragmatics [62–65]. The whole body of music and emotion studies as well as studies on the effects of music and its inductive power are likely to provide substantial empirical grounding for this approach [54, 66, 67].
4. New perspectives on musical sense-making: biology and embodiment
The pragmatic approach brings us to some new perspectives on musical sense-making which are characterized by the conflation of scientific disciplines and levels of semiosis. They can be summarized as belonging to one of the following explanatory theories: (i) the ecological approach to listening, (ii) the biosemiotic approach, (iii) the biological and embodied approach to musical sense-making, (iv) the enactive approach to musical semantics, and (v) the experiential approach and the inductive power of music.
The ecological and biosemiotic approaches have been described already above. They revolve around the concept of affordance and the construction of an internal model of the sounding world as the outcome of interactions with this world. The concept of musical affordance is really important here. It means that we should try to understand music in terms of what it affords to us and not merely in terms of its acoustical qualities . The question, however, is what these musical affordances are? There seem to be four major possibilities: (i) the exploration and investigation of sounding material and the production of musical instruments, (ii) the exploration and mastering of techniques in order to produce musical sounds, (iii) the shaping of the sound by using modulatory techniques, and (iv) the phenomenon of musical entrainment and motor induction.
The whole history of musical instrument building is typical of the first. It can be considered as one prolonged search for applying craftsmanship to raw materials in order to obtain musical sounds. About all kinds of materials have been scrutinized for what they afford to human ears from a musical point of view. Playing techniques, as a second possibility, are also related to this search for sounding materials, but an additional focus is laid on the sound-producing actions, which encompass singular actions like hitting, stroking, kicking, and blowing as well as more complex or compound ones. Examples of the latter are drumming a rhythmic pattern or sliding up and down a melodic contour. But even the metaphors used in talking about music may refer to sound-producing actions (slow, fast, up, down, etc.), and the same applies to musical terms like martellato, leggiero, tenuto, and legato . The shaping of the sound is a further extension of the second possibility for sound production. It is exemplified most typically in string players, wood players, and singers. Strings, e.g., can be plucked or bowed, and within such action categories, there is even a whole spectrum of techniques for further modulation of the sound. The same holds true for a singer who shapes the sounds that result from the air supply provided by the lungs. Singing, in fact, is not merely reducible to the production of vowels and consonants but involves also aspects of intonation and ways of emotional expression such as timing, articulation, dynamics, tone onsets, and vibrato. It embraces a whole gamut of sentic modulation [70, 71], i.e., a general modulatory system that is involved in conveying and perceiving the intensity of emotive expression by means of three graded spectra—tempo modulation, amplitude modulation, and selection of register—somewhat analogous to the well-known rules of prosody.
A last interpretation of music in terms of affordances, finally, is more manifest and involves musical entrainment and motor induction. It calls forth the possibility to move in reaction to the sounding music. Music, then, is a stimulus for movement and is perceived in terms of its motor induction capacities. The movements can be specific and articulate, but they can relate also to more general levels of motor induction, as forces and energies that are inherent in musical structures, which, in turn, account for our perception and imagination of tension, resolution, and movement.
It is thus possible to conceive of music in terms of its activity signature with at least five major possibilities: the sound-producing actions proper, the effects of these actions, the possibility of imagining the sonorous unfolding as a kind of movement through time, the mental simulation of this movement in terms of preconceptual bodily experiences or bodily based image schemata, and the movements which can be possibly induced by the sounds .
All these examples are musical affordances that refer to the level of sound production. It is possible, however, to go beyond the mere productive level and to conceive of affordances at the level of experience as well. To conceive of music in terms of experience involves at least an aspect of egocentricity, in describing subjective experiences in terms of bodily resonance or motor imagery that projects our bodily movements to the music. Affordances, in this extended view, embrace perceptual qualities, mood induction qualities, and socio-communicative qualities, invoking aspects of sense-making, emotional experience, esthetic experience, entrainment, and judgments of value [72–74].
The biological approach is an extension of the ecological and biosemiotic approach . It revolves around the biological concept of adaptation and the possibility of coping with the sounding world. As such, it takes as a starting point the concept of sensorimotor integration, which questions the origin of structural solidarities and functional cohesions that are to be found in the individuation of biological systems and the interdependency of an organism and its environment. The organism enriches, in a way, its repertory of genetic adaptations with acquired dispositions that are the outcome of its capacity to control present activities in terms of personal experiences that are the outcome of previous activities  (p. 925).
This brings us to the embodiment hypothesis of cognition, which understands perception as “perceptually guided action” and conceives of sensory and motor processes as being inherently inseparable, mutually informative, and structured so as to ground our conceptual systems  (p. 173). It is a point of view, which argues for a biological interpretation of the experiential world, allowing observers to explore their environment with their bodies and their senses. As such, the mind is not to be seen as a passive reflection of the outer world, but as an active constructor of its own reality with cognition and bodily activity implying each other to a high degree. The fundamental building blocks of cognitive processes, in this view, are not disembodied propositions and representations but control schemata for motor patterns which arise from perceptual interactions with the environment .
Musical sense-making thus calls forth processes of sense-making and engagements that allow the listener to “enact” a musical experience and to react even bodily to the sounds . The claims are closely related to the embodied and enactive approach to cognition which defines it in terms of “nonobjectivist semantics.” It is a promising area of research that defines cognition not as the representation of a pregiven world by a pregiven mind but as “the enactment of a world and a mind on the basis of a history of the variety of actions that a being in the world performs”  (p. 9). Understanding cognition, then, is not taking the world naively—this is the claim of naive realism—but seeing it as having the mark of our own structure, which we are cognizing with our mind  (p. 16). Knowledge, thus defined, is the result of an ongoing interpretation that emerges from our capacities of understanding—this is the claim of cognitive realism—which are rooted in the structures of our biological embodiment but which are lived and experienced within a domain of consensual action and cultural history  (p. 150). Such a view is a “nonobjectivist” orientation to semantics that views cognition as enaction and that is consonant with the “experimentalist” approach to cognition. It is, in fact, a cognitive semantics that accounts for what meaning is to human beings, rather than trying to replace humanly meaningful thought by reference to a metaphysical account of a reality external to human experience ( (p. 120), see also [76, 79]).
The epistemological claims of experiential and enactive cognition define meaning as a matter of human understanding. They are highly dependent upon structures of embodied imagination and highlight the dynamic and interactive character of meaning and understanding  (p. 175). As such, they are typical examples of “nonobjectivist semantics”: they do not take the world naively, i.e., objectively, but conceive of it the result of understanding, imagination, and embodiment.
The embodied claims have received a lot of attention in recent developments in cognitive science, with a move toward the inclusion of the body in the understanding of the mind as exemplified most typically in the experiential approach of cognitive linguistics (see [29, 78–87] for musical applications), which states that the fundamental conceptual representations in the human cognitive system are schematic perceptual images extracted from all modes of experience.
Cognitive scientists, further, have begun to infer connections between the structure of mental processes and physical embodiment . This viewpoint, also known as embodied or situated cognition, treats cognition as an activity that is structured by the body which is situated in an environment that shapes its experience. It calls forth a conception of embodied action which is closely related to theories of cognitive organization which treat cognition as an activity that is structured by a body which is immersed in an environment. Cognition, in this broadened view, depends upon experiences which are based in having a body with sensorimotor capacities that are embedded in an encompassing biological, psychological, and cultural context.
Such a theory of cognitive organization challenges the propositional approach to sense-making. Rather than thinking in lexico-semantic terms, it reconceives the nature of linguistic meaning by stressing the role of metaphor as a basic structure of understanding. Embodied cognition, in particular, stresses the role of the body in providing “cross-domain mappings” as metaphors make it possible to conceptualize an unfamiliar domain (the target domain) in terms of another more familiar domain (the source domain). The human body, in this view, can function as a primary source for this kind of mapping.
5. The role of sensory experience and real-time interactions with the sounds
The enactive approach to music cognition is a challenging new area of research. It provides a useful theoretical framework for setting up a full-fletched program of empirical research. This holds true especially for the study of musical affordances (see above) but also the study of real-time listening can benefit from this approach. It can even be subsumed under a broader area of research which is related to the musical experience and the way listeners make sense of sounding music (see [57, 88, 89]).
Starting from a definition of music as a temporal and sounding art, it seems arguable to revalue some older contributions by pragmatic philosophers as Dewey  and James , who elaborated already extensively on the subject of having an experience. As Dewey states: Experience in the degree in which it is experience is heightened vitality. Instead of signifying being shut up within one’s private feelings and sensations, it signifies active and alert commerce with the world; at its height it signifies complete interpenetration of self and the world of objects and events.  (p. 19)
Experience in the degree in which it is experience is heightened vitality. Instead of signifying being shut up within one’s private feelings and sensations, it signifies active and alert commerce with the world; at its height it signifies complete interpenetration of self and the world of objects and events.  (p. 19)
This heightened vitality has adaptive value, as exemplified in the life of the savage man who is in danger in a threatening environment. Observation, for him, is both “action in preparation” and “foresight for the future.” They are not merely pathways for gathering material that is stored away for a delayed and remote possibility, but they function as sentinels of immediate thought and outposts of action  (p. 19).
A musical experience, accordingly, is not basically different from an auditory experience at large. It is continuous with the natural experience or experience proper with a difference in degree rather than in quality. Esthetic perception, and musical-esthetic perception in particular, should be characterized by a rich and full perceptual experience, contrary to the objects of ordinary perception, which mostly lack this completeness. The full perceptual realization of just the individual thing we perceive is then cut short and replaced by the identification of something that acts as an index of a specific and limited kind of conduct, replacing the act of exploring and experiencing by mere recognition.
A somewhat related approach was advocated by James [91, 92] who dealt with the tension between “concept” and “percept.” In his little-known but very important doctrine of radical empiricism, he stresses the role of knowledge by acquaintance, which he defines as the kind of knowledge we have of a thing by its presentation to the senses. The significance of concepts is not relinquished, but it always consists in their relation to perceptual particulars. What matters in this “empirical view” is not propositional knowledge, but the fullness of reality which we become aware of only in the perceptual flux: We extend our view when we insert our percepts into our conceptual map … but the map remains superficial through the abstractness, and false through the discreteness of its elements […]. Conceptual knowledge is forever inadequate to the fullness of the reality to be known. Reality consists of existential particulars as well as of essences and universals and class-names, and of existential particulars we become aware only in the perceptual flux. The flux can never be superseded.  (p. 245)
We extend our view when we insert our percepts into our conceptual map … but the map remains superficial through the abstractness, and false through the discreteness of its elements […]. Conceptual knowledge is forever inadequate to the fullness of the reality to be known. Reality consists of existential particulars as well as of essences and universals and class-names, and of existential particulars we become aware only in the perceptual flux. The flux can never be superseded.  (p. 245)
Conceptual knowledge is needed only in order to manage information in a more “economical” way. There is, in fact, a difference between the recognition of a sounding object or an event as a discrete entity and the experience proper of its sonorous articulation through time. In the recognition mode, we stop acoustical processing of a sounding event in favor of conceptual processing which allows us to conceive of it in a propositional way. Such kind of processing is much quicker and less demanding as it is much easier to select and delimit events and to pick them up in an act of episodic attention than to deal with them in an act of sustained attention. This is, in a nutshell, the core assumption of cognitive economy. It holds true, of course, also for listening to music, which is both an experiential and a conceptual matter. Consisting of sensory realia as well as of their symbolic counterparts, it embraces both perceptual immediacy and conceptual abstraction.
The experiential framework can be easily applied to music, but it can be extended further by introducing also the conceptual tools of deixis and indexical devices. This means that we should locate epistemic transactions with the sounds with the listener being considered as the origo of something that happens in a “here” and “now,” thus providing a kind of anchoring in a referential exchange.
The very concept of deixis goes back to Bühler  (see also ) who drew an explicit analogy between gestural and linguistic means for showing direction or place. Conceiving of two basic types of linguistic expressions, which he called deictics (or “pointing words”) as opposed to symbols (or “naming words”), he presented as a main thesis that deictic expressions refer to a deictic field of language whose zero point (the origo) is fixed by the person who speaks (I), the place of utterance (here), and the time of utterance (now). Deictic terms, accordingly, are words that pick out or point to things in relation to the participants in a speech situation  with terms as “this/that,” “here/there,” “I/you,” “my/your,” etc., as typical examples. They are related to the notion of indexicality  and the notion of pointing and its correlates . Pointing words, further, act as a source of reference: they locate individual elements in context rather than simply tagging them. They have their origin within the speaking—or listening—situation, with the meaning of deictic expressions depending crucially on when, where, and by whom they are used, allowing each ordinary referential exchange to be systematized in terms of personal, spatial, and temporal deixis (the so-called socio-spatio-temporal axes). Deictic expressions, therefore, make it possible to provide an operational description of space/time moments and their relations to the position and time of utterance and to define an utterance with reference to the referential exchange, its participants, and its settings.
The deictic approach, as applied to music, favors an experientialist as against a merely conceptual-symbolic approach to music. Rather than creating distance and polarization between the listener and the music, it argues for a dynamic-vectorial and directive approach, stressing the field of pointing rather than the symbolic field of meaning. As such, it holds an empiricist position which stresses the first-hand information in perception rather than relying on second-order stimuli. It means that we should conceive of “music as listened to” and “music as perceived,” rather than thinking and conceptualizing of music merely at a symbolic level without any connection to the music as it sounds.
Real-time musical sense-making, in this view, needs the transition from the symbolic to the deictic field of meaning, with the actual now moment of sounding music as the context for locating epistemic transactions with the sounds. The field of pointing, in that case, provides an interesting frame of reference for the assessment of the listener’s making sense out of the perceptual flux. It calls forth the role of interaction with the sounds, either at the actual level of real sounding music or at the virtual level of imagery and representation.
In order to provide a concrete example, Figure 1 depicts three representational formats of the “andante grazioso” from Mozart’s Sonata No. 11, KV 331. The upper pane shows a waveform notation of a larger section (about 2 min and 15 s), the left lower part depicts the first four bars (about 11 s) in standard notation, and the middle and lower right parts depict the same bars as a waveform (middle pane) and a spectrogram (lower pane). It is immediately clear that the standard notation is discrete symbolic: it subdivides the continuous sonorous flux in “discrete” elements that stand for themselves and that are separated from each other. As such, they facilitate cognitive decoding by stressing their “symbolic role” of referring to conventional pitches. What a listener actually hears, however, is not a succession of distinct and separate pitches, but a continuous flow, as exemplified in the waveform and spectrogram notation. As is obvious from the figures, there are no cuts and no blank spaces between the notes, which clearly shows that the discretization is imposed by the listener’s mind. Listeners, moreover, are free to mentally point to the sonorous unfolding and to delimit at will focal points or zones in this unfolding. Standard notation may be helpful here, as it provides already a discretization of the flux, allowing listeners to direct their attention to some of its elements (the notes). Figure 2 provides a rather obvious example. It shows how listeners can select deliberately the most prominent notes of the accompanied melody of one of Schubert’s Impromptus for piano. It is up to the listener, however, to decide which elements are selected for giving them semantic weight.
Focal attention, moreover, is not limited to notes. It can be directed to other structural features such as timbre, dynamics (a crescendo, a diminuendo), harmonic sequences (a succession of chords), aspects of voice leading (simultaneous organization of vocal or instrumental voices), etc. It can even be extended to musical gestures with or without melodic contour. Figure 3 depicts an example of an alternative notational system (lower pane), by replacing the discrete symbols of score notation by a kind of contour notation that combines segmentation (separate figures) and continuity. It is a hybrid notation as there is an almost one-to-one relationship between the figures (continuous) and the notes (discrete). The example, however, is merely illustrative of a possible translation of a discrete symbolic system to a more intuitive gestural approach. It shows clearly the possibilities of focal epistemic interactions with the sounds with a lot of freedom and subjectivity for each individual listener.
The field of pointing and the symbolic field, finally, are not necessarily opposed to each other. Listeners, involved in real-time listening, are constructing music knowledge, which relies both on sensation and representation, proceeding both as a moment-to-moment history and giving way to a kind of synoptic overview that is constructed in imagery and representation. Dealing with music, in this view, holds a view that balances between actual sensation and conceptualization and representation, between focal and synoptic allocation of attention, and between in-time and outside-of-time processing of the sounds. The former holds track with the unfolding through time; the latter can take some distance with respect to the sounding flux by dealing with music merely at a level of representation that is not dependent on the inexorable character of the unfolding of time.
6. Conclusion and perspectives
The role of the musical experience has for a long time been marginal in existing musicological research. There are psychological studies and music reception and cognition studies, but musicology as a discipline is still waiting for a comprehensive and theoretically grounded framework that explains the idiosyncrasies and commonalities of real-time musical sense-making. There is, however, a considerable body of older theoretical writings that have dealt extensively with the topic of having an experience. These writings, however, did not yet receive much attention in musicological research as they dealt with experience in a rather general way. This holds true, also, for this contribution, which describes the musical experience in a rather theoretical way.
It is interesting, therefore, to look for empirical findings that can support the claims. Much is to be expected here from the ecological approach and from music and brain studies. The ecological approach to musical sense-making, in particular, has a lot of operational power. It considers psychology as being continuous with the natural sciences and has been elaborated in depth by the Connecticut Tradition (Center for the Ecological Study of Perception & Action at the University of Connecticut) (see [98, 99]), which aimed at identifying general principles at the ecological scale of action and perception within an interdisciplinary framework. There is, in addition, a growing body of neurophysiological research from the growing field of music and brain studies which offers a vast body of empirical grounding for the theoretical framework that is related to having a musical experience (see  for an overview).
Many of these disciplines, however, have been working in isolation with only little connections to the domain of music. As such, there is a need of an interdisciplinary approach that brings together contributions from different fields that are all related to the process of dealing with music. Such a common field is not yet established as an official research community with institutions, official journals, and academic positions. There are, however, some emerging research communities which focus on a kind of common paradigm revolving around four major claims: music as a sounding art, the process of dealing with music, the role of the musical experience, and the process of sense-making while dealing with music.
All of them are exemplary of the pragmatic turn in musical semiotics with a major shift from a nominalist tradition in semiotics to a realist position that takes the real experience as a starting point. It brings us to old medieval discussion about nominalism and realism: should we conceive of musical entities as mere words that are the products of abstraction by our intelligence (nomina) or do they refer, on the contrary, to real material things (realia)? The discussion seems to be a hot topic in current research. It is challenging as it brings together philosophical and empirical claims, revaluing and broadening to some extent the old dreams of Leibniz and Descartes to realize a kind of all-encompassing science (mathesis universalis). The same idea, moreover, has been taken over by more modern thinkers as Carnap, Morris, and Tarski, who argued for a universal framework that would bring together all sciences in their search for a common problem consciousness. It is arguable to think that music, as a temporal and sounding art, could be helpful in bridging the gap between the conceptual approach of propositional semantics and the dynamic-vectorial approach that is typical of the deictic approach to cognition [101, 102]. Much research, however, is still to be done.