Towards a Semiotics of Visual Music

Abstract

‘Visual Music’ has a long history, much longer than most people realize. Over time, several eminent scientists, musicians, and artists have tried to establish correspondences between sound and vision. Some of these efforts were based on scientific principles, some on genuine synesthetic experiences, while others were more obviously creative aesthetic choices. With the resurgence of interest in this field, the argument presented here is that now is the time to re-evaluate this canon of knowledge, to identify more clearly, and expand the core concepts at its center. A selection of works, by pioneering film makers from the twentieth century, are examined from a semiotically informed perspective, to reevaluate some of the history of Visual Music alongside new ideas from the adjacent fields of science, psychology, and neuroscience. To this end a range of principles/parameters are outlined that arguably constitute the fundamentals of all creative approaches that translate between sound and vision going forward.

1. Introduction

‘The possibility of communication in terms of symbol, in terms of typography, in terms of language and in terms of music, [and] art, all begin to have potential on this great … medium of the twenty-first century, not today”. (John Whitney) [1].

‘Visual Music’ has a long history, much longer than most people realize. As Maura McDonnell explains [2], the starting point resides well before 1900, as far back perhaps as 570–495 BC, when Pythagoras established the mathematical laws of harmony, which were then applied to both music and color, and so the first color-tone correspondence was established. Arguably, the principles of harmony, and their mathematical equivalents, still have much to offer the field today. However, it is widely acknowledged that the correspondence that Pythagoras developed using only four colors, (white, black, red and yellow) leaves much to be desired, both in theoretical and practical terms.

Over time, several eminent scientists, musicians, and artists have tried to establish correspondences between sound and color. Some of these efforts were based on scientific principles, some on genuine synesthetic experiences, while others were more obviously creative aesthetic choices.

For example, Sir Isaac Newton attempted to relate the frequency of light and sound waves in physics [3]; Goethe developed romantic theories of correspondence between color and sound; Castel invented the Occular Harpsichord and Krüger, Guyot, Bishop and Rimington devised other musical/color instruments of a similar nature; Kandinsky’s essay “On the Spiritual in Art”, sets out his description of synesthesia in relation to making paintings related to music [4]; The artists, Robert Delaunay, Paul Klee and Piet Mondrian which were all concerned at some point with the idea that art and music were deeply linked in some fundamental way [4]; Perhaps most importantly, the twentieth century filmmakers Len Lye, Oskar Fischinger, Mary Ellen Bute, Norman McLaren, Jordan Belson and John and James Whitney, arguably realized ‘Visual Music’ in its purest form for the first time [5].

There are of course others that have continued, and still continue, to explore the boundaries of visual music (the author being one of them) but to a certain extent it has been forced underground, superseded by developments in technology and culture in our digital age that have occupied the mainstream media. However, it is evident that we are seeing a resurgence in these ideas. Indeed, McDonnell, along with Steve Gibson (in particular), Leon McCarthy, Paul Goodfellow (and others), provide an excellent and thorough going explanation of the history of “Live Visuals” up to the present day in the first part of the recent book of the same name [6]. Unfortunately, there is little space to discuss the depth and breadth of this book here but suffice to say it is exhaustive and timely in describing this old but newly resurgent field as it is right now.

Very simply, we can ostensibly say that there are four phases in the evolution of Visual Music. Phase 1: Early history, which focuses on color tone relations. Phase 2: The evolution of color-tone relations from music into two-dimensional static art. Phase 3: The transition from static to time-based visual art and film. Phase 4: Multimedia experiences, both analogue and digital associate with art-rock happenings, rave culture, and the rise of the VJ. It goes without saying that each phase builds on the experiments and achievements of those that came before. However, it is also clear that fundamental questions remain at the heart of the field in terms of how sound/vision correlations take place. Furthermore, there is still much to be done in terms of establishing some basic principles that enable approaches, developed across continually evolving technical forms of media, to be correlated effectively.

With the resurgence of interest in this field, the argument presented here is that now is the time to re-evaluate this canon of knowledge, to identify more clearly, the core concepts at its center. In short, now is the time to grasp the fundamentals of the correspondence between sound and vision in a way that has previously not been possible. Interdisciplinary approaches to research within twenty-first century academia have the capacity to bring knowledge from a range of other areas to bear on these issues in ways that have not previously been possible. This chapter attempts just that. It brings relevant understanding from, not only art and science as before, but also from psychology, neuroscience, and semiotics to cross reference findings about how we experience sound and vision. The aim is to shift our understanding of Visual Music, away from the arbitrary aesthetics of an experimental art form, towards a kind of language that can be grounded in theory and described more formally.

2. Color/pitch correspondence

It is almost impossible to discuss Visual Music, without first dealing with the relationship between color and pitch. Indeed, while the main thrust of this chapter lies in exploring theoretical input from a range of other different domains beyond this issue, it is necessary to be clear about this point first and foremost, due to the long and complex nature of the history associated with it.

In science, sound and light are often considered as being very different things. Sound is generally understood to be waves that are generated by some external force that acts on a medium, i.e., some initial energy input generates movement in the particles of a gas, a liquid or a solid that behaves like a wave, with a particular frequency (think of a drummer hitting a drum). Light, on the other hand, is generally thought of as particles or photons that sometimes exhibit wave like behavior, sometimes not. Significant debate remains about this dual particle/wave behavior of light [7, 8, 9]. Current theory suggests that all particles have this dual behavior. The main difference between the two seems to be one of scale. Sound can be described as energy of a certain frequency making the molecules (larger conglomerations of particles) move at a rate depended on the frequency input that propagates through the medium. Light, on the other hand, is often described as packets of energy at certain frequencies that are embodied as photons (particles in their own right) at a subatomic level moving themselves, even through a vacuum.

What is key is that, in both cases, we have energy, frequency and movement, which can result in wave like behavior. Arguably, this is best described in terms of the electromagnetic spectrum, which positions both sound and light at very different points on an energy scale determined by frequency/wavelength. To the human ear, sound occupies the frequency range of 20 Hz to 20,000 Hz. Whereas light, as visible to the human eye, occupies the frequency range of 4x10¹⁴Hz to 7x10¹⁴Hz.

Beginning from this set of assumptions allows us to return to the ideas established by Pythagoras in terms of his mathematical principles of harmony. For sound, especially music, the concept of harmony is particularly important. The vibrating string of a guitar or violin, for example, produces sounds of certain frequencies that have been ascribed to certain notes depending on where they are held (i.e., C1, A2, E3 etc.). This is possible because of the mathematical principles of harmonics that divide strings into ratios producing the harmonic series. There are multiple different ways to make these divisions from Pythagorean tuning to 12TET tuning and other more exotic forms. Regardless of their differences, they all stem from the same idea; the division of strings under tension into ratios that result in changes in frequency of sound I.e., changes in pitch.

An excellent example of this is the common place notion of the octave in music, take any fundamental note, say A2, divide its string length in half and you will find the 1st harmonic of that note which happens to double the frequency of the fundamental. This harmonic would be A3 in our case, and thus there is a 1:2 relationship between A2 and A3 in terms of frequency. The peculiar thing about this is that it has led to the introduction of naming conventions that revolve around the repetition of note names between A-G, with the appropriate sharps, flats and cents thrown in depending on the tuning system in play. This naming convention (specifically the western 12TET piano based system) has become engrained in musical nomenclature and notation and tends to obfuscate the significant differences in frequencies between notes. Technically A2 and A3 are completely different pitches but they are both named A. This repeats across a very large frequency range (10 octaves from C0 to C10).

This is not the case in terms of light, where the range of frequencies, although much higher, is much narrower. This would suggest that octaves within the visible spectrum, would seem improbable, at least in terms of equivalence to the range within audible sound. However, recent papers exploring the use of lasers have shown that frequency doubling (essentially, the creation of octaves) in light is more than possible, even if it does stray into U.V. territory [10]. This point is made here to underline the similarities between sound and light in terms of frequency and to open the way for the development of a primary relationship between sound and light that is based on the fundamental principles of harmony, mathematics, and physics. Essentially, if we agree that all these principles are in play, then it is only a small step to see that the light equivalence of sound is attainable through harmonic principles of transposition. This enables us to establish a relationship between any musical note on any scale (if we know it’s frequency) and its equivalent color based on the notion that the corresponding frequency will be 40 octaves above in terms of frequency. No doubt debates about the properties of light and sound make this problematic on one level (i.e., waves versus particles as discussed previously). However, by overlooking these issues in physics (which are by no means resolved) and concentrating on mathematics, frequency and harmonics. We can find a way to translate between the perceptual experiences of two essentially different media.

This has been problematic for a very long time with many scientists, artists and musicians establishing their own (competing) sound/color correspondences, often in very arbitrary ways. Debates about the color of A or the color of C and where the correlation begins have plagued the field for centuries. This frequency based harmonic approach allows us to establish a very simple mechanism of translation from one medium to another. An approach that can be replicated across other parameters as we move beyond those debates about color and pitch and on to consider more interesting debates about what else needs to be established within this re-emergent field.

3. Synesthesia and cross-modal processing

Early forms of Visual Music are inevitably connected to the concept of Synesthesia. Indeed, issues around color/pitch correspondence most likely derive directly from this concept.

In medical terms, Synesthesia is a neurological condition in which two or more senses are involuntarily and consistently linked together, leading to unexpected sensory experiences. The artist, Wassily Kandinsky, wrote extensively about synesthesia near the start of the twentieth century, which has had an enduring effect of the development of Visual Music as a whole. While it remains debatable as to whether he experienced such a thing himself, his understanding of the subject had a profound effect on his approach to painting. Indeed, the very idea of synesthesia and the correspondence between senses has been a theme for many artists over the years, and in some cases, there are clear examples of people with synesthesia making art about their experiences [11].

For Kandinsky, as for many artists in the first half of the twentieth century, it was the relationship between art and music that inspired him most. His paintings, particularly of his later period, are well known as increasingly abstract improvisational representations of music. The boundary between these two areas of creative endeavor have been crossed many times, both formally and informally, in terms of exploring the territory of correspondence (think simply of how many great rock musicians came out of art schools from the 1960s onwards).

Arguably, creative people seem to a have a propensity to want to explore the boundaries between senses, as to whether this is a feature of their psychological make up is up for debate, but the concept of cross-modal processing is well understood within the domain of psychology. Cross-modal processing refers to the way in which the human brain makes connections between different perceptual inputs allowing us to integrate and co-ordinate our experiences of sight, sound, touch and taste. It enables us to perceive the world in a cohesive and meaningful way, rather than as a series of isolated sensory experiences.

There is much debate within the scientific community about the extent to which synesthesia represents a special case of normal cross-modal processing, or whether it is a distinct and separate phenomenon. Some researchers have argued that synesthesia results when the brain fails to properly control cross-modal connections that are commonplace, others suggest that it may be caused by specific differences in brain function or organization.

Some recent studies have shown evidence of increased activation in brain areas associated with cross-modal processing in people with synesthesia, suggesting a ‘special case’ of normal processing. Other, structural neuroimaging studies, have found evidence of differences in the connections between brain regions in people with synesthesia, suggesting a hard-wired organizational predisposition is necessary for the condition to emerge [12]. Interestingly, Ramachandran and Hubbard [13, 14, 15] have proven without doubt that synesthesia is indeed a perceptual effect and not a cognitive or memory driven experience, as was once thought.

Research of this type has used brain-imaging techniques to establish the locations of stimulation when subjects are having synesthetic experiences. Findings have established that synesthesia is likely caused by cross activation in neurons in adjacent areas of the brain, which are responsible for slightly different perceptual or cognitive activities.

Ramachandran et al. have proposed that this might occur because of genetic mutation whereby the dysfunctional pruning of neural linkages during brain development leaves connections that normally would not be there. This in turn is reinforced by learned experience, which cements connections in the brain between normally unconnected areas. This opens the possibility of a resolution to the debate between the ‘unity of the senses’ thesis and the opposing ‘modular’ thesis camps [3, 16].

Ramachandran and Hubbard go on to suggest that this mutation could be quite common and spread throughout all brains in a patchy way across both lower perceptual and higher cognitive functions. They also consider how this may play a role in understanding metaphor, creativity, and the development of language. Additionally, they also suggest that this distributed patchy ‘cross wiring’ of the brain is why we see more examples of synesthesia in artists, poets, and other creative people.

So, why is all this neuroscience relevant to us in terms of developing an approach to understanding Visual Music? Well, fundamentally it lets us move on from many of the debates that have been at the heart of Visual Music for years. Questions about whether synesthesia is real or not, how it occurs and whether it is at the heart of the creative endeavors that explore the boundaries between sound and vision, become obsolete. The science essentially answers those questions for us. We can take it from the experts in those fields that cross-modal processing is a normal aspect of perceptual sense making and that synesthesia, although a special case for individuals that experience it, is derived from those normal cross-modal processing brain mechanisms that we all have.

This is important because for the first time it shows that developing correspondences between these seemingly different domains of knowledge based on two seemingly different perceptual components of existence is a completely natural part of how we make sense of the world around us. The very fact that our brains are built around these cross-modal perceptual mechanisms, allows us to take the next step in terms of understanding what theses mechanisms are. If they are also implicated in terms of understanding language, metaphor and creativity then attempting to also think of Visual Music in these terms makes even more sense (E.g., lexicons, grammar, translations, transpositions). Furthermore, it opens the opportunity for deeper interdisciplinary investigation in terms of developing the parameters of cross-modal translations between mediums. By drawing on additional ideas from related fields.

4. Sound symbolism and its implications for visual music

Sound symbolism, also known as phonosemantics, is a field of research within psychology that is exploring the way in which certain sounds seem to carry meaning beyond their primary role as simple (sometimes arbitrary) units of sound in a particular language [17]. Specifically, researchers are investigating the ways in which the physical properties of the sounds themselves, such as their duration, pitch, or intensity carry that meaning. As such it is an investigation of the formal properties, or parameters of sounds used in speech rather than the meaning associated with particular words (which Saussure characterized as arbitrary) [18].

The obvious form of sound symbolism that we would all recognize is onomatopoeia, i.e., words that mimic the sounds they represent (such as “hiss” or “buzz”). These types of words are found in all languages and are thought to play a key role in the development of language in children. More interestingly, a significant thread of research on sound symbolism has investigated the way in which certain sounds are consistently associated with particular meanings across different languages and cultures.

Arguably, the most well-known example of this is the Bouba/Kiki effect, first tested in the 1940s [19]. The experiment investigated the associations participants make between these two words and two different shapes, presented to them. A round blob shape and a spikey asymmetrical star type shape. Findings showed that more-often-than-not, participants associated the rounded shape with the word Bouba and the spikey shape with the word Kiki. The implication being that the sound of the word and the quality of the shape have some correspondence at a deep psychological level [20].

This experiment has been repeated many times. A recent version that explored the effect over 25 different languages (9 language families and 10 writing systems) found strong evidence of the effect, essentially establishing once and for all that: “The Bouba/Kiki effect is rooted in cross-modal correspondence between aspects of the voice and visual shape … the strongest demonstration to date that the Bouba/Kiki effect is robust across cultures and writing systems” [21]. The importance of these findings cannot be overstated in terms of verifying that our perceptions of both sound and vision are deeply interconnected at some neurological level. Moreover, other related studies have found that additional relationships exist in a similar way.

The words ‘Mil’ & ‘Mal’ have been shown to correspond with different sized (small, big) circles and the words ‘Wee’ & ‘Woo’, similarly seem to correspond to changes in aspect ratio such as (tall/thin) or (short/fat) ellipses [20]. A recent large study by Johansson et al. [22] looked at sound symbolism across a multitude of lexemes and languages pointing towards the embodied/experiential basis for language formation. Macro-concepts such as hardness, softness, smallness, roundness and flatness were clearly identified as cross-modal, suggesting that certain sound types are correlated to these particular embodied (physical and visual) experiences. Interestingly, Ramachandran and Hubbard have also shown that “associations between shapes and sounds are absent in individuals with damage to the angular gyros, suggesting that this is a robust neuropsychological phenomenon” [13, 14, 15].

The upshot of all this essentially means that we are, at least to some degree, hardwired to associate certain sounds with specific kinds of visible phenomenon and that our language, is in part at least, built upon that principle, perhaps explaining why we can describe experiences of both light and sound as ‘bright’ or ‘dull’.

Arguably, this is a ‘Rosetta Stone’ moment in the symbiotic relationship between sound and vision and potentially has far reaching implications for those working in the expanding field of Visual Music. Not only do the outcomes of these studies help make sense of the kind of underlying synesthetic principles that have underpinned Visual Music since Kandinsky’s first musings on the subject. They also offer a much more robust and scientifically grounded approach to mapping relationships between sound and vision, which up until now have arguably been quite arbitrary. We will see this when we examine some key examples from the Visual Music film cannon in Section 6.

5. Psychoacoustics and sound symbol oriented semantics

Psychoacoustics, a branch of psychophysics, is essentially the study of auditory perception, which includes both the sensory and physical aspects of the phenomenon [23]. Its practical implementation has found a home in product and industrial design, as Sound Quality Evaluation (SQE), where psychoacoustic principles are used to measure and evaluate the effects of sounds generated by products and machines on the psychology of human beings, usually in terms of safety. Over the years SQE has established several standardized parameters of sound description that provide a measure of sound quality using semantic differentials and Likert scales [24]. These parameters include:

• Loudness: Understood as a parameter of amplitude or volume ranging from very quiet at one end to very loud at the other.

• Duration: The length of a sound measured over time i.e., from short to long. Additional parameters may also be applied to duration in psychoacoustic descriptions, such as attack or decay, like the attack, decay, sustain and release, (ADSR) parameters of synthesizers.

• Pitch: The change in tone of a particular sound ranging from high pitched to low pitched physically measured by frequency. Musical scales are composed of distinct notes of pitches that go up and down the scales relative to changes in frequency.

• Sharpness: A measure of how bright or metallic a sound is. Sharp sounds refer to those that are high pitched and dull sounds as those that are low pitched. Higher pitched sounds are associated with powerfulness or aggressiveness, whereas lower pitched sounds are considered a bit rounder or softer.

• Roughness: A feature of temporal variation over time, particularly around the 70 Hz frequency, that produces a kind of warbling or texture to the sound.

• Fluctuation: Like roughness, fluctuation is a temporal modulation in frequency, that gives a feeling of repetition, beating, or tremolo. It becomes evident below 16 Hz and particularly at a round 4 Hz. As time increases further, fluctuations become more like echoes or delayed repetitions of a sound.

Susini et al. [25], discuss the relationship between psychoacoustic descriptors and auditory attributes, focusing on the way in which sematic differentials across multi-dimensional bipolar scales can cover a range of overlapping terminology (e.g. sharp-dull, pure-rich, cold-warm, colorless-colorful), to capture the often, complex nature of the sounds or soundscapes being investigated.

It is interesting to note that these bipolar scales bear a strong resemblance to the kinds of bipolar relations explored in sound symbolism experiments. While the standardized components of psychoacoustics may not relate directly, the lower-level specifics of the semantic differentials used to make evaluations have striking similarities. For sound symbolism the exploration is in relating the sounds of words to experiential equivalents (e.g. Bouba/Kiki) which tend to differentiate between poles. For SQE its about describing sounds within polarised parameters described by words. What is interesting is that in both cases the relationship between words and parameters is in play and it does not take a great deal of effort to theorize that there may be some significant cross overs, where the parameters being identified are applicable to both sound and vision at the same time.

This is important for the purposes of developing our theoretical understanding of Visual Music because, thinking carefully about the wording of semantic differentials that help frame descriptions of sounds, with reference to what we know about sound symbolism, could help to establish a set of parameters that enable the direct translation to a visual counterpart. In other words, it is entirely possible to start thinking about piecing together a set of fundamental principles of Visual Music that map the parameters of sound to those of vision. This is underlined by the fact that key concepts in psychoacoustics, such as sharpness or roughness are words that operate metaphorically, being that they are ‘already borrowed’ from the sensory domain of touch and operate within the realm of sound and music and no doubt have a correlate within the visual domain.

The missing piece in all of this is an investigation of the parameters that are in actually in play within the field of Visual Music. It is to this that we now turn, taking these theoretical ideas with us.

6. A brief semiotically informed exploration of visual music

There is not the time, or the space, here to perform a full-blown semiotic analysis of the entire canon of Visual Music. Indeed, it is only possible to discuss a selection of works by pioneering film makers from the twentieth century, from a semiotically informed perspective. However, while what is presented here is not an exhaustive review of seminal works, it does provide an overview of the early approaches to the making of such works in terms of the theory we have reviewed so far. In doing so the evaluation makes efforts to try and identify where these early experimental film makers were clearly and successfully connecting sound and vision together in terms of trying to establish the structural parameters of this cross-modal medium. In many cases this was an arbitrary approximation or interpretative process driven by the personal aesthetics of the film makers. In others there are much more direct mappings and processes that connect sonic and visual elements together in a more denotative manner where sound and vision truly begin to signify one another.

Moreover, the criticisms levelled at the film makers in this section is in no way meant to be derisory or devaluing. On the contrary every single work produced by these pioneers is held in great regard by the community and by the author. However, it is in the nature of applying a critical lens to these works that a separation between ‘successful’ and ‘less successful’ films (for want of better expressions) emerges, in terms of how close these early works get to really grappling with a visual language that could claim to be an equivalent to music.

Such an analysis, though cursory, and which belies the much greater need for a systematic semiotic analysis of the whole canon of these films, moves us much closer to a better theoretical understanding of Visual Music, not just in terms of film but for newer interactive digital forms, that are wrestling with the same issues today.

6.1 Len Lye

A review of the early works by film maker Len Lye, such as, “Kaleidoscope”, 1935 [26] and “A Color Box”, 1937 [27], reveals a very experimental approach to film making, which is rich, beautiful, and mesmeric. These films consist mainly of montages of shapes, squiggles and blobs of paint and ink either painted or printed directly onto filmstock. The very painterly marks bear no direct relation to the music played alongside them, which tends to shift between rhythmic drumming and jazz.

There is a vague sense of timing that is more coincidental than carefully planned. The visual elements are rough, splodgy and fluid but vibrant in terms of color and the sense of movement, which is lively. From a semiotic point of view we could only really say that there is a vague correspondence between the visual and musical elements in the film. There is no real precision that could denotatively align a particular shape, color or movement to a particular beat, note or instrument. The visual elements are simply loose approximations or interpretations of the accompanying music that connote a sense of rhythm in general. There is a very informal messy quality to them and Lye’s style has clearly had a significant impact on rock music culture, through the happenings of the 1960s to prog rock music visualization of the 1970s and on into the punk rock and grunge music videos of the 1980s and 90s. It is the roughness and textural qualities of the works that really stand out and the connect perhaps theoretically with SQE parameters of roughness. It would be interesting to take more time for example to consider both the visual and sonic content of the films from the point of view of establishing where they sit in terms of parameters such as: rough-smooth, lively-calm, rich-impoverished, colorful-colorless.

6.2 Oskar Fischinger

By contrast Oskar Fischingers early visual music films, such as “An Optical Poem”, 1938 [28] were much more structured. Perhaps because he was working with symphonic classical music or perhaps because intentionally at the outset, he was trying to find a stronger visual correlation to the musical element. Indeed, at the start of An Optical Poem there is a statement: “To most, music suggests definite mental images of form and color. The picture you are about to see is a novel scientific experiment – Its object is to convey these mental images in visual form”. So, from that we can take that there is a very clear intention present in Fischingers work to represent the sensation of music visually.

The film itself is full of simple visual forms (mostly circles, and some other shapes) carefully synchronized to the timing of the music, for example it is clear from the first few bars that there is a strong intentional link between the string section and the color, size and position of the circles that represent it semiotically. However, in those first few bars there is nothing to represent the horn section (this seems to appear later). Importantly from semiotic point of view, there is evidence of an intentional link between dark-colored circles representing sounds with low pitches and light-colored circles representing higher pitches. This mapping is not precise or consistent in terms of specific pitches, colors or shades but there is a definite general rule at work here.

As already mentioned in relation to the string section there is an attempt to connect shape and movement to the phrases played by that part of the orchestra, but it is not consistent and at times it gets difficult to connect specific shapes with specific instruments. It all feels a bit arbitrary but there are clear attempts at relating shape and color to a particular type of instrument or timbre.

On screen position also suggests a visual relationship with pitch. Visual elements nearer the top of the screen seem to represent higher pitched sounds and elements lower on the screen seem to represent lower pitches. This works to reinforce the connection between light/dark pitch relationships. Additionally, there is a strong sense of a three-dimensional space or field of view within the work where different sizes of shape come towards the camera or recede away from it. This appears to be connected to volume where larger shapes are louder sounds and smaller shapes are sounds fading away, receding into the background and disappearing.

Semiotically, this feels like an intentional mapping of parameters, but it is not consistent and does not always work as expected. What is most significant about Fischinger’s works is that they exhibit some of the first real attempts to establish parameters that visual elements might share with sound. Most notably attempts to correlate visual size and audio volume; dark/light shades of color with lower/higher pitched instruments or different types of instruments (e.g., strings vs. trumpets); screen position of visual elements with note pitches.

Unfortunately, from a semiotic point of view, where we are looking for clear denotative correspondences between visual signs of music Fischinger’s films fall slightly short, largely because he is working with Symphonic music which is so vast in its complexity that it is almost impossible, given his chosen medium of animated paper cutouts, to achieve the kind of precision necessary to make such precise connections. His films are, however, remarkable in terms of how far they get down that road, paving the way for many who come after him.

6.3 Mary Ellen Bute

Contemporary to Lye and Fischinger, Mary Ellen Bute was also making films that were experimenting with the relationship between music and a visual equivalent. Her early films such as “Rhythm In Light”, 1934 [29], “Synchromy No 2”, 1935 [30] and “Parabola”, 1937 [31] are shot in black and white, and so immediately there is no color element to relate to sound. However, Bute’s take on the relationship between sound and vision seems more akin to interpretive dance than a search for a direct correlation between sonic and visual elements.

These early works are built around abstract montages of overlaid imagery that move in and out of focus across the screen and are vaguely coordinated with the tempo of the music. There are no strong relationships formed between the visual elements and the sonic content. Indeed, from a semiotic viewpoint is seems as if the experimental dreamlike imagery, that plays with chiaroscuro lighting of unusual physical objects, seems to slip in and out of view as an accompaniment to the music, that echoes, the general feel of the music rather than attempts to signify it. At times the imagery is beautiful, but it is so abstract and referent to the ‘physical’ content that it is hard to really think of these as visual music films.

This is in complete contrast to some of her later work, particularly the works she directed with Norman McLaren as the lead animator. Works such as “Spook Sport”, 1939 [32] and “Tarantella”, 1940 [33] are shot in color and immediately there is more vibrancy and connection to music in some ways. However, it is the sinuous line and, in “Spook Sport” particularly, that the semiotic content begins to connect more strongly.

At its outset, the film clearly identifies the ‘characters’ within the movie, and these characters (symbolically animated representations of bats and ghosts) go on to dance across the screen space to great effect. Unfortunately, in terms of visual music the outcome is again more like interpretive dance than a visual version of music. The semiotic characterization linking more to traditional film and theatre in terms of developing a narrative, rather than connecting with the sound directly. However, McLaren’s input really does show in both films. “Tarantella” is the more abstract of the two, drawing heavily on Russian Modernist forms and colors to drive a quasi-narrative forward to some degree, but the drawing in “Spook Sport” connects more directly to McLaren’s own work, which takes the semiotic mapping of sound and vision to a much deeper level.

6.4 Norman McLaren

In Mclaren’s work we get a much more direct relationship between sound and vision but arguably it is not as direct as it might have been. McLaren took the ingenious step of using the sound portion of film stock to create his own ‘sound effects’ by drawing/painting sequences of marks directly on to it in order to use it as a sound generating instrument alongside the images he drew on the visible element of the film. In such a way he was able to precisely control the sounds that were made, when they were made and ensure that the visual and sonic elements were carefully synchronized when the film played. This is a major technological breakthrough in terms of realizing the semiotic potential of the medium to embody a genuinely interconnected visual music for the first time.

Before McLaren, music either came before or after the concerns of film making. McLaren revolutionizes the approach by making sound and vision with the same set of tools, thus bringing the two elements closer than ever before. However, from a semiotic point of view, what is interesting is that while he drew a series of lines or blobs on the sound portion of the film to control pitch, volume and softness [34]. He did not draw the same thing on the visual side of the screen. Indeed, with more screen space to play with he drew more complex visual forms that fold and unfold, explode, and reform on screen in time with his abstract soundtrack. The synchronization is pinpoint precise, but questions remain from a semiotic point of view as to whether the visual elements directly represent the sounds or if they are once again arbitrary representations of things that have sounds, rather than of sounds. If McLaren had painted the same thing on screen as he did on the sound portion of the film there would have been a direct relationship between the two, but this in itself brings significant limitations and is perhaps visually much less interesting than what McLaren actually drew.

“Dots”, 1940 [35] is an excellent example of this. It is an incredibly simple little film but there is a very strong relationship between the visual and the sonic elements thanks to McLarens special technique. The size of the dots seem to represent aspects of pitch. Small dots have higher pitched sharper sounds and larger blobby dots having lower softer sounds. The same is true of the relationship between size and volume, again smaller shapes seem to have lower volume and larger shapes seem to be louder. This approach to parameter mapping is reminiscent of Fischingers work (which McLaren was more than aware of), and his work seems to reinforce these ideas apart from the fact the position on screen does not correlate with changes in pitch as Fischinger’s tended to do (as seen in musical notation for instance).

Interestingly, Mclaren’s later works seem to move away from the tight correspondence between soundtrack and visual material. Works such as “Lines Horizontal”, 1960 [36] for instance, where the animation is a visual interpretation of music by Pete Seeger. The horizontal black lines seem to bear absolutely no direct relationship to the guitar or the flute. They do move up and down the screen, multiplying as they go and it does feel like there is a timing relationship to changes in the music, such as the introduction or removal of instruments in the mix. However, the only thing on screen are black lines and occasional background color changes. This is very different to ‘Dot’ for example.

Likewise, “Synchromy”, 1971 [37] is also quite different. While it does have a carefully orchestrated sense of timing between visuals and soundtrack, it’s not as direct a relationship as his earlier work. Visual elements arrive on screen and move in relation to complex rhythm patterns. These change all the time and sometimes there seems to be several different moving strips in different horizontal screen positions that correspond to different instruments that are being played. The multiple rhythms are complex, and it seems that this is where the synchronization is aimed but it’s not entirely clear as to which ‘track’ of rectangles relates to which sound element. It is also clear there are size and shape patterns at work that imply a relationship to the sound but as to whether they relate to beat, volume, or timbre for instance is difficult to establish. This perhaps is the difficulty of working with an external sound source rather than working with the sound portion of the film, which he expertly did in his earlier films.

6.5 John and James Whitney

The work of John and James Whitney is legendary in the Visual Music community, particularly John for his contribution towards the development of early computer graphics. Again, their contribution to the field began early, in the 1940s. They worked with bespoke film and audio production equipment, customized from military ballistics systems, which enabled them to create complex films. Like McLaren, both the audio and visual elements were carefully synchronized by scoring and animating directly onto film. This resulted in carefully constructed sequences of color and shape compositions based on modernist composition theories using cut out masks that cover and reveal light directly rather than reflecting it (like the cutouts of Fischinger do). “Five Film Exercises”, 1943–1944 [38, 39, 40], winner of the Brussels Experimental Film competition in 1949, are perhaps the best examples of this way of working. While the timing of the shifts in shape and the strange experimental noises are clearly coordinated, there is little that really connects the shapes or colors directly to the noises being made in a cohesive, nonarbitrary way. The visuals tend to operate alongside the sounds. For instance, size and shape are not consistently correlated with volume or timbre and color. The films are extraordinary in terms of a quality in both sound and vision that seems quite fluid but not linked in any stable way.

As well as working together the brothers produced films separately. John really pushed the boundaries of the medium, moving inexorably towards computers over time. “Permutations”, 1968 [41] for example predates his experiments with digital computers, showing off the levels of complexity he was able to achieve by pushing his analogue ballistics system in combination with hand drawing series of dots on frames of film. The sound and the visuals are not directly connected in any way, being that the soundtrack is composed of rhythmical drumming and the visuals are visual representations of mathematical figures based around ‘rose’ equations. The swirling visuals are utterly mesmerizing and the synchronization between them and the sound is strong despite the lack of direct connection. The resultant output is reminiscent of Mary Ellen Bute’s work in that it feels like the dots on screen are dancing and shimmering along with the music but not directly tied to it in a strong denotative way.

Likewise, “Arabesque”, 1975 [42], operates in a similar vein. This does demonstrate Whitney’s early computer programming experiments, and as a result is an order of magnitude more complex than “Permutations”. Once again though, the visuals coincide with the music, which is Indian tabla/sitar music, rather than being directly derived from it or used to produce it.

Representing some of the first attempts at the precision manipulation of on-screen shapes using mathematical formula, it is a masterpiece of sequential programming, that clearly demonstrates the power of the computer in coordinating such complex movements. However, and this is controversial, it’s not the strongest in terms of demonstrating clear thinking about the relationship between the music and the related visual structure. One clearly informs the other but inevitably the programming seems to take precedent over the music, presumably because it simply took so long to produce frame by frame, making it difficult to connect directly with the soundtrack. All the computer-generated films are like this. One wonders, if it were possible to revisit the mathematical underpinnings of the works, as to whether an alternative soundtrack could be generated by slaving the equations to some form of sound synthesis alongside the visuals using today’s equipment.

Criticism aside, the contribution to the field made in these films is a monumental step forward, which perhaps unwittingly via the math, takes us closer to the concepts of harmony outlined right at the start of this chapter. No doubt there are mathematical equations at work that demonstrate complex ratios and relationships (e.g., rose figures, arabesques, polar co-ordinates etc.), it’s just that they are not systematically linked to the principles of the harmonic sequence or frequencies of pitch that come directly from the music, but they could be in future.

7. Discussion of analysis

This brief review of some key seminal works from the Visual Music Canon reveals several important things. Firstly, it shows the way in which some of the early pioneers of film and animation were grappling with the benefits and limitations of the media. It is obvious that the time-based nature of film offered a temporal space for the visual, which was equivalent to music, for the first time. The early films seem to be more like visual interpretations of the music or visual dances in response rather than direct correlations (Bute’s early, and Whitney’s later, work for example).

In contrast, the works of Fischinger and McLaren (in particular) hint at technical solutions to timing that clearly link the visual and sound elements together in a precise way. This aspect of timing is the first clear parameter that both elements share. Establishing a tight connection between the two seems to be fundamental to being able to translate between sound and vision or to engage with both elements as if they were one.

The second parameter that begins to emerge from this analysis is that of the relationship between visual size and audio volume. To an extent this appears quite strong in Fischinger’s work but is again strongest in McLaren’s. His technique of drawing the sound on the film alongside the visual elements meant he was able to precisely time the changes in both. So not only do we see a synchronization of duration of noise and duration of shape, we also so see a synchronization between volume of noise and size of shape. So now we have two parameters working together in both the sonic and visual dimensions. This is also consistent with theory from sound symbolism such as ‘Mil & Mal’ in terms of size differences and ‘Wee & Woo’, changes to aspect ratio, which play out in animation as squash and stretch.

Additionally, there is a third aspect of correspondence that is built into this in terms of the type of sound and type of shape. The kinds of noises that McLaren was able to make often varied between high frequency sounds and low frequency sounds and the shapes he associated with these two sounds were often simple circles (high) or more blobby type shapes (low), the simple shapes being smaller and the blobby ones being bigger.

It cannot be overlooked that the pitch of the sound McLaren created was determined by the frequency and thickness of the lines that McLaren painted onto the celluloid. Many thin lines created high frequency sounds and fewer rounder lines created softer lower frequency sounds (Wee and Woo again). It is of course logical to follow that principle onscreen. Had McLaren painted the same configuration of lines on the visual part of the film as he had in the sound portion, there would arguably have been a one-to-one relationship between the two.

At this point we are beginning to see multiple correlations overlapping with one another and integrating into compelling Visual Music experiences. Again, the relationship between sound and shape connects with theory from sound symbolism. In this case specifically the Bouba/Kiki effect, where lower frequency sounds connect most strongly with rounder shapes and higher frequency sounds connect with thinner spikier shapes. So, quality of sound or ‘timbre’ potentially has a correlate within the visual component of shape. That quality of sound might be considered in relation to its frequency signature (its combination of fundamentals and harmonics) in terms of where it sits on the frequency spectrum or musical scale.

The quality of the related shape then seems to operate on a scale from soft and round to sharp and spikey. However, there are other visual correlations that could potentially come into play here. Two spring to mind. Dark vs. light and color mapping. Color mapping has already been dealt at the beginning of this chapter. But it is worth noting that in most of these examples, while color is an important element in most of them, few of the film makers, with the exception of Fischinger, seem to consider developing the relationship between color and pitch in a denotative way. Most of them seem to prefer arbitrary or loose connections, perhaps because strong connections are problematic and complex to establish in the medium of film.

The Dark to light correlation is much more straight forward. It is commonplace to describe different tones and feelings within musical passages as either dark or light and often this difference in darkness or lightness is a feature of the pitch of the notes being played. Again, higher frequency notes tend towards the lighter end of the scale and lower frequency notes tend towards the darker. Some would call this darkness vs. brightness and it would be easy to associate trumpets with bright sounds while cellos would be dark. To a degree this could be related to timbre but in fact it is likely more strongly a feature of the pitch range of those instruments and where they sit in relation to the bass or treble clef in orchestral music.

Examples of this ‘dark to light’ correspondence between sound and vision, is to a certain extent evident in Fischinger’s work. Perhaps because he worked with orchestrated music but most probably because he was sensitive to those sonic differences and was genuinely attempting to find a visual correlate to it. It is most obvious in the difference between the brighter (whiter and smaller) circles that occupy the upper part of the screen and the darker (bigger) colored circles that occupy the lower part of the screen in his work.

This brings into focus another parameter at play in relating the sonic to the visual that of the screen space itself. The placement of the visual elements in Fischinger’s screen space demonstrates a direct correlation to the musical stave, i.e. the positioning of notes on a page in terms of pitch to relative height, the higher the note the higher the position on the stave. It is of course obvious that this is derived directly from musical notation. However, it is interesting to consider that this correlation on the scale of pitch (frequency) to height again is probably derived from an embodied experience related to sound symbolism regardless of its conventional nature. It seems logical then, that vertical positioning on the screen should correspond to pitch to some degree. The thing that is unexplored though is the relationship between the horizontal axis and the sonic. Certainly, in the making of these early films there would have been no such thing as stereo, and the concept of surround sound would have been unimaginable. However, it stands to reason, following the logic developed so far, that a relationship between the stereo field of sound and the stereo field of view is a reasonable one to suppose. While none of the films could exhibit this, projecting forwards, it is easy to imagine a direct mapping of the visual screen space in terms of stereo left and right.

8. Conclusions

This chapter has attempted to reevaluate some of the history of Visual Music alongside new ideas from the adjacent fields of science, psychology and neuroscience. These new ideas offer the opportunity to build better theoretical foundations for Visual Music and all other creative approaches that translate between sound and vision. To this end a range of principles/parameters are outlined here that need further exploration (see Table 1).

Pitch to color	Sound frequency over one octave C0–C1	Spectrum of color	Transposition from sound octave to Hue in HSV color space (~40 octaves above C is green)
Pitch to Brightness.	Sound frequency over 10 octaves C0–C10.	Value in HSV color space from 0 to 255.	Low pitches, dark colors. High pitches, light colors.
Volume to Size	Audible changes in amplitude (dB)	Visual changes in size.	Mapped directly but modified by multipliers to fit visual space.
Timbre to Shape	Frequency and /or sound symbolism	Geometry, cymatics or visual equivalent to sound symbol	Low pitches are rounder. Higher pitches are spikier. Build a lexicon of sound/shapes.
Sound Space to Visual Space	Stereo field + sound frequency.	Center X, zero Y.	X axis represents the stereo field left and right of center. Y represents up and down of pitch.

Table 1.

Proposed fundamental visual/music parameters.

8.1 Pitch and color

The concept of harmony and the harmonic (and subharmonic) series provide fundamental building blocks for both music and visualization from a scientific and mathematical perspective. This has far reaching implications for the visual space on several levels. Applying the principles of harmony to the issue of color mapping results in the concept of sound and color relations operating via the mechanism of octaves. Light is approximately 40 octaves above sound and thus equivalent correspondences can be mapped between sounds/music and color based on frequency.

8.2 Pitch and brightness/darkness value

One problem of this harmonic approach is that there are approximately 10 octaves within audible sound and only one within the range of visible light, thus C0 and C2, or C10 would be equated to the same color. This is logical but does not consider differences in how we experience low and high musical notes. It is common practice within the sphere of music to describe sounds with low frequencies (bass notes, and the lower octaves) as ‘dark’ and those with higher frequencies (treble and higher octaves) as ‘bright’. Phrases like ‘low’ and ‘high’ end are also applied to this phenomenon. Thus, two complementary parameters arise that can be applied to sounds in terms of visual correspondence. The first, dark to light, can be made manifest in color correspondence through the mechanism of the HSV color space, that provides an axis of brightness (value, V) alongside hue (H) and saturation (S). Thus, the octaves of sound can be mapped to variations in brightness or saturation, alongside the corresponding color frequency of hue. This provides a very satisfactory mechanism for taking the experiential aspect of sound into account alongside the harmonic. The second parameter is spatial (see Section 8.5).

8.3 Volume and size

It is clear in relation to our investigation of Visual Music examples that there is some evidence of both size and shape being used in a way that is consistent with ideas emerging in the study of sound symbolism. It is also clear that the most logical mapping of the size of visual element is related to volume. Changes in the volume of sound from quiet to loud correspond very clearly with our experiences of smaller and larger sized things. As a general rule, small things (including animals, e.g. rodents, insects etc.) make quieter (usually higher pitched) noises. Larger things (including animals such as tigers, bears, elephants etc.) tend to make louder, lower pitched noises. One might also equate the two in terms of closeness. Close noises are louder than the same noises further away and close objects appear larger than the same object further away. So, it follows that our natural mapping of loudness corresponds with the visual equivalent of size. Thus, a triangulated relationship emerges between loudness of sound, size of shape and closeness (or depth of field on screen).

8.4 Timbre and shape

Shape itself inherits this aspect of size but it also has other features that correspond to a different aspect of sound. The Bouba/Kiki experiment, as already mentioned, reveals the relationship between round shapes-softer words and spikey shapes-sharper words. To a degree this must be a feature of frequency (pitch) in that rounder, softer, (darker) sounds have more low-end frequency and higher, sharper (brighter) sounds have more high-end frequency. Arguably, it is no accident that the concepts of ‘flat’ and ‘sharp’ in music also correspond with shifts in pitch that change in frequency accordingly. It follows then that shape relates to this element of frequency but arguably it is more complex than this. Timbre or type of noise is a consequence of the variability in the harmonic content of the instrument not just the pitch of the note. Thus, a C2 played on a guitar sounds very different to the same note played on a trumpet. Each of these instruments has a different harmonic signature, which are variations in the levels of the harmonics that develop from the fundamentals of the note being played attributed to the physical nature of the instrument being played. Pure sine waves from a tone generator have the cleanest harmonic structure and it is no coincidence that sine waves are also mathematically equated to the generation of circles. As a baseline relationship then, pure tones seem to correlate directly with circle shapes and changes in volume correspond to changes in size. More complex timbres will have more complex shapes, but a great deal of work needs to be done to establish how the harmonics of sound translates to an equivalent shape in terms of our perception. Potentially the harmonics of geometry has a lot to offer here and it does make sense that the Bouba/Kiki effect will play a part in this as our understanding develops.

8.5 Sonic space and screen space

While there are surround sound systems and 3D visual experiences that are prevalent today, most of our media experiences are still consumed in terms of 2D screens and stereo sound. Therefore, as a starting point it makes a great deal of sense to establish a baseline relationship between the stereo space of sound and the 2D visual space of the screen. Thus, the pan position of the stereo space and X axis of the screen line up across the horizontal. The Y axis on the other hand relates back to the difference between ‘low’ and ‘high-end’ frequencies. Musical notation is spatially oriented both in terms of left to right across that page (a feature of reading and time) and more importantly up and down upon the stave. Thus, it is logical to link vertical positioning on the Y axis with the octave structure of harmonically organized sound (i.e., pitch). So, pitch is double mapped to both brightness and height in relation to experiential descriptions of sound alongside color, which is purely based on octave equivalence.

8.6 Future work

A great deal more work needs to be done to solidify these foundations. A more rigorous semiotic evaluation of the Visual Music outputs (not just film) would be invaluable. Additionally, basic psychology experiments to verify the relationship between abstract sounds, words and shapes would also be helpful. For instance, it is not beyond the realms of conception, for example, to hypothesize that such experiments would provide evidence that low frequency pure tones correlate with the word Bouba (and its rounded shape) and high frequency pure tones correlate to Kiki (and its spiky equivalent).

This kind of experimental approach to verification has not yet happened, as far as the author is aware, but the wealth of evidence from the range of domains explored alongside the evaluation of Visual Music examples strongly suggests that it follows from what has been discovered so far.

Future work should involve the development of an interdisciplinary approach to research that combines experimental psychology with Visual Music. This would bring many benefits to both psychology, in terms of furthering our understanding of cross-modal processing, while at the same time helping to verify the parameters presented here, that describe Visual Music more fully as it evolves into its next phase, This is particularly relevant for those practitioners that are creating and making artworks and performances at the boundaries of these two mediums (Figure 1).

Figure 1.

The author performing with his live visual music system, based on the parameters discussed in this chapter.