Literature Review on Intermedial Studies: From Analogue to Digital

Abstract

This chapter traces an overview of the evolution of the research on Intermedial Studies in the last two decades. It expands the research presented in the InTech volume Comparative Literature. Interdisciplinary Considerations, in a chapter entitled ‘Intermedial Comparative Literature: from the Sister-Arts Debate to the twentieth century Avant-gardes’. The literature review offers a description of the major interdisciplinary contributions that have shaped the field of Intermedial Studies, with areas such as media and communication studies, art history, and the visual arts, including theater, dance and performance, sequential art (comics, graphic novels), photography, radio, film studies, electronic literature, videogames and Artificial Intelligence.

1. Introduction

The word ‘medium’ has its roots in classical Latin to refer to the middle of something or an intermediate course of action. The Oxford English Dictionary (OED) defines the term in a dual way, as the means or agency through which communication takes place, often synonymous with channel, and as the physical vehicle of expression employed in a representation, ranging from general categories of artistic, technical forms or modes of expression (e.g. photography) to specific materials, tools, and methods. In linguistic discourse, the concept may variously refer to language; speech or writing (i.e. the phonic medium and the visual-iconographic medium), including, for instance, distinctions between handwriting and print. In communication, ‘medium’ often refers to a specific technical form for interpersonal exchange, for example, via post, telephone, or computer-mediated communication. In its plural form, ‘media’, it names the instrumental use in mass communication (i.e. newspapers, television, etc.) that appears in everyday practices, hinting also at the interactions among different medial forms.

The variety of meanings attached to the notion of ‘medium’ shows that it stands as a boundary mark that signals the crossroads between its mediating (semiotic) agency and its material format (the physical vehicle or technological artifact used to inscribe any type of representation). In this regard, it is important to consider that relations in the world are simultaneously material (between things) and semiotic (between concepts). Thus, a given medium ‘mediates’ these relations. A medium is more than a channel for the transmission of information; only potentially meaningful. It may refer to a specific material (including nonvisible aspects such as the wave dimensions of the phonic medium that characterizes oral speech) or a technology, more or less complex.

The materiality of media is physically and culturally encoded. The term ‘multimodality’ denotes the perceptual modes at play in perception and in the use of the medium, whether given in nature or technologically created. It also refers to the affordances enabled by the medium. For instance, while the page of a printed book might afford a visual perceptual experience to most people, those with impaired vision require a page printed in Braille for a tactile experience.

Lars Elleström’s opened a debate to explore the differences between concepts like multimodality and intermediality. While the first referred to sensory configurations, intermediality was a complex cluster that involved not only perceptual modes but also technological, sociological and cultural aspects. The sociocultural context ‘qualifies’ media, causing a particular technical apparatus to be developed, accepted and popularized [1]. Elleström went on to refine his classification of intermedial and intramedial relations that can occur among ‘qualified’ media as well as among media products from similar or different, also ‘qualified’ media, noting that these categories are not borders but, rather, plastic contact zones. He also tackled the issue of transmediality [2, 3] (more on this below).

The history of Intermedial Studies is impacted by an entire ecology of communication and productions that can be tackled from a variety of disciplines. These include the material medium, closely related to changing technologies and apparatuses, the affordances of medium specificity, delivery mechanisms and channels, and circulation and reception practices, embedded in particular institutional and sociocultural contexts.

While Dick Higgins’s contribution to the concept of ‘intermedia’ is by now well-known in academia, not many scholars have noted that the term ‘intermediality’ was also introduced in 1990 as part of the project ‘Edge’90: Art & Life in the Nineties’, A Biennale celebrated in the UK and curated by Bob La Frenais and Tracy Warr. Edge’90 included installations, performances, sculpture, a video library and a two-day conference. Inspired by Higgins’ and George Maciunas’ Fluxus, the exhibition aimed to express the collusion between intermedia artworks and the personal spaces associated with everyday life [4]. Thus, since its inception, intermediality has been contemplated as defining the relationships between art forms, technologies, and spaces of meaning. The combination of radio, static images (photography) and moving images (film) to conform cinematography and later broadcast television show that technical media follow a sort of cumulative process of engineering. Each new medium, communicates by transforming multimodal experiences, conforming a novel phase of practices as well as associated creative industries with their own mechanisms of circulation, discussion forums and distribution.

Communication and representation include the mobilization of different medial forms; from the vocal cords used in oral speech, the instruments used in painting and writing, to the more complex technical media employed in cinema, and, more recently, in computer-mediated communication and new media forms such as videogames. These mechanisms communicate by producing a sensuous embodiment of ideas or thoughts. Architecture, painting, printing, photography, film, all the arts exist in the combination of media, perceived through different perceptual modalities. The material medium, whether given in nature or artificially made, as well as the human agents and their perceptual/cognitive abilities, interact in ways that mobilize and organize ideas, giving them shape and inscribing them, that is, fixing them in some kind of material substratum. This organization of knowledge is also a social act, situated in a particular spatiotemporal context with particular characteristics that evolve over time.

This chapter offers a panoramic of some of the key concepts present in Intermedial Studies from the perspective of the various disciplines that have shaped the field in the last decades of the twentieth century. In the 1960s, studies were mostly grounded on linguistic aspects, following the tradition inaugurated in Europe by Ferdinand de Saussure. The impact of Harvard professor of pragmatism, logic and semiotics, Charles S. Peirce gave equal prominence to other non-linguistic signs (i.e. images, gestures, etc.), helping extend the debate on the multimodality of human communication. The 1990s saw the emergence of an increasing range of methods for intermedial analysis, including neohistorical analysis, iconography, semiotic analysis, ethnomethodology, poststructuralist, gender or postcolonial approaches. Another fundamental aspect is the relevance of narrative aspects in Intermedial Studies, and the similarities and differences between intermediality and transmediality, a key issue in contemporary gaming theory. The following lines mention some of the most prominent studies coming from several of these areas.

2. The evolution of intermedial studies in the last decades of the twentieth century

One of the first systematic and comprehensive accounts of the grammar of visual design was introduced by Gunther Kress and Theo Van Leeuwen in their 1996 volume, Reading Images. Dress and Van Leeuwen drew on a large amount of examples, from textbook illustrations, photo-journalism to fine art. They built their methodology on M. A. K. Halliday’s meta functional theory [5]. Their subsequent research focused on a wide variety of formats, from texts to photographs, magazine pages and film. In Kress and Van Leewen Multimodal Discourse: The Modes and Media of Contemporary Communication they included TV programmes, films and the interaction of sound, music, language and images [6]. Meanings they argued were created through complex semiotic interactions which Kress expanded the debate about multimodality and intermediality to also cover a growing number of sign systems such as traffic signs [7].

In 1959, Roman Jakobson had published an article investigating intersemiotic translation and opening the path to the study of translation in intermedial environments. Jakobson differentiated between intralingual (rewording), interlingual (between two natural languages) and intersemiotic translation. This last one encompassed the interpretation of verbal signs by means of signs of nonverbal sign systems ([8], pp. 260–6). This research was expanded by Claus Clüver [9], Heinrich Plett [10], Umberto Eco [11], Henrik Gottlieb [12], Regina Schober [13], or Daniella Aguiar and João Queiroz [14], among many others. Theater and film adaptation, for example, have been considered intersemiotic translations. Other transformations, such as graphic novel into a film, a film into a theatrical performance or opera, opera into ballet, or a computer game into a film, might constitute a change of modality – a change of channel of perception in the recipients’ perspective – but not a translation, a term with strong associations to the transfer between different verbal languages. Thus, the notion of media transformation would be more appropriate in these cases.

In the later 1980s and early 1990s, scholars like Ernest B. Gilman [15] questioned the centrality of language in giving voice to images and discussed the imperialism of language as central to inter-art comparisons, emphasizing the rivalry between the arts and disciplines. Grant F. Scott also saw early studies on ekphrasis (see López-Varela for more information [16]) as evidence of appropriation of the ‘visual other’ and as an attempt to ‘transform and master the image by inscribing it’ ([17], p. 303). Walter Ong [18] and Walter Mignolo [19] explored the evolution of cultural memory, from indigenous oral forms to written national and colonial histories. Unlike Ong, who claimed that oral cultures did not fade with the spread of writing and later the printed press, Mignolo saw the disappearance of oral cultures in relation to the role of written literacy in nation and empire building. In his studies on the Mesoamerindian oral legacy, Mignolo unveiled the ways in which indigenous oral forms were silenced and destroyed ([19], p. 58, 62).

Jerome McGann studied the changes in the form of mass production during the eighteenth and nineteenth centuries and the sociocultural tensions between oral and written forms [20]. W.J.T. Mitchell began by exploring William Blake’s illuminated poetry, and moved on to research the ideological motivations behind the word/image debate. For Mitchell, the divisions in art and literary genres may have fulfilled the function of segregating the arts in ‘an imperialist design for absorption by the more dominant, expansive art’ ([21], p. 107) . In his ground-breaking Picture Theory: Essays on Verbal and Visual Representation, Mitchell argues that although visual and verbal media are different at the level of sign-types, forms, materials of representation, and institutional traditions, in terms of ‘expressing intentions and producing effects in a viewer/listener, there is no essential difference between texts and images’ ([22], p. 160). In particular, he turns to Laokoon in order to show how Lessing’s conception of the differences between painting and poetry is permeated by a rhetoric of gender; with images attached to female narrative and abstract symbols to superior male discourses, ‘like the masses, the colonized, the powerless and voiceless everywhere’ ([22], p. 157) .

2.1 From symbols to icons

Sensory modalities determine the way signs are decoded in intermedial configurations. Unlike Saussure, who gave prominence to the linguistic sign, Peirce noted the equal importance of three types of signs: indexes, icons and symbols. In Peirce’s classification, the visual modality can distinguish between a gesture (an indexical or iconic sign, or both, depending on the type of gesture), an icon (signs that resemble the real such as maps, diagrams and images) and symbols (abstract and arbitrary signs in a community of users, for example, human discourse). Peirce’s classification is based on how signs are related to their objects. An icon signifies its object by virtue of shared qualities, an index by virtue of a causal relation, and a symbol by virtue of an action ruled by a norm or habit, without resemblance or real connection to the denoted object but agreed by convention (for more information see López-Varela 2023) [23].

Indeed, only a historical perspective, as proposed by Jürgen E. Müller [24] can help unveil the changing sociocultural aspects that ‘qualify’ media. For instance, what Mitchell termed ‘the pictorial turn’ displaced logocentric criticism and established an iconic logic where the picture might speak to us; a time for the image to strike back, observes Bernd Stiegler in ‘“Iconic Turn” et réflexion sociétale’ [25]. This approach also included a semiotic turn performed by scholars such as Mieke Bal [26] who paved the way for the shift from intersemiotic transposition to the notion of reciprocal interaction between image and text, stressing the need to move beyond the theoretical deadlock and academic partitions which originated in the binary model of the sister-arts [16].

Following these trends, Birgit Neumann and Gabriele Rippl opened postcolonial studies to the discussion on visuality, the power of images and visual esthetics, examining writers such as Michael Ondaatje, Derek Walcott, Salman Rushdie, David Dabydeen and NoViolet Bulawayo [27]. The verbal-visual relations examined alternative modes of representation that reflect critically on hegemonic visual regimes.

This theoretical turn, which included aspects of cultural studies, visual ethnography as well as narrative theory, triggered the shift from intersemioticity to intermediality and the emergence of cross-boundary concepts, countering the logic of contemplating Intermedial Studies as a closed system. According to Rui Carvalho Homem and Maria de Fatima Lambert, ‘the present currency of a relational nexus, as theoretically averse as it is to binary oppositions, entails a reading of the intermedial that underscores notions like contamination and hybridity’ ([28], p. 11, 13). From painting to photography, from stage to the screen, modernity and avant-garde experiments have continued to inspire much of the research in the first decade of the twenty-first century. Martin Heusser ([29], pp. 221–236), for instance, has explored visuality in the relationship between text and image, noticing how different historical periods approached this relationship in different ways. Heusser believes that the context that led to the visual development of poetry at the turn of the twentieth century has to do with a turn from anabolic poetry (synthetic, complementary or constructive oriented) to catabolic poetry, where the visual component subverts the linguistic meaning. According to the author, these two tendencies seldom intersect, as they do in the modernist technique of montage. In the avant-gardes, catabolic poetry occurred far more frequently and, in its extreme forms (i.e. Lettrism, Dadaism), the resulting text was devoid partially or entirely of semantic content. Heusser connects this experimentation to a desire for a new metaphysical order and language during the inter-war period in Europe.

2.2 Extending ekphrasis

In 1992, Murray Krieger offered a rereading of the ‘ekphrastic principle’ in relation to twentieth-century experimentation. The author conceived it as ‘a verbal description of something, almost anything, in life or art’ ([30], p. 7), a device to ‘interrupt the temporality of discourse, to freeze it during its indulgence in spatial exploration’ (Ibid.) [30]. Krieger includes an analysis of poems that emulate the pictorial or sculptural arts by achieving a kind of spatiality [30]. Likewise, Wendy Steiner defines it as a description of a ‘pregnant moment in painting,’ that is, as an attempt to imitate the visual arts by describing a still moment and thereby halting time ([31], p. 41).

Attempts to redefine ekphrasis include James A. W. Heffernan’s comprehensive historical approach in Museum of Words: The Poetics of Ekphrasis from Homer to Ashbery [32], or Claus Clüver’s work, which emphasized that ‘contemporary ekphrastic practices have subverted the traditional relation of the representational visual text to its verbal representation, even to the point of discontinuity’ ([33], p. 30). With the onset of digitalization, the conceptualization of ekphrasis as a verbal representation of a visual representation has been increasingly perceived as too narrow (for the historical origins of this discussion, see López-Varela 2023 [16]). Margaret Persin expanded the range of ekphrastic studies by discussing ‘uncanonical art forms such as television, photography, comics, and cinematography’ ([34], p. 19). Other examples of intermedial crossings between photography and literature appeared in Julia Breitbach’s exploration of novels by Don DeLillo, Michael Ondaatje, and Ali Smith [35]. Jørgen Bruhn’s The Intermediality of Narrative Literature: Medialities Matter (2016) is an excellent study of Anglophone texts ranging from Nabokov, Chandler and Tobias Wolff to Jennifer Egan that demonstrates that literary fiction often includes significant amounts of extra-literary material which expose not just ekphrastic exchanges but also intermediality at large [36].

Regarding ekphrastic encounters in the field of musicology, Siglind Bruhn [37] has explored synaesthesia in the intersections between music, words, and images. Bruhn also explains that very often, composers of instrumental music are inspired by other artworks and try to convey their essence, features and message in their own medium of music. In these cases, a given medium thematizes, evokes and sometimes imitates elements and structures of another medium in order to stretch semiotic levels to their limits, modify perception and conceptual imagery, and increase immersion and esthetic response. An example of these crossings is a novel structured as a fugue, as is the case of William H. Gass’ The Pedersen Kid (1961) or the section entitled ‘A Fugue’, from his novel The Tunnel (1995). Both show how rhetorical and prosodic devices, such as rhythmical and sonorous patterns, enable the polyphonic techniques that musicalize these works and impact upon emotional aspects of reception [38]. Another example is the influence of John Cage’s experimental music in Katherine Norman e-lit piece, Windows (2012) [39].

Werner Wolf’s The Musicalization of Fiction, a title inspired by Aldous Huxley’s homologous term from his 1928 novel Point Counter Point, defends that intermediality describes a transfer process of structure and esthetic intention from music to fiction. Wolf describes modern and postmodern esthetics in terms of harmony-driven structures, finding that avant-garde experimentation not always created meaning. The book contains excellent analyses of modern British experimental fiction modeled on music [40].

2.3 The intertextual grounding of intermediality

Werner Wolf’s contribution went beyond the categorization of musical-textual relationships. He began to distinguish between ‘overt’ and ‘covert’ intermediality. In the first type, ‘qualities are immediately discernible on its surface’ ([40], p. 40). Wolf’s categories referred to the distinction between combination and integration of media on the one hand and transfer and transformation of media on the other. In the case of media transformation, the German scholar spoke of ‘implicit intermedial reference’ (imitation), in which fiction mimics the musical form, usually as repetition or counterpoint, to thematization or ‘intermedial reference’ (texts that thematize, quote, or describe other media, like a novel devoted to the career of an artist). Music can be thematized through ‘associative quotation’ ([40], pp. 67–69). ‘Intermedial transposition’ are adaptations from one medium to another. For Wolf, the underlying connection seems to be the possibility of a narrative structure or ‘transmedia’, that is, phenomena that can be represented in more than one medium because of their narratological basis.

In an article from 2002, Wolf proposed another intermedial typology with a distinction between ‘intracompositional’ and ‘extracompositional’ [41]. According to Wolf, a phenomenon such as a novel transformed into an opera, conceptualized as ‘intermedial transposition’, should rather be considered as ‘extracompositional intermediality’, while the musicalization of fiction, labeled ‘implicit intermedial reference’ (imitation), should be considered ‘intracompositional intermediality’ ([41], pp. 27–29). Yet, both types of intermediality presuppose an awareness of pre-existing media products or media qualities (a certain narrative or musical characteristics) and hence point to something beyond the ‘intracompositional’, claims Elleström ([2], p. 14). Wolf’s research was further expanded in The Metareferential Turn in Contemporary Arts and Media [42], with many of his works collected in a celebratory 2017 volume [43].

In some of his studies, Wolf uses the term intermediality to detect implicit and explicit discourses, similar to intertextuality, and this prompted a number of scholarly publications directed to this inquiry. This is the case of Jürgen E. Müller’s work [44]. Peter Wagner distinguishes ekphrasis from iconotexts or ‘the use of (by way of reference or allusion, in an explicit or implicit way) an image in a text or vice versa.’ His definition of intermediality is still tied to pre-digital constrains. He considers that it is ‘the “intertextual” use of a medium (painting) in another medium (prose fiction)’ ([45], p. 17).

A similar exploration is offered in the collection coordinated by Valerie Robillard and Els Jongeneel In Pursuit of Ekphrasis: An Intertextual Approach (1998). Claus Clüver, who participates in this volume, explains that ekphrasis retains a certain degree of Aristotelian energeia and defines it as ‘the verbalization of real or fictitious texts composed in a non-verbal sign system’ ([33], p. 49).

These approaches maintained a certain emphasis on the contents of the information transferred, rather than on its form and material aspects. Some of the ideas that contributed to enriching the discussion on the materiality of intermediality came from film studies but, mostly, with the onset of widespread digitalization in the 1990s, from hypertext and hypermedia theorization, as well as from the field of semiotics. Bernhard F. Scholz also recognized that the concept is a ‘complex multi-dimensional multi-faceted semiotic phenomenon’ ([46], p. 75) with ties with intertextual relations ([46], p. 74).

Following Wolf, Irina Rajewsky proposed several categories outlined in her book-length study Intermedialität and in subsequent articles. ‘Media combination’ (Medienkombination), where it is an inherent quality of a certain genre of media, like theater or film, exemplified with opera, comics, etc. ‘Medial transposition’ (Medienwechsel) or transformation of a given media product, where intermediality manifests itself in how a media product emerges, as in the case of adaptation. ‘Intermedial references’ (intermediale Bezüge) evoke or imitate entities of a medium in a different medium, e.g. painting or cinema in literature, exemplified with phenomena such as ekphrasis, narrativization of music and references to painting in film ([47], pp. 15–17, [48], pp. 51–53). Thus, Rajewsky’s categories run from mere contiguity of two or more material manifestations of different media to genuine integration.

These categories are not closed structures, so film adaptation can be classified as a media combination (of theater and photography) and medial transposition (of a literary text). It is also interesting that she also considers contextual aspects such as the production and specificity of material media patterns, which have changed over time, and she goes on to relate their ‘intermedial’ qualities to their use and reception. In the case of film adaptation, Rajewsky explains that the viewer receives the original literary text, not as something on which the film production is based, but as another nucleus that produces crossed relations on the horizontal (not vertical/hierarchical) intermedial level. Rajewsky’s approach has the advantage of theoretically distinguishing between intramedial (and thus intertextual) and intermedial references. Within the first, a medium evokes and generates an illusion of another medium’s specific practices, as mentioned above. Intermedial references, however, constitute themselves in various complex combinatory ways in relation to another medium (monomediality) or several media (plurimediality), as in the case of dance theater. In these cases, a given medium thematizes, evokes and sometimes imitates elements and structures of another medium in order to stretch semiotic levels to their limits, modify perception and conceptual imagery, and increase immersion and esthetic response.

2.4 Intermediality, transmediality and the impact of narratological studies

Elleström’s summarizes the types of media transformation in two: ‘transmediation’ and ‘media representation’ [2]. They resemble Rajewsky’s distinction between ‘medial transposition’ and ‘intermedial references’ only at a superficial level. For Elleström, ‘transmediation’ is ‘transfer of media characteristics’ ([49], p. 5), ‘a central part of intermediality, which is an even broader concept based on the proposition that different media types are interrelated in all kinds of ways’ [49]. Transmediation includes previous categories of ‘implicit intermedial reference’ (imitation) and ‘intermedial transposition’. Elleström explains that some perceived media characteristics of the target medium are, in important ways, the same as those of the source medium, ‘which is to say that the media characteristics of an initial medium are perceived to be represented again by another kind of medium ([2], pp. 20–27). Thus, ‘transmedial narration’ ‘should be understood to refer to all varieties of transmediality and transmediation where narration is a media characteristic that is significant enough to be observed’ ([49], p. 6).

The second category formulated by Elleström, ‘media representation’, resembles Wolf’s ‘explicit intermedial reference’ (thematization) ([41], pp. 27–29). However, Elleström’s notion of ‘representation’ and ‘mediation’ are different. He describes ‘mediation’ as a pre-semiotic phenomenon based on the material properties of objects in general and technical media in particular, perceived through the senses in a particular spatiotemporal configuration. Many analogue processes are sequential, including human language; sequential sign systems can easily turn into narrative forms, whether in oral or written discourse, in melodic music, and even in the case of iconic signs, as in motion pictures. These properties are potentially meaningful when sensory configurations are activated by cognitive import, thus becoming meaningful. This is what Elleström describes as the semiotic phenomenon of ‘representation’. Any object or media, for example, the page of a printed book, stimulates the perception of a sensory configuration through the visual mode. In Braille, the activation would take place through the tactile modality. The page can contain a literary piece, a musical score, or a printed image or diagram. The musical score or the poem previously seen on the page can be then transmediated by another modality or sensory configuration, in this case, the voice as the piece is read or sung, as well as mediated by another kind of technical medium ([2], p. 19). Elleström mentions here the differences between the book page emitting photons or sound waves generated by vocal cords, explaining that he does not conceive the ‘technical medium’ necessarily as a machine or a device for production or storage but rather as a form of distribution that disseminates sensory configurations. In this sense, for Elleström, transmediation is not only remediation in another technical medium (in Bolter and Grusin’s 1999 sense) [50] but also includes the possibility of multimodal translation of content from one sign system into another, triggering different sensory configurations. The second category theorized by Elleström, ‘media representation’, involves the notion of one medium representing another medium. This case involves transmediation as well as repeated mediation of equivalent sensory configurations.

While Elleströms’ inquiry comes from the field of Communication Studies and Semiotics, Marie-Laure Ryan, and Marina Grishakova have explored intermediality from the perspective of narratology. Drawing on Dan Sperber’s notion of ‘metarepresentation’, in the volume Intermediality and Storytelling (2010) coordinated with Ryan, Grishakova expands her differentiation between two forms of intermedial metarepresentations; metaverbal (an attribute of verbal texts that evoke images) and metavisual (an attribute of images that reflect on the incomplete nature of visual representation). Grishakova explains that ‘the metaverbal text (e.g. an ekphrastic text, cine-novel, or graphic poetry) reflects on the incomplete nature of the verbal medium by probing the limits of verbal representation and appealing to the visual forms (graphic elements, real or virtual film shots, works of art, dreams, hallucinations, mental imagery, etc.).’ On the other hand, ‘the metavisual text reflects on the incomplete nature of visual representation by juxtaposing image with verbal message and revealing their discrepancy’ ([51], pp. 313–315).

In Picture Theory, Mitchell had already noted that he sought ‘to experiment with the notion that pictures might be capable of reflection on themselves, capable of providing a second order discourse that tells us—or at least shows us—something about pictures’ ([22], p. 38). Marie-Laure Ryan’s introductory piece expands this research and by, comparing images and texts, she concludes that images tent to be decoded as less fictive. Some years before, in the volume Intermediality and Performance, edited by Freda Chapple and Chiel Kattenbelt [52], Peter M. Boenisch had also tackled a similar issue. For him, mediatized representation always takes the form of a second-order reality, ‘whether as idealist representation of a crude reality, in terms of the beautiful and the sublime, as a realist rendering of the actual thing, or as a surreal invention in Science Fiction.’ Boenisch affirms that in all cases, the experience is ‘present’ to audience, and ‘whether factual or fictional, would be felt in the same way because it is authentic in terms of its spatiotemporal effects.’ ([53], p. 110). Indeed, contributions from semiotics show that iconic signs (images, graphs, and diagrams) acquire primacy because they require lower levels of decoding ([54], pp. 204–5). Discourse typically has a lower sense modality than photography, television or film, judged to be more realistic because of their higher image content.

Jan Baetens’ important work on sequential images and word/image combinations has focused on a range of narrative types, from roman-dessiné and comics to film photo-novels [55, 56]. His research has contributed interesting insights into transmedial thematology. Sequential montage of images organizes as a series drives a certain visual movement, a sort of chain of sign patterns that can be interpreted as a narrative framework. Film photo-novels have a hybrid format where pictures with captions and dialog balloons are used to re-create a cinematic story. These lowbrow film novelizations appeared in weekly and monthly magazines and had a significant social impact, particularly in Latin America and the United States in the first half of the twentieth century. The history of graphic narrative and its remediations to various formats has also contributed to the study of transmedial narratologies (see, for instance, [57]; for comics, [58, 59]).

2.5 Intermediality, the performing arts, cinema, television, and beyond

A number of studies on intermedial exchanges fall under the scope of theater and drama. Before the invention of cinematography, drama and dance performance where possibly the environments with greater intermedial scope. Technical skills in drama production involve a wide range of knowledge, encompassing design and architecture, drawings, sketchbooks and photography [60]; illustration of stage instructions; text-based instructions such as notes, scripts [61, 62], prompt and cue lists, and so on [61, 62]. Edited by Michael Bull, The Routledge Companion to Sound Studies (2018) is an extensive volume presenting the workings of sound in culture and the role that sound media play in the construction of experience [63]. Samples of textiles, furniture and all kind of model artifacts are also used in performance [64]. While puppets were very popular in the past, today, electric shadows as well as video might be projected including digital technologies [65, 66, 67]. All these aspects enhance intermedial combinations, explored in a wide range of works like those by Chapple and Kattenbelt [52], Brown, Hauck and Larrue [68], Sager [69], Drumm [70], and Bay-Cheng, Kattenbelt and Lavender [71], among many others.

On the topic of intermedial adaptation from theater to television and cinema, we can mention Mancewicz [72], and Wyver [73]. The presence of cinema upon Intermedial Studies owes much to the collection of books by Edinburgh University Press entitled ‘Edinburgh Studies in Film and Intermediality.’ In the last 10 years (2012–2022), the collection has published 19 volumes that approach the topic from a wide range of angles; for example, conversations between filmic techniques and literature, photography, theater, painting, architecture or videogames (see for instance, Ingham [74]. Susanne Foellmer, Maria Katharina Schmidt and Cornelia Schmitz [75] deal with dance and performance art as well as opera to explore the transitions among media as well as the dynamics of transfer between the performing and visual arts. Miriam de Paiva Vieira tackled topics spanning narrativity, various modes of literary expressions, intersemiotic translation and multimodal communication [76] (see also [77, 78]).

Research on the intermedial aspects of cinematography is extensive. Jeffrey Geiger’s collection, for instance, examines the relations of cinema to literature, photography and other modes of media, from the magic lantern, the zoetrope, or the flick-book, to the iPhone and the computer [79]. Other works on cinema and Intermedial Studies include those by Mariniello [80], Gaudreault, Russell and Véronneau [81], Pisano [82], Nasta and Huvelle [83], Altman [84], Ágnes Pethö [85, 86], De Giusti [87], Gaudreault and Barnard [88], Vieira and Rio Novo [89], Giuliani and Negri [90].

Examples of historical media conjunctions are the combination of radio, static images (photography) and moving images (film) into cinematography and later into broadcast media like television, with the recent rebirth of the serial formats that has accompanied the emergence of subscription streaming services. The 1970s and 1980s were the era of national television. TV content varied considerably from nation to nation, broadcasted in analogue devices with normally one TV screen per household. After the 1950s, magnetic tape video recording became a major contributor to the television industry, via the first commercialized video tape recorders, with two standards, VHS (the Netherlands) and Betamax (Japan), which entered the homes in the 1970s. They joined forces in developing a new digital format for audio recordings, later known as compact disc (CD), which replaced magnetic tapes in the 1990s [91].

The success of TV was not only due to the number of users but in how its content was present in other media, like newspapers, magazines, advertisements, radio etc. ([92], p. 118). John Ellis also demonstrated that television was constructed by means of interrelationships between other audiovisual media, mainly radio, cinema and video recording [93]. Thus, while television was created by previous media, which were adapted to the new forms, it is also important to consider their relative power, for instance, the relationship between radio and television, as well as the significance of other media that contributed to circulate information from the TV screen to other media industries, as the tabloid press did. This bond between media industries continues today in Internet platforms that offer background information to the news, for example, or the lives of the performers who appear on television. This cooperation also existed between TV music channels, like MTV. This cable channel was launched in 1981 to promote music videos and interviews with artists. It became a huge marketing machine able to operate in conjunction between the music industry, television and other forms of promotion such as concerts, creating a media event able to distribute its brand into other marketable products (T-shirts, souvenirs, etc.), distributing to consumers via different media at different times. Although different from the transmedia phenomenon, in which a story unfolds across multiple media platforms, with each new text making a distinctive contribution to the whole, ‘transmedia storytelling’ [94] could even be seen as a result of a brand narrative expressing itself in different media and business areas [94].

The collection edited by Paul N. Reinsch and Laurel Westrup offers a fresh approach to the albums that accompanied the promotion of films, television shows, or video games and the audiovisual relationships that drive the encounters between music and media studies [95]. Louise Harris interrogates the relationship between audiovisual work in diverse media, including for instance audiovisual sculpture [96]. Likewise, for years music videos have helped promote artists and circulate their works widely. With the proliferation of digital technologies, the advent of YouTube in 2005 and social networking sites, music videos visualize the music industry. The collective volume by Lori A. Burns and Stan Hawkins includes a range of essays that explore issues related to the sociocultural impact of music videos as well as their intermedial implications [97].

The 1990s were a turning point for wireless telecommunication and, as mention, artistic trends, like Fluxus video art, took new directions with digitalization. Various technical media saw innovations. For instance, a textual data extension called ‘teletext’ was added to analogue television, and digital stereo broadcasting systems were also introduced. A full convergence was achieved before the twenty-first century when MIT introduced a computer-compatible digital high-definition television and video (HDTV) system with progressive image scanning that revolutionized the future of television. Digitalization also accelerated changes towards streaming of video via broadband Internet. Nowadays, using the Internet to watch television is common, although young users prefer personalized experiences on Youtube or following the channels of their favorite influencers. The evolution of television captures an ongoing negotiation between emerging technical media, future expectations, past experiences and old content, out of which new things emerge as in flux.

Intermedial Studies with a focus on sound have also been explored from the point of view of the cultural. Mediations: Body Sound Technology (2008), edited by Carolyn Birdsall and Anthony Enns, recognizes sound as ‘a co-participating agent in cultural practices and performance’ ([98], p. 3). Each chapter of the book offers a case study coming from diverse disciplinary backgrounds, such as musicology, film and media studies, art history, comparative literature, philosophy, performance, and Intermedial Studies [98]. Karin Bijsterveld’s [99] collection studies the ways in which ‘staged sounds’ might present the changing identities of cities, like Amsterdam, Berlin and London. Their soundscapes, as recorded in historical documents, radio plays and films, help bring back a rich cultural heritage. Similarly, to ponder the reciprocity between intermedial relations, creativity, different sensory systems and memory [100]. Sarah Durcan’s [101] Memory and Intermediality in Artists’ Moving Image, contributes to intermedial and memory studies by examining the mid-1990s transitional moment that coincided with the impact of digitalization upon video art and experimental film [49, 99]. These works highlight the ways in which the political and the sensory are linked directly to modalities of representation belonging to a given cultural moment. In this regard, it is important to mention the work developed at CRIalt, heir to the Centre for Research on Intermediality (CRI), established in 1997 in Montreal, Canada. The centre starts from the principle that we cannot study each media in its autonomy because all media interact and define each other in their interactions [101].

As I have pointed out, it is important to contemplate intermedial relations and their semiotic, historical, sociological, cultural and political implications. In this regard, research by Asun López-Varela mainly focuses on semiotic and cultural aspects [102, 103], including the recently published Palgrave Handbook on Intermediality (2023) [23].

3. Intermediality and the digital turn

As seen, the evolution of Intermedial Studies owes much to the peculiarities of the medium, and digitalization has further complicated this interdisciplinary field. In analogue media, the concept of intermediality means that separate elements, taken from different technical media and that have evolved historically (i.e. static images-photography turning into moving images-film), are interrelated in ways that their heterogeneous forms of expression come together and produce one integrated effect upon reception. Thus, after the onset of digitalization, intermedial inquiries were initially directed towards the study of similarities and differences between analogic domains, conceptually expressed by means of metaphors and metonymies among sister-arts; that is, in terms of physical phenomenology, like the human analogic perceptual apparatus.

In digital environments, different technical media are integrated within a complex technical system where their distinctions are blurred. Interrelationships are no longer grounded on phenomenological transformation. Unlike their mechanical analogue counterparts, digital machines include layers of mathematical code in combination with human languages. Computing processes simulate, that is, create a physical resemblance on the superficial layers and on the so-called ‘interface’. The differences and technical requirements of previous analogue media are dissolved in the inner workings of computation. Experts have spoken of media convergence. This convergence involves not only technical media, but also their associated industries, services, and work practices [23, 94, 104]. This raises the following question: in the digital scenario where the transformations of analogue technical media are dissolved, can we speak of intermediality?

Our answer is a resounding yes. We can still speak of intermediality because the technical medium is just one of the cluster of aspects involved in the multi-layered dimensions of intermediality. In the case of digital media, we may speak of complex forms of ‘media representation’ ([2], p. 19) , at least on the superficial interface level. In other words, the digital machine simulates exactly those transformation processes that occurred in analogue media, activating equivalent sensorial configurations. Thus, in transmediation, in the sense used by Elleström, we can talk of a simulation of transformative processes which include the repeated mediation of equivalent sensory configurations by another kind of technical medium [49, 105, 106, 107].

3.1 From hypertext to cyberspace

Since the origins of hypertext in the 1940s [108] and the development of Tim Berners-Lee’s World Wide Web, digital advance has multiplied. The debate on what a medium is and how it constructs culture has often followed from Marshall McLuhan’s pioneer work [107]. In the 1990s, the field of electronic literature began to grow. The first pieces of e-lit included only hypertext and could be read along diverse modular paths, like Borges’ short story ‘The Garden of Forking Paths’ (1941). Gradually, text-based e-lit gave way to digital art and Net. Art, which began to include a growing number of images, sound, video to be manipulated on the screen (see the collections of the Electronic Literature Organization ELO). As in cinema and photomontage, this accumulation blurred, even more, the linearity that conforms to traditional reading patterns, accumulating sign systems into a sort of spatial density. The ordering principles of the logic of print formats and analogue technical media were broken. Everything on the screen could be manipulated, from the words of a text to the images that accompany it. Within years, E-lit and Net. Art pieces included even more links to external ‘social’ websites that allowed commentary from the readers/users. The term ‘social media’ refers to technologies, platforms, and services that enable individuals to engage in communication from one-to-one, one-to-many, and many-to-many. This was possible through the development of Web 2.0, which in the first years of the twenty-first century aimed to be user-focused.

Hypertext continued to evolve into a very intricate network. The very essence of the World Wide Web challenges linearity. Undoubtedly, although there is a legacy of the analogue world in the digital, many things have changed. For instance, in analogue electronic media, signals constituted a linear continuity. In digital machines, signals are not continuous but compressed into discrete data units known as digits (1 s and 0 s at the lowest level of machine language, representing voltage bands). These correspond to the two values of ‘true’ or ‘affirmative’ and ‘false’ or ‘negative’ in binary logic and in the Boolean algebraic domain.

In her description of computer signs as ‘flickering signifiers’, Katherine Hayles draws a similarity with the film [109]. In analogue projectors, film advanced frame by frame, each frame standing still in front of the lens for a portion of the frame cycle (typically 24 cycles per second) and rapidly accelerating, flickering, advancing, and decelerating again. This intermittent motion is fused by means of a device known as the Geneva drive, first implemented in the world-famous Swiss mechanical watches. It is also known as the ‘Maltese cross mechanism’ due to its visual resemblance when the rotating wheel has four spokes (a spoke is one of some number of rods radiating from the centre of a wheel). With this mechanism, flickering is no longer perceived by the human eye, and the discontinuous images are seen as fused. The ‘flickering signifiers’ of digital media also mean that while the analogue figurative is always affirmative and true, resembling the real, the digital allows for a reversible switching mechanism that includes the false and the negative (1 s as well as 0 s; see also [110]). Lev Manovich has stressed that machine languages and the logic of the computer could be expected ‘to significantly influence the cultural logic of media’ ([111], p. 46).

One of the most important characteristics of digital machines is that they have screens that function as interfaces with the analogic world of human perceptual experiences. In this sense, they resemble the ‘cinema effect’, as Sean Cubitt put it in his 2004 homonymous volume [112]. Screens are optical media that, like mirrors, allow a very close similarity with the real. Cinematic representation configured three-dimensional space as well as the various elements that constitute sensual phenomenological experience –light, shape, and texture –what Cubitt termed ‘spatial effects.’ Digital media turns represented space into a simulation. It is a ‘heterogeneous space’ that features windows not as windows onto the world but as windows that ‘open onto other representations or other media’ ([113], p. 329) [50]. In doing so, it provides a mechanism to transform analogue information into a schema or map, drawn in binary code at its lowest level; this is almost like the neuroplasticity that the brain performs when filling in the gaps in mental images.

The illusion of transparency, a feature of all screens, including those of digital media, blinds viewers to the part played by the internal aspects of the technical medium in constructing experiential worlds. In digital media, sense modalities are increasingly combined to produce the greatest effect of authenticity (hyper-real as coined by Bolter and Grusin [113], p. 24) [50], beyond the mere similarity of the mirror. Indeed, ‘the logic of immediacy leads one to erase or automatize the act of representation,’ for it ‘acknowledges multiple acts of representation and makes them visible’ (1999, p. 329) [50].

3.2 Navigating the digital

George Landow’s first exploration of hypertext noted the modularity and non-hierarchical status of sign units, whether text, image, video, or sound, and explained that it yielded a completely different reading experience [114]. Within texts, sign-words acquire meaning in relation to the entire chain in which they occupy a particular position –subject, verb, object, and so on. In digital environments, distributed on the screen space, sign-positions, ruled by mathematical coding, also become signs, with links anchored to texts, images, parts of images, video, or any other webpage. Thus, digital screens do not provide a specific reading direction. They comprise all directions. In conventional narrative structures, which follow temporal paths with a beginning, a middle, and an end, the readers’ capacity to envision the end determined meaning. The reading experienced followed a dynamics based on suspenseful prospection, curiosity-driven retrospection and surprise-generated recognition. Any gaps lacking information were filled by inferences. In intermedial multimodal configurations, there are more paths, and the heterogeneity of signs, ruled by the underlying code, problematizes the making of inferences [115].

In trying to explain the reading process in cybertext, a term that in the late 1990s began to substitute hypertext, Espen Aarseth used the word ‘ergodic’, from ancient Greek ‘ergon’ (a unit of work or energy) and ‘hodos’ (threshold as well as path) [116]. Aarseth’s concept signalled the spatiotemporal fusion that takes place in the digital technical medium. The prefix ‘cyber’ used ‘cyberspace’, came from in ‘cybernetics’, in ancient Greek ‘kubernētēs’ meaning pilot or steersman. All these terms came to emphasize the navigational skills required in online omnidirectional environments. The temporal features present in print reading environments are reshaped from unidirectional patterns into multidirectional ones. One of the first examples of multilinear storyworlds [117] is Michael Joyce’s Afternoon, a story published by Eastgate Systems in 1990. The story includes various paths that the reader/user can navigate by clicking the hypertextual links. The order of the events in the story is not fixed. They appear in the order that the reader activates when clicking on them. In 1992, Stuart Moulthrop created Victory gardens, also published by Eastgate Systems. Upon entering the work, the reader is presented with a series of choices as to how to navigate the story. The reader may enter the text through a variety of means: the map of the garden, the lists of paths, etc. With no end to the story, the multiple node-links provide a certain sense of closure.

Mark Bernstein provided a summary of the different patterns that links could take, thus influencing the online reading experience: ‘cycle, counterpoint, mirror-world, tangle, sieve, montage, split/join, neighborhood, missing link and feint’ [118]. Some of these hypertextual structures, he argued, interweave different narratological voices (in Bakhtin’s sense) of equal (or nearly equal) weight within a single exposition (see, for instance, counterpoint). The ‘mirror-world’ establishes a second voice that separately parallels (or parodies) the main statement. The ‘sieve’ and the ‘split/join’ explicitly guide a user’s path and may allow users to experience different episodes or points of view. The ‘split/join’, however, might be misleading in the diversity of viewpoints represented because no matter which path a user takes through the landscape, the exit is always the same. Bernstein suggests that each structure suits different rhetorical purposes, so argumentation, for example, utilizes a ‘cycle’ by repeating points or modifying them, and a ‘tangle’ might disorient users in order to make them more receptive to a new argument or an unexpected conclusion.

Leonardo Flores [119] studied electronic poetry (e-poetry) as a model of digital textuality in order to explain how mathematical code languages enabled different forms of engagement on the part of readers/users. For instance, some software features allow the continuation of different statements of text, images, and other types of multimodal content [119]. Recursion or repeated execution of patterns creates loops and subroutines. As systems of layers, each of them programmed in different mathematical languages, digital environments allow great mobility of units. They can be assembled and re-assembled into larger-scale objects and continue to maintain their separate identities ([111], p. 30). Manovich indicates that the major difference between old and new media is the fact that old media involves a manual assembly of visual/verbal elements into a composition or sequence, whereas digital media generate many different versions and variations accomplished by automation and random conditions ([111], p. 36).

Along these lines, Jay David Bolter and Richard Grusin’s [113] Remediation, brought forth the idea that a medium appropriates the techniques, forms, and social significance of previous media and attempts to rival or refashion them. The authors discussed how new media develop in relation to medium-specific modes of previous media, designed to improve previous shortcomings. Thus, remediation in Bolter and Grusin refers to the historical interplay of older and newer media, paradoxically trying to erase the old and multiply the new media ([50], p. 5). Unlike Bolter and Gusin, Elleström conceives ‘remediation’ as repeated mediation only concerning the physical realization of entities available in perception. Online digital formats show the dominance of certain sensorial modes. In printed texts, the gaze moves the narrative forward until the reader turns the page. In digital multimodal formats, there is a complex intermediality that synaesthetically engages touch through the movements of the cursor enabled by keyboard or mouse, eye-tracking that follows the multimodal cues provided by links and which access various types of content –whether text, image etc. This complexity was explored in a paper co-authored with e-lit artist Serge Bouchardon [120]. Other synesthetic aspects in multimodal cueing in virtual environments have also been explored to show how changing technological patterns introduce ontological and cultural implications into the discussion on intermediality, a field that continues to expand [121].

3.3 Visual narratology

As mentioned, one of the fundamental issues raised in Intermedial Studies after digital media convergence is related to narratology. Jens Schröter’s research on visual narratology as tertium comparationis between different artistic expressions and transformations from analogue to digital formats was expanded in the 2008 volume he edited with Joaquim Paech [122]. In his 2011 article ‘Discourses and Models of Intermediality,’ Schröter identifies four models of discourse. Synthetic intermediality is a fusion of different media. Formal (or transmedial) intermediality is a concept based on formal structures not specific to one medium but found in different media. Transformational intermediality is a model centred around the representation of one medium through another medium, leading to the postulate that transformational intermediality is located in the processes of representation. Finally, ontological intermediality is a model suggesting that media always exist in relation to other media. Schröter mentions a fifth model of virtual intermediality that he expands on in his 2014 handbook, published in German. In agreement with his category of ontological intermediality, a medium defines its own ontology by relating itself to another medium; thus, it seems, just as with humans and their others, that it is not possible to define the specificity of a medium in isolation except through a dialogical encounter with another medium [123].

The first decade of the twenty-first century saw the publication of a large number of studies on intermediality. For instance, the edited volume by Herkman, Hujanen, and Oinonen [124] places intermediality as a theory as well as a methodological approach. Some articles provide a critical reflection on media, while others deal with intermediality, the analysis of representative cases, and media institutions and professions [124]. Also, in 2012, research by Bernd Herzogenrath discusses digital photography, comics and graphic novels, performance art, techno, hypertext, and video games. The volume also explores how intermedial interactions shape techniques of representation, strategies of communication, and forms of reception [125].

Among the histories of digital media, Gabriele Balbi and Paolo Magaudda examine the interactions between political, technical and sociocultural elements [126]. Alongside Baetens’ work, already mentioned, Brianne Cohen and Alexander Streitberger’s [127] provide an exploration of the ‘photofilmic’ looking at the different ways in which art, cinema, and other forms of visual culture respond to a digitized and networked world. The essays in this collection deal with various practices, ranging from panoramas, drone surveillance, tableau-vivant, press coverage, and computer-based editing [127].

3.4 The contribution of non-Western studies to Intermediality

The field of Intermedial Studies is in continuous expansion. This paper has not been able to cover research from non-Western sources. However, we can briefly mention Yvonne Spielmann who, during her visits to Japan between 2005 and 2009, discovered the hybridity of Japan’s media culture, a blend of digital-analogue connections, Western and non-Western media esthetics. From Japanese fondness for precision and functionality to the poetics of unobtrusiveness and detail, Spielmann examines pioneering mixed media artists like Masaki Fujihata, Seiko Mikami, Sota Ichikawa, Toshio Iwai, or Tatsuo Miyajima [128].

The monograph by Jihoon Kim also considers experimental film and video, essay film, gallery-based installation art, and digital art and explores technical issues such as stillness, movement and indexicality as well as forms of storage and archiving under the influence of digitalization. The author explores over 30 artists and filmmakers, including Jim Campbell, Bill Viola, Sam Taylor-Johnson, David Claerbout, Fiona Tan, Takeshi Murata, Jennifer West, Ken Jacobs, Christoph Girardet and Matthias Müller, Hito Steyerl, Lynne Sachs, Harun Farocki, Doug Aitken, Douglas Gordon, Stan Douglas, Candice Breitz, among others [129].

3.5 Postdigital performance and installation art

Catherine Elwes [130] also focuses on film and televisual installations through architecture, examining galleries and museums, as well as painting, sculpture, performance, expanded cinema, and countercultural film and video from the 1960s, 1970s, and 1980s. Elwes, herself is critic, curator, and practitioner in British feminist art politics, examines political and ideological positions in the works of practitioners such as Anthony McCall, Gillian Wearing, David Hall, Janet Cardiff, Peter Gidal, and Maya Deren [130]. Sally Johnson and Astrid Ensslin’s collective volume tackles issues of identity, gender, youth, citizenship, politics and ideology from a sociolinguistic perspective, exploring communication across a range of mediums including television, radio, newspapers, magazines and the Internet [131].

Gabrielle Jennings, Kate Mondloch Abstract Video, also explores video experimentation but also Net. Art, installation, new media, expanded cinema, visual music, and experimental film, including celebrated artists such as Jeremy Blake, Mona Hatoum, Pierre Huyghe, Ryoji Ikeda, Takeshi Murata, Diana Thater, and Jennifer West, alongside emerging artists [132]. Claudia Benthien, Jordis Lau and Maraike M. Marxsen [133] explore the literariness of media art starting from the framework of Russian Formalism, where the term ‘literariness’ was coined. The work of renowned artists such as Chantal Akerman, Mona Hatoum, Gary Hill, Jenny Holzer, William Kentridge, Nalini Malani, Bruce Nauman, Martha Rosler, and Lawrence Weiner is explored in this volume that analyses experimental film, video performance, moving image installations, and other media-based art forms from the perspective of intermediality, remediation, and postdrama [133].

Digital Theater (2020) by Nadja Masura examines de interaction of digital technology and performing bodies exploring live theater performances which incorporate video projection, animation, motion capture, robotics and Virtual Reality (VR). The volume includes practitioners like George Coates, the Gertrude Stein Repertory Theater, Troika Ranch, David Saltz, Mark Reaney, The Builder’s Association, and ArtGrid [134]. Luis Campos & Fiona Jane Schopf [135] explore the topic of intermedial performance of electronically controlled sounds, embodiment and sound on the stage, and musical speech. Among the works explored, Iannis Xenakis’ reworking of ancient Greek in Oresteia; trans-genre adaptation in Strauss’s Der Rosenkavalier or Philip Glass’s Cocteau trilogy, musical comedy in Ligeti’s Le Grand Macabre [135]. Body Knowledge: Performance, Intermediality, and American Entertainment at the Turn of the Twentieth Century (2013) by Mary Simonson is a comprehensive monograph on the part played by female performers in North-American onstage and on-screen. The volume explores the ways in which intermedial experiences in early twentieth-century art are the product of a dialog between technological developments and new modes of perception, involving novel understandings of the body and the self, and shifting conceptions of gender, race, and sexuality [136].

Activism and Postdigital Performance by Liam Jarvis and Karen Savage [137] traces how identity can be created, developed, hijacked, manipulated, sabotaged and explored through performance in postdigital cultures. Considering how technology is reshaping performance, this collection of essays examines the artist-activist as a producer of avatars engaging in performance practices that problematize identity politics. Using a range of examples in theater, film and Internet-based performance practices, the essays examine mediatized digital cultures, expanding the notion of intermediality. Case studies include James Cameron’s Avatar, Blast Theory’s Karen, Ontroerend Goed’s A Game of You, Randy Rainbow’s online videos, Sisters Grimm’s Calpurnia Descending, Dead Centre’s Lippy and Chekhov’s First Play, and Jo Scott’s practice-as-research [137]. Theater maker and senior lecturer Liam Jarvis had previously explored the field of ‘virtual embodiment’ in performance, installations, theater/art-making, arts-science and VR, applied to a wide range of contexts where the human body becomes part of computer-generated experiences. His 2019 volume on immersive embodiment included an interesting discussion on the complex experiences in healthcare, for instance [138].

3.6 Transmediality and storyworlds

much contemporary e-literature and e-poetry play with the graphic patterns of letters, words, or symbols rather than by the meaning of words in conventional narrative arrangement, like Concrete Poetry did in the twentieth-century avant-gardes.

digitalization has highlighted the debate between intermediality and transmediality, and its storytelling foundations. Transmedia storytelling has become an influential mechanism in the field of media and entertainment, enabling the transformation of books into films, TV series, videogames, advertising, etc., telling stories across multiple platforms. One of the first scholars to theorize the field of transmedial narratology was Marie-Laure Ryan. In 2014, she published Storyworlds across Media together with Jan-Noël Thon [116]. The volume explores how media, old and new, give birth to various types of storyworlds and provide different ways of experiencing them. The volume critically assesses the cross- and transmedial validity of narratological concepts such as storyworld, narrator, representation of subjectivity, and fictionality. Drawing on her previous work, Ryan notes that narratives exist simultaneously as representations or mirrors of another world and as a cognitive mode or structure for understanding out world. As a cognitive mode, it relies on the technical medium that displays it. Thus, print narratives are different from film narratives and from computer worlds, determined not by the text itself but rather by the experience of the text; its manipulation, in the case of digital storyworlds. Ryan claims that ‘the convergence of media around a common center that we may call “narrativity”’ is ‘a center that is itself organized around a storyworld’ ([117], p. 3). Thus, the book explores media convergence and transmedial storyworlds by examining forms of storytelling based on multiple media platforms. This book offers a valuable contribution to the literature in the areas of transmedia storytelling, narratology, digital fiction, electronic literature, locative storytelling, performative writing, digital culture studies and human geography.

The collection ‘Routledge Advances in Transmedia Studies’ includes a number of volumes that explore the concept of transmediality mainly from the perspective of communication studies and journalism. For example, the volume by Renira Rampazzo Gambarato, Geane Carvalho Alzamora, Lorena Tárcia [139] explores transmedia dynamics in various facets of fiction and nonfiction transmedia studies. Moving beyond the presentation/definition of transmediality as a field of study, the authors examine novel advancements in the theory, methodological development, and strategic planning of transmedia storytelling [139].

From a sociocultural perspective, Christina Meyer and Monika Pietrzak-Franger [140] examine early transmedia practices in British Victorian media, including books, theater, advertising, newspapers and games, thus tracing a historiographical account of transmedia narratology. The volume also explores nineteenth-century forms of audience participation and the role that social agents –authors, publishing houses, theater producers, games manufacturers, as well as audiences – played in the production, distribution, and consumption of Victorian media. It considers such examples as Sherlock Holmes, Kewpie Dolls, media forms and practices such as cut-outs, theater broadcasting, as well as a number of authors [140].

3.7 Videogames, animation and virtual worlds

Finally, it is necessary to mention videogames as a growing field within Intermedial Studies. Videogames are played in an increasing range of technical media, including home consoles, mobile devices (smart phones, tablets) or computer games. The can be played individually or online with multiple participants. Markku Eskelinen [141] is one of the first theorists on the situation of videogames within the intermedial panorama together with Susana P. Tosca [142]. Many game scholars have examined relations between the visual arts, musicology, cinema, and videogames. In text-based adventure games, the progress follows traditional narrative lines, although the game might include several levels that depend on the inner programming. Elverdam and Aarseth [143] proposed a model of games according to which the game object is divided into a mechanical layer (the rule-based engine) and a semiotic layer, stressing that these layers exist independently of each other. The semiotic layer is the game interface and that which is perceptible to the player. The mechanical layer contains the rules of game manipulation, which include restrictions and affordances. In adventure games that follow a narrative pattern, the game space is accessed by an avatar or game-character, under the control of the player. Rune Klevjer also characterizes avatars as a sort of prosthetic tele-extension that may take on the player’s point of view (there are also games that use third-person panoramic point of view) [144]. Unlike singleplayer games where the player can traverse through a single sequence of events, in multiplayer environments, each player makes diverse choices and follows different paths with various outcomes. In these cases, there are different story lines within the same shared fictional world (see also [145]).

Animation has played a key role in television, cinema and videogames. Christopher Holliday and Alexander Sergeant [146] study the relationship between fantasy cinema and the medium of animation. The term ‘machinima’, a portmanteau of the words machine and cinema, refers to the use of computer graphic engines to create a cinematic production [146]. This practice of using graphic engines from video games arose from the animated software of the 1980s as an alternative to traditional frame-based animation (see [147]). Holliday and Sergeant’s collection provides a range of essays on subjects including Disney, Pixar, and Studio Ghibli, filmmakers such as Ralph Bakshi and James Cameron, and film and television franchises such as Dreamworks’ How To Train Your Dragon (2010) and HBO’s Game of Thrones (2011).

Other scholars have explored how games also remediate, simulate and integrate other qualified media. Adam Lowenstein Dreaming of Cinema: Spectatorship, Surrealism, and the Age of Digital Media is a wonderful example of how surrealist techniques and ideas can be used as a tool to understand new media [148]. The author focuses on ‘enlarged spectatorship’ (the artwork enlarged with the spectator’s associations), ‘interactive spectatorship’ (this part on Roger Caillois’ Man, Play and Games 1961, the exploration of David Cronenberg’s eXistenZ 1999, and contemporary gaming are particularly interesting). Finally, Lowenstein examines fan videos and a YouTube channel in order to explore ‘collaborative spectatorship’.

In their collection, Metamorphoses of (New) Media (2016), Julia Genz and Ulrike Küchler examine media uses within social discourse and esthetics covering digital storytelling and videogames [149]. Games employ multimodal sign systems that correspond to different sensorial qualities, so that their content can be explored through the lens of adaptation, transmedial storytelling, and convergence culture.

Michael Fuchs & Jeff Thoss’ volume Intermedia Games (2019) studies transmedia storyworlds by exploring their remediation in film, television, and literature [150]. As already mentioned, the term ‘storyworld’ was proposed by Marie-Laure Ryan to differentiate between the spatiotemporal world context and the mode of representation used to mediate it (discursive, iconic, etc.). Like Jan Baetens, Ryan has been exploring the phenomenon of ‘transmedial narratology’ ([151], p. 35) in an attempt to explain how only certain forms of temporal unfolding of a series of events present in a narrative could be transferred among media [151]. This research can provide an answer to the problem of considering videogames as events that can engage actions while films play on visual images and a sort of internal interpretation.

SusanaTosca and Lisbeth Klastrup have explained that transmedial worlds have three core features: the mythos or backstory of the world, that is, ‘the central knowledge one needs to have in order to interact with or interpret events in the world successfully’ ([152], p. 4); the topos, which describes the spatiotemporal setting of the world, and the ethos, that describes the codes of behavior and connects the characters to the logic of the world [152]. Although some e-lit, e-poetry and Net. Art tries to break narrative conventions, in general, transmedia is intimately connected to artworks that are built around a story as well as to brands and franchises that spread content across media by corporate conglomerates (e.g., Star Wars, The Matrix, Harry Potter, Disney franchises, etc.) as well as with the entanglement with fanfiction etc. Although transmedia storytelling predated digitalization, media convergence has greatly extended its possibilities.

It is important to note that Elleström argues that ‘transmedia storytelling’, that is, narratives in different media types working together to form a larger whole, requires that narratives can be largely transmediated. It is in this sense that he formulates his wider notion of ‘transmediality’ and interrogates its limitations, since media types open possibilities as well as pose constraints that can impact upon narration (for an extensive catalog of studies that explore different media ‘affordances’, see Elleström [49], pp. 7–10, [49, 105, 106, 107].

Worth mentioning is the question of online platforms. The MIT Platform Studies Book Series highlights this neglected aspect that includes research on physical platforms (i.e. arcades, consoles, card decks, and boards), physical interfaces (i.e. mouse and keyboard, controllers, and VR headset), their relations to computational codes and shells (i.e. operational systems and software frameworks), the mechanical layers (set of actions users can perform to change the state of the game) and, finally presentational layers, with objects whose ontological status is context-dependent and may change over time. These platforms are also subject to different levels of regulation such as public or private use, age access, level of audience active participation, and so on.

In the era of Virtual Reality (VR) and Artificial Intelligence (AI) Intermedial Studies face new challenges. The scopic drive to enter contemporary 3D spaces could be compared to the study of filmic close-ups performed by Laura Mulvey ([153], p. 14–26). The first recorded use of the term ‘virtual reality’ occurred in a collection of essays entitled Le Théâtre et son double (1938) authored by French avant-garde playwright Antonin Artaud who used it to describe the illusory nature of characters and objects in theater. The concept of ‘scopic absorption’ was coined by media theorist Vivian Sobchack [154] to describe how viewers become fully absorbed to the point of losing themselves in the visual experience with great emotional intensity. New media, like AI, continues to reinforce aspects already pointed out after the emergence of cinematography, for example, in Sobchack’s scopic or Mulvey’s ‘male gaze,’ which emphasized the ways in which cinema reinforced traditional gender roles. The benefits of AI might be many, but the logic of scopic absorption is already a danger. The fact of looking is justified and exhausted in itself, without giving a chance to understand the real story or the fake behind it [155]. This goes to show the intricate temporal border crossings in the study of intermediality, defending the need to look at the field in a diachronic perspective, exploring shifts in tech-material considerations as well as the conceptual and sociocultural configurations that qualify media.

4. Conclusions

As this chapter has shown, Intermedial relations are a ‘complex and highly dynamic set’ ([156], p. 10) of alliances and relations amid various agents – creators, readers/users, institutional drivers, qualifying aspects, and so on. They stage what Higgins, the father of the term ‘intermedia’, contemplated as a ‘conceptual fusion’ ([157], p. 30). In all cases, the conceptual and semiotic level is always conditioned by the material substratum. The tools and media we employ play a significant role in knowledge management, information transmission, and artistic creation. From analogue to digital constraints, these elements can function as enablers as well as barriers in the semiotic processes of communication and knowledge transfer ([158], p. 252).

As the chapter has shown, over the past 50 years, numerous significant studies on media have been conducted by scholars. The literature review presented has shown that intermediality is very much an interdisciplinary pursuit, a cross-breeding between the arts and other cultural practices ([159], p. vii). The main goal has been to illuminate the development of the field of intermediality in the light of adjacent areas of inquiry, such as semiotics, narratology, and visual studies, including, for instance, performance, cinema and television, and more recently, game studies, among others.

The chapter highlights the network of interconnections among media products, practices, and concepts within the broader media culture of the last decades, after the digital computer became a metamedium that concentrated all previous forms of human communication and representation – speech, writing, image-making, sound and radio, cinema, television, and so on – in one single cyberspace, enabling the integration of digital technology into our portable screen devices and, thus, facilitating human engagement with information at any time and from any place. The process of digitalization emphasizes that media not only represent but also provide resources for engaging with and influencing reality, aspects of intermedial agencies that still require further exploration [23].

Within Intermedial Studies, transmedia storytelling poses numerous challenges for narratology, including the need to develop skills in multimedia semiotic analysis. In the context of constantly evolving and expanding stories, a narratological approach must extend beyond studying narrative representations within a single medium, recognizing the rich complexity of natural and artificial languages. Exploring the histories and diverse manifestations of intermedia and transmedia is crucial in order to foster a deeper comprehension of their interconnections.

On the premise that ‘all media are mixed media’ ([160], p. 260) and that ‘everything is intermedial’ [161], the main takeaway is that the field of Intermedial Studies can still offer important insights into the nature of meaning-making, being pertinent even after digital media convergence.