Designing a Participatory and Interactive Opera

Abstract

Opera as an art form should lend itself to participatory design, given its wide variety of theatrical and musical components. Opera, however, imposes its own constraints and challenges. The problem as laid out in the early stages of the project was to design a short one-act interactive opera that could be managed within a modest budget. The source material was a science fiction manuscript currently undergoing publication by a small independent publishing house. Our research and production team included a composer, several singers/musicians, an engineer, a designer, and a choreographer/dancer as well as the writer. Furthermore, several of these had done extensive work with interactive and immersive environments. In addition to the opera itself, which incorporated both live and virtual elements, we recycled the 3d virtual designs as the substratum for an online co-creation environment that could be used to elicit public participation in the future development of our operas. We discuss both the diverse challenges involved in creating the opera and the co-creation environment, and highlight projected future work.

1. Introduction

At its beginnings, dramatic art as practiced, for example, by the ancient Greeks [1], was understood to be a participatory activity. Audiences were coached and guided to participate in the chorus [2], audience members were often invited to act out a part, and the cathartic effects of drama were designed to be used as a form of social engagement [3]. Over the centuries, especially in the period leading into the Enlightenment, audiences became more passive and a separate class of professional performers emerged who were tasked with ensuring that formal performance esthetics were achieved [4], especially in music [5], although there were also contrary tendencies [6]. Today, although this remains the de facto modus operandi of most public performances, there is a growing demand on the part of a broader public to be re-integrated into the dramatic production process. The lines between professional and audience productions are blurring again. Hence video and audio recordings are “remixed” by adventurous user communities, and “mashups” that combine elements from several sources are also created and disseminated [7]. Fan fiction has become a huge industry, with countless individuals writing their own versions of popular stories, whether these remain purely literature or have been transposed from film or television series, or even stories confined to the gaming universes [8]. Ways are being found to integrate audience productions within live drama presentations, for example by integrating live camera footage of audiences into stage productions [9].

It has been noted that opera as an art form should lend itself to participatory design [10] given its wide variety of theatrical and musical components, but efforts to do so are still relatively limited. Furthermore, if one were to succeed in making opera more participatory, then this success should spill over into other performance arts where the modes of representation are more constrained. Participatory design in a broader sense has been, of course, an established practice now in research circles for several decades, addressing problem-solving in relation to a wide variety of application domains [11, 12]. Furthermore, participatory design is increasingly being used within art production practices [13, 14, 15], including the design of television episodes. Opera, however, imposes its own constraints and challenges. In our context, we had neither the budget of a television studio nor that of a major production house—instead, we had a small seed budget from the Canadian Social Science and Humanities Research Council (SSHRC) for developing the opera (the Insight Development program). How, then, to not only design a participatory opera, but also to develop a collaboration protocol that would support the design process?

As a team, we had not all worked together before, which posed its own challenges. However, some of us (particularly Edwards and Bourbeau [16]) had extensive experience working with multidisciplinary teams. Several members of our team were also highly polyvalent, that is, masters of more than one discipline (see Table 1 which shows the different forms of expertise).

Team member	Basic expertise	Complementary expertise
Geoffrey Edwards	Senior researcher	Writer (librettist, virtual reality, augmented reality, smart garments, geomatics, motion capture, fashion/costume design)
Jocelyne Kiss	Researcher in music	Virtual reality, singer (soprano), non-verbal communications
Ernesto Morales	Architect, designer	Rehabilitation, co-design
Cora McLaren	Choreographer, dancer	Rehabilitation, nursing
Serge Lacass	Composer, singer	Network communications applications
Sophie Stévance	Singer, theorist	Critical theory
Juan Nino Falcon	Student, mechatronics	Virtual and augmented reality, animation, voice
Marie Louise Bourbeau	Singer, coach	Immersive installations
Jonathan Proulx Guimond	Student, visual design	Avatar design, 3D design
Antoine Guérette	Student, architecture	3D design
Alicia Lamontagne	Student, architecture	3D design
André Dorval	Sound engineer	Experience with film, stage and TV production, concerts and studio recordings

Table 1.

The production team.

The opera project formed a natural and organic extension to the creative work of the original author and librettist for the project, Edwards [17]. His writing activity sprang out of a lifelong interest towards improving the lives of other humans that has also underlain his career as a research scientist. The themes his writing addresses bear witness to this interest—his work raises issues of tolerance and its relationship to conformity and peer pressure, gender fluidity issues, bullying and violence, disability and mental health among other thematic areas. The engagement of a broader public within his creative work is one manifestation of this abiding interest, and one that he finds particularly compelling, in that it opens the possibility to carry the writing out “into the world” in ways that are not usually accessible to writers.

2. The challenge

The problem as laid out in the early stages of the project was to design a short (20–30 min) one-act interactive opera that could be managed within our modest budget. The source material was a science fiction manuscript that Edwards wrote in 2012, which was itself the first volume of a fifteen volume opus which is now undergoing publication [17]. We had on board for the project (see Table 1), a composer (Lacasse), several singers/musicians (Kiss, Stévance, Bourbeau, Falcon), an engineer (Falcon), a designer (Morales), and a choreographer and dancer (McLaren) as well as the writer (Edwards). Furthermore, we had several people who had done extensive work with interactive and immersive environments (Kiss, Edwards, Bourbeau and Falcon primarily).

Hence, there were several tasks to be addressed:

• Find an appropriate scenario.

• Determine how the story should be interactive (e.g., what mix of augmented and virtual reality should be incorporated? How should individual interactions be engendered, and how should group interactions be facilitated? etc.).

• Embellish the scenario with the other components (stage design, music, choreography, and lyrics).

• Bring the elements together into a performance for presentation to diverse publics.

Figure 1 shows an example of how we viewed this taking place. Some of the details of the flow are subject to discussion, but, overall, this was our planned process.

Figure 1.

Original concept of the design process.

What is important about this conception of the process, is that although there were areas where understanding was lacking, the plan followed a kind of general procedure for developing performance-based productions. The diagram also incorporates a number of assumptions: that a scenario needed to come first, and that following the scenario, the libretto and the music had to come next. In our case, the libretto was viewed as preceding the music because the two steps had to be carried out by different individuals (when carried out by the same individual, historical evidence suggests the process is much more iterative, with a constant back and forward shift of attention between music and libretto [18]). Furthermore, within our original discussions about interactivity, the presence of augmented or virtual reality avatars was implicitly assumed, although exactly how these would be integrated was unclear. The diagram incorporates these assumptions.

We also viewed the process as having an addendum—in parallel to, or following, the development of the opera, we would develop an online environment for co-creation, but in our original conception, these tasks were viewed as separate. In the next section, we shall see how the design and production process actually occurred and what the differences between plan and realization taught us about these processes.

3. The production process

3.1. The libretto

Anyone who has worked on a screenplay for a film or a TV episode will attest to this fact—there is nothing “linear” about the process of developing a viable scenario in a group production enterprise. The process took about eight major iterations to arrive at something workable [19], several of these after we had made substantial progress in other areas of the production. Indeed, the scenario and its attendant libretto were still being modified during the final stages of the production, although the changes at that point were limited. The initial problem was to take a 150 page novel and extract from it a scenario that could be presented in a dramatic way within a 20 min span of time. Also, there is very little documentation that exists to guide anyone attempting to develop an opera or write a libretto. The only scholarly text we could find was a small memoire written in 1914 [18] that drew on analysis of existing, mostly nineteenth century operas to make a few suggestions, although there is also information to be gleaned from discussions concerning whether the text or the music should have primacy [20] as well as the process of bringing meraviglia into opera [15].

The original story was focused around the character of Oreph Sodenheim, a young teenager who is the main protagonist of the novel Pinnacle: The First Book of Eng. Oreph is a central figure for the larger cycle of 15 novels that collectively forms The Ido Chronicles [17]. However, in Pinnacle, he passes through a series of events that lead to the destruction of his family. The subject matter, overall, is therefore depressing. In addition, the main theme of the story is concerned with being bullied, both by conservative factions within the society of the planet called Pinnacle, and also by mysterious groups that belong to the larger civilization called the Humanitat.

As Edwards was working on the compression of the original story to its essential elements—about eight scenes were identified as key, but this remained too long for the planned production—an opportunity came up to develop a short story based on what was originally a secondary character in Pinnacle, that of Oreph’s girlfriend, Relliana Sapuro. The short story that resulted from this effort provided a much better framing for the opera, and iterations #2 and #3 involved recasting the original story to include Relliana (iteration #2), and eventually to focus exclusively on Relliana’s story (iteration #3). This led to a much more compact scenario, now limited to four scenes, and a story which was overall much more uplifting (although also involving conflict) and which better reflected the title Pinnacle.

In the revised version, Relliana when first encountered is complaining about a set of unusual aches and pains she is experiencing in her body, while also engaging with a friend, Aki Oneya, who is a “gender morph,” that is, a person who is bi-gendered. Aki is being bullied by a group of young conformists. Later, Relliana encounters one of the bullies, Mandra Lakso, as it is revealed that her body is undergoing some sort of profound transformation, although we still do not know what this is leading to. During a performance event at the end of the story, Relliana’s transformation completes itself and a dragonfly-like winged creature emerges in Relliana’s place—the first stage of a new “splinter species” of the human race.

By this time, a stage design had been completed (see Section 2.2 below), and the next iteration led to a reorganization of the scenes so that all the different exchanges could take place within the same physical setting (iteration #4). In addition, the discussions concerning interactions (see Section 2.3 below) had singled out the idea of a finale that would be affected by audience participation, and so the climax of the scenario/libretto was re-written to accommodate a multiplicity of endings (iteration #5).

It should be noted that although in principle the scenario and the libretto are distinct, in practice, given the time constraints of the production schedule, a draft version of the libretto was created as early as iteration #2 of the scenario. The scenario may be viewed as the general description of unfolding events and interactions, whereas the libretto meant the full verse realization of the text to be sung or declaimed.

Interaction elements were also introduced earlier in the scenario, and it was realized that an introductory segment would be useful. As a result, an introduction was written and incorporated into the libretto (iteration #6). During the work on the music, it was realized that the libretto lacked an intimate contact with Relliana and her difficulties—as a result, a new aria was introduced into the libretto (iteration #7). The lead-up to the interactive component in the final scene also needed some adjustments (iteration #8).

The libretto itself was written in metered verse, but drawing on principles such as harnessing the natural rhythms of the English language rather than slaving the verses to its metric structure. In addition, although rhyming structures were used, they were not consistently maintained at all times, again, to break up the regularity of the resulting text. Because the scenario was science fiction, a certain amount of exposition was required—this made for relatively dense lyrical structures, that is, arias that contained significant amounts of information.

The meter and the length of lines were, however, varied across different arias and recitatives as a reflection of the emotional intensities and information being conveyed. Voice range was determined by the available singers—that is, each aria was developed as a function of the vocal range of the singers. This probably follows traditional practice in operatic production—historically, many arias were designed with particular singers in mind, although well-established composers could also pick and choose at will, permitting greater flexibility. Additional small changes were made to the lyrics during the musical composition stage (see Section 3.4 below).

Figures 2–4 show some examples of the libretto lyrics that highlight these considerations.

Figure 2.

Lyrics sung by Aki, the “gender morph” (bi-gendered person). “Moelen” is the term given to the fold of skin that can curl to form either a pseudo-penis or pseudo-vagina as the situation calls for. The text highlights the use of metric verse and near but not perfect rhymes to break up the regularity of the poetry (e.g., “frayed” is a near rhyme to “sage”).

Figure 3.

Example of the lyrics sung by the conformist bullies—the first two verses of the song. Here the meter changes between the snappy chorus and the more substantive main verse. The rhyme scheme for the main verse is abcadeddd, so, again, some regularity and some variation, whereas the chorus is much more structured.

Figure 4.

These lyrics are part of the additional aria added for Relliana to sing, to explain her own feelings about the transformation she was going through.

3.2. Stage and 3D design

As outlined earlier, the original concept was to develop a physical staging of the opera that would incorporate virtual reality or augmented reality elements in support of interactions with users. The design was developed and organized by Morales and his students based on a series of initial discussions about the scenario/libretto as well as the larger opus that framed the opera. Indeed, Morales and Edwards led several co-design sessions held at the CIRRIS¹. The design was initially organized around the opening scene of the opera, which consisted of a ledge on the side of the airborne city called the Orr Enclave. The Orr Enclave was conceived as being roughly cubical in shape (Figure 5), but with numerous extensions and indentations down its sides, one of which formed the stage for the opening scene of the scenario. It became evident that the cost constraints of the production meant that we needed the one stage to support all scenes. For this to work, in addition to the ledge, we needed the main area to have sufficient depth to accommodate bully dancers, we needed a kind of gate-like area that might represent a school playground or patio, and we needed a round “portal” that would provide access to images of diverse types. One of the most important challenges was to decide which parts of the staging should be real and which virtual. We initially assumed that the production would involve both elements within an augmented reality type staging. To do this, the virtual elements needed to be fit perfectly and seamlessly with the real stage objects. We developed and tested virtual components that would operate in this way. In addition, we conceived of the use of VR goggles that could be provided to the public, and which would allow users to select among a range of display options. During the co-design sessions, Morales and Edwards generated many drawings to ensure staging ideas were clear for everyone involved. In addition, we were conscious that a complex stage would be costly to build, so we were looking for something simple.

Figure 5.

The Orr Enclave, the floating city that serves as the main setting for the events that take place in both Pinnacle: The First Book of Eng (the original novel) and Pinnacle: A One Act Participatory Opera (the opera).

The early designs therefore incorporated these different factors (see Figure 6).

Figure 6.

The basic stage design. The set consisted of a ledge on the side of the city, an oval area onto which images could be projected, a fabric structure (the curved form), and steps from the main stage to the protrubing lower ledge where the nemo plants were to be situated.

Since it is supposed that the city itself was composed largely of fabric materials along with some structural, weight bearing elements (that is, because the city floats in the atmosphere, it must be made out of light materials), the final stage design incorporated fabric, cord and aluminum struts into an elegant design that could be relatively inexpensive to construct. Different variants of the ledge structure and upper area were proposed—only one of these is shown in Figure 6. Further along we shall show the staging area used in the final production.

The production almost stalled when it was realized that our budget was probably not adequate to cover the stage construction costs. It was at this point in the development that the team decided to go with a virtual staging which was much easier to manage within the context of the grant monies, even though the final costs may have been roughly commensurate to those involved in a physical staging. The difference was that the costs of the virtual production went primarily towards paying students rather than into the construction materials themselves, which would have required storage, which needed to be dismantled and then rebuilt and so robust against such manipulations, and so forth. No doubt a more experienced production team would have known how to circumvent these issues more effectively, although the budget was fixed and rather modest.

The advantage of shifting the production towards a virtual staging meant that the two initiatives, which were initially viewed as separate, could be merged together into common development protocols—that is, the construction of the opera and the construction of an online co-creation environment. The latter effort had been focused towards the creation of a 3D version of the “cube city.” Edwards had initially developed some rudimentary 3D models of the city (see Figure 7) that showed both its internal structure and the locations of key scenes. Drawing on the architectural expertise of students such as Alicia Lamontage, under guidance from Edwards, a rough design for the top of the “cube city” was developed. Within this structure, several key features that were present in the original novel (Pinnacle: The First Book of Eng) were given a detailed structure—the Agora space in which major events took place, the sunken Concourse area where the final performance was originally planned, the Portal building, where information about other planets and habits could be accessed, and the threader stations where the spindle-like transport vehicles, which moved within dedicated tubes, could be boarded. In addition, for the online co-creation environment, an additional location was eventually introduced, a monument space along the front edge of the city top (Figure 8).

Figure 7.

Early design for the top surface of the cube-shaped city of Orr Enclave; the cleft is the location of the concourse, the blue dome at the right is the Agora and the arched shape at the left is the monument.

Figure 8.

(Above) The monument site and the threader transport system (tubes), in the early design, and (below) as later implemented within Unity in the co-creation environment, along with the user avatar (labeled) and the dancing avatars.

The disadvantage with the shift from a physical to a virtual production was that we needed to work harder to find ways to bring “embodiment issues” back into the production of what would be essentially a virtual opera (see next section). We felt from the beginning that the opera needed to be “embodied”—part of both Edwards’ and Kiss’ larger research programs concern how to enhance embodiment effects within environments that may be partially or wholly virtual. Indeed, our use of live singing/declaiming was also an attempt to enhance the feelings of presence and embodiment ([21], see also Section 2.4 below).

Furthermore, to return to the opera staging, the 3D city design allowed us to stage the opera on the ledge on the side of the city. Rather than redesign this area, the design of the physical stage was simply incorporated into the 3D city design, resulting in the final “stage” for the virtual opera (Figure 9). This elegant solution meant that if, at some future occasion, we should want to physically stage the opera, we could do so with no substantive changes to the overall set design. Furthermore, the approach used aligns with cognitive design principles [22, 23].

Figure 9.

Virtual stage for the opera, incorporated into the 3D city design. Shown is the scene with the bully dancers, with Aki (in blue) and Relliana in gray and brown next to hir.

The final virtual rendition of the stage was carried out by a visual arts student, Jonathan Proulx Guimond. As we shall discuss further on, Jonathan also did extensive development, in tandem with our engineering student, Juan Nino Falcon, of the more extensive virtual environment used for the co-creation endeavor (see Section 2.10 below).

3.3. Interactive components

As might be expected, designing and incorporating interactive elements into the opera posed enormous challenges to the production team. The problem was to allow the audience to influence the unfolding story in manageable ways that were thematically compelling in the frame of the story itself. Hence, for example, many potential interactions discussed were discarded either because they would have been too costly to do effectively or alternatively, because they allowed game-like interactions that were, however, poorly integrated with the story line.

Any attempt to use the audience’s interactions to modify the story potentially runs into the problem of combinatorics. If a choice of two outcomes is introduced, then two stories need to be constructed. If a second choice of two outcomes is introduced, at another point of the story, then four versions of the story need to be determined. For a third decision point, we are now at eight versions of the story, and so on. This quickly becomes unmanageable. Therefore a “management strategy” needs to be developed to allow variation without leading to such combinatory effects. For example, audiences could be given the ability to modify aspects of the staging that concern appearance (i.e., esthetics) but which do not substantially influence the plot itself.

Early on in the design process, even during the stage when we were considering a physical staging with additional virtual elements via augmented reality techniques, we developed an approach where each audience member could have their own “access device” within which they could be given more flexibility for customization than in the common staging, whether the latter be physical or virtual. Possibilities discussed for these devices included introducing virtual flora and fauna that could be influenced by the audience member (Figure 10), superimposing a photograph of the audience member’s face on one of the actor’s (or avatar’s) faces so that the audience member could experience an enhanced identification with a character of their choice (early experimentation showed this to be possible—see Figure 11), and introducing new (virtual) avatars that moved between the actors and which could be influenced by audience members. Furthermore, Kiss believed the possibility existed to incorporate voices from the audience members as well, although this possibility was never fully fleshed out. … In addition, even for a physical staging of the opera, we realized that it would be possible to “add in” the side of the city around the physical stage using augmented reality techniques—again, early experimentation showed this could be achieved sufficiently well to create the impression that the action was taking place on the side of the floating city.

Figure 10.

The nemo plants (blue) emit particles that enhance airborne buoyancy and in addition that respond to audience input (via either voice or breath).

Figure 11.

Superimposing a photographed face onto an avatar; above—the software used to capture the image of the face; below—the rendered virtual face using the soda can as the directional reference object (normally, the reference object would be invisible or more discrete!).

Of course, once we abandoned the idea of doing a physical staging, the integration of virtual elements into the staging was greatly facilitated—we no longer had to worry about anchoring virtual elements into the physical stage (even though we had demonstrated to our own satisfaction that this was feasible).

There were concerns even from the early discussions about the intrusive nature of “viewing technologies,” whether these be expensive devices like VR or AR glasses, or cheaper devices such as tablets or smartphones. Among alternatives we discussed (and tested) were the use of simple smart garments—and in the end we did develop and test an interactive belt device.

Another interest was to generate interactive possibilities that allowed audience members to work together to affect the opera, and not merely opportunities for individual customization. Finding an interactive modality that would not corrupt the opera’s story line remained a challenge throughout these discussions. Any attempt to provide users with some form of access to or control over avatars on the stage seemed to mean disrupting the way the story evolved. Of course, one option could have been to abandon any effort to control the storyline, and this might have been an interesting endeavor, but having taken the time to select a scenario and construct a libretto, this seemed self-defeating. We therefore sought mechanisms for interaction that preserved the story-line or allowed only for limited variations.

Our solution to the problem of allowing audience members to introduce elements that could affect the story in limited ways that nonetheless felt “organic” was to focus on the flora and fauna that form an essential background to the stage setting (in fact, to the world being presented via the staging), and could be allowed to influence the unfolding of the story even if such influences had not been present in the original version (the source novel). The world portrayed in the larger opus (The Ido Chronicles) is situated in the far future, at a time when ecosystems are essentially constructed by humans. These constructed ecosystems are designed to provide utilitarian functionality to support a range of human activities as well as to function with each other in ecological ensembles. Among the plants and animals described in the Chronicles are plants called “nemos” and animal hybrids such as “doskies” (combining features of flying squirrels and dogs) or adapted animal species such as “jonahs” (modified sperm whales). Nemos are plants derived from sea anemones that bush outwards via tendril-like leaves that collect water droplets from the air, while “spiners” are adapted from spiders to spin structural elements for buildings. Doskies are a kind of flying or gliding domestic pet. Jonahs have been modified to function in interstellar vacuum and can transport people or freight between planets or habitats inside a solar system.

For the purposes of the opera, we decided that nemos could emit some kind of particle that would provide buoyancy to someone attempting to float or glide in the air. In addition, we assumed they could sense and mimic vocal tones, and that they would emit their buoyant particles following vocal stimulation. By populating the ledge on which the stage is set with nemos, we could introduce virtual elements that could be controlled by humans. We explored several interaction modalities—as suggested in the previous sentences, vocal activation was one of these, and the one we eventually retained. But in an earlier effort, we also developed a belt that activated the nemos via a person’s breathing. Both of these interaction modalities were attempts to re-introduce embodied interactions into the opera.

We modified the scenario to accommodate these interactions. In the original scenario, a group of bullies were intimidating the two principal characters, Oreph and Relliana, in the opening scene, and these would eventually be pushed off the city. However, they both wore “windplanes,” a kind of glider that allowed them to escape falling to the distant ground. In the early scenario modifications, we determined that if the nemos were activated enough, then the two characters would not fall at all, they would fly up. Later, however, as we simplified the story line to fit it into a sufficiently short time from (less than 30 min), we used the same idea in the final scene of the opera. Now, however, it was the transformed Relliana, with her nascent wings, that was to be supported by the audience interaction. If they successfully supported her, she flew away upwards. If they failed, she eventually fell downwards.

In addition, we posited the existence of a plant species we called “phramae,” which could organize into physical forms that mimic, in a ghost-like manner, the movement of real creatures, human or otherwise. These phramae were used in the bully sequence to make a number of additional dancers than the small number of actual bullies, and they were also used in the on-line co-creation environment to provide added control to users to “create” their own dancing avatars (see Section 2.10 below).

3.4. Music

Developing the music for this opera posed several challenges. For reasons that are too complex to go into here, Lacasse was not as available to do as much composing as had been originally planned. The libretto, which as outlined earlier went through many different versions, incorporated relatively early on a “bully song,” to be sung by a main “bully” and his/her troops as they underwent a “bully dance.” Lacasse developed quickly a successful musical rendition of the bully song, drawing on both pop culture musical ideas as well as more traditional opera constructions. The result was compelling in interesting ways. The bullies in the scenario are complex individuals in the sense that they are not simply “bad guys.” They promote a kind of conformism, but they are in some sense supported by the ambient culture, and their ideas are not unsympathetic. Lacasse captured these ambiguities in his musical rendition, giving them a positive energy offset by the negative sentiments to some extent expressed in the lyrics.

Following this initial effort, however, Lacasse was no longer available to develop the remainder of the songs or the musical introduction. Kiss, who has no claims to be a composer, drew on a process of combining public domain royalty-free samplings, some recordings she had from an earlier collaboration with violincellist Haruko Kido, and a knack for bringing disparate elements from the creative commons together into a musical rendition which while not as “characteristically individualistic” as Lacasse’s work had been, resulted in a musical soundtrack that supported the opera in appropriate and interesting ways. She and Edwards worked in tandem on this, Edwards for overall direction and Kiss for the creative musicality of the ensemble. In addition, we consulted with Marie Louise Bourbeau, another collaborator with extensive experience in lyric singing and musical production, who helped guide us through some of the difficult choices we had to make at that point in the production.

3.5. Choreography

Once Serge Lacasse had produced the music for the bully song, we solicited our choreographic collaborator, Coralee McLaren, to develop a musically-responsive dance work. At that time, we were still orienting towards a physical staging, so we were looking for movement that could be performed by a small group of dancers on stage. However, we also imagined using virtual avatars in addition to actual dancers. As such, we organized a movement session with Motion Capture equipment available to us through Edwards’ engagement with the Centre interdisciplinaire de recherche en réadapation et intégration sociale (CIRRIS), a research center in rehabilitation. Dr. McLaren flew from Toronto to Quebec City to carry out this work. In preparation, she created a number of movement phrases that complimented Lacasse’s rhythms and melodies. Drawing on these phrases, McLaren responded physically to spontaneous feedback called out from the team, and modified/refined her movement in response to fast changing, verbal cues. This resulted in an unparalleled collaborative movement experience and invaluable methodological insight, with each “audience” member providing input that reflected his or her own sensorial expertise or movement preferences (e.g., as singer, stage designer, writer, composer, etc.). The final outcome was a dance that enhanced Lacasse’s dynamically rich score, and a subsequent Motion Capture sequence of movement was digitally applied to the avatars during the virtual staging (Figure 12). Furthermore, as we shall discuss below, the dance sequences were included within the co-creation environment in interesting ways.

Figure 12.

Matching the motion capture recorded movements to the avatar movements.

3.6. Animation

Once we had opted for a purely virtual staging, the major issues became the process of developing and animating the avatars that would represent the characters in the story. There were a number of issues that affected this process. As mentioned earlier we were experimenting with the idea of applying a user’s face to the avatars, and so, at least in the preliminary version of the opera, we used our own faces. There are three main characters in the opera, and we applied an image of our faces to each of these characters—Kiss’s face was applied to Relliana, Falcon’s face to Mandra, the leader of the bullies, and Edwards’ face to Aki, Relliana’s friend. Aki is a bi-gendered character—neither female nor male. Therefore, we chose a neutral morphology for the avatar, used Edwards’ masculine face, and chose dress-like clothing which resulted in an avatar which is successfully ambiguous in terms of gender. In follow-up work on the opera we propose to modify the user interface so that the process of assigning the face to the avatar could be carried out by any member of the audience for their private interface. The calibration process that allows this to occur has already been developed, tested and programmed (see Figure 11).

The Unity game engine underwent several improvements over the course of the project which allowed us to improve the animations. Initially, the work was quite onerous. In general, gestures, postures and movements were chosen from banks of animation data in appropriate formats that were publically available and these were looped or integrated with other animations to generate movements that would appear (we hoped) as natural as possible. Originally, all the timing of the animations had to be implemented manually. However, after our first public performance, Unity installed a new module called Timeline that enabled a more organic and systemic control of the timing between different events, greatly facilitating the subsequent animation efforts. A second major challenge was the development of appropriate lip-synching. Finding a reliable and robust method to develop lip-synching of the avatars with the singing proved to be a substantial challenge, and was never solved to our complete satisfaction.

Another problem with the use of avatars and a virtual reality environment, of course, is that the sense of embodiment, of embodied actors is largely lost. Since the themes incorporated into the opera were at least partially resonant with issues of embodiment, this was a significant deception for the production team, and we worked hard to re-introduce elements of embodiment to compensate for this loss.

The fourth element that needed to be mastered to create a compelling operatic experience, beyond the appearance of the avatars, their programmed movements, and the lip-synching, was the development of camera zooming and panning sequences that could also be used to enhance the drama. Because the opera was presented in a “streaming” Unity environment, in principle the means to control the camera could have been left in the hands of the user/observer. However, since camera movements can significantly enhance or detract from dramatic tension, we retained control of these elements in our first version of the opera.

3.7. Bringing the components together

Additional elements that needed to be managed included the opening sequence, the closing sequence, and the introductory remarks and organization. Indeed, because the opera presented the audience with a number of new elements, explanations needed to be provided along with some basic “training.” We organized the material into several formats for different presentation protocols. In most of our presentations, we performed the singing live in addition to the prerecorded music.

Hence, during our first presentation, we used belts that tracked the singer’s breathing and used the breath as a way to “operate” the interactive plants so that they opened wide under deeper and more expansive breathing and spread more of the buoyant particles into the air. The belts were designed using “flex bands,” that is, sensors that convert elastic stretch into a capacitance measurement that can be read off by a computer chip. However, although we achieved one working prototype, the second failed to operate correctly. Given the difficulties ensuring robust measurements from the belts, we developed a second interaction method which used a centrally placed microphone to capture the audience singing and used the intensity of the voice to open the nemos and create more buoyant particles. This solution was much more robust and repeatable, and so we eventually adopted this solution within our final production.

3.8. Public performances

The initial staging was done using the Unity Game Engine in interactive mode projected onto a shared screen, following a predetermined script that determined avatar movements, virtual camera movements, music soundtracks, and that tracked voice production on the part of the audience. In addition, the two main live performers, Kiss and Edwards, sang (in Edwards’ case, declaimed) the lyrics along with the pre-recorded lyrical tracks. The second and third times we presented the opera, we used a video recording of the scripted Unity staging along with the live performers and the voice-interaction module. This allowed us to incorporate subtitles in either English or French translation to assist audiences in understanding the sung or spoken text, as well as facilitating the complexity of the staging. In the final public presentation, we used a different real-time streaming version designed using the 360 view capabilities made available in the more recent version of Unity, with a view to giving the user greater flexibility using VR glasses to change view directions. For this fourth presentation, we had refined the avatar and camera movements, and had done more work on the lip-synching. This fourth staging could, in principal, accommodate changes to scripting possibilities and potentially allows for multiple endings. In the second, third and fourth stagings, we trained the audience to sing a small melodic line that could be used to guide the protagonist towards her more hopeful outcome.

3.9. Post-performance discussions

Each time we presented the opera to a public, we engaged in a post-performance discussion to both assess reactions on the part of audience members concerning content and staging, but also to discuss technical aspects of the production that might interest audience members. One of the goals of the development of this opera was to engage audiences around content issues, that is, the need to be more tolerant towards others who are different from ourselves, especially in the light of peer pressures towards greater conformity, but also to recognize that even tolerant communities may include individuals or groups who are less tolerant, and that communities that may at first appear to be intolerant may include elements who are more tolerant if one only paid more attention.

Overall, audiences reacted favorably to the opera as staged, and during the discussion it became clear that the conformity-tolerance issues were accepted and interesting discussions ensued. These were mostly scientific audiences, although they included practitioners from a broad variety of public health contexts, including disability studies, mental health, social justice, and so on.

3.10. The co-creation environment

Originally conceived to be a tool to enable public engagement in the process of producing the opera (and this is still our long term goal), it was necessary to simplify to some extent the design of the environment to fit for the budget and time constraints of the project. Essentially, we had funding for 3 years and the process of developing the opera itself took most of the first 2 years, leaving a little over a year to develop the co-creation environment (talesfromthehumanitat.com) with, by that point, greatly reduced funding.

We used the 3d design developed for the opera as the substratum for the co-creation environment—although we also substantially embellished the environment to provide more virtual experiences in much greater detail, drawing on both information provided by the original novel and extended notes by the author (Edwards). While the setting for the opera itself was a platform on the side of the floating city, we used the top surface of the city to define four areas for the co-creation environment. Each of these areas showcases a different set of co-creation tools. Table 2 presents the main features of these four sites.

Site name	Plant/animal	Site building	Co-creation skills
Dance zone	Phramae	Monument	Dance
Concourse	Nemos	Concourse	Singing
Agora	Jonahs	Agora	Video
Portals	Spiners	Portal	Text

Table 2.

Co-creation stations.

In addition to the sites themselves, we implemented a transport system. This was part of the fictional world on which the opera was based. Transport vehicles are called “threaders”—they are elongated spindle-like structures that move in closed tubes. The user of the co-creation environment may move by foot between the four sites, or may use a transport.

Furthermore, we framed the experience for the online co-creation environment within a game. Hence the user, upon entry, is presented with the task of collecting five medallions, representing the five different factions (EngFax, DeoFax, UmaFax, EcoFax and IdoFax), each of which is located at one of the co-creation stations. Note that the images used for the five medallions were designs developed by Morales at the request of Edwards in support of the novel cycle. The first medallion is provided to begin with, and the user must visit each of the co-creation stations to find the other medallions. Furthermore, they must activate the co-creation mechanisms to gain access to the medallion. The location of the medallion is indicated once the station’s particular co-creation modality is engaged by a rising stream of red particles. Once all five medallions are collected, the user must find the final stream of particles to bring the five medallions together, and this liberates a final surprise interaction, which consists of access to an oracular service.

Hence, in a typical gaming encounter, a user would visit the dance zone and create a choreography with as many dancers as they chose (Figure 13a), would visit the Concourse and capture sung or whistled melodies that would be reproduced by the nemos when avatars passed nearby (Figure 13b), would visit the Agora and interact with the jonahs through their breathing, (Figure 13c) and would visit the Portals and either view/read or post a document of their choosing (Figure 13d), and would do these activities in any order. Moving between the sites could be done either on foot or via the transport system. Having visited and activated each of the four sites, the user could then access the oracular system, which is based on another of the 15 volumes that make up the Ido Chronicles (Figure 14). The interactive music components incorporated design principles developed by Kiss and her students [24] for the VR staging of music.

Figure 13.

The four stations of the online co-creation environment. (a) The dance zone; (b) the concourse; (c) the Agora; and (d) the portals.

Figure 14.

Accessing the oracular function of the online co-creation environment.

In addition to ensuring the esthetics of the 3d design of the co-creation environment, our visual designer, Jonathan Proulx Guimond worked on optimizing the rendition to make the complex 3d structure digitally more compact so as to ensure faster loading times and better dynamics. For example, objects were sorted in terms of visibility for each scene, and objects not visible in any of the principal scenes were suppressed. Furthermore, details for objects that were far from camera positions were also suppressed. Objects were also grouped together where possible to diminish rendering times for complex structures. The colors were processed to incorporate atmospheric absorption and the effects of distance were also included in the quality of the rendering.

4. Discussion

The participatory opera we designed and implemented has both strengths and weaknesses. On the positive side of the ledger, we developed a viable staging from a complex story and setting that is understood and appreciated by audiences (given the far future setting and the science fiction components, this is itself a net achievement). The story is perceived as being engaging and intense, as well as having undeniable poetic and esthetic qualities. Furthermore, as we desired, the opera raises interesting and nuanced questions and elicits discussion about tolerance and conformity.

On the other hand, none of the current (virtual) staging efforts was fully successful. Indeed, despite several attempts to re-introduce embodied elements into the opera via interactive components, for example, audience identification with the avatars remains weaker than we would have preferred. For example, the bully dance, which was brilliantly choreographed, remains of muted impact because the dance movements are applied to the small figures of the animated dancers. To have more impact, the animations of individual dancers should vary slightly from one dancer to the next, and means should be provided to view “up close” the dance sequences. The integration of live lyrics with the pre-recorded or scripted scenarios helped give the opera a more effective impact [23], but the integration of the live, virtual and pre-recorded components remains a challenge.

The incorporation of subtitles into the staging was easier to do using the pre-recorded scenarios than in the scripted scenarios. Although the scripted staging offers in principle greater flexibility, in practice most members of the audience were not even aware of the difference between the two options, and the pre-recorded option is much easier to manage since it consists only of running video footage (rather than triggering Unity scripts that have to work together).

The interaction via the plant ecosystem appears to be a successful strategy for engaging audiences within the unfolding action. It allows the story to be influenced in “organic” ways without these being overly intrusive. However, the actual interaction as carried out remains of limited scope (affecting the final success of the protagonist’s flight). We would like to provide more varied opportunities for interaction—again, this was part of our original concept but our resource limitations made full implementation of more complex interactions difficult to achieve, even within the virtual staging.

We also successfully implemented a multiplayer online co-creation environment, although we have as yet not fully harnessed the environment to support the ongoing work on the opera and its potential sequels. The online environment allows users to input and play with melodies, movement and text scenarios with a view to influencing future development of related operas, as well as providing the means to explore other aspects of the “world building” that contribute to the success of the Ido Chronicles universe.

Validation remains an issue that has not been adequately addressed. The effort to develop the opera was itself so demanding that we had little time or resources left over to work out a viable evaluation protocol. Coming up with assessment modalities is part of the ongoing work we are looking into. Indeed, assessment needs to address several contexts: the success of the opera as an artistic achievement, its success in raising pertinent issues for discussion, and the success of particular technical features of the opera design.

In the introduction, the value of the opera project to the writer (Edwards) was described. Although the project is by no means finished, and the full engagement with the public has yet to be achieved, the work to date has led to the opening of significant areas for further development. In particular, Edwards has begun work on a second context, that of the passage of trauma across generations, within the context of the impacts of the Second World War. Drawing on the existing work within the opera project and its online co-creation environment, we are now beginning the development of an additional co-creation environment dealing with this other context, hence creating extended value for engendering transformation within broader publics.

Furthermore, the story of Pinnacle: The First Book of Eng was substantially modified to support the development of the opera script and its corresponding libretto. Given that the Ido Chronicles itself is still undergoing publication [17], this presents an opportunity to “update” the story of Pinnacle to bring it into alignment with the opera. Note that one of Edwards’ goals in writing the Ido Chronicles was to allow for a certain level of inconsistency to exist between the different versions of the same story, that is, emphasizing that multiple truths are needed to describe human experiences, not all of them consistent with each other. Nonetheless, the opera represents a dramatic change and hence some effort to reduce the level of inconsistency is being attempted. Pinnacle: The First Book of Eng required revision in any case, for other reasons, but the rewriting will include paying attention to the need to bring the text within the broad framework provided by the operatic version of the text. Hence the work on the opera has enriched the writing of the source story in substantive and interesting ways.

5. Conclusion

Our team remains convinced that participatory opera is viable despite skepticism on the part of some practitioners [25]. Our project, in many ways, only scratched the surface of what could be achieved. Participation could potentially involve many different modalities, up to and including direct insertion of audience members, e.g., as avatars, into the unfolding scenario of the opera. However, such direct insertion poses challenges for managing the story line that results. We found a more organic means of having audiences influence the unfolding action by allowing them to control flora and fauna, that is, peripheral elements within the current set design that nonetheless may affect the story line.

We also noted that it is possible to provide a kind of “local” interaction that individual observers may see—through, for example, a virtual or augmented reality device without affecting the shared perspective—that is, they may control the virtual appearance (e.g., esthetics) of many of the scene elements without necessarily affecting the story line itself. These esthetic considerations/interactions will also affect the final perception by the participant of the opera. Indeed our work highlights these two distinct modalities for participation, one involving the private customization of the esthetics of the presentation via personal devices, and the other collecting reactions (i.e., gestures, voice) so as to ensure their cumulative effect on the shared action as it unfolds.

Although we remain convinced that augmented reality productions are a viable performance modality, our experience of augmented virtuality constitutes a useful and significantly less costly performance alternative. We use the term “augmented virtuality” to highlight the fact that although our opera was virtual, it included live performance elements that enhanced what would otherwise be a pure VR experience [26]. For opera designed to be presented to small audiences in, for example, educational settings, virtual opera is a feasible product, where a fully developed stage production or even a reduced augmented reality staging would be prohibitively expensive.

Now that the opera and the online co-creation environment are both complete, we will be using these to reach out to a range of audiences in order to address issues of tolerance, change, conformity, disability and other relevant issues raised by these productions. We are also planning to extend the capabilities of the online co-creation environment and to encourage its use as a means to engage audiences in the production process, that is, the development of new operas, or in support of other collective ventures. Finally, it is worth noting that as a collaborative effort this was an extraordinary adventure at the intersection between artistic creation, technological innovation and the science of audience engagement.

Acknowledgments

We gratefully acknowledge the financial support of the Canadian Social Sciences and Humanities Research Council (SSHRC) via its Insight Development Program, and also we thank Sophie Stévance, Erin Manning, Adel Elmaghraby, Alicia Lamontagne, Antoine Guérette, Afnen Arfaoui and André Dorval for their contributions to this work.