Plenum a la Mode - Augmented Reality Fashions

Abstract

Inspired by ideas portrayed in science fiction, the authors sought to develop a set of augmented reality fashions that showcased scenes from a science fiction novel recently published by the principal author. The development team included artists and designers, a programmer, and the writer. Significant technical challenges needed to be overcome for success, including fabric construction and manipulation, image enhancement, robust image recognition and tracking capabilities, and the management of lighting and suitable backgrounds. Viewing geometries were also a non-trivial problem. The final solution permitted acceptable but not perfect real-time tracking of the fashion models and the visualization of both static and dynamic 3D elements overlaid onto the physical garments.

1. Introduction

As a writer of science fiction, one of us (Edwards) has written about the use of dynamically changing skin-tight garments in a future civilization. In the scenario worked out in this fictional world, the first volume of which is called Plenum [1], these membrane-like garments protect their wearers from the hardships of the vacuum of space, but they also incorporate nanotechnologies that allow surface patterns to change and swirl across the surface of the skin, or even to extrude into the third dimension.

Inspired by these ideas, and also by earlier work as a fashion designer on the part of Edwards (see collections at www.geoffreyjenedwards.com/fashion-design), a small multidisciplinary group was pulled together from different backgrounds and abilities to prototype garments that do this. Rather than using nanotechnologies, which aren’t advanced enough to permit this kind of application yet, we are working with augmented reality (AR) technologies instead. Michaud is a student with training in mathematics who has retrained in the textile arts, Proulx Guimond is a visual artist with training in 3D design and animation, and Caron-Roberge is a programmer.

Augmented reality is beginning to be used by certain avant-garde fashion designers and artistic visionaries [2]. Like many emerging technologies, the technical challenges are considerable. The field of augmented reality capabilities has progressed rapidly in the past decade, but there are still major issues that are not easily resolved to support real-time AR visualization in a robust artistic performance setting.

In this paper, we shall describe the application context, the particular choices made, and the technical challenges we overcame in order to present a real-time fashion show in front of a live audience, showcasing our designs.

2. Augmented reality technology

Augmented reality is achieved by providing elements in a scene which can be recognized by computer vision software and tracked over time [3]. Once recognition has been obtained, the coordinates associated with the recognized image, including its orientation in space, then become the anchor point for overlaying 3D virtual reality elements in relation to this anchor point. Often AR involves careful design to ensure that the virtual elements merge seamlessly into the real elements from the observer’s perspective, so that it becomes difficult to distinguish what is real from what is virtual. That is the magic of AR, that it provides an enhanced or enriched image of the real world.

The task of creating computer vision software to do the recognition and tracking is a formidable one. Fortunately, a number of APIs now exist that do a lot of the basic work, and instead the programmer can focus more on the interface design and visual integration, which is also a challenging problem. We selected the Vuforia™ API after studying several alternatives [4]. This API is sufficiently mature to support robust work in AR. On the downside, however, it offers few possibilities for adjusting the recognition parameters. In a sense it must be treated as a black box.

In earlier work, we used artificial high contrast patterned targets, such as the one shown in Figure 1 [5]. Indeed, following numerous studies and trials in the early years of work on AR, these patterned targets had become the de facto standard for ensuring reliable recognition and tracking [3]. However, although these patterns still provide the best targets, the APIs now support non-standard targets - in fact anything that offers high contrast in an asymmetrical pattern can act as a target. Furthermore, Vuforia offers a cylindrical recognition mode. Essentially, instead of recognizing a flat target, the software will recognize a cylindrical target, which can be tracked even as it turns, thus allowing virtual elements to be added in relation to the cylinder which follow as it turns. For applications involving garments, this is ideal.

Figure 1.

Example of our initial effort using a printed pattern target.

We use the Unity Game Engine™ as our programming or interface environment. Unity provides a stable, robust, integrative framework for immersive reality projects, and interfaces directly with Vuforia [4]. We have used it in the past for both virtual reality [6, 7] and augmented reality applications [5].

3. The fashion event

As indicated earlier, we conceived of an event that would both showcase the fashion designs and also the science fiction books which inspired the creation of these garments. As a fashion designer with several collections, Edwards had experience mounting fashion shows. We therefore conceived of an event built around the idea of a fashion show, that is, with a runway, models, and successive reveals of different garments, combined with music, an audience, and lighting ambiance. The event, however, also sought to integrate elements usually associated with a book launch, that is, excerpts to be read, copies of the book available for purchase and a book signing activity. Finally, we also undertook to develop additional representations of the event via video, and an app that could showcase some of the designs developed. These could serve both to document the work achieved and also act as marketing tools for the event and for the books that inspired the event.

Edwards selected several excerpts from Plenum that both acted as examples of the writing but also highlighted settings which could be visualized. We therefore conceived of four scenes (initially three - one was added later in the development) that would be presented. For each scene, we designed a different belt using colors and textures that matched those of the scene. Some of these original design choices were modified however, over the course of the work, as a result of changing conditions external to the work itself (and in particular, constraints imposed by the Covid19 pandemic, which struck about six months into the work).

The three initial scenes selected were (a) the approach by the main character to an artificial habitat build around a small moon or moonlet, (b) the climactic final scene which takes place in the vicinity of a sun, and (c) the large spaceborne platform called the Annex within which the main character spends most of the time. The scenes also include two kinds of ship, artificial ships and organic ships called jonahs, which are descendants of Earth-bound sperm whales. The fourth scene eventually added was to showcase the kind of dynamically shifting skin textures described in the books. For this we selected a fractal pattern.

Here are examples of the first two excerpts [1]:

Excerpt #1: “The Rock, as its residents named it, was some fifty kilometers in diameter … Naively, one would have expected vertical structures pinned onto the Rock, but the Yard was structured horizontally rather than vertically. Instead of spires, it consisted of folded sheets through which poked a large, rounded section of the Rock. It had slightly more gravity than a true microgravity environment, but at humorn tempo, objects still settled quickly enough to notice. If you left things floating, they drifted downwards at a stately pace, but the oblique angle of the sheets often made the downwards direction something other than the apparent vertical!”

Excerpt #2: “Vanu boarded the flyer, and piloted it across the short distance to the access port for the Wellhead. This was an egg-like structure which looked to be tethered to the star by a gossamer thread that dropped away beneath it. The construction looked ungainly, a beached whale thrown up by the seas it served, although Vanu had to dampen the brightness of hir visual field to even perceive it properly. Vanu also had a momentary thought about what the Wellhead represented. It was like a guardian, or a gatekeeper, a Saint Peter at the pearly gates, not just to support those who went down into the Core, but also to screen out the unworthy.”

4. Fashion and textile choices

Although our end goal is to create full body AR fashions, for technical reasons this is still too ambitious an undertaking. Obtaining one set of AR elements to follow the garment is already a significant challenge. So we have focused on the creation of a set of belts. One of our reasons for choosing belts is a consequence of the technology we are using, that is, the cylindrical recognition feature of the Vuforia API. However, it took considerable experimentation to work out how to do this efficiently. For example, although we earlier used patterned targets printed on paper (Figure 1), we wanted to develop fabric-based targets, but fabrics have a stitch structure that acts to effectively degrade the contrast and resolution features of the target images.

Tests with different woven fabrics were undertaken first to determine which features are best for ensuring recognition by the Vuforia software. The image used must be non-repetitive and asymmetrical as well and must favor sharp linear edges rather than curves. Experimentation showed that high contrasts are required for the Vuforia recognition and tracking software to function effectively. Furthermore, recognizable scene elements must be evenly distributed across the cylinder to ensure the belt can be tracked adequately over time and body movement. Some occlusion, for example from the arms and hands, can be tolerated without losing the tracking lock. We initially tested printed images such as the one shown in Figure 2.

Figure 2.

Example of a printed image used to test the cylinder recognition process.

Although the Vuforia API supports a range of possible cylinders, for our application we needed a wide, relatively narrow cylinder, since it was to be worn as a belt and remain flat in its lengthwise dimension, thereby keeping its design stable. After some experiments varying the width of the belt, we found that it needed to be at least 15 cm wide to provide a reliable recognition lock. This is wider than most belts, but there are many examples in fashion where belts of that width are used. We also discovered that the Vuforia API was happier if the belt had a high contrast edge (see Figure 3).

Figure 3.

The finished moon belt with its black edging.

Images can be pre-tested in Vuforia before they are definitively used. Essentially, Vuforia in its test mode can provide a map of recognized points (Figure 4, corresponding to a segment of the belt shown in Figure 3). Hence if the density of recognized points is too weak, the image can be further manipulated to enhance recognition, but only up to a point. Furthermore, in the case of a woven belt, such as shown in Figure 3, modification poses a challenge. Although we tested the scene recognition initially using printed images (e.g. Figure 2), the final tests had to be done with the finished weave, as the Vuforia software is sensitive to small deformations in the image. However, once the weave, itself a time- consuming step, is finished, further correction is difficult. Since some contrast is lost with the move to the woven image, we had to be creative to seek better recognition at that stage of development. We did attempt to further enhance the contrast using embroidery techniques, but ultimately the solution was of a different nature (see the discussion below concerning the background).

Figure 4.

Recognition results for the moon belt.

Weaving complex scenes such as the image of the lunar surface shown in Figure 3 could be done accurately on a Jacquard loom. Although there are a number of Jacquard looms in Quebec City, access to them proved to be difficult (and would have been virtually impossible once the pandemic arrived). Fortunately, Michaud had completed a degree program in the textile arts and was able to use a range of different techniques to achieve similar results.

Considering the restriction of available equipment, Michaud used her own four-shaft jack table loom to test three fabric designs; overshot, shadow weave, and a double ikat-like technique (without resists). Cotton was used, as it is easy to find, does not stretch, is affordable, and can be dyed if necessary, with thread counts of 16/2, 8/2, and 8/4. Overshot in black and white, with the bigger thread 8/4, proved to be the most effective. However, the results were deemed too abstract for the project, so a compromise was made for a more pictural solution, using a double Ikat technique [8, 9], which offers a way of painting the threads before they are woven. Tests were undertaken to determine the appropriate weave density and to validate the painting techniques.

The second belt produced used an image of the solar surface (Figure 5). Recognition was generally stronger than it had been for the lunar surface (Figure 6). The belts for scenes 3 (space habitat) and 4 (fractal textures) are still under development. More details will be given below, since other parts of this work are in a more advanced stage of development.

Figure 5.

The finished solar belt with its black edging.

Figure 6.

Recognition results for the solar belt.

Also, in order to ensure recognition, the belts needed to be reinforced to make them stiffer. Otherwise, they would bend to conform to body shape and thereby distort the image viewed by the Vuforia software, disrupting recognition. A canvas backing was therefore added to the inside of each finished belt. The belts are 32 in. in circumference, that is, they were designed for the slender physique of most fashion models, and an elastic closure was added so that they can be adjusted to different sized waists. Experimentation has shown that the presence of a small gap in the image at the back does not appear to disrupt the recognition and tracking, nor does a small overlap.

Each of the four scenes is characterized by a different color palette. This was, of course, intentional, to give a distinct character to each scene/belt combo. The lunar scene presents yellow and orange elements, while the solar scene is more red and orange. The platform itself is gray, but the nebula is predominantly blue and green with splashes of red, purple and yellow. For the fractal scene, we decided to produce the patterns in blue and white rather than black and white. This gives its corresponding belt a distinctive color.

5. 3D virtual elements

As illustrated in the excerpts given above, the scenes depicted are highly visual in nature. Proulx Guimond is a visual artist with whom Edwards had worked previously. Indeed, he developed the cover art for the novel, Plenum [1]. He created 3D scenes for each of the four situations. The scenes were designed so that they would integrate with the visual appearance of each belt. Hence the lunar scene was anchored to the belt which presents the surface of the moonlet itself (Figure 7), showing as extrusions of the artificial city constructed around the moonlet (in yellow). For each scene, we planned three distinct elements, first a static 2D visualization, second a static 3D visualization and finally a dynamically changing 3D visualization. The dynamic elements added to the depiction of the moonlet and its artificial extrusions were the arrival of starships which would dock with the city.

Figure 7.

3D image constructed for the lunar scene.

In the second scene, the heliocentric orbital station is modeled (in exaggerated scale, since it would normally be too small to see at the scale of the sun) along with the surface of the solar photosphere with its sunspots. Furthermore, the dynamic elements included the jonahs and a solar flare made for a dramatic and flamboyant event within the whole sequence. To provide the fashion representation with more drama, the orbital station was situated on a diagonal with respect to the solar surface.

In the third scene, the main focus was the spaceborne platform itself, again, oriented obliquely with respect to the model’s body and extruding from this (Figure 8). The platform includes a dock and two rotating cylinders used to provide pseudo-gravity to its occupants. Here, again, jonahs are seen moving past the structure, along with an octopus who is also one of the main characters in the story. The platform is viewed against a colorful emission nebula in the background, and here the dramatic event is the swirling movement of the nebula under time acceleration, the spinning of the cylinders and the movements of the jonahs and octopus.

Figure 8.

3D image constructed for the space platform scene.

The 3D graphics were created in 3DS Max™ and eventually transferred to the Unity environment. Efforts were undertaken to make the render of the visual as light as possible to ensure real time updating of the scene elements. This was achieved by reducing the complexity of the scene to its bare minimum, that is, incorporating only a small number of simple elements, and limiting the number of different materials and textures used. In addition, the animated sequences were also kept to a minimal complexity.

The visual integration of the virtual scene elements with the physical belt and, indeed, the body of the models required further adjustments once all the elements were in place and fully integrated. The integration, as outlined below, posed significant challenges since the majority of the work was carried out individually under lockdown conditions over the course of 2020 and the first half of 2021, that is, during the midst of the Covid19 pandemic. Indeed, at one point we considered doing the fashion show as a purely virtual event. However, the initial impetus for the project was for a show with live audience, and we remained convinced that the full impact of the AR technology consists of its use in real time in the presence of a physical audience. We therefore decided to delay the event itself until it could be performed live. As a consequence, however, this paper is being completed before the final presentation event.

6. Garment tracking

Testing of our efforts to integrate the tracking software with the belts revealed a succession of challenges. We found that the software was temperamental in its ability to recognize the belt patterns, even though initial efforts had been successful. Good lighting turned out to be an important element in ensuring this recognition. Once recognition was achieved, and the virtual elements reliably overlaid on top of the optical image, it would retain the lock for a certain set of manipulations, but eventually the lock would be lost when viewing conditions became less than ideal (for example, if the person wearing the belt moved too far away from the camera). The model would then need to move in close, or image viewing parameters otherwise manipulated, until a recognition lock could once more be obtained. At first, the virtual image was stable only for short intervals. We introduced some persistence into the visualization, so that even when the recognition failed, the virtual objects would persist a second or so. This sometimes results, however, in a jerky movement of the virtual elements in relation to the model’s movement.

Effective integration also required introducing occlusion effects. Hence, we used the belt’s cylindrical shape to create an occlusion model and used this to hide virtual elements as they ostensibly moved behind the body. Without intentionally occluding the virtual elements in this way, they were perceived as always being in front of the model, breaking the illusion.

As the model moved, the location of the virtual elements would often lag behind the model. We used a cube-like envelope to test these issues. We eventually discovered that we had been testing the performance of the visualization against “busy” backgrounds. Much of the early testing was done in a living room or kitchen viewed on Zoom. When we met and did tests outside, the same problems persisted until we chose a uniform background. At that point, many of the tracking problems diminished. After that, we chose uniform walls for testing. Feature-filled backgrounds clearly confused the recognition software, causing it to lose its lock more frequently. Once this problem was identified, lock persistence, although not perfect, became acceptable.

At one point we also sought to use an accelerometer and Inertial Motion Unit (IMU) that could be integrated into the garment as another source of data to help stabilize the imaging. Our efforts in this direction failed to generate the necessary stability, however, and the effort was abandoned.

Another strategy we eventually developed to ensure longer lock persistence was to restructure the viewing geometry using a second camera feed. Essentially, we had been using the same camera image as the source for recognition and to serve as the support for the visualization. Our realization that Vuforia must degrade the image somewhat during processing, a realization that followed its lack of sensitivity to difference resolution webcams, led us to segment out of the image the region surrounding the belt and provide this image segment to Vuforia in a separate camera feed instead of the full image (Figure 9). This resulted in better recognition at a wider range of distances. Hence, for example, in our earlier attempts a distance of about a meter was required to ensure a recognition lock, but after the segmentation step was added, a reliable lock up to three meters could now be achieved.

Figure 9.

Image of the finished lunar belt in situ along with image subset used for recognition.

Recourse to a more complex set of processing options such as described here meant, however, abandoning our commitment to permitting the use of a tablet or smartphone for viewing the AR scene. In the final configuration, we need access to a full computer, although a laptop is also acceptable.

7. Final integration (including music and staging)

Each belt is used to anchor its corresponding scene—indeed, each scene was designed so as to align the final appearance including the moving model with the esthetics of the 3D images. In principle, this means that several belts and animations could be viewed simultaneously. In practice, however, since each set of animations requires significant computing power to render in real time, we preferred to keep each presentation separate. Furthermore, the belts are worn with a white top to provide a canvas over which the 3D visuals can be overlaid.

The final integration of the 3D graphics with the physical belts are shown in Figure 10 for the lunar belt and Figure 11 for the solar belt. As indicated earlier, two additional scenes are under development, one involving a large space platform and the other a dynamically changing costume incorporating fractals. The 3D images are almost complete and the belt development is in its final stages. With the relaxation of lockdown restrictions currently underway, the organization of the event itself is also now possible.

Figure 10.

3D image for the lunar scene integrated with the physical belt in situ.

Figure 11.

3D image for the solar scene integrated with the physical belt in situ.

A staging scenario was developed for the presentation in front of a live audience. This script included information on the timing of each segment. This was necessary also to assist in the development of the music accompaniment. At this point in the development of the event, the music has still not yet been finalized. However, we are proceeding as we have in previous projects, which is to mash together public domain samples of music into a coherent and dynamically interesting whole [6].

The total duration of the final show is estimated to be about 40 minutes. Following an introduction, the presentation of each of the four belts cycles through the three levels of visualization (static 2D, static 3D and dynamic 3D), along with the reading of the text excerpt, and hence each belt takes about ten minutes to present. After the individual belts are presented each in turn, the four belts are presented together in a kind of finale, without the full dynamics so as to render the processing tractable and with an additional excerpt. Following the conclusion, a book signing session is expected to be announced.

Our plan is also to stream the event for virtual participants. Indeed, the book publisher is located in Boulder, Colorado, and both collaborators and other interested parties are located there. Streaming became a de facto standard with the pandemic, and we retain that here. In addition, we are developing an app which will allow the 3d scenes to be viewed independently of the belts, as a value-added product to be made available to individuals as part of the marketing of the book.

8. Conclusion

In summary, our Plenum A La Mode event constitutes a unique amalgam of different ideas and technologies that straddles the artistic, scientific and technological worlds. The technical challenges in terms of obtaining reliable recognition of the fabric belt patterns, tracking the belt over a complex set of manipulations and movements in real time, and visually integrating both the optical scene with virtual three-dimensional enhancements were considerable. Furthermore, the project required the close collaboration of several individuals with different areas of expertise to achieve the desired results, including the writer, event and fashion designer (Edwards), 3D graphics artist (Proulx Guimond), textile artist (Michaud) and programmer (Caron Roberge). Another colleague with expertise in the development of music is also working with us, Jocelyne Kiss.

Among the challenges addressed and overcome were the introduction of occlusions using the belt cylinder as a model, so as to create the illusion of virtual elements passing behind the model’s body, the provision of stable and good lighting conditions to ensure robust target recognition, the use of uniform backgrounds so as to limit confusion for the recognition software, the use of a second camera feed so that the tracking could be carried out independently of the scene enhancement, and the introduction of persistence to enhance the stability of the visualization of the virtual elements and prevent loss of the recognition lock.

The final result of these efforts provides acceptable, but not perfect, real time augmented reality image enhancements. The virtual elements experience a small amount of jitter while the software tries to track the belt, and if the movement is too rapid, the model passes behind a visual obstacle, or simply moves too far from the camera, the recognition lock is lost and the virtual elements disappear until a new lock is acquired (generally by moving towards the camera and holding still). The public staging requires the use of a desktop computer of relatively strong power, loud-speakers and a webcam. Our expectation is to project the enhanced image onto a screen as the model parades along the runway. The belts are themselves esthetically interesting as well as serving as target patterns for the augmentation.

Ultimately, we believe that more complete full body augmentations could be achieved using several targets integrated into the garment design. These might include flat targets in some areas, such as across the chest or back, as well as additional cylindrical targets for, example, around the thighs and arms. The augmented images would need to be matched to each other and integrated visually into a seamless whole, which will pose many additional technical challenges to the ones we encountered. Details of the final fashion show event will be made available through the principal author’s website at www.geoffreyjenedwards.com.