Smart Garment Design for an Augmented Reality Body Mapping Experience

Abstract

Based on previous work, we created wide handmade belts as support for a live augmented reality (AR) event. We describe how the garments were made to best meet the constraints of the AR design and how they were used as part of a fashion show and book promotion event. We determined the kind of features the AR software can detect and then created prototypes based on those characteristics. The belt design followed several steps: 1. Computer image preparation; 2. Thread preparation; 3. Thread dyeing; 4. Weaving; 5. Assembly; 6. Photographing the result and retouching the image. The lack of adjustability to deformation was a concern when dealing with clothing and fabric. We resolved this issue by stiffening the fabric. We discuss belts performances well during the live event; other textile techniques as substitutes (e.g. patchwork) and future research prospects.

1. Introduction

Plenum: The First Book of Deo [1] is a science-fiction novel written by Dr. Edwards. It presents the coming-of-age story of Vanu Francoeur, a novice in a monastic religious order who lives on a space station. In this distant future, society determines everything from birth—status, gender, and profession—and leaves little to individual choice. The novel also depicts numerous references to clothing and examples of visually changing garments, to reflect the wearer’s emotions or state of mind. Our work here was to recreate four scenes from the book, to share how a smart garment can change one’s body perception, that is, in a manner like what is sought in traditional body mapping, and as a complement to the use of the technology to promote the book itself.

We were inspired by the principle of body mapping [2], which is an art-therapy tool used to help patients express their feelings by drawing on a body-shaped cardboard that represents their own body. Drawing is a great substitute when words are hard to use, and extensive research has shown the utility of the approach [3, 4, 5]. However, this tool was designed for flat, 2D surfaces. Could something similar be achieved in 3D?

Augmented reality (AR) offers one possible solution. It mixes real and virtual elements to produce an immersive 3D experience. Drawings can be positioned as objects in the environment, and even their animation is possible. AR expands the realm of expressiveness available to patients. Even so, contrary to traditional body mapping, a camera is mandatory to get a view of the whole AR scene displayed inreal-time on a screen. This process has been described in previous studies by our team [6].

As visual AR is based on pattern recognition, it needs an anchor, a visual reference or “target.” Considering the many references to clothing in the novel, we chose a garment to act as an anchor. We focused on the development of a wide belt as this is worn close to the body, can adapt to many shapes, covers the torso through a complete 360° rotation, thus is located on a relatively stable area of the body when walking, compared to the limbs or the head. To preserve the idea of having a garment that will evoke the novel and at the same time an impression of closeness and authenticity, we decided to create a handmade fabric. It was also easier to meet the technological constraints related to the use of AR if the belt was made specifically for this project.

Our main objective was to assess if our handmade fabric patterns could be recognized by Vuforia©, an AR software, and if we could generate stable images. There were many technical issues regarding the design of the belts; this is the focus of the paper. The main question was: To what extent is it possible to ensure efficient and robust pattern recognition using a handmade fabric as the target for the AR software? In the first part of this chapter, we present our methodology. We then describe how we proceeded to create the images that would be incorporated into the belts, create the fabric itself, and then constructed the belts. Next, we show how these innovations were used during a live streaming fashion show session. We conclude with a discussion of the technical limits of our process and future research perspectives.

2. Methodology

Vuforia is an AR software that detects target patterns, then allows augmented images based on this detection to be introduced. The images that appear in the AR mode, to be recognized and produced, must undergo a preparation phase with an online tool called Target Manager, available on the Vuforia website. Vuforia is then plugged into Unity3D, a virtual environment development software. We used Unity3D to display a 3D image or animation when the appropriate target was recognized, using trial and error to arrive at good results.

The first task was to determine the kind of features Vuforia can detect, the second was to create prototypes based on those characteristics. We tested first with paper templates printed from the Vuforia website to understand how the system works. We then created our own printed designs and tested these. After adjusting these to ensure good recognition capabilities, we experimented on handwoven fabrics. In the next section, we show step by step the process that led to the development of the first prototypes.

3. Garment design

As reported above, an earlier study [6] showed that AR can be achieved with a handmade fabric. The fabrication of the latter needs to comply with a set of rules to ensure that it can be used as an anchor for the animations. These rules are summarized below. Following this enumeration, we present the augmented images we developed.

3.1 Detection characteristics of AR software Vuforia

Vuforia is a software that can detect image features called targets; that is, it takes as input a camera image and looks for a specified target pattern. It then provides the possibility to overlay another image on top of those targets (or at a fixed offset), which produces the augmented image. Vuforia can be used with many platforms, including Unity.

Several image features were determined necessary to ensure good target detection:

• Asymmetry

• High contrast

• Clear edges

• Square image

• Many angled edges

• Black border

• No deformation.

Testing the different features on paper helped conclude that the best target is a black-and-white square image with many geometric straight-edge shapes on a rigid support.

The lack of adjustability to deformation from Vuforia is a concern when dealing with clothing and fabric. We resolved this issue by stiffening the fabric, first with cardboard, then with fusible interfacing. More details are provided in the following sections.

3.2 Prototypes

3.2.1 Weaving techniques and thread choices

The next step was to create and test fabric samples designed to maximize target recognition while at the same time serving the thematic focus of the project, that is, presenting scenes excerpted from Dr. Edwards’s book. Many weaving techniques exist to create a fabric with colors that is asymmetric. These include jacquard, multi-layer, shadow weave, and crackle weave, the latter two of which require tweaking to avoid symmetry. Since the goal was to produce a figurative image, based on elements taken from the book, the jacquard weaving technique was deemed the most appropriate, as it offers the most versatility. It needs a specialty loom, however. There were two such machines to our knowledge in Quebec City. Unfortunately, the Covid-19 pandemic struck at that moment, rendering access to the jacquard looms difficult, so we sought another method.

One of us had a four-shaft weaving loom at home. This simple loom limits the kind of techniques one can achieve in a timely manner, but there are ways to produce a working fabric with the application of some creativity. It was thus decided to use the four-shaft loom to create samples. As for technique, both shadow weave and overshot were used to produce big and edgy patterns.

We also had to choose which kind of thread to use. Since we had to stiffen the fabric anyway, we decided to go with cotton, as it does not stretch, is easily available, and is cheap. Cotton of different counts was ordered, namely, 4/8 (coarse), 2/8 (medium), and 2/16 (fine), in black and white.

3.2.2 Weaving and finishing protocol

The thread was warped to obtain an alternation of black and white. Overshot threading was used with different setts ranging from 18 to 48 ends per inch with the same 12 dents per inch reed in all cases. The same number of picks and the same treadling were used for each of the two series of patterns: diagonals and semi-concentric diamonds. The first and last picks were woven into tabby. The initial test patterns were designed to combine the advantages of overshot and shadow weave, that is, to produce thick and contrasting outlines (Figures 1 and 2).

Figure 1.

Draft of diagonal pattern.

Figure 2.

Draft of semi-concentric diamond pattern.

After light washing and drying, to allow the cotton fibers to take their final shape, the samples were fixed on a sturdy piece of cardboard to prevent the image captured by Vuforia from being distorted when handling the fabric (Figure 3). The choice to use the front or the back of the weave was guided by aesthetics. The weave was then photographed using a digital camera with natural light and no flash against a white background. The images were color-encoded in a JPEG format with a resolution of 180 pixels per inch. They were then cropped to keep only the area with the patterns (Figures 4 and 7). A static 3D object, a white cat, found in a free resource bank for Unity3D, was used to test the image augmentation (Figure 6). The detected points in the target patterns are shown as yellow crosses on the relevant images (Figures 5 and 8; Figures 21–24).

Figure 3.

Fabric of the diamond pattern after washing and fixing onto a cardboard backing, 4/8 thread, sett 18 ends per inch.

Figure 4.

Cropped image used with the target manager, diagonal pattern.

Figure 5.

Detected targets, diagonal pattern.

Figure 6.

Screenshot of augmented image with a digital white cat, diagonal pattern.

Figure 7.

Cropped image used with the target manager, diamond pattern, 4/8 thread, sett 18 ends per inch.

Figure 8.

Detected targets, diamond pattern, 4/8 thread, sett 18 ends per inch.

3.2.3 Target detection results

The Target Manager tool indicated that these weaves made solid, highly recognizable images with a multitude of well-distributed targets and a detection score of 5/5 for almost all weaves. Despite a rectangular aspect ratio (1:2), they performed well in augmentation tests with Vuforia on Unity3D. Lower detection scores were found when using medium and fine yarn at low densities, possibly because the aspect ratio was higher. These results allowed us to decide in favor of the thicker 4/8 thread. The diagonal pattern detection score was 4 out of 5, which is very good (Figure 5). The diamond pattern, 4/8 thread, 18 thread count detection score was 5 out of 5, which is excellent (Figure 8).

4. AR system and smart Belt design

Dr. Edwards chose four scenes from his novel that served as inspiration for the design of the belts. These scenes were: 1. the surface of a moon whose resources are exploited by a space base built upon it (Figure 9); 2. a scorching sun with a space elevator (Figure 10); 3. A fractal drawn from the Mandelbrot set that acts as a dynamically changing body covering in the novel (Figure 11); 4. a space station within a nebula (Figure 12). Note that the lunar and solar surfaces were image subsets from public domain NASA photographs obtained via the internet, the nebula scene resulted from an artist’s rendition, and the fractal image was produced using a fractal algorithm running in Unity.

Figure 9.

Lunar belt original image.

Figure 10.

Sun belt original image.

Figure 11.

Fractal belt original image.

Figure 12.

Nebula belt original image; drawing by J. Proulx-Guimond.

The belt design followed several steps: 1. computer image preparation; 2. thread preparation; 3. thread dyeing; 4. weaving; 5. assembly; 6. photographing the result and retouching the image. Note that we encountered difficulties with the detection algorithm applied to the nebula belt, due to a lack of contrast and the blurred outlines of the shapes. From a methodological point of view, this point caught our attention. This highlighted the importance of reinforcing the quality of the image and the color balance before its reproduction on the threads.

We used a double ikat-like dyeing technique [7] rather than the previous weaving techniques so that the whole set was more aesthetically consistent with the scenes in the book, even though the AR support does not need to have a conceptual link with the augmented image. We chose the ikat technique because it allows a great deal of precision in the dyeing process, as the warp and weft threads are painted separately. It produces an impression of both blurry and pixelated effects that lend themselves well to the aesthetic sought for this project and to the recognition constraints of AR. Ikat is a dyeing technique that consists of dyeing the threads before weaving. Everything must be well measured and calculated in advance to obtain a good reproduction of the patterns. However, when the preparation is liquid and therefore not very viscous, it tends to diffuse. A thickening agent, such as sodium alginate, agar-agar, and even starch, helps give a more precise contour.

The colors of the first three belts were chosen so that they stood out visually from each other. The blue color for the fractal belt was chosen when dyeing the threads to produce a chromatic contrast with the first two. The final belt was more colored to represent the diversity of colors that can be found when the nebula acts as the background.

4.1 Computer image preparation

At the very beginning of the AR process, the chosen image must be prepared on a computer. To do this, the image is cropped at a ratio of approximately 6:1 for width versus height. Used for a cylindrical shape, this ratio gives a rather square image, which is most effective for Vuforia. In practice, the height of the resulting belts was approximately 15 centimeters, which is the maximum that can be comfortably worn by a model just above the hips. Therefore, the length is about 90 centimeters, resulting in a diameter of 14.3 centimeters. This implies that our current belts can only be worn by slender models.

For the sun belt, sunspots were added where there were few elements to detect. The background was standardized, and the number of colors was reduced to four to accentuate the contrasts between light and dark areas. These modifications made it possible to obtain the most stable belt for an augmentation. The image was then adjusted to 12 pixels per inch in height and 12 pixels per inch in width, the same measurements as used in a test sample. The image was then printed at its actual size in two single-sided copies and one copy mirror-reversed over its length.

4.2 Thread preparation

One of the single-sided copies of the image was placed on a flat surface previously covered with a tarpaulin. It was then covered with a transparent plastic. This served as a reference for placing the colors later. The warp consisted of 100 threads, of which the first 12 and the last 12 were black and the others white. The black threads were not visible on the final piece and served as a guide for sewing the bias during finishing. They also served as a buffer zone when the dye diffused too much into the weft yarn. Following this, the warp was stretched across a flat surface over the printed image. The threads were spaced according to the planned sett, which was 12 ends per inch (Figure 13).

Figure 13.

Warp dyeing, threads stretched over a table, fractal belt.

For the weft, a rigid cardboard support was used, the measurements of which corresponded to the width of the weave, considering stiffness and shrinking. Single-sided and mirror images were laid on either side of the cardboard, aligning them together. The weft was wound continuously around the support, taking care to respect the final picking (Figure 14). The weft ends needed to be clearly identified. The warp and weft yarns were soaked in 2% sodium carbonate solution and left to dry. The sodium carbonate served to increase the substrate pH, which allowed the chemical reaction with the dye.

Figure 14.

Weft dyeing, wrapped around the cardboard support, fractal belt.

4.3 Thread dyeing

The threads were dyed with reactive dyes as these are ideal for cellulose-based fibers. The dye was mixed in high concentration, between 4% and 8%, with urea water. The urea water was used to keep moisture in the fiber, long enough so that the chemical reaction can take place. To prevent the dye from spreading too much on the yarns, a thickener was also used. This was especially handy for adding small details. A dark dot was added on each end of the weft to indicate pick changes.

A tarpaulin, laid so that it did not touch the threads, was placed over the whole setup during the time the dye was soaking in. After that, the threads were removed from their holders, washed, rinsed, and dried.

4.4 Weaving

The warp was beamed onto the loom and threaded. A tabby section in white 4/8 thread of about two centimeters was woven at the beginning, and the end was overcast to prevent fraying; this was hidden when sewing the lining during finishing (Figure 15). Some unweaving was necessary to align the patterns.

Figure 15.

Loom weaving, fractal belt.

4.5 Assembly

A horsehair interlining was used to give rigidity to the belt. This was heat-bonded onto the back, taking care to align the horsehair in the belt’s height direction. The interfacing was inset by 1 centimeter at the top and bottom of the dyed area, that is, on the black border, along the width, and on the areas of white tabby at the weave’s start and end. The excess fabric was folded over onto the back. A black polyester lining was sewn by hand onto the back. This lining ensured a finished look and was softer than the horsehair. Finally, a black polyester bias was also sewn in by hand (Figure 16).

Figure 16.

Belt assembly diagram.

A strip of black jersey about 30 centimeters long was sewn by hand to one end of the belt to serve as a fastening system. It could be fixed at the other end with pins. Located on the back of the user, this stretchy material facilitated breathing and adjustment to the body.

The bottom figure represents the hand-woven fabric ready for assembly. Above in beige is the horsehair interlining. Next comes the black lining, which covers everything and gives a finished look. At the very top, the black frame represents the bias that evens out the black border and completes the finish. The weaving is rectified using a square and a steam iron. One of the sides must be chosen to be the visible side. This is an aesthetic choice.

4.6 Photographing the result and retouching the image

Once the belts were constructed, it was necessary to photograph them and import the resulting image in the Target Manager software. For this, natural light was preferred. A white sheet, on which the belt was laid, was stretched out on a flat surface in front of a large window. The photo was taken vertically, avoiding shadows, and placing the belt as straight as possible. The image was imported into a photo editor to be cropped close to the black border. It was sometimes necessary to rework the border, such as for the lunar belt. In this case, the image was cropped flush with the dyed area and a new border was added (Figures 17–20). The retouched image was imported into the Target Manager website using the cylindrical model option. All images of the belts were placed in the same package that was imported into Vuforia (Figures 21–24).

Figure 17.

Finished and retouched lunar belt image.

Figure 18.

Finished and retouched sun belt image.

Figure 19.

Finished and retouched fractal belt image.

Figure 20.

Finished and retouched nebula belt image.

4.7 Software and digital development

Alongside the belt fabrication process, other team members carried out other parts of the project. Jonathan Proulx-Guimond was the animation designer. In particular, he drew the scenes from the novel and added several details that were not in the descriptions to make the scenes more complete. These details included the shape and texture of the ships, the look of the lunar docking station, the overall shape of the nebula space station, and the solar elevator. This creative work was carried out continuously, thanks to weekly meetings, which made it possible to adjust the project so that it formed a coherent whole. For example, for the moon belt, he was inspired by what D. Michaud, who did the weaving, had made beforehand to give a color theme and a texture to the moon, which was itself inspired by the images provided by Dr. Edwards. And conversely, the design of the nebula belt reflects the concept imagined by Dr. Edwards and Mr. Proulx-Guimond.

For the software component, Jonathan Caron-Roberge modeled the 3D scenes in Unity. He first created an application using Unity for users of body mapping integrating AR [2]. This application presents a character, male or female, standing in the common living area of a house. The character is seen in a third person view, without any characteristics, essentially a silhouette. In this version, the patient sees themself from the outside at first. The patient is then guided by a resource person mastering the application to add visual elements to their body map. The elements that can be added are mostly simple geometric shapes, such as cubes and spheres, but also flames and rain. It is possible to customize the color, size, and quantity of shapes as well as their position in relation to the body. Shapes can also move through space. Mr. Caron-Roberge then created another app for the show (see the next section). He incorporated Mr. Proulx-Guimond’s animations and carried out the port of the project from Windows to iOS. This made it possible to use a smartphone to film the belts, transmit the information to a laptop computer to manage the image recognition and the augmentation, and send it back to the phone screen to view the rendering.

5. Live fashion show streaming

The design team carried out a fashion event showcasing the belts in real time, which was broadcast live on the internet, on April 23, 2022, from 2 p.m. to 4 p.m. (Quebec City time). This event, under the overall leadership of Dr. Edwards, was called Plénum À la Mode¹. It brought together literature and fashion in an unprecedented hi-tech mash-up. During this event, our design team was almost all present.² We present the pictures of the belts taken that day (Figures 25–28). The resolution of the images is low as these are the images that were live streamed by Vuforia.

For the augmentation to be visible, the camera needed to be placed about 1 meter from the target, which can be difficult if the person wearing the belt is moving. This is a technical limitation of the recognition process. Therefore, the models were asked to stand still. Also, for stable augmentation rendition on the screen vis-à-vis the target, the latter needed to have a high detection score, which was not the case for all belts. Indeed, during preliminary experiments, it was determined that tracking the movement of a mannequin with a handheld camera, such as a smartphone, could lead to abrupt jumps in the final image. To counter this, we could have choreographed the movements. Unfortunately, this was not possible due to restrictions related to the Covid-19 pandemic. The different confinements and the limitations on the number of people who could gather in the same place meant that it was not possible to carry out rehearsals. As a result of these limitations, the models were instructed to stay put and maintain a relatively fixed posture.

6. Discussion and conclusion

The design and development of belts for body mapping produced with the assistance of a team and the organization of a live fashion show based on the technology demonstrated that AR is feasible using hand-woven fabric as the target pattern. Indeed, we achieved our objective, namely, to manufacture a garment linking virtual events with the bodies of the participants.

The sun belt performed the best. We believe this is due to the high contrast between paler and darker areas, the large number of target points all around the body, and a highly asymmetric pattern resulting from better pre-processing of the initial image (Figure 22). The black dye was the same for all the belts, but it was probably more concentrated on this one. The relatively uniform background orange color helped to make the black spots stand out better. The fractal belt was a close second, as far as target ratings for the Target Manager were concerned (Figure 23). The weft painting was quite precise and made for great contrasts despite a busy design. The moon belt did well, but contrasts were not as good, and some areas lacked targets (Figure 21). Granted, it was the first belt created and the whole production process was still under development.

Figure 21.

Detected targets, sun belt.

Figure 22.

Detected targets, fractal belt.

Figure 23.

Detected targets, moon belt.

The nebula belt, while the most pleasing visually, did not perform well with Vuforia (Figure 24). It was the last one made, and we did not have time to determine which elements were not working. The most possible reason is the lack of contrasts and the blurred edges. The design for this one was a deliberate stretch regarding target detection. Even though we experimented earlier with black-and-white samples, we wanted to try a more elaborate blending of the decorative elements. Regarding the cylindrical shape of the belts, it is worth noting that the augmentation was possible even though the models’ torsos were not of the same shape or dimension as the one inputted to the Target Manager. The software seems to allow the 3D digital models some ease in terms of deformation (Figures 25–28).

Figure 24.

Detected targets, nebula belt.

Figure 25.

Model 1 wearing the moon belt with the moon station animation.

Figure 26.

Model 2 wearing the solar belt with the solar flare animation.

Figure 27.

Model 1 wearing the fractal belt with the fractal animation.

Figure 28.

Model 2 wearing the sun belt with the nebula animation and its station.

It is also worth remarking that the patterns that can be created with a basic loom can be rich enough to be detected by a tool like Vuforia. The use of other weaving apparatus, such as a multi-frame loom or a jacquard loom, should also lead to useful and interesting results. Furthermore, the use of weaving as a base technique makes the virtual medium more concrete and the images more tangible.

Other textile-related techniques could be tested to achieve similar results. Patchwork is a sewing technique that consists of assembling small patches of fabric together. It is probably the best alternative to weaving as it can be done with little equipment. Embroidery involves sewing threads and other embellishments on fabric. We did try this on a few samples but not enough to get conclusive results. Knitting and crochet are unfortunately not an option, as they produce stretch fabric and hence induce deformations.

The next step could be to promote the fact that augmented reality on a textile support offers a promising avenue for art therapy via body mapping. This could take the form of a proof of concept with patients to measure the usability and usefulness of the setup. The integration of our first body mapping app, developed by Mr. Caron-Roberge, with the smart belt could also be undertaken. At the very least, this is a research project that we wish to explore in a future study.

Several uses are possible for such a technology. In addition to body mapping, consumer clothes might incorporate the technology to create interesting fashion statements. Some designers have started to explore this idea [8, 9]. It could also be possible to create tailor-made clothes, to create AR costumes for virtual universes of the mixed reality type to fit into gaming contexts or to share clothes between users of the same universe. We can also imagine using the technology for real-time performances, on a stage, for example, or for cosplay enthusiasts. In the field of entertainment, Disney Enterprises has recently developed an AR project that makes it possible to juxtapose an image of a digital costume on a person [10]. The associated patent describes how the image of the costume makes it possible to adjust to the static pose of a person, while respecting their personal morphology. There are other initiatives in this direction, namely, to superimpose a costume on a person. One blog post even defines this kind of application as “augmented costume” [11]. These AR applications have the advantage of allowing users to try on a costume, either for a one-off photo shoot or to visualize how it might fit their figure before making the physical garment. All these ideas have the disadvantage of requiring a stationary subject and a still pose. Our solution is innovative because a camera can move around a still-standing person and add animation to the other augmentations.

Modifications to the system as presented could also take different forms. For example, instead of only modeling the belt in the form of a cylinder, it would be interesting to represent the complete body of the user, as is the practice in traditional body-mapping techniques. A full body animated in real time would allow the augmentation to be adjusted to follow these movements. For this, the use of VuMark on the whole body (Vuforia’s proprietary technology in the form of hexagonal targets), anchoring points on the garment to reveal virtual images, might be appropriate. This might also allow the use of other textile-manufacturing techniques, such as stretch fabrics and non-regular meshes.