How to Use the Advantages of AR and VR Technique to Integrate Special Visual Training Strategies in Non-Verbal Communication Skills Training for Children with Autism

Abstract

Social communication is the basis of human interaction, as people transmit messages, their emotions, and some symbolic social content through oral or non-verbal communication. However, it is difficult for most children with autism to master, or understand, these non-verbal social communication skills. In recent years, Augmented Reality (AR) and Virtual Reality (VR) technology (A/VR technology) have been widely used in the social training of children with autism. This technology creates many learning states that are difficult to achieve in the traditional social training of autism, and they are mainly achieved through the sensory advantages of A/VR technology in the visual presentation and simulation of surreal state situations. This technology can provide more attractive interactive games and play strategies to help autistic children learn social reciprocity skills. Therefore, this paper discusses how A/VR technology can be used to construct a surreal visual training strategy in the non-verbal social training of autistic children and, by using different teaching strategies and visual training applications, to help them to improve their mastery of non-verbal social communication. Based on the findings, future researchers are offered some suggestions and references for the application of A/VR technology in the social training of autism.

1. Introduction

1.1 Congenital social communication obstacles in children with autism

The common obstacles for the social skills of children with autism include the following [1]: they have difficulty in their social reciprocity with others, they have difficulty in maintaining mutual social relations, they are unable to share their feelings with others, they are unable to think like, and understand the emotions, of others, from their perspective [2, 3], and they have a lack of empathy [4, 5]. Children with autism find it difficult to establish effective social relations due to their social skill barriers [6, 7, 8, 9], which lead to them to feel uneasy and to retreat when interacting with a crowd [7], and indirectly lead to social isolation [5]. A person’s social skills refers to his/her ability to successfully and independently participate in social interaction, to establish and maintain social relations with others [10, 11], and to meet the social reciprocity conditions and communication skills required in different situations [12]. Having social skills is one of the important abilities that people need, in order to get along with others in society. It includes the ability to observe and listen to others, to make basic eye contact with people [13], to give social body feedback [14], to have complex social interactions and to manage their emotions [15]. Social skills are often constructed on the basis of some non-verbal social communication behaviors [16], such as facial expressions, head or hand movements [2], posture and eye expressions, etc. However, children with autism generally lack these important social communication skills [17, 18].

1.2 Social communication skills for children with autism

The social communication skills that autistic children need to master are mainly divided into various levels: (1) social initiation, (2) social reciprocity, and (3) non-verbal social communication [19, 20]. Social initiation refers to the ability to actively interact with others and to initiate social content. Social reciprocity refers to the ability to have good social communication and interaction with others, to be able to have appropriate behavioral responses, and to engage in a dialog. A person’s non-verbal social communication ability refers to his/her ability to understand the deeper meaning, or symbolic metaphors, of other people’s body language and expressions in non-verbal communication [21]. Among them, the ability to reciprocate socially summarizes most of a person’s comprehensive ability and behavior performance in social skills, and it is also considered as the most important comprehensive ability in social contact [22, 23]. For example, people can judge the environmental background and social situation of the other party, while understanding the other party’s emotion and discourse and making an appropriate social response [24]; it includes the ability to understand other people’s emotions [25], their deep feelings, and their implications [26]. Therefore, social reciprocity is often regarded as the focus of learning social skills for autistic children.

1.3 Non-verbal social orientation covered by social reciprocity ability

The DSM-V [27] of the fifth edition of the diagnostic manual of the American Psychiatric Association (APA) points out that one of the most important symptoms of autistic patients is the behavioral disorder of social reciprocity. The key to this behavior is whether they can master some more subtle non-verbal social communication clues, including facial expressions, eye contact, social body movements, social ceremony actions, or emotional gestures and eye expressions [17, 28]. These social features cover a large number of slight visual characterizations and non-verbal communication cues, which provide the basic elements of strong social reciprocity for ordinary people to interact with others, to share their feelings, to think and understand their behavior, from their perspective, as well as their intentions, emotional feelings and empathy [3]. Moreover, this ability requires considerable visual information for grasping and making social judgments [17, 18]. However, this ability is very difficult for autistic children, as they often have difficulty in handling non-verbal social communication proficiently, and they cannot carry out effective social interaction and communication with others. Similarly, it is a challenge for them to actively make eye contact with others and to make appropriate social responses [29]. Such symptoms will cause autism patients to become estranged from people and will make it difficult for them to effectively integrate into society [30].

1.4 Social training methods and teaching strategies are widely used in autism nowadays

There are many social training methods [31] and teaching strategies to help autistic children learn to judge more “social situations”, to understand different “non-verbal social cues”, and to make an appropriate social reciprocal performance; these include behavioral interventions [32], pivotal response training (PRT) [33], scripting [34], story-based interventions, etc. [35].

Among them, the most common training strategy is the use of Social Stories™ [36], or the social intervention training method, based on situational stories [35]. The Social Stories™ method can help children with autism to better understand and follow social rules and routines. In addition, it can promote a better self-awareness, help them to gain insight into the perspectives of others, and to understand how their behavior impacts others. This model appropriates social interaction by describing a situation by using the relevant social cues, it gives the perspectives of others, and it suggests an appropriate response. Generally-speaking, social stories or scripts are created to test for autism and they are used to assess the effects of multiple-baseline designs, across the different participants of Social Stories™, for intervention or training [36]. By using the social story method, combined with role play, autistic children can be guided to handle a situation and understand the relationship between social objects and other social communication states [37]. This includes how to respond with appropriate social reciprocal behavior, according to different social objects, and to grasp social skills [38]. For example, according to other people’s non-verbal social cues (such as their posture and eye expressions, facial expressions, or all kinds of social body movements), they will try to figure out the social intentions of the other person, or to carry out a role exchange and a role disguise game [39], which is the same as a psychological intervention training strategy [21]. This kind of teaching method can help autistic children to master the situation, as well as the interpersonal interactions in social stories, and it guides them to observe, and familiarize themselves with, the social behavior and skills that an adult society should have [36]. Through targeted social behavior intervention training [32], the social reciprocity and judgment of autistic children in a social situation can be gradually improved, and their ability to communicate socially with others can be enhanced [17, 40].

1.5 Current limitations of social training for autism

The existing social intervention training, such as role-playing, using the social story method, or interpersonal skills on empathy, requires considerable imagination and pretend-game skills [41] to guide autistic children to figure out and understand “social relations” and “social behavior cognition”. Among them, empathy is the ability to understand and comprehend the emotions and behavior of another person. This skill, which can be trained, facilitates interpersonal relationships.

However, children with autism are reluctant to participate in the situation guidance of intervention training, when they cannot see the scene, or anything to help them understand the situation or game [42]. For example, starting from general social training, game intervention activities, such as role-playing or perspective-taking, are training methods that are used for autistic children to perceive the feelings of other people and to generate empathy. However, due to their inherent social defects and weak imagination, they cannot be effectively implemented, which leads to the exclusion of autistic children from teaching and training, making it even more difficult to attain effective training results [43].

In addition, the ability of role play or transposing their thinking is a very difficult skill for autistic children, and they cannot understand their social relationship with others, from the perspective of the others, by using empathy or their imagination [43]. Moreover, they are often unable to identify the non-verbal social cues in different complex social situations. It is difficult to effectively teach autistic children to understand special and complex social situations, or to develop further empathy. It is a complicated and abstract social structure for them, which makes it challenging for them to acquire such concepts and social skills [17, 18].

1.6 Key factors of good social intervention training strategy

For children with autism, the key point of a good social intervention strategy is to provide powerful visual media in this training game, to attract their attention and maintain their interest [17]. Compared with oral expression, or the written communication performance, image information is one of the best ways of learning. Temple Grandin, who is one of the most well-known and accomplished adults with autism [44], once described in his work that all the words and dialogs presented before the eyes of autistic patients are like pictures, which represents the unique visual learning approach and learning ability of autistic patients.

However, past research has pointed out that the traditional way of matching a picture with a text is not attractive to autistic children, and the effect is not significant [21]. The reason for this phenomenon is that traditional teaching strategies lack interaction and a clear teaching framework, and it is difficult for them to present or demonstrate an appropriate interactive situation. For example, traditional storybook combines social storytelling methods for teaching; however, storybooks can only provide fragments of situational pictures. (Situational pictures, like the normal photographic capture of a real scene, which we call a “montage” or a “stop motion video”, include some people’s interactions and social reciprocity behavior in a specific place. This material is used to help the therapist to teach the children and to indicate their roles and their interaction with it). However, it is difficult to show continuous details from situational pictures, and traditional storybooks lack an interactive mechanism, and many details can only be imagined, which causes autistic children to get bored or to lose interest.

2. A/VR technology’s great breakthrough in a visual sensory experience

With the intervention of Augmented Reality (AR) and Virtual Reality (VR) technology, these training problems have been improved, to varying degrees. Due to an autistic child’s innate visual and learning advantages, A/VR technology provides users with a great breakthrough by providing a visual sensory experience [45]. This includes the most basic interactive game operation for complex multi-person perspective exchanges, self-reference imitations, and a variety of situation simulations [42, 43, 46].

2.1 Overview of AR and VR technology and their benefits and differences

AR technology is an interactive display platform that superimposes virtual objects in a real environment. Using the computer, it generates 3D image display, sound, text and animation effects to enhance the user’s visual sensory experience. Therefore, it is good for pretend-play and symbolic play games to use AR technology, because they can augment the virtual 3D materials by overlaying them on the real objects to make it look like another thing. For example, by using AR technology, we can add the 3D model or animation overlay to real matchboxes to symbolize cars (Figure 1). By using the AR app, the matchboxes will look like the shape of a car on the screen, which can help autistic children to pretend and imagine that the matchboxes are cars. This concept can be easily understood from Bai’s (2014) research article [47].

Figure 1.

AR technology is used to help children with autism to elicit pretend-play and imagination skills, such as imagining a square matchbox as a car. The original picture was produced and re-drawn from Bai [47].

Another VR technology provides a more immersive and realistic environment in which people can experience the environment. VR provides visual images on the media to help children with autism to have an in-depth experience, to master their imagination, and to experience visual sensory stimulation. Thus, VR technology has another benefit in that it lets the children with autism become immersed in another world and to empathize with another person’s perspectives, feelings and thoughts. Therefore, the goals of AR and VR technology can be very different in the way that it presents the virtual environment, by either enhancing or changing the perceptual world.

The biggest difference between AR and VR is that AR integrates with real objects (Table 1), while VR is completely immersed in a virtual environment. AR presents an interaction with the objects in the actual field, such as the social skills training that is used in pretend games or iconic games, while VR strengthens the experience of simulation and the environment of situational feelings, which means that they have different functions and characteristics in social training for autism. Therefore, we will show some examples in the following chapter, and explain how AR and VR technology are applied in the social skills training of autism.

		AR technology	VR technology (including VR CAVE)
1	Technical Description	AR technology can add additional visual information on the surface of specific objects, space environments, or image information, and its application in teaching has been effective	Enhancing the perceptual state of an immersive perception environment.
2	Technology core	AR technology can help autistic children with their expressions through self-reference and watch their social status with others through pretend and symbolic games, or from the perspectives of different persons	VR can provide different situations for autistic children to speculate about their current immersion feelings and to master environmental information through the immersion environment.
3	Execution strategy	Additional virtual information is superimposed on physical space objects, e.g. the 3D virtual car model is overlaid on the box, making it look like a real car. It is usually combined with board games or pretends to operate.	A head-mounted display is usually used to exchange perspectives in immersive situations, to strengthen and experience empathy
4	Learning mechanics	Pretend-play and symbolic play (imagination training), self-reference and imitative learning	The role perspective exchange of the first person and the third person (taking a perspective and empathy construction), and social situation simulation
5	Social Skills Enhancement	Strengthens the mastery of (1) pretend play and symbolic play (imagination training), (2) self-reference and imitative learning.	Strengthens the behavioral cognitive training of (3) role perspective exchanges of the first person and the third person (taking a perspective and empathy construction), and (4) social situation simulations

Table 1.

The differences of AR and VR technology in social skills training for autism.

2.2 Advantages of A/VR technology in the social intervention training of autism

A/VR technology for the social intervention training of autistic children has developed to another different sensory level and visual cognitive stage and, from the general game interactions, different teaching interaction strategies have gradually developed. One by one, they correspond with the different social aspects of autistic patients, such as teaching them how to pretend to be others and to symbolize things through AR technology games [47]. In addition, empathy, imagination, and non-verbal social communication skills are the abilities that ordinary children should have when they are involved in social interaction with adults. These abilities will also be indirectly reflected in the performance of their social skills. Therefore, A/VR technology has considerable advantages in the social training strategies of autistic children, not only because it can reproduce and simulate different situations, social roles, and other perspectives, but also because it is advantageous for the different psychological levels of vision. For example, VR can provide different situations for autistic children to speculate about their current immersion feelings [43, 48], and the 3D modeling technique in AR technology also allows them to practice social reciprocity with specific characters through pretend-playing and role-playing [43]. In addition, AR technology can help autistic children to express themselves and watch their social status with others by playing pretend and symbolic games and seeing things from the perspective of different people [49]. A/VR technology has greatly broken through the traditional training framework, but at the same time, it has extended the training on the basis of the existing teaching strategies and theories, giving it a certain theoretical basis for development, as well as a learning framework [50].

2.3 AR technology solves the problem of a weak imagination in children with autism

Because AR technology can add additional visual information onto the surface of specific objects, space environments, or image information, its application in teaching has been effective [51]. For autistic children, AR technology can help to overcome their weak imaginations, as it is capable of presenting visual information beyond words or pictures [52]. Moreover, the superimposed information can make autistic children perform the operation and teaching application of disguise and symbolic games. Below are some special teaching and training cases to illustrate and analyze the follow-up research.

2.4 AR technology can provide autistic children with a training situation by using the content of social games in disguise and symbolic play

In the process of learning how to interact and play with others, impersonation and symbolic games play an important role for normal children [53]. The children can try to figure out different situations and things and even have a tacit understanding between each other, by pretending to be in a game [54, 55]. For example, in the game of pretending to have a specific role or occupation, such as a doctor, nurse, or salesman, normal children can use the available items at hand as accessories, in the process of their diagnosis and treatment, or they can use the items on display in stores, such as using bananas as telephones [56], using matchboxes to symbolize cars (Figure 1), using sticks to symbolize a king’s scepter, or using a bottle as a microphone. In general, normal children can achieve the operation of each other’s games by defining different objects, formulating game rules and interactive methods, and even developing them into more complex game mechanisms. For example, children are often “playing house “or “playing grown-up” games (playing house, is a traditional children’s game. It is a form of make-believe, where players take on the roles of a nuclear family). They are game forms that help children to understand each other’s gestures and social interactions and to grasp metaphorical symbols. In the process of playing these games, they need to use a lot of imagination, to follow the rules of the game, and to have the ability to understand the concept of the symbolic form and the ability to guess the intention [37, 52].

These games not only retain specific symbols and rules, but they also integrate many different ways of interaction. In addition to the game, some more subtle interactive information helps children to learn and familiarize themselves with more complex social skills in the process of the game. These techniques can convey different information through oral or non-verbal communication and they can also involve social and emotional connections. People will use some conventional objects or gestures to replace the content of complex dialogs. For example, certain body movements, including nodding, shaking hands, hugging, high-fives, and shoulder-clapping, are used to express the participants’ concern and interaction. Moreover, different body movements will naturally convey the distance and intimacy of their social relations. However, there is a certain degree of complexity that is required to implement such a social approach with autistic children, by using traditional teaching strategies. Because autistic children find it difficult to understand complex and abstract social communication codes, the symbolic concepts and social cognition content need to be translated [57, 58].

Therefore, this is an opportunity to use AR technology to help autistic children to understand and participate in pretend-games and try to use their imagine to decode the social signals, because pretend games usually require an imagination to figure out some state that does not exist on actual objects, for example, imagining a square matchbox as a car (Figure 1), or imagining some actions and processes in the game, such as using a toy car on the table to imagine a real car running on the road. In this way, AR technology can today superimpose the 3D model and animation of the car and overlap it on the square matchboxes, to let the matchboxes look like real cars. In this way, it can help autistic children to master the skills of pretend-play, by using visual methods.

3. How to enhance social interaction skills in children with autism by visualization

3.1 How to help children with autism translate complex and abstract non-verbal social communication codes

Non-verbal social communication ability covers a wide range of social cognitive levels and sensory ability integration, including judging the other party’s emotional expressions, their body gestures, empathy, and the ability to pretend to play games and imagine things [59]. These non-verbal social cues convey the way people communicate and their ability to convey emotions [60]. For autistic children, such content needs to be visualized, or even structured, to achieve a better performance of social details and social connections. How to translate abstract and complex non-verbal social cues is a research topic that makes researchers curious, because non-verbal social communication often represents some symbolic social communication symbols and behavior performance, which are conveyed to others through body movements or facial illustrations that involve the relationship between physical and social communication symbols [21]. For example, when a man hands a red rose to a young woman, the series of hand and body movements and their facial expressions convey each other’s behavior, intentions, social relationship, and even their emotional feelings [28] (Figure 2). The use of situational pictures can facilitate and develop a child’s ability to adapt to social relationships and to understand the overall interactive situation described in this series of story plots, by repeated learning and guessing. However, these abstract non-verbal communication expressions appear to be difficult for autistic children to understand [61], as they are unable to deconstruct the information content and social communication codes conveyed by such non-verbal communication. They need to make an extra effort and give extra attention, in order to detect these different social context fragments and non-verbal social cues.

Figure 2.

When a man hands a red rose to a young woman, the series of hand and body movements and their facial expressions convey each other’s behavior, intentions, social relationship, and even their emotional feelings [28].

3.2 How to help autistic children deconstruct visual media and focus on specific non-verbal social communication codes

As described in a previous study, attention is one of the key factors in social cognition and in the deconstruction of social communication codes [62]. The study points out those autistic children often find it difficult to detect and perceive these key non-verbal social cues. However, traditional social training, whether by using video modeling [4, 53] or specific social story methods [36], did not have effective outcomes or help autistic children to deconstruct and understand the social cues in situational pictures. In such training, therapists or special education teachers guide autistic children to observe specific situational pictures or play the videos repeatedly, through which they gradually become familiar with the social details in the story. After multiple practices, autistic children can gradually grasp such social information and apply the information in their daily lives. This kind of teaching strategy is a very common approach in social training; however, it is considered to be a rather passive method and does not attract the interest of an autistic child, even with films or picture storybooks.

3.3 How to help children with autism to master the key to visual guidance and the contextual information behind it

On the basis of our previous study [21] extracted non-verbal social cues from a series of dynamic life films and compiled them into a social storybook. Through the fixed visual structure of the storybook and the sequential page-by-page learning framework, autistic children can clearly grasp the visual pictures and clues of the occurrence of the social situations. A fixed and structured visual information deconstruction method, which researchers call Stop Motion Video (SMV), can guarantee a simultaneous and stable visual interpretation, just like traditional social storytelling [18].

Although the key frames are structural, they are quite fragmentary [28]. Such social storybooks lose most of the social context clues and story content. At the same time, static images do not easily attract the attention of autistic children and encourage them to read. Therefore, by combining them with AR technology, researchers can use the key social context clues in social storybooks as visual primers, and they can also use the explicit learning framework and visual structure of social storybooks, to help autistic children deconstruct and master the specific non-verbal language in a series of complex social films. For example, the social interaction state, a handshake, a hug, or special eye contact, and the expressions of male and female protagonists in the film, are employed to guide autistic children to master the key events of the complete story by using a visual primer that is constructed on the fixed screen. The situational information that is guided by the back of the protagonist can then be played in a dynamic video and superimposed on the social storybook, for autistic children to refer to and read repeatedly, thus making up for the lack of information in social storybooks. At the same time, focusing on non-verbal social cues by using AR technology, successfully increases the attention and motivation of autistic children. The film that is triggered by AR technology promotes the opportunity and vision for autistic children to understand the more complex stimulation of social cues [21].

3.4 How to help autistic children master the relationship between non-verbal social communication messages and symbols

Non-verbal social communication covers a wide range of levels, including the structural relationship between people and symbols. Different body movements that correspond to the extraction of different symbols, also endow the situational story with a specific social narrative framework and clues [63]. For example, a man with a bunch of flowers kneeling on one knee before a woman may represent a marriage proposal, while taking out a diamond ring represents a wedding ceremony. These social cues and symbols will be continuously expanded upon in the process of a child’s growth, as a part of their acquired learning. Generally, children obtain more social experiences when interacting with adults, which help them to apply the non-verbal social communication information and signs, and to establish a connection between such information and signs [28].

As indicated by the playing-house game, children become familiar with the strategy of the disguise and the symbolic game, which leads them to deliberate on the situation and to feel during the process. However, these social information and symbols are full of metaphorical and social content for autistic children, who find it difficult to decipher the invisible and abstract social communication codes. Therefore, additional visual aids can help them. For example, the abstract concepts, such as honor, friendship, peer recognition, praise, or respect for others, could help them to understand some social implications and emotional components, and thus to eventually develop the ability to handle the mechanism of social interaction [37]. Generally, children can figure out these skills through symbolic and pretend-game strategies, such as giving a badge to represent honor, bowing to each other as a symbol of respect, offering a handshake as a symbol of friendship, giving high-fives for peer recognition, or giving a thumbs-up as a symbol of praise. Such content often appears in children’s game stories. However, the question is whether such ways can trigger and motivate autistic children to devote more attention, and their imagination, to specific non-verbal social cues and to successfully link them to a symbolic ceremony.

3.5 What are the benefits of AR on social skills training on children with autism?

Nowadays, AR can play a very important role [37]. The relevant literature points out that it offers a significant breakthrough in the disguise and symbolic game strategies, because the additional visual information given by AR can enhance an autistic child’s attention and mastery of specific social cues [21]. At the same time, it also gives some abstract symbolic content that is different from any visual sensory interpretation, such as giving a visual perception of abstract symbolic content at the moment when the perceptual acousto-optic feedback of an event is triggered [37]; for example, when courage is acquired, AR systems provide animated responses for children with autism.

3.6 AR can help autistic children to comprehend the abstract and metaphorical social relationships

We can understand the benefits of AR technology from different perspectives. For example, in metaphorical social relationships, it is difficult to explain to autistic children about the relationship between you, me, and him. We have always called them pronouns, which are a common part of speech, but they are the concepts that make children with autism most likely to encounter confusion. Autistic children often reverse their status of use, which may be one of the reasons why it is difficult for them to change their role status or feel empathy for others.

Moreover, it is not easy to explain the relationship between social intimacy and intimacy. Usually, when one teaches ordinary children about the differences between the concept of family members and their relatives (such as uncles and aunts) and even neighbors and friends, we can easily convey a distant relationship by using an oral description or some other explanations. However, autistic children find it difficult to grasp such a concept, because of their relationship with society; so they become obstacles in the transformation of the concept of communication. An oral description lacks a clear visual framework and image structure to help them understand. Therefore, it is necessary for autistic children to deconstruct different social concepts and situations by using visualization and images. One can use a Concept Map (CM) plus AR technology to deconstruct different social concepts and situations, by linking different role objects (Figure 3). Autistic children can then understand such social connections through visual aids. In addition to the social content that can be taught, AR provides auxiliary content, so that the visual interface on static images can be extended and generate more visual stimuli, which can subsequently attract autistic children to invest in more attention and observation [21, 49].

Figure 3.

A concept map plus AR technology provides additional visual aids to help autistic children to construct abstract concepts and enhance their learning motivation and attention [17].

Through AR technology, we can give different visual interpretations to abstract social concepts. In the past, autistic children could only understand social content through role-playing and social stories. However, with the intervention of AR technology, some social content can be presented and interpreted in different ways, and can extend the sensory level given by static images or inanimate objects, such as teddy bears, dolls, or a doll in human form (Figure 4). The therapist uses these entity dolls to explain the situational dialog and emotional feelings between different characters to the autistic children, by using disguise and symbolic game strategies. However, in the process of a pretend-game, the dialog content, body movements, and the palm of the eyes are included. This becomes a very difficult task for autistic children, because their imagination is inherently weak, and they do not easily associate and interact with each other. The rigid thinking mode makes it difficult for them to understand each other’s communication, as well as the real intention behind the words and the social meaning that they hope to convey.

Figure 4.

Personification is when something non-human is given a human characteristic/personality. However, personification is sometimes given a broader definition. Therefore, in this section, we focus on enhancing the autistic person’s empathy towards others and trying to think, feel, and understand the emotions of another person, from their perspective. The AR system will turn an inanimate object into a real creature.

3.7 AR technology can be used to give new life to role-playing games

Through the superposition of 3D animation and situational sound, AR technology can easily solve the problem of a weak imagination in autism. A therapist can easily make a lifeless doll appear to be alive (Figure 4). Through the man–machine interface and context design, the social interaction content can be successfully deduced [64]. In addition to increasing the motivation and attraction of autistic children in learning, the whole process also involves the sensory connection of entity interaction, which could help them to acquire this social concept. When autistic children play with the entity doll, they can link the social cognitive relationship between the senses and the body, which reinforces their understanding of a social relationship. In addition, AR technology gives a deeper feeling to the physical interaction and sensory manipulation required by the disguise game. From the perspective of the third person, the social interaction relationship between the disguised role and AR technology can become a more complete social training mode and make the social training of autistic children, from a visual sense, gradually push to the level of a psychological feeling. Next, the researchers will explain how to further use AR technology to transform the transposition thinking ability of autistic children and to trigger their inner ability to show empathy.

3.8 AR technology can help autistic children to transform and achieve transpositional thinking and empathy training

Empathy is seen as the ability to perceive another person’s emotions, intentions, and feelings [65]. In the growth process of ordinary children, it is an innate ability and can indirectly become a skill for social interaction between people [66]. If they have this ability, people will understand another person’s feelings through transpositional thinking and then be able to predict and adopt his/her behavior representation and social communication mode [67]. However, this ability is difficult to achieve in autistic children and it is closely related to non-verbal communication cues. In general, normal adults can understand the feelings and intentions of others through their facial expressions, body movements, voice intonation, and even some of their gestures and their eyes, which are all non-verbal communication clues that could transmit abundant social information. People can use them to perceive the emotions and intentions of others and to make further judgments. Through the accumulation of experience and learning, people’s judgments thus become more accurate and effective.

3.9 Shortcomings of traditional transposition thinking training

In the teaching strategy of the transposition thinking ability training for autistic children in the past, facial pictures [68], or asking autistic children to look in the mirror, was often used for repeated judgment training [69]. However, this method has not been effective, especially when autistic cases enter a real-life field, whereby such training is difficult to implement and the effect is not obvious. The researchers pointed out that it is necessary for autistic children to be proficient in this skill by using more flexible and authentic sensory stimuli. This also shows that it is difficult for autistic children to use simple images and integrate their creative imagination into a more profound on-the-spot experience, and such a perceptual environment can only be achieved in real life. However, some autistic children have a fear of the real environment, and the complexity of the information in the environment makes those with poor communication more afraid and likely to retreat, which means it cannot really achieve the purpose of training [70]. Therefore, transpositional thinking skills and empathy (also known as defects in the Theory of Mind (ToM) - the ability to interpret another person’s interests, intentions, and emotions) has always been the focus of researchers in the training of autism [71]. However, there are many difficulties in the training and implementation of a person’s transpositional thinking ability. The first difficulty to be faced is how to let autistic children figure out how to take on another role? It is very difficult for them to show their abilities, because they cannot understand why therapists ask them to show the emotions and feelings of another character, or even to pretend to be another character, because they have a weak imagination and rigid logical thinking [72]. This is a rather difficult task for autistic children to understand. These communication methods are not in line with their real identity, they are abstract, and cannot be interpreted through a visual mode. For autistic children, who understand things through vision, it is only when they can truly see, or feel, the social status of different roles that they can solve the current training problems [43].

3.10 AR technology can help autistic children to realize a social interaction experience from multiple perspectives and senses

With the application of AR technology, a person’s transpositional thinking ability has been greatly improved. AR can simulate the facial expression of patients through self-reference (Figure 5) [46], Unlike the traditional training strategy, where autistic patients are asked to look into a mirror to guess, and learn about, emotional expressions, which is especially futile for them [69], AR technology can give correct facial expressions and superimpose them on the faces of autistic children [46]. The training methods listed above (Figure 5), whether they are looking at their own facial expressions in a mirror or imitating their own social actions, are collectively referred to as self-referential imitation, especially the visual reference of social cues (such as facial expressions, body movements, voice intonation, etc.). Whether viewed from their own first-person perspective, or from the perspective of others (a third-person perspective), it is a very special visual sensory experience for autistic children [46].

Figure 5.

AR technology can achieve a self-face reference training strategy to solve the dilemma of autism in social skills training [46].

3.11 The difference in the implementation of AR and VR for helping children with autism to empathize and improve their transpositional thinking

AR can help children with autism to implement empathy skills, by using board games. The intervention method of AR technology is usually to superimpose the content of different media on an object. For example, AR can superimpose different 3D virtual facial expressions on the autistic patient’s face to help them think about the different kinds of emotions that are present in themselves. We call this as self-facial modeling (Figure 5), which was previously impossible to achieve, but it can be done by using AR technology. In addition, AR can provide some virtual 3D decorations for children to play with, or it can augment some virtual 3D animations on their body by self-reference, which are shown on a screen, to help a child pretend he is a king or a specific character (Figure 6). In this form, AR will become a tool to help autistic children to imagine and pretend, and they will gain the ability to empathize from it. This strategy lets autistic children play the game of empathy through the mechanism of pretend-play. Most of the AR methods are used to manipulate or passively perceive the comparison between real objects and the virtual content, in order to obtain the feeling of empathy.

Figure 6.

Using AR technology to help children with autism to elicit pretend-play and symbolic play (a crown means a king, or courage). An autistic child can also play a self-role game; its meaning is different from a table game, because participants can join this scene face-to-face and play with others who are also self-imitating.

From another perspective, VR provides a completely immersive picture experience, but it has different effects, compared to AR. The current VR technology can change a person’s judgment of the senses, which allows one to fully integrate another person’s perspectives, and this method can make one see the world through another person’s eyes. In addition, through the exchange of different perspectives, the individual can even see the state of interaction between himself and another person, through the eyes of others, or he can experience and observe the type of social interaction from a different personal perspective. This approach greatly changes and breaks through the VR of the past, as a mechanism for on-site experience, but with the performance of empathy, and it helps users to think from the perspective of others and to master the skills and abilities of empathy. In addition, the empathy experience provided by VR is felt from the first-person perspective and is achieved through the mechanism of exchanging perspectives (Table 2).

		AR technology	VR technology (include CAVE VR)
1	Viewing angle (perspective view)	From the third-person perspective, a person must try to figure out the feelings of the posing object and watch the superimposed animation to understand the interactive content.	The game experience method, from the first-person perspective, helps autistic children to engage directly in face-to-face social reciprocity activities with virtual characters, and to master skills.
2	Special visual situation to achieve empathy and transpositional thinking skills	Visually, it can be transformed into a state of empathy in a fake game, by comparing the relationship between the virtual and the real state. For example, through the teddy bear doll in the hand and the realistic animation of AR, the doll becomes a real bear and one must to try to figure out his feelings.(Figure 4.)	Visually, it can switch to the perspective of different characters, such as viewing things from the perspective of friends, or viewing yourself from the perspective of others, and feeling the feelings of others.
3	Teaching and implementing the mechanism	Self-imitation, self-modeling, pretend-play game, symbolic game operation,	A multi-person perspective exchange, direct face-to-face interaction

Table 2.

Differences in the implementation of empathetic and transpositional thinking skills between AR and VR.

4. Some specific situations in VR presents

Similar concepts can also be presented in the creation of science and technology art. For example, the work is called the inter dis-communication machine created by the Hachiya [73]. The Dis-Communication Machine is a communication system that is aimed at transmitting and receiving sensual experiences used by two people wearing head-mounted displays. The ‘machine’ projects one wearer’s sight and sound perception of the environment into the other one’s display, thus confusing the borders between the identities of ‘you’ and ‘me’. Through the exchange with the world, from the perspective of the different role objects, the participants can understand and feel the sensory world visually [73]. In the past, this visual sensory effect was achieved through the exchange and transmission link of entity lens images. Today, VR technology can achieve this surreal sensory experience. The case subjects can view others (first person) from their own perspective and interact with virtual characters, or view themselves from the perspective of others, or even watch their interaction with others (third person) from the perspective of another person (Table 3). These switching perspectives successfully achieve the visual thinking mode of transpositional thinking in the virtual world. For autistic children, the empathy training strategy can promote the physical sensory experience synchronously. Whether immersed in a specific virtual environment, or watching themselves interact with virtual characters, from God’s perspective, they have achieved different visual sensory experiences, thus forming another training application of self-reference. This training framework has also been preliminarily verified by experiments, which proves that autistic children can still achieve the common perception of psychological and physiological perceptions, with the help of technology. Autistic children can effectively perceive and express empathy, which encourages them to use the mechanism of social reciprocity, and enhances their social skills [43].

Table 3.

The self-reference application of a synchronous visual sensory world achieved by CAVE-like immersive 3D technology (a-autistic participant, A”-(participant’s virtual avatar captured by front or back camera), B-virtual character).

4.1 Unique visual and sensory advantages of VR technology

Researchers believe that VR technology has some unique visual advantages in the training of autistic children, which can form surreal visual and sensory benefits [43], such as a role perspective exchange of the first person and the third person (taking a perspective and empathy construction), and social situation simulation (Table 3). Compared to the existing teaching strategies, VR technology has considerable differences and has made major breakthroughs [45]. As a compensation mechanism of visual sense and additional training advantages given to the visual characteristics of autistic children, VR technology can encourage them to face more social objects and to observe the behavior and reactions of others. Moreover, these external stimuli can increase the possibility and opportunity of real social interaction between autistic children and their peers. Therefore, expanding these non-verbal communication training units, as well as visual assistance and additional sensory compensation, can help autistic children to reinterpret and solve their problems, even though they have a non-verbal social disorder and a weak imagination. The unique visual and sensory advantages of these VR technologies have repeatedly reminded researchers that they should make good use of VR technology, so that it can be applied and brought into play in these unique special education fields. In the next section, we will further explain and analyze the visual and sensory advantages of these VR technologies.

4.2 VR technology can help autistic children to widely connect to a real social environment and different social situation stimulations

As described above, the past training in social reciprocity behavior would make autistic children come into wide contact with real social environments and different social situations, which can promote their skills in social reciprocity with others. VR technology has sensory advantages in situation simulation and social training, especially in the early stages of its application. It is often used to solve the problems that autistic children are afraid to face in the real world, and to help them to simulate and experience different situations, such as going to school, taking busses, shopping, or to solve different levels of social fear problems [70]. However, most of these training styles are still in the stage of scenario simulation, which is also the most common and best operating mechanism of VR technology. Situation simulation solves most of the social training problems that cannot be repeated in real situations, such as helping autistic children learn how to complete a certain task under different situations (such as customers checking-out in front of a cash register, or ordering meals for customers by using an ordering machine). These training mechanisms are mostly aimed at solving the problems that autistic children need to face and overcome in real society, and to encourage them to be able to enter the workplace, in order to integrate better into the society [30].

Most researchers recognize that VR technology has several major advantages. For example, the situation simulation training developed by VR technology can be used for repeated situation training and task operations and can help the children to overcome some situations, which may be difficult to do in the real world. Another example is that it allows autistic children to experience social interactions by using different roles. However, it is undeniable that social interaction in the face of a real life environment cannot be replaced, and so VR technology is generally recognized as an early-training framework that can help autistic children to integrate into a real-life environment. When an autistic patient is unable to successfully face a real social field, he is considered to be in need of training through role-playing and placing him in social situations. He can then carry out situation experience and social simulation training under the game mechanism of specific social situation guidance and role exchange. However, the inborn social defects of autistic people often make them resist such game mechanisms. The main reason is that they are not interested in the interactive content, or that it is difficult to make a social conjecture of game interaction. These problems are related to their innate weak imagination, their inability to think transpositionally, or their own rigid behavior problems. Therefore, it is necessary for VR technology to move from the framework of situation simulation and to gradually cross over to the psychological and sensory level, because then it will obtain better results. The following section will provide some suggestions for AR and VR, respectively.

5. Discussion

5.1 AR technology can be used to guide social visual cues and provide additional extended information

Whether in non-verbal social communication or in different visual image information, AR technology is considered to be a scanner. Through a hand-held tablet, PC or other imaging devices, any image information can be converted into a scanning icon that is suitable for AR application (Figure 7). Moreover, through lens sensing in front of a hand-held tablet or PC, the AR app can overlay dynamic information images (such as 3D animations, additional visual information assistance, videos and audio media, etc.) behind the static images. Researchers call this a framework approach for visual cue guidance, which means that it constructs an autistic child’s sensory cognition of the visual guidance through a fixed visual framework (they are usually static images, such as social storybooks, which are commonly used in autism training). With a fixed visual cue framework and sequential story guiding the content, children with autism can quickly grasp the main structure of social stories through visual images and they can then extend to more complex and diverse social content, or context, details. This division of labor can greatly reduce the cognitive load of autistic children in their understanding of information, through the hierarchical information of AR technology, and it can also attract the attention of autistic children with the help of the audio-visual effects of AR technology (Figure 7). The information provided by AR includes different layers of visual information, and complex social information is deconstructed by using a visual architecture. Moreover, with the support of AR technology and visual cues, the extended social information content is supported and improved [21, 28].

Figure 7.

AR realizes hierarchical visual indexing. Firstly, the fixed visual structure is used to help autistic children master the social context framework. Secondly, AR technology is used to link further videos and audio information [21].

5.2 AR technology can help autistic children to master complex social content through a structured visual architecture and an entity interaction strategy

Visual information can be deconstructed, and then presented by linking CM (Figure 8) or sequence learning [38, 74]. Since people’s reception of sensory stimuli depends on the interaction of their vision, hearing, and body senses, autistic children are endowed with a touchable “concept of objects”, which can help them to clarify the symbolic meaning that is conveyed “behind the things”. Moreover, such a form can also be used to teach autistic children to understand the state of social stories. For example, the storybook itself is a symbol that can help to construct structured concepts. By reading the situational stories in the storybook, we can help autistic children to construct the visual state of situational concepts and the sequence of events. Just like operating tools with people, we can construct a smooth operation mechanism between the mind and body, through repeated practice. Although the mechanisms of tactile, visual, and physical perceptions in sensory acceptance are different, autistic children can use their common sensory memory to grasp complex social content and sensory information, so that they could learn to deconstruct continuous social actions and interaction concepts.

Figure 8.

Concept mapping can help autistic children to form visual and abstract social relations and help them to understand and master the content of a social situation [17].

Through the interaction strategy of a structured visual concept and entity, a visual framework, or visual primer, can be established to help autistic children understand and deal with abstract social relations, such as the distance between relatives in social relations, the social connection between different roles, or role playing and the speculation of transpositional thinking (Figure 8) [49]. In addition, the 3D character animations provided by AR also have the advantage of multiple viewing angles. Autistic children can watch the social relations of different character objects by different role identities and then speculate on the game. At the same time, through the game’s dolls in the board games, such a learning framework can become a key training strategy for understanding social games, or disguising games, with the aid of the concept mapping and visual content that is superimposed by AR technology [50]. By combining the game operating mechanism with the board game concept, players can easily pretend to take on the role of God (or the third-person perspective). In this way, we can at the same time solve the problem of the weak imagination of autistic children. As AR animation directly makes up for the visual sensory information, which is difficult to show in the operation content, it provides effective social animation content to help autistic children understand the social status among different roles (such as hugging family, waving at friends, schoolmates clapping encouragement), supplemented by fixed visual guidance, which constitutes a more stable and dynamic social interaction structure [17].

5.3 VR technology can provide an alternative mechanism of different multiple viewpoints, an immersive virtual environment, and an immersive theater game environment

VR technology is a great breakthrough in the field of autism social training. It can change the visual training strategy under different sensory conditions and through multiple viewpoints (‘multiple viewpoints’ means that there is a perspective shift of being able to look at something through another person’s eyes. One of the biggest influences that multiple viewpoints have is that they enable us to see how others view the world, which enables one to experience another person’s sense of sight face-to-face; of course, we just let the autistic person wear the head mount display, in order to display the other person’s viewpoint via a camera), and an immersive theater game environment. Moreover, it contributes to the scale, from being a table game size to a sensory experience with the whole immersion environment, and even to the development of a semi-immersive Mixed Reality (MR) training mechanism. Such content has a different training purpose and different game strategies on the basis of training. Through visual experience and fixed visual guidance, researchers can enlarge the scale of board games to the framework of immersive theater games. Using the same visual concept strategy, we can develop a social interaction mode, with the first-person perspective and God’s third-person perspective, and then, through to the immersive and semi-immersive interaction experience, we can construct the participants’ on-the-spot sensory experience of the social interaction objects. These visual senses can be used to guide and induce empathy through the visual sensory mechanism provided by VR technology, and they are applied in the following situations, as described previously: (1) first-person and third-person role perspective exchange (empathy and empathy construction), and (2) social situation simulation. The operation of this mechanism requires more situational guidance and sensory conditions, combined with different game and entity interaction strategies, to achieve the training effect, and such a training form needs to be designed and interpreted through a situational script. Through such a mechanism, autistic children can gradually acquire social skills and non-verbal communication and it will equip them to live their own lives.

6. Research limitations

In real daily life, children with autism have to face many obstacles, with regard to social reciprocity, most of which are reflected in the mastery of some non-verbal social behavior. These social actions and reciprocity behaviors are not impossible to achieve in real life, but because of the innate social barriers of autism, they cannot master social skills in a more natural or appropriate way. In such a state, it is difficult to use traditional methods, such as VM or storybooks, to train them. However, AR or VR, as a visual method and teaching strategy, helps them to “see” the situation, to feel, and to have a status in these states. It even helps children with autism to understand the feelings, the states, or social reciprocity of others by multi-person perspective exchange, self-reference imitation, and a variety of situation simulations. These social actions and reciprocal behaviors usually have to be felt and communicated through empathy and inner mechanisms, and are not tangible skills (because the skill of empathy is not like riding a bicycle, as long as you can master it with proficiency, it needs the ability to function effectively in a dynamic environment in which multiple, and substantively different, perspectives must be maintained). Because empathy is an inner state and a social cognition mental skill, A/VR technology provides an attempt to help autistic children to master this ability, but not all feelings can be expressed. Children with autism still need to rely on repeated training, or to learn this skill with the growth of life experience, so this article is only a stepping-stone to help researchers to think of other ways and other methods of thinking about social training issues, to get rid of traditional memorization method, and to become a more flexible training strategy (Table 4).

	Difficult to achieve non-verbal social skills training in the real world	Main technical assistance	Non-verbal social skills to be strengthened (training purpose)	Teaching strategy description
1	Self-facial expression reference and limitations	AR	Master emotions and facial expressions by observing the face and facial expressions, such as head movements; (refer to the Figure 5)	It is very important for children with autism to observe the correct expression state that they should make through AR technology, and, especially when they cannot master the correct emoji, AR can solve this problem
2	Give life to items in the pretend-game	AR	Master the state of social and physical interactions through pretend-play games.; (non-verbal social communication skills, such as eye contact, social body movements, social ritual behaviors) (refer to the Figure 4)	In the real world, some inanimate objects cannot truly interact, but with the assistance of AR technology or VR technology, these inanimate objects can become real characters for interaction (for example, an inanimate teddy bear becomes an alive role)
3	See the relationship between each other through virtual information	AR	Improve the grasp of social relationships with the aid of visual information (refer to the Figure 3)	There are many intangible social relationships and statuses in the real world. In the AR interface, the social distance between each other can be presented visually. For example, through a combination of the AR and CM strategies, the relationship between each other can be represented by images.
4	Observe the detailed state of different body movements from all angles	AR/VR	Use visual information aids to enhance social interaction and behavior control (facial expressions, social actions); (refer to the Figure 7)	In the real world, there are many details relating to body language. Only in the virtual environment can you repeatedly figure out and observe from different perspectives.
5	Watching and practicing social body movements repeatedly	AR/VR	Use visual resources to enhance the mastery of social reciprocal behaviors (social actions, including nodding, shaking hands, hugging, high fives, and shoulder clapping)	It is impossible to practice social action repeatedly in the real world, but the virtual mechanism can.
6	Socialize as someone else	VR	Grasp the state of empathy through different personal perspectives; (refer to the Table 2)	In the real world, it is difficult to see things as another person, such as thinking about things in different roles, and autism can only be taught through visual images (thinking from the perspective of others). VR technology can achieve this function.
7	Instantly see your social interaction status with others	VR	Grasp the state of empathy through different personal perspectives (refer to the Table 2)	In the real world, only through video recordings and video referencing can you see the state of interaction between yourself and others at the same time, but it can be achieved in real time under the framework of VR.

Table 4.

Non-verbal social skills training that is difficult to perform in the real world.

7. Conclusion

The situational script design and social story context are often two of the most critical and characteristic training materials in the training process. The situational script involves many important training items, including the setting of social reciprocity behavior, the social context content, non-verbal communication body action, the conversation content, the symbolic metaphor story, the role interaction mechanism, and so on. Through the interpretation and visual deduction of A/VR technology, we can achieve the training framework on a visual level and then promote the transformation of social behavior in learning. This depends on a different scenario script design and social story writing. In the execution, subjects are often obtained from the participants’ daily life and applied to script writing. Therefore, A/VR technology has become an interactive platform and game carrier for social training and it also gives these social materials and scripts another visual interpretation. Through the interposition of A/VR technology, autistic children can pay attention to the information details of different non-verbal social cues and transform their communication feelings in the visual images, so that the visual information can be deconstructed and presented sensitively. This kind of training strategy breaks away from the traditional story-book and pure film training mode and expands the understanding level of sense and perception. It has been released in the training of autistic children and has become another new training framework.

The visual training strategy of A/VR technology has gained its initial achievement in the non-verbal social training of children with autism, and the application of this technology has solved many limitations in traditional teaching. However, in the application and practice of this new technology, researchers must adopt a more rigorous experimental design and take a cautious attitude. Especially for autistic children with mental development defects, the use of this technology must consider more clinical and experimental evidence and it needs to be evaluated by more therapists and physicians. Therefore, we look forward to the breakthrough and development of A/VR technology in the field of special education. At the same time, we are also careful to apply the various visual effects of A/VR technology on social interaction stimulation (e.g., body language, social situations, social reciprocity, and relevant non-verbal communication in social interactions). We hope that it will be used better and applied in the future and that its advantages in working with these special children with autism will be displayed.

Acknowledgments

This work is supported by the Ministry of Science and Technology of Taiwan (107-2218-E-027 -013 -MY2/109-2221-E-027-069 -MY2). The authors thank the referees very much for their valuable comments and suggestions on this paper.