How to Create Suitable Augmented Reality Application to Teach Social Skills for Children with ASD

Abstract

Autism spectrum disorders (ASDs) are characterized by a reduced ability to appropriately express social greetings. Studies have indicated that individuals with ASD might not recognize the crucial nonverbal cues that usually aid social interaction. This study applied augmented reality (AR) with tabletop role-playing game (AR-RPG) to focus on the standard nonverbal social cues to teach children with ASD, how to appropriately reciprocate when they socially interact with others. The results showed that intervention system provides an AR combined with physical manipulatives and presents corresponding specific elements in an AR 3D animation with dialogue; thus, it can be used to help them increase their social interaction skills and drive their attention toward the meaning and social value of greeting behavior in specific social situations. We conclude that AR-RPG of social situations helped children with ASD recognize and better understand these situations and moderately effective in teaching the target greeting responses.

1. Introduction

Autism spectrum disorders (ASD) are characterized by social interaction difficulties, communication challenges, and tendencies to engage in repetitive behaviors [1]. In particular, social reciprocity defects are one of the core deficits in social interaction for people with ASD [2]. Social reciprocity depends on the ability to empathize with others, to be aware of emotional and interpersonal cues, and to respond appropriately [3]. In addition, ASD is characterized by an impaired ability to engage in social relationships and can result in serious deficits in the ability to make friends or interact with others [4]. Typical deficits include an absence of appropriate greeting behaviors and a failure to acknowledge the presence of a familiar person [5]. Such deficits in social greeting appear to be common among individuals with ASD [6]. This impairment has far-reaching consequences for social interaction, communication, and imagination [7].

In contrast, typically developing (TD) children are socially interacting with people effectively, including maintaining eye contact, initiating interactions, responding to the initiations of others, sharing enjoyment, reading body language or nonverbal cues of others, and adopting the perspective of another person. However, most children with ASD seem to find it difficult learning how to engage in the give-and-take process of everyday human social interaction. Even in the first few months of life, many children with ASD do not interact, and they avoid eye contact [8]. They seem indifferent to other people and often seem to prefer being alone [9].

2. Related works

2.1. Current training methods for social interaction skills

2.1.1. Social stories™

Social Stories™ (http://carolgraysocialstories.com/social-stories/) is a promising strategy that has been used to teach social interaction skills for individuals with ASD [10, 11]. A Social Stories™ intervention involves creating brief stories that describe social situations, what others are thinking or feeling, and how to behave in the specific situation. As part of describing how to behave, a good social story would also highlight what social cues the person should look for and how to respond to others [12]. Specifically, teaching through the use of a Social Stories™ strategy with a storybook is widely used [13, 14, 15]. Social Stories™ provides a catalyst for change, providing children with other perspectives and options for thoughts, feelings, and behaviors [16]. Previous studies examining the effectiveness of Social Stories™ interventions have yielded varying results, but overall Social Stories™ appears to have a positive effect [11, 17]. Social Stories™ interventions have targeted a range of skills: initiating verbal greetings [10], initiating and responding to interactions [18], and maintaining appropriate social engagement [19, 20].

2.1.2. Video modeling

In addition, another strategy that has been used to teach social skills to people with ASD is video modeling (VM) [21]. VM entails showing the participant a video segment that demonstrates how to perform a task or behavior. The participant is expected to learn by observing the instructional video segment and repeating the modeled behavior [22, 23]. VM has been effective for teaching a variety of social skills: social initiation [24], social language in play situations [25], social engagement [22], and expressive behaviors, such as intonation and facial expressions [26]. Therefore, some studies combine Social Stories™ and VM [27]. This strategy can effectively enhance the child with ASD to recognize and understand emotions in themselves and to generalize them to other situations. Social Stories™ can increase awareness and understanding of social situations; simultaneously, VM can also provide the video for children with ASD to mimic and pretend character’s gesture [28]. In addition, VM has been therapeutically effective for teaching functional, social, and behavioral skills to children with ASD [22, 29, 30]. That is why many experiments adopt this strategy to teach their students with ASD.

2.2. Primary problem with current training

However, most studies in Social Stories™ combined with VM have only a unidirectional way to teach children with ASD to mimic behaviors that they see, but they cannot directly trigger spontaneous social events because children only watch videos and do not directly interact with others. Moreover, most VM presents all flat displays, and children with ASD can only imitate actions presented on single-perspective 2D views of scenarios to understand everything that is going on in the scene; nor do they understand three-dimensional (3D) facial expressions and body movements or how to reciprocate [31]. In addition, although VM is advantageous for promoting the motivation of children with ASD to learn, children still have difficulty dynamically adjusting the size of their attentional focus and switching the locus of their attention [32, 33, 34], especially in patterns that include dynamic, repetitive, or social stimuli [8]; however, that decreased multiple object tracking (MOT) performance is not due to deficits in dynamic attention but to a diminished capacity to select and maintain attention on multiple targets. Therefore, several problems that may occur, such as a lack of progress, could be due to a lack of reinforcement of sustained attention, poor video content, or a lack of prerequisites [35], while children with ASD may engage in pretense when instructed to do so, and they still find it difficult to develop creative extensions [36].

2.3. Benefits of AR combined with physical manipulatives

Fortunately, there are other assistant tools to help children with ASD promote their social skills and also help focus children’s attention on specific target social cues. In some specific cases, AR technology has been proven effective for teaching social interaction skills. For example, AR was effective for helping people select and maintain their focus on the 3D model and helping them better judge the correct mental task [37]. In addition, AR can effectively reduce cognitive load and increase participants’ interest in training [38]. Other studies also indicate that physical object manipulatives can support children with ASD to collaborate and communicate in new ways [39]. Additionally, AR with physical manipulatives provides physical behaviors and aids children’s active learning [40]. Foundational evidence from LeGoff [41] suggests that interaction with physical manipulatives supported children with ASD to collaborate for extended periods of time by helping channel children’s attention and providing a common context for sharing objects and ideas [42]. Physical manipulatives might be particularly well suited to children with ASD because they take advantage of children as active learners, whose experience is grounded in the body and improved through sensory awareness. The kinesthetic learning experience might be ideal for the development of social skills because tangible interface offers expressive activity, programmability, and the construction of moving objects with structural integrity [43]. Hence, most AR combined with physical manipulatives can bring beneficial opportunities that children can simply imitate and as they pretend to do the modeled behaviors, without actually facing the conflict activity in a real situation. For example, Chen et al. [31] used AR technology with tangible facial masks as physical manipulatives to enable three adolescents with ASD to become aware of facial expressions observed in situations in a simulated school setting. They provided 3D animations of six basic facial expressions overlaid on participant faces to facilitate practicing and to pretend emotional status and social skills. They indicated that AR technology can allow children with ASD to pretend their own facial expression correspond to occurring events and can also let the children with ASD pay attention focused on this specific facial part of social cues. Moreover, a physical manipulative interface also enables people with ASD to control the system in an intuitive way and provide immediate haptic feedback during social interaction, and this helps develop social and language skills [44]. In addition, AR environments can emphasize learners’ participation in different roles and can improve the sense of presence, immediacy, and immersion [45].

2.4. The gaps in these studies

In this study, we used AR technology combined with the tabletop role-playing game (AR-RPG) as training platform that focused on social greetings. This system encourages imaginative social activities and allows children with ASD to interact directly with physical role props, as they do in natural circumstance. An AR system can guide children with ASD to express what they understand and feel in an easy and interesting way. We summarized some differences between our research and that of previous studies. First, we focused on social reciprocity behavior training (i.e., greeting behavior), not on work skills or basic daily life skills (e.g., washing dishes, eating food, getting dressed) [46]. Second, our training materials were created using the Social Stories™ strategy and related to each child’s daily life situations at home, in school, and in the community, whereas previous studies created their materials using textbooks or treatment manuals without Social Stories™ strategy. Third, our research focused on children with ASD, not on TD children; most previous studies [47] used AR for children without ASD. Fourth, our AR system was based on markerless 3D tracking technology that uses Qualcomm’s Vuforia to increase its realism. This allowed children with ASD to directly sense which marker is their role-play photo image, which reduces the confusion that many different role-play markers produce. Fifth, our AR system had more advantages than traditional VM and conventional AR systems do (see Table 1). Sixth, our study focused on greeting gestures and related social interaction, not only on facial expression pretend training [31].

Conventional VM

AR-RPG system

1. 2D flat image and video; user can watch content only on monitor; few direct interactions; 2. Content is fixed and cannot change in real time; 3. Primarily single-user based to watch; no interaction; 4. Lack of real manipulation can help facilitate mental skills such as pretend play skills; 5. Most video has single development context; lacks concept of visualization network; and cannot enhance learning experience incentive or motivation; 6. Provides only single view angle from the video; multiple viewing angles are not allowed; 7. User cannot obtain additional spatial information from their surrounding environment to help users recognize the setting

1. Realistic full 3D virtual objects with tangible manipulation controller; user can directly manipulate and interact with setting; 2. Content is not fixed but can change in real time; 3. Multiple-user based; makes learning interesting; 4. Rich entity operability can increase users’ understanding of the relationship between roles and help them to pretend play.; 5. AR with Social Stories™ can establish their social network concept map and also enhance children’s learning incentive and motivation; 6. Multiple viewing angles are allowed; users can see gestures and facial expressions from different angles; 7. Gives extra real spatial information as a reference to help users recognize the setting

Table 1.

Comparison of conventional video modeling (VM) and augmented reality (AR) tabletop role-playing game (AR-RPG) system.

2.5. Developing the AR-RPG system

AR is a live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, or 2D and 3D graphics; thus, AR enhances one’s current perception of reality [48]. It is for developing instructional content that needs to present augmented spatial information to reference and other parts of its features. AR creates mediated space that exists between the mind and physical space by overlaying real space with flexible virtual objects. AR-RPG system is like a “miniature theater” in which children with ASD pretend, using role-play, to participate in specific social events. This role-playing game platform uses a small 3D space where virtual and real spaces are tightly linked together and can accommodate single or multiple users (see Figure 1). We used several settings, a living room, a classroom, and a community (see Figure 2), all of which are familiar places to our participants. We also included virtual furniture or some other objects to make the space more realistic.

Figure 1.

The therapist taught the children how to use the AR-RPG system and how to manipulate the cardboard avatar to observe and pretend (role play) the avatar’s situation.

Figure 2.

Different scenes with corresponding scenario events (classroom and playground in the community).

The AR-RPG takes a series of scenario stories created as training materials that portray the participants’ everyday life activities to focus on training their greeting behavior as it might happen in school, home, and community. For example, we got some information from their daily experience: (e.g., greeting parents, greeting friends, and greeting strangers). Those scenes are useful for children with ASD to be trained while pretending to make appropriate responses to greetings. In addition, we focus on teaching the six types of greeting behaviors of social etiquette, including (1) nodded head and tiny smile, (2) handshakes, (3) waved hand and said hello, (4) kisses their head or face, (5) hugs, and (6) shrug your shoulders; this greeting is often seen in Taiwan’s daily life and also useful to train children with ASD social interaction skill. Beyond the greeting, which may involve a verbal acknowledgment and sometimes a facial expression, gestures, body language, and eye contact can all signal what type of greeting is expected. Therefore, we also add the social story and dialog contents with text description within our greeting scenario, to let the specific greeting behavior situation present more closely to the real status.

2.6. Operational settings and facilities

The researcher created 20 scenario stories after discussing suitable scenes and scripts with each participant’s parents and special education teacher. We designed those stories based on the child who would do the role-play, the greeting response to be learned, the conversation content, the role’s intention, and its emotional status. We let them read the script or watch the scene, and we ask participants to mimic and role-play with therapist, who would help them comprehend. We used AR to construct our tabletop role-play game platform because it allowed us to quickly and easily change scenes. The AR-RPG system has two functions: (a) it is a tangible scene simulator with a physical manipulative function, and (b) it is a vehicle for providing virtual content. In addition, we created each child’s pattern printed on tangible cardboard as avatars to represent each one’s identification (ID) (see Figure 3). Children with ASD can directly handle their own cardboard avatar when they interact with others (e.g., peers, parents, or other characters) or when different characters contact each other at the different situations (e.g., home, school, or park). Our system will trigger the event’s related AR 3D greeting animation based on the background scene (see Figure 2). For example, when a role-playing child sees a neighbor appeared on the street corner, he will say “Hi! How are you?”, wave at the neighbor, smile, etc. By providing a wide range of scenarios that reflect their everyday life and analyzing stories suitable for them, therapists might better understand what actually happens in the lives of their child clients and how the children feel about it.

Figure 3.

Operational scenario and facilities (different cards represent different pattern IDs, e.g., the role itself and others people’s cards, decision about which greeting behavior card, and scenario card to choose).

3. Method

3.1. Participants

We recruited three participants. The three children with ASD (two boys and one girl (Oliver, Peter, and Jane), all pseudonyms to guarantee anonymity) (mean age = 7.73 years old; age range 7–9 years; intelligence quotient (IQ) scores: (a) full scale IQ (FIQ) = 100.67±3.79, (b) verbal IQ (VIQ) = 100.33±4.04, and (c) performance IQ (PIQ) = 101.33±2.08) were enrolled in the mainstream school with other TD students (see Table 2). The inclusion criteria for this study were (1) a clinical diagnosis of ASD based on DSM-IV-TR criteria, (2) no other specific disabilities, (3) not taking medications, (4) no physician-diagnosed comorbidities, (5) not undergoing any other interventions at the time of the testing, and (6) no intellectual disability (a full scale IQ > 70). Oliver, Peter, and Jane had additional assistance from a teacher’s aide in general lessons. Once a week, an occupational therapist provides educational services to their class. The teachers reported that the three students with ASD did not greet or acknowledge the teaching staff or other adults at the school. The parents also said that their children did not greet or acknowledge others across different contexts. Their teacher and therapists believed that it would be appropriate and desirable for them to learn how to greet familiar people at school in order to promote greater socialization and improve their communication skills. The participants’ sensory abilities were within the normal range; however, their parents also reported that they usually engaged in unusual responses to greetings: hand-wringing, slapping their own face, hand tapping, or rubbing, especially when their parents force them to greet others. For example, they do not understand why some of their relatives greet them by hugging and kissing them. Those behaviors make them feel awkward and confused.

	Information
Participants	Age	FIQ	VIQ	PIQ	Diagnosis
Oliver	7.2	98	104	99	ASD
Peter	7.6	105	96	102	ASD
Jane	8.4	99	101	103	ASD
Mean	7.73	100.67	100.33	101.33

Table 2.

Summarized demographic information of the participants.

3.2. Instruments

Data on the participants’ intelligence, sensory abilities, and social and communication skills are based on multiple information sources: parental interviews, teachers’ reports, verbal IQ scores (Wechsler Intelligence Scale for Children), and levels of functional language and social adaptation levels (based on clinical observations or behavior and adaptation scales). All participants had a disability identification card issued by a medical institution in Taiwan and had been counseled in special education schools and institutes in Taiwan. The relevant university ethics committee approved the study, and parental consent was obtained for the students’ participation. We used Social Stories™ ([49, 50]) to create the Social Story trial (SSTs) tests to evaluate the AR-RPG training effect: pre- and post-intervention scores were compared using Kolmogorov–Smirnov test (KS test) to determine whether a particular intervention method improved their social greeting skills. Each scenario was associated with a different event, but each was consistent in length and at a similar level of difficulty. The entire scenario stories pretend that the process that was used followed the same rules for content creation and discussion with a special education expert and their teacher. The SST used for the intervention phase was different from those used for the baseline and maintenance phases; they were counterbalanced to reduce bias.

3.2.1. SST tests

Social Stories™ provides a standard strategy that has been used to teach social interaction skills to people with ASD [11, 51]. A Social Stories™ intervention involves creating brief stories that describe social situations and what others are thinking or feeling and how to behave in a specific situation. We created the SSTs using the rules dictated by Baker [49, 50]) for different phases (baseline and maintenance phases). A good scenario not only describes how people behave, but it also highlights what social cues the person should look for and how to respond to others [12]. Those SSTs were created by the school’s ASD therapist and reviewed by two other experts with experience in implementing Social Stories™ interventions. Social Stories™ provides a catalyst for change and provides children with other perspectives and options for thoughts, feelings, and behaviors ([16]). Therefore, we had all our participants take the standard SSTs in the different phases according to multiple baselines across participants’ design. We arranged 20 SSTs to occur at random in our SSTs to reduce boredom. Each question was different, but they were all at the same level of difficulty to reduce the test–retest effect.

3.2.2. Test evaluation

The SSTs used for the intervention phase were different from those used for the baseline and maintenance phases; they were counterbalanced to reduce bias. One test question was asked per short scenario script, and there was no prompting for answers in any session. We provide the six basic fundamental greeting behavior answer options that most happen in Taiwan’s daily life for them to choose, including the (1) nodded head and tiny smile, (2) handshakes, (3) waved hand and said hello, (4) kisses their head or face, (5) hugs, and (6) shrug your shoulders; those behaviors presented in AR animation also add with the dialog and extra detail information (e.g., facial expressions, communication, and dialog’s tone tone); yet, in the channel choice, we use the physical paper card (it also used as the AR marker for camera to detect their choice) with text description for them to choose. In each session of the experimental, 20 SSTs were given in three phases. After the test, we recorded the answer to determine the correct response rate. For example, if a participant got 10 correct answers in 20 SSTs, their correct response rate was 50%. The answers were checked by therapists and researchers who tested for normative answers.

3.2.3. Role-play evaluation

All of the participants role-played the greeting behavior after each SST question in each session with the therapist and their special education teacher. The therapist evaluated their greeting behavior feedback to evaluate their learning performance. The therapist evaluated performance on a 5-point Likert scale (5—Strongly Agree, 4—Agree, 3—Undecided, 2— Disagree, 1—Strongly Disagree) to evaluate the children’s role-playing performance. Performance included (1) gestures, (2) emotional responses, (3) interactive methods, (4) intention emotion, and (5) conversation. These separate channels were evaluated by the therapist and two social behavior experts’ score for expert assessment to ensure that each evaluation of role-play is accurate.

3.3. Intervention program

3.3.1. Procedure

In this study, we focus on teaching children with ASD how to appropriately reciprocate when they socially interact with others. Therefore, we arrange the different scenarios for them to judge and ask them to consider appropriate reciprocate in each scenario, and they need to imagine role’s situation and did the appropriate reciprocate judgment (choice one of the six greeting behaviors with facial expression and dialog). And, after the judgment, they watch the 3D animation, and then they need to role-play and mimic the character’s status; our therapist will evaluate their learning performance.

3.3.2. Experimental conditions

One certified occupational therapist with more than 7 years of experience working with children with ASD conducted all the sessions and taught all the children how to use the AR-RPG system. The experiment consisted of three phases: (a) the baseline phase, in which baseline information on the children was collected; (b) the intervention phase, in which the AR-RPG system was used to obtain the performance data for assessment; and (c) the maintenance phase, done 6 weeks after the intervention had been completed, in which the performance of the children was assessed. During the intervention phase, the AR-RPG system was used twice a week for 1–1.5 months to train all of the children in social greeting skills. The three children in our study group had a congenital condition that manifested differently in each; therefore, we used a single subject in a multiple baseline across subject design [52] to confirm whether the intervention was effective in single participants, despite their being ostensibly members of a group of similar participants. This is regarded as a standard and evidence-based method in many computer-based treatments used in special education. It is a fundamental experimental method for research in the field, and, in actual practice, it does not require control groups or many participants. The multiple baseline design is a style of research that involves carefully measuring multiple persons, traits, or settings, both before and after a treatment. This design is used in medical, psychological, and biological research and is especially relevant for ASD studies [53, 54, 55]. Because the manifestations of ASD are different in each individual, the purposes of the research were to see whether the intervention was effective by documenting whether and how each individual had improved.

3.3.2.1. Baseline phase

In the baseline phase, the therapist (a) first explained to the children the meanings of the greeting behavior that they would be asked about. (b) The ASD therapist and the children’s special education teacher created the series of greeting behavior scenarios into the SSTs, which were reviewed by two other experts with experience in implementing Social Stories™ interventions. (c) The scenarios were presented on the treatment room’s computer. (d) The therapist then asked the children some questions about the scenarios. (e) After the children had completed the scenarios, the therapist showed them a picture of six real greeting situations and asked them to determine the proper response to each. (f) The children chose one of the six greeting behavior pictures from the target pictures that they thought best reflected the correct response in this scenario. (g) Then the therapist asked them to role-play the scenario with appropriate gestures, conversation, intention, and facial expressions of emotion. Correct answers prompted the therapist to guide them to the next scenario. Correct and incorrect answers were identified and recorded, and the rate of correct answers was determined.

3.3.2.2. Intervention phase

In the intervention phase, the children were required to use the AR-RPG system to activate their understanding of the social reciprocity contexts and judge the questions about greeting behavior. (a) In the first session of the intervention phase, the children were instructed by the therapist on how to operate the AR-RPG system and how to perceive cues, to ensure that they felt comfortable using the AR technology. The instruction time was 40–45 min. (b) The children began the experimental sessions by following the therapist’s instructions to use the AR-RPG system. (c) In each scenario, the therapist shows the SST questions to ask the children with ASD. (d) After the SST questions, the children had to choose the appropriate greeting behavior card. (e) When the choice was correct, the AR-RPG system showed the avatar’s 3D greeting behavior animation and triggered the “Correct!” sound signal. When the choice was incorrect, the system showed the error signal and triggered the “Incorrect!” sound. (f) Then the therapist asked them to role-play the scenario with appropriate gestures, conversation, intention, and facial expressions of emotion. (g) After the children had completed this part of the training, the therapist taught them the correct response and reconstructed their greeting behavior map. In the intervention process, each child was given the same training by the same therapist, but each was separately trained for different durations, as required by the individual child.

3.3.2.3. Maintenance phase

Between the intervention and maintenance phases, there was a 6-week hiatus to reduce recall interference in order to determine, using the baseline phase procedure, but not the AR-RPG system intervention materials, whether the children had maintained the skills that they had acquired.

3.4. Experimental setting

All sessions were conducted in a quiet 3.5 m × 6 m room of the day treatment center at the school. The room contained a table and chairs, an Intel Core i7 laptop computer, a web camera, several tangible role-play markers with avatars, and a 37-inch LCD display set up in front of the children. The therapist sat on the right of the participant and guided the training process; the researcher operated the computer, set up the environment, and helped the children use the AR-RPG system (see Figure 2). To begin the intervention test, the therapist showed the test sample to the children, taught them how to use the AR-RPG system, and ensured that they felt comfortable using the AR technology. The children started controlling each step of the test after the first trial task was presented. During the test, they used their hands to control the cardboard avatars to make interact with other avatars that appeared in the scene. After they had chosen the appropriate greeting, the next task trial was presented. The system ran the AR application in the background and showed the image on the LCD screen. The children were able to practice pretend play and mimic the greetings with their therapist, which allowed them to feel comfortable doing this experiment.

4. Results and discussion

Oliver, Peter, and Jane could perform tasks when supervised. Results from the Vineland II Adaptive Behavior Scales [56] indicated a moderately medium adaptive level. They all could engage in typical conversations about daily activities and things they liked. However, they had few friends, and they were reported to interact appropriately only with their parents or relatives. The researcher who examined the procedural reliability of this study was the same certified occupational therapist who conducted all of the tests. We followed the related experimental methods used in other studies [57] to train and test the participants’ ability to identify the correct greeting behavior response. Those separate channels were evaluated by occupational therapists and special educational experts. We used a checklist in the test procedure to follow standard operating procedures for a therapist to ensure consistency in the processes and related controls. We also used the same AR role-playing strategy and context design to control the consistency of each story event to ensure that there were no unclear or emotionally confusing parts.

4.1. Training effects of AR-RPG system

The purpose of this experiment was to examine the differences in answers and greeting behavior responses between baseline and maintenance. We used a multiple baseline design across single subjects. The baseline phase consisted of four sessions for Oliver, six for Peter, and eight for Jane. The intervention phase consisted of eight sessions for each. The maintenance phase consisted of eight sessions for Oliver, six for Peter, and four for Jane (see Figure 4). All three children started with low scores (error rate range, 66.88–70%) during the baseline phase (see Table 3). All three scores rose significantly (p < .05) and dramatically (error rate range, 26.88–29.38%) during the intervention phase and remained significantly higher than at baseline (error rate range, 18.75–25.63%). The most dramatic error reduce rate was for Jane, who started with 66.88% and ended with 18.75% (p < .05).

Figure 4.

The error rates of the participants during three phases.

	Training effects of AR-RPG system (error rate)
Participants	Baseline	Intervention	Maintenance
Oliver	70.00%	29.38%	25.63%
Peter	68.33%	26.88%	20.00%
Jane	66.88%	27.50%	18.75%
Mean	66.40%	27.92%	21.46%

Table 3.

Summarized results for the participants.

The three curves in Figure 4 indicate that the correct assessment rates of all the children significantly improved (p < .05) (Kolmogorov–Smirnov test) after training and that the children retained in the maintenance phase the social expression and social skills that they had learned in the intervention phase. In addition, the mean difference in the performance level between the baseline and maintenance phases was significant (p < .05).

4.2. Their overall expression performances of role-playing

The training effect was also evaluated by the therapist using a 5-point Likert scale (see Table 4). The means of each phase of the score were recorded. All three children started with low scores (mean range, 1.5–1.75) during the baseline phase. All three scores rose significantly (p < .05), dramatically (mean range, 3.43–3.73) during the intervention phase, and remained significantly higher than at baseline (mean range, 3.57–4.18).

	Role-playing performance
Participants	Baseline	Intervention	Maintenance
Oliver	1.75	3.73	4.18
Peter	1.53	3.43	3.57
Jane	1.50	3.48	3.70
Mean	1.59	3.55	3.82

Table 4.

Summarized results for the participants.

After the training, the curves indicate that the overall role-playing score was significantly (p < 0.05) higher in the maintenance phase than in the baseline phase.

Data collection and test reliability.

5. Conclusions

The present study’s findings provide new insights into the innovations and technologies that promote social greeting behaviors for children with ASD. Children with ASD who participated in the study improved significantly. We found that, when closely monitored by an experienced therapist, our AR-RPG system was useful for teaching children with ASD how to recognize and understand the social greetings of others and how to appropriately respond. AR was important for helping the children learn how to properly greet people.

There are a few possible reasons for this. First, AR uses 3D virtual animation to show social behavior and real-time interactive scenarios. AR holds unique and promising potential to bridge between real-world activities and digital experiences, allowing users to engage their imagination and boost their creativity. Traditional VM cannot do this. Second, AR combined the cardboard and 3D avatars to help the children understand the scenarios, and the multisensory methods can help integrate the senses, such as focused attention, understanding situation boundaries, kinesthetic learning experience, and watch change. Third, using the physical manipulative avatar increased the children’s learning motivation and comprehension because they could directly manipulate and observe the scenario’s interactions. Fourth, AR and the related environment setup can help children with ASD quickly understand the situation in a specific place that is happening in a living room, classroom, park, or supermarket. This process can foster the development of more direct communication by employing techniques such as shared role-play situations than only reading a text description or watching a video.

In addition, many studies [58, 59, 60] support our perspectives that 3D animation with real spatial information in scenarios helps facilitate children’s mental skills because they can directly see the characters’ gestures and imitate them their feeling and action, and it does not require the user to image the scenario from 2D flat image [31]. What is most important, however, is that AR reduces a learner’s cognitive load and increases their interest in being trained because they can directly see the scenario [38, 61, 62]. AR has other advantages. For example, AR interfaces can directly provide real-time 3D visual support and promote spatial visualization, which is related to cognitive ability [63]. We designed an AR platform that combined real avatars that used their real facial expressions. It allows users to manipulate a real tangible object to trigger different events in the setting. It is easy to change the scene and add some real tangible object such as a table, chair, etc., inside the AR system to enhance the user’s immersion experience and motivation.

5.1. Feedback from participants

We also found clear changes in the social behavior of the three children before and after the training. Our therapist reported that, when children with ASD encounter a new situation and an unexpected greeting scenario, they first shrink back and look around for help or they focus on noncritical clues like clothing decoration patterns and other things that they are interested in. After they had been taught and trained using the AR-RPG system and 3D animation, they began to observe gestures and facial expressions. For example, when they used the AR-RPG system, they seemed quite excited that the content was a 3D animation that brought the scenario “to life,” and they began, without prompting, to ask the therapist a series of questions about the characters’ facial expressions, gestures, and related social greeting activities. In addition, when students in their class were noisy, the teacher put a hand in front of his mouth to ask them to be quiet. Or, when classmates shrug their shoulders, it might have meant “I have no idea,” “I can’t help you,” or “I don’t care”; their teacher found those participants actively trying to observe those people’s behavior differently by noticing target character’s gesture, dialog’s tone, and background situation to understand their body language. Although we cannot say that children with ASD easily learn to understand social situations, sometimes they still make wrong judgment. However, we can say that they learned to imitate social etiquette that they had been taught.

The children’s parents also said this on their questionnaires and in interview feedback reports. They said that when their children met their teacher or other people at school, and when they met their neighbors or other adults in the community, they would more frequently respond appropriately. Sometimes, they would ask why some people did not wave or say goodbye or why some would cry and cover their face with their hand. When unexpected situations occurred, the children were unable to respond in real time to response, but they were trying to notice the social clues that they had learned. Furthermore, therapist also found that our AR-RPG system gives them a chance to not only pretend different role situations but also cause their pretend play habit frequency to increase, and the therapist said that their students with ASD always want to pretend they are different roles (e.g., pretend they are teacher) and sometimes will disturb their peer or other people. For example, they will pretend some situation that they learn some specific situation from our system and repeat to do this with their teacher or classmates (e.g., hugs to their teacher); they feel that different responses from those people are fun and ridiculous. However, this behavior might bore others, but, in a positive perspective, AR-RPG system takes advantage of children as active learners to social interaction with others.

5.2. Limitation

This study has some limitations. First, because this is a fairly new intervention strategy for children with ASD, it was difficult to recruit participants to join the study; moreover, the participants had limited time for the tests because many had their routine school’s homework or family gatherings to take part in. Thus, it would be advantageous to recruit and enroll larger samples and extend the experiment period to provide stronger evidence. Second, it was difficult to determine whether the social skills of our participants had actually improved because it included many complex reciprocal social behaviors (e.g., effective modulation of eye contact, sharing affect, nonverbal reciprocity), not easily to separately measure each part of whole social behavior. Our positive findings indicate that children with ASD might change their behavior when they are aware of being observed; however, this will require a great deal of prospective observation and a long-term study. Third, we focused on the improvement of the greeting behavior as a whole. Future research might also take into account individual differences, e.g., eye contact and facial expressions, and how each individual uses the AR technology.

5.3. Future work

AR-RPG system made the way we children with ASD interesting and fun for them and, therefore, more efficacious than traditional methods. Our intervention system was effective for helping the three children with ASD maintain their focus on greeting behavior clues. It triggered the children’s learning incentive, encouraged them to observe nonverbal social signals, and improved their social interaction skills. In future studies, experiments involving more participants of all ages with ASD and AR technology should be included to spur research in this area. In addition, training materials for ASD need to be more complete and more reflective of real life. Finally, we hope that our findings will encourage new research projects on how to reinvent visual media to increase in adolescents and others with ASD the recognition of nonverbal social reciprocity cues in social situations.

Conflict of interest

The authors declare that they have no conflicts of interest.

Informed consent

Informed consent was obtained from all individual participants included in the study.