Augmenting Reality with Intelligent Interfaces

Abstract

It is clear that our daily reality will increasingly interface with virtual inputs. We already integrate the virtual into real life through constantly evolving sensor technologies embedded into our smartphones, digital assistants, and connected devices. Simultaneously, we seek more virtual input into our reality through intelligent interfaces for the applications that these devices can run in a context rich, socially connected, and personalized way. As we progress toward a future of ubiquitous Augmented Reality (AR) interfaces, it will be important to consider how this technology can best serve the various populations that can benefit most from the addition of these intelligent interfaces. This paper proposes a new terminological framework to discuss the way AR interacts with users. An intelligent interface that combines digital objects in a real-world context can be referred to as a Pose-Interfaced Presentation (PIP): Pose refers to user location and orientation in space; Interfaced means that the program responds to a user’s intention and actions in an intelligent way; and Presentation refers to the virtual object or data being layered onto the perceptive field of the user. Finally, various benefits of AR are described and examples are provided in the areas of education, worker training, and ESL learning.

1. Introduction

Researchers have long foreseen that Augmented Reality (AR) and Virtual Reality (VR) technologies have the potential to play a large role in our future daily lives, education, and business operations [1, 2]. The ubiquity of sensor-rich technologies in the pockets of the masses, the rapid pace of technological progress for such devices, and widespread interest in mixed-reality interfaces by both researchers and corporations, are harbingers of that foresight. Since it is clear that we are moving toward a future where the virtual will play a big role in our real lives, how we best interface with the computers in our natural environment will become an increasingly important consideration.

An intelligent interface is one that learns about the user and can adapt to serve them better through a minimally invasive presentation. An interface designed to overcome challenges faced by specific user groups by responding to the context of the user can be called intelligent [3]. Mixed-reality applications should be created to assist people and to be responsive to a diverse range of needs in a variety of contexts. This paper will discuss the rise of Augmented Reality as a means to interface with data in an intelligent way by discussing some examples of AR designed specifically for the needs of different learner groups: workers, students, foreign language learners, and immigrants.

This literature review to parse the current state of Augmented Reality for Education was done between October and November 2017 using ERIC ProQuest, the Social Sciences Citation Index (SSCI) database, and Google Scholar. The search terms ‘augmented reality’ OR ‘mixed reality’ OR ‘augmenting reality’ + ‘education’ OR ‘learning’ were used. Search results were limited to articles and conference papers in English. Much of the foundational research and history of AR was discovered through reading the most widely cited works on AR according to Google Scholar’s reverse citation search function. The purposes of this review are to place AR in the broader historical context of technologies that inhabit the spectrum of mixed reality within education technology research and to paint a compelling picture of the current state of AR for enhancing learning.

2. The spectrum of mixed reality

In the early 1990s, Tom Caudell and David Mizell were exploring the possibility of a heads-up display to enable workers to more efficiently create wire harness bundles during aircraft manufacturing at Boeing. They developed a see-through visual interface to identify, through the use of virtual cues, which real wires should be bundled together. They published the first academic paper that used the term ‘Augmented Reality’ [1] and are often credited with coining the term. Since its inception, AR has been widely studied in many contexts, while its underlying technologies have progressed by leaps and bounds [4]. The general principle of a continuum between the real and the virtual still exists [5] and is becoming an increasingly poignant consideration in our daily lives.

Augmented Reality is not a specific device or program; it is a type of human-computer interaction that occurs through a combination of technologies that superimpose computer-generated content over a real-world environment. Encompassing a broad range of technologies and components, AR has been historically defined as any system that: (1) combines the real and the virtual, (2) is interactive in real time, and (3) appears three dimensionally [2]. AR overlays virtual objects into the real world. These virtual objects then appear to coexist in the same space as objects in the real world for the purposes of interacting with the user in some meaningful way [6].

Reality and virtuality act as two opposite ends of a continuum [7]. Milgram and Kishino [7] proposed that there were various ways that reality and virtual content could be mixed and presented. Reality could begin as a location in physical space or could be computer generated. In a VR setting, reality becomes entirely virtual and replaces natural reality to create an immersive experience. Inversely, the goal of an AR system is to “enhance reality with digital content in a non-immersive way.” ([4], p. 79). A mixed-reality program can be to varying degrees exocentric or egocentric [7]; the user can feel as if reality is situated within the program, such as when playing an immersive 3D video game with a head-mounted display [HMD], or the program can seem to be layered on top of reality and situated within it, as when viewing context specific information about your environment on a smart phone or interacting with a virtual tour guide in a museum (Figure 1).

Figure 1.

Modified version of Milgram’s virtuality continuum [8].

The real environment acts as a substrate for virtual modifications. In AR, the user can maintain a direct view of the substrate background upon which the virtual is layered, as with HoloLens [9], or alongside the substrate, as exemplified with smart glasses [10]. Alternatively, the substrate can be processed by a camera and presented on a display; this is what smartphones currently do. A seemingly more futuristic option is that the virtual object can be projected directly into reality, such as a spatial AR or hologram [11]. The focus of this paper is the exploration of AR interfaces for the benefit of different learner populations, but the points made herein can be applied to the VR or AV ends of the continuum as well. The exact mix of how much reality is injected into the virtual (or visa-versa) is worthy of consideration when designing and studying mixed-reality applications.

3. Pose-Interfaced Presentation: A classification framework

Klopfer and Squire [12] resist using technological features to define AR; they claim AR occurs in “a situation in which a real-world context is dynamically overlaid with coherent location or context sensitive virtual information” ([12], p. 205). Initially, many researchers have tried to craft an exact definition for AR (e.g., [2, 7]), but in Klopfer and Squire’s opinion, any definition would restrict the exact meaning, so any technology that combines real and digital information could be considered to be augmenting reality. An intelligent AR interface requires the combination of tracking input, to find the pose of the user, with displaying output to combine the real and virtual world together in order to support the user. Therefore, this paper proposes a new terminological framework with which to discuss the way AR interacts with users: an intelligent interface that combines digital objects in a real-world context can be referred to as a Pose-Interfaced Presentation (PIP):

• Pose—user location and orientation in space is accurately tracked and sent to the program;

• Interfaced—the program responds to a user’s intention and actions in an intelligent way; and

• Presentation—virtual objects or data are layered on to the perceptive field of the user.

The following three sections discussing the current state of the field based upon the above PIP framework.

3.1. Pose: tracking on smart devices

The keys to a successful device running an AR interface are (1) the accuracy of the tracking inputs to accurately overlay virtual objects in the field of view, (2) speed of the processor to be able to layer more complex virtual objects (such as videos and 3d models) onto the field of view, and (3) ease of portability, so the user is free to move around in the real world unencumbered. An important consideration for successful AR devices is that they have an array of highly accurate sensors in order to establish and maintain detailed data about the environment. The application will make use of the data to situate a virtual output in real space. In a mobile phone, a visual AR interface uses the built-in camera to send image data to the application, which is programmed to find some pre-determined pattern (e.g., a QR code, picture, or building) that exists in the real world to determine user and device pose. The application layers a virtual object over a real-time display of the environment, presented on the screen. A location-based AR interface uses GPS co-ordinates and compass, while inertial tracking uses a gyroscope and accelerometer, rather than the camera, to track pose. Data triangulation through multiple-sensor input is required to eliminate drift in visually-established pose [4]. While AR visual, location-based, and inertial sensors are by far the most common methods of tracking pose, AR could also mean sensing sound, electromagnetic fields, radio waves, infrared light, or temperature. Hybrid pose tracking methods, combining input from multiple sensors at once, are far more reliable [13].

Pose tracking accuracy, and sensors in general, will continue to develop, as smartphone technology investment shows no signs of slowing. As of May 2016, the global average user range error for GPS receivers was ≤2.3 ft. 95% of the time [14]. GPS enhancement technologies such as Real Time Kinematic, which uses the GPS signal’s carrier wave for measurement, have the potential to improve accuracy to the centimeter level. This increased level of sensitivity will allow AR-based mobile applications to be able to establish exactly where the device is located in space, and therefore make much better use of location information in context-aware applications.

Another area of rapid progress that will improve AR is visual sensor technology. 3D camera systems are already appearing in flagship smartphones and should continue to proliferate [15]. The use of multiple cameras can provide depth information in order to greatly improve pose fidelity. One well-known approach for real-time 3D environment modeling is the Microsoft KinectFusion system. Newcombe et al. [16] describe KinectFusion as using visual data obtained from ‘structured light depth sensors’ to create real-time 3D models of objects and environments, and these models are also used for tracking the pose of the device and the user in the environment. By combining tracking of the environment with tracking of the users’ limbs, KinectFusion allows for a highly accurate gesture-based human-computer interface that allows users to interact with a detailed 3D model of the environment. This type of ‘natural user interface’ [8] has enough fidelity to even track facial expressions [17] and may end up becoming as common in future mobile devices as multi-touch screens are today. We can expect pose tracking to improve in both speed and accuracy over the coming years.

3.2. Interfaced: responding to human intention

Once the user pose is known, the user needs to be able to interact with the program. AR programs, by their physically situated and reality-referenced nature, tap into an intrinsic human affinity for physical rules and spaces. The interface between the virtual and the real world is accomplished through physical gestures, voice, movement through space, and gaze [18]. That interaction should be intelligent; that is, it should allow interaction with computers in a minimally invasive, intuitive, and efficient way that minimizes excessive perceptual and cognitive load [19, 20]. An intelligent interface should allow a program to be able to accurately determine the wishes of the user and respond to their needs with minimal input and data display. The ultimate goal is to create an interface that is functionally invisible to the user and makes interacting with the virtual world as natural as interacting with real-world objects, removing the separation between the digital and physical. Augmented Reality is one of the first technologies to make this type of intelligent, reality-referenced interface possible [4].

Interfaces should support users’ plans and goals in order to be effective. An intelligent interface presents information clearly and is easy to understand [19]. There is a lot of overlap between ‘intelligent’ interfaces and what could be called ‘effective user-experience design.’ Sullivan et al. [19] state that the requirements of intelligent interfaces are essentially those of human-computer interaction research in general. AR is a form of human-computer interaction that facilitates intelligent interface design because of its ability to make complex interactions feel grounded in reality. Situating the experience of computing in physically referenced space is a good first step toward creating a type of human-computer interface that is responsive and feels meaningful.

3.3. Presentation: Augmented Reality displays and devices

Our reality will be increasingly interfaced with virtual information; the trillion-dollar question is how the digital world should best be interacted with and presented to users. Research firm Global Market Insights [21] predicts that the global market for AR products will surge by 80% to $165 billion by 2024. Giant US blue-chip corporations have recently invested heavily in AR, in products such as Microsoft’s AR headset and enterprise feature suite HoloLens [8], Apple’s AR platform (ARKit) [22], Google (project X, formerly Glass) [9], Android’s AR platform (ARCore) [23], Facebook’s VR headset (Oculus Rift) [24], and ambitious startups such as Magic Leap [25] and Meta [26] who seek to create their own entire AR ecosystems. Although sentiment on VR softened in the first half of 2017, corporate investors are now actively looking for mobile AR investment opportunities. Digi-Capital’s new AR/VR Investment Report reported that $1.8 billion was invested across 27 AR/VR investment sectors in the first three quarters of 2017 [27].

4. Current and future trends in Augmented Reality

The future of AR lies within the usage and adoption of this new human-information interaction (HII) paradigm, not in the AR devices themselves. Technology will evolve over time, but fundamental improvements to HII and intelligent interface design will survive those mutations. Although head-mounted AR display technologies such as Meta Vision, Magic Leap, and HoloLens are grabbing all the media attention, they are merely stepping stones toward the widespread proliferation and integration of AR into mainstream culture. These flashy new devices are still too large and expensive for daily consumer use. HMDs have limited field of view and image fidelity as well as practical limitations such as battery life and limited product lifespan. That being said, HMD AR devices have demonstrated utility within commercial and industrial applications, as is discussed in Section 4.2 of this chapter.

As technology improves, mobile devices are becoming ubiquitous and capable of running AR applications. It used to be the case that immersive reality augmentation would require expensive, stationary computers and bulky HMDs. The first portable AR interfaces involved LCD screens tethered to desktop computers that provided the necessary tracking and processing power [28]. The sensors required for establishing the position and orientation (pose) of the user and device relative to the environment have also rapidly evolved since mobile phones received their first cameras in 1997 and Mohring and Bimber demonstrated the first mobile phone-based AR application in 2004 [4]. Today, smartphones can easily and smoothly run the software required for AR and are outfitted with a plethora of useful sensors/receivers (e.g., gyroscope, compass, accelerometer, light sensors, cameras, GPS, WIFI radio, near-field communication or NFC) which allow programs on smartphones to accurately track pose and make use of situational data that can be layered on top of the environment to augment the reality that the user experiences. Thanks to the development of more advanced smartphones and widely available software platforms such as Apple’s ARKit and Android’s ARCore, seamless dialog between reality and virtuality can now be accomplished much more easily than ever before, and this bodes well for the future of mass AR adoption.

We are progressing toward a technological future where our daily reality is intermixed with digital information that is presented to us in a real-time, user-friendly way. Earlier in this paper we outlined the PIP framework for the discussion of AR; our smartphones are mature, widely used devices capable of the Pose-Interfaced Presentation required for advanced AR interface. Adoption is not going to happen all at once, but many areas of our lives will be noticeably changed over the coming years as AR slowly replaces more traditional means of human-computer interaction. Entertainment, marketing, commerce, online social interaction, and professional practice are all likely to see major technological changes due to AR.

Major corporations are placing a lot at stake in AR. Taking into account these recent trends in the AR development space, it is important to ask how this emergent technology can best be applied, and who might benefit from the paradigm shift of layering data onto reality rather than alongside it. AR development should consider how best to leverage this new form of human-computer interaction to empower people who need support. In order to do this, it is useful to identify what AR is especially good at doing in real-world contexts in order to help focus design efforts and maximize potential benefits.

As early as 2011, Yuen et al. [29] predicted five directions for AR and provided multiple examples. While many current AR educational projects have focused on scientific inquiry (e.g., astronomy, mathematics, architecture, and engineering), educators have been working on AR projects in fields such as such as language, art, political science, sport management, textiles, fashion merchandising, and food and nutrition. AR learning tools allow students access to capabilities and resources that can dramatically increase the effectiveness of individual study and discovery learning [29]. For example, many historic sites supply overlay maps and different points of historic information for their visitors. AR will allow visitors to experience an event such as a treaty signing or a battle while they visit an historic location. Medical students around the world will be able to easily and inexpensively work on digital cadavers or dummies that can be reset after each use. Through the situated learning implicit in AR, learners will be better able to transfer what they learn into authentic situations.

With the continued development of AR-enabled educational books and games, combined with AR models of the real world, AR has the potential to make learning ubiquitous, allowing learners to gain immediate access to a wide range of location-specific information from many sources. In the areas of non-formal and informal education, museums and other public organizations are only beginning to explore how the technology can be used to improve experiences. For example, the Franklin Institute in Philadelphia has created an AR exhibit about the Terracotta Warriors [30].

For many teachers and students, the task of creating 3D models for AR is too difficult, since it requires significant technical knowledge. However, easier-to-use development kits are the goal of many firms investing in AR, so these problems should ease with time. Both Apple and Android are continually improving their native AR platforms to provide greater functionality, easier integration, and better performance. Flagship phones from most manufacturers are more than capable of running these AR platforms. Third party software developers and social media companies, such as Facebook, are working toward incorporating Augmented Reality into their platforms. These developments will accelerate the development of new applications in education and other fields.

4.1. Augmented Reality as a tool for enhancing learning

One consistent promise of AR has been in its application to formal and informal learning settings [31, 32, 33]. Indeed, several canonical learning theories describe the potential benefit of an augmented version of reality that is crafted with the intention to give context-specific, just-in-time information to a learner in an unobtrusive way. As a teaching and learning method, AR aligns well with both situated and constructivist learning theory. These theories frame the learner within a real-world physical and social context. AR affords the type of scaffolding and social context that constructivism values. It allows for participatory and metacognitive learning processes through authentic inquiry, situated observation, peer connections, coaching, and reciprocal peer learning, in line with a situated learning theory-driven view of education [34, 35].

The Cognitive Theory of Multimedia Learning can also be used to discuss the potential benefits of AR in education [36]. This theory states that there are multiple channels (or pathways) for sensory input to be brought into working memory. People learn best by using combined words and patterns, such as images, rather than text alone [37]. In this theory, learning is an active process of filtering, selecting, and using input from different sensory channels. Since each channel can become overloaded, it is important to reduce extraneous load, which is reflected in van Merriënboer and Sweller’s notion of cognitive load reduction through mixing of input channels [38]. AR can accomplish this input channel mixing in an elegant way by overlaying intelligently presented interface data onto a view of the actual environment. Germane (as opposed to extraneous) cognitive load can be achieved through a smart mix of real and virtual inputs in order to free up working memory. Limiting extraneous cognitive load paves the way for more effortless learning.

In light of its conceptual alignment with multiple learning theories, AR has been increasingly investigated by educational technology researchers. One of the most widely cited systematic reviews of AR in education is by Wu et al. [39], who reviewed 54 different education-related studies of AR that were indexed in the Social Sciences Citation Index (SSCI) database from January 2000 to October 2012. They found widely-reported positive effects on students’ learning, improvements in collaborative learning, and increases in student and teacher reports of motivation. By 2013, a consensus among educational researchers was building that AR was going to play a significant role in the future of instructional design. Researchers also agreed that, at that moment, the research into instructional design using AR was still in its infancy, but AR would become one of the key emerging technologies for education over the next five years [39]. In the past five years, there has been growing scholarly interest in the application of AR in educational settings.

In the SSCI database, Chen et al. [40] found 55 papers published from 2011 to 2016 on the topic of AR in education. Of those 55, only 16 were published between 2015 and 2016, which indicates that the literature search likely took place early in 2016. The authors found widely reported increases in learning performance and motivation; deepened student engagement, improved perceived enjoyment, and positive attitudes were associated with students who learned with AR-based instruction. However, this meta-analysis suffered from some transparency and methodological issues: learning performance was not defined, and data was not presented on how categories were coded or how much support existed between categories. Though they claimed to have searched for papers published from 2011 to 2016, the authors did not report their inclusion criteria or when their search took place. They claimed to have used the search term ‘Augmented Reality’ and searched through 2016, but in light of other researchers’ findings, one would expect many more papers to have been found for 2015 and 2016.

Akçayır and Akçayır [41] performed a more transparent and exhaustive review of AR research related to education written in English from 1980 to January 15, 2016. They located and then analyzed all of the published studies in the SSCI journal database (to the end of 2015) that addressed educational uses of AR technology. 102 articles were discovered, 68 of which were determined to be relevant after applying the inclusion criteria detailed in the paper. They found that improved learner outcomes were the most common finding, with ‘enhanced learning achievement’ at the top of the mentions list. Next was motivation, followed by understanding, attitude, and satisfaction. Decreased cognitive load and increased spatial ability were interesting finds. Pedagogical contributions were the second major category of reported findings, with enhanced enjoyment and engagement mentioned most frequently. Pedagogical advantages included both collaboration opportunities and ‘promotes self-learning,’ which seems a bit contradictory.

The present paper undertook an update to the above literature search to see if the trend has continued since Akçayır and Akçayır’s [41] search. Using the same terms and inclusion criteria in December 2017 that they used in early 2016 (‘augmented reality’ OR ‘mixed reality’ OR ‘augmenting reality’) yielded 74 relevant results: 34 from 2016 and 40 from 2017. A search on EBSCO Education Source using the same terms from 2016 to 2017 yielded 145 English scholarly journal results. The trend of articles reporting increases in student learning and motivation continued, and researchers seemed to agree that AR remains one of the most important developments on the educational horizon. AR is expected to achieve widespread adoption in two to three years within higher education, and four to five years in K-12 education [42].

Although all three overviews of the field presented here found similar positive educational outcomes associated with AR, there are some gaps in research that should be addressed. Wu et al. [39] caution that the educational value of AR is not solely based on the technologies themselves but also on how AR is designed, implemented, and integrated into formal and informal learning settings. In light of the fact that most AR research has been short-term, quasi-experimental, mixed-methods, or design-based research using small sample sizes, Wu et al. recommend that more rigorous studies are needed to examine the true learning effects.

Akçayır and Akçayır [41] found that some studies reported that AR decreases cognitive load, while others reported that it can actually lead to cognitive overload. They note that reasons for this difference are unclear, but it is likely to be due to design differences across studies. Similarly, there are conflicting reports of usability issues. “Whether there is a real usability issue […] that stems from inadequate technology experience, interface design errors, technical problems, or the teacher's lack of technology experience (or negative attitude) still needs to be clarified” ([41], p. 8). These discrepancies are emblematic of the broad array of design strategies implemented across different iterations of AR-based learning and of the lack of long-term, rigorous studies that compare larger samples of learner populations. Future research should aim to develop empirically-proven design principles for intelligent AR interface design that eliminate cognitive overload and facilitate ease of use.

Another continuing trend is that educational AR research has largely ignored diverse learner populations. Wu et al. [39] and Akçayır and Akçayır [41] both note that educational AR should be extended through design for different educational purposes, such as assessment and tutoring, as well as for students with special needs. It will be important, moving forward, to design AR with a focus on diverse learner populations in order to maximize its potential benefits, especially when one considers the potential power of AR as an assistive device for those with language barriers and learning disabilities [43, 44].

4.2. Augmented Reality for worker training

Education does not only occur in the classroom. Industrial training is an area that has received considerable attention by educational technologists; to wit, AR itself was first conceived of as worker training system [1]. Tools that provide customized instruction for workers have been repeatedly shown to significantly improve task performance [45]. Although seemingly futuristic, the application of AR interfaces for educating and guiding workers has been developing for quite some time. Nearly 20 years ago, AR displays were being shown to significantly improve performance on manual assembly tasks [46]. For example, the ARVIKA consortium was a German government-supported research initiative with 23 partners from the automotive, aerospace, manufacturing, and industrial research sectors that ran for 4 years from 1999 to 2003, aimed at developing AR technology to improve manufacturing [47]. AR has been shown to aid industrial and military maintenance-related tasks by increasing the speed at which components and schematics can be compared by over 56%, compared to traditional information display methods [48].

Today, the use of AR interfaces to superimpose useful training and technical information onto the real-world view of workers is gaining widespread adoption [49]. AR enables people to visualize concepts that would otherwise be challenging, in order to make use of abstract concepts and interface with complex systems [50]. One salient example of an AR interface used for abstract concept visualization is the CreativiTIC Innova SL [51]. The Innova SL can recognize individual circuit boards and overlay useful information about their components, aiding workers in industrial settings or functioning as a lab guide for engineering students. AR can also overlay instructions over real-world object targets, thereby increasing accuracy and speed [47]. Westerfield et al. [45] explored the use of an intelligent AR tutor to assist with teaching motherboard assembly procedures, rather than simply guiding people through the steps, and was able to show a 40% improvement in learning outcomes in a post-test for students who used the AR intelligent tutor.

Technically complex assembly tasks are well suited to AR interfaces. General Electric (GE) has leveraged the use of consumer-grade AR glasses, powered by custom learning software, for wind turbine assembly [52]. With the glasses, workers can view installation instructions and component diagrams in the same visual frame as the actual task, thereby increasing efficiency and reducing the number of errors. GE reports a 34% increase in productivity from the very first time the technician uses the AR interface, compared to non-augmented controls.

Many HMD AR devices, such as the Microsoft HoloLens, are being used in enterprise education settings. Thyssenkrupp Elevator have developed a custom AR interface for 24,000 service engineers using the Microsoft HoloLens, and the application has reportedly reduced the average length of service calls by a factor of four by allowing technicians to access interactive 3D schematics of the elevator systems they are working on without looking at a separate screen [53]. Technicians can also communicate and share field of view directly with support staff, in order to give situationally specific advice.

It is often the case that industry takes the reins of research for emerging assistive technologies in order to develop them for profit. This is certainly true for Augmented Reality devices; for-profit industry has driven the development of AR technology precisely because of its demonstrable benefits to their learner populations. The issue with industry-led technology development is that it usually occurs behind closed doors, out of the light of scrutiny of competitors and of researchers who might be able to benefit from their study. Because neither GE nor Thyssenkrupp has made this research public, it is unclear whether there are any potential downsides to AR interfaces in complex assembly tasks, such as the potential for distractions and workplace accidents relating to visual field obstruction. There is clear evidence of the distracting nature of virtual objects [54]; in fact, virtual objects can be so visually captivating that they have even been used to distract patients enough to reduce the perception of chronic pain and phantom limb pain in clinical settings [55, 56]. Poorly designed AR interfaces have also been shown to be distracting to drivers [57]. In light of such findings, it is reasonable to ask if layering virtual interfaces over complex assembly tasks could lead to dangerous distraction. It will be increasingly important for AR interface designers to consider the cognitive load of information that is presented to users in order to make them as minimally invasive as possible as we move toward a future that promises an increasingly augmented view of reality.

4.3. Augmented Reality for language learners

Although the vast majority of Augmented Reality systems have been designed to assist in industrial settings and STEM-related education [40], AR has also been shown to work well in other disciplines, since it is an effective tool for situating knowledge in real contexts [58]. One promising educational application of AR technology, both inside and outside the classroom, is facilitating language acquisition [59, 60].

English is arguably the most popular second language (L2) in the world [43]. For most students who study English in their home countries, English is learned as a foreign language (EFL). Unfortunately, not everyone has the means or opportunity to study abroad in order to reap the benefits of immersion. In countries like Taiwan, Japan, and Korea, English is not widely spoken in everyday life, yet learning English remains an important priority for a large number of people there due to its prominence as the language of international commerce, entertainment, and higher education. In EFL classrooms, English teaching is not connected to daily reality. Traditional L2 instruction in these countries usually relies more on knowledge acquisition than on skill or fluency, although simply repeating words, explaining grammatical rules, and reading irrelevant text leads to low motivation and poor learning outcomes [61]. Compounding these issues, EFL students often have few opportunities to practice English outside the classroom.

Contextualizing learning improves EFL performance [62]. Augmented Reality has been shown to contextualize information [63] while providing increased opportunities for language practice and serendipitous language learning [64]. AR has also been shown to increase long-term retention in vocabulary acquisition tasks [58]. This makes AR a fitting platform for EFL learners, who lack exactly what it may be able to provide: real-world context, motivation, exposure to vocabulary, and opportunities to practice what was learned.

AR for language learning has only recently begun to receive attention in research. Ho et al. [65] studied AR as a tool for ubiquitous learning (U-learning) in EFL at the University of Taiwan. U-learning is a concept closely related to the same proliferation of powerful mobile devices that was discussed earlier in this paper, which is facilitating the rise of mixed-reality applications. Essentially, U-learning refers to any learning environment, combined with pervasive technology, which is responsive to user needs. These environments support sharing and context-aware content and deliver personalized content to the user [66]. U-learning is a paradigm made possible by technologies like Augmented Reality that operate in a ubiquitous computing environment.

Some research on EFL U-learning using AR has been shown to improve learner outcomes. Santos et al. [58] used multimedia learning theory as a framework to apply AR to EFL settings in order to reduce cognitive load during vocabulary learning tasks. Building on the design work of previous AR language researchers [67, 68, 69], they found significant student improvements in Filipino and German vocabulary retention when implementing situated vocabulary learning by overlaying relevant words and animations over real objects found within the learner environment. Students in the study also reported higher satisfaction with their own learning outcomes and had an easier time paying attention to lessons. There appears to be promise in teaching vocabulary through the relationship between virtual data and real objects in AR, through its ability to situate information in context.

Augmented Reality language education extends beyond EFL settings; for the ever-increasing number of voluntary migrant learners and involuntary refugees, rising to the challenges of sociocultural integration and language education are of utmost importance for their success [70, 71]. Recent work by Bradley et al. [72] suggests that language learning on mobile devices can assist in easing the sociocultural transition of migrant learners. This view is supported by previous researchers [73, 74], who affirm that AR platforms allow migrant learners to practice language in context and to interact with socially normative language usage. It is also widely held that AR can be designed to increase social interactions by requiring the learner to use real-world information and locations [41, 64].

Augmented Reality games are emerging as a powerful tool for language learning. Godwin-Jones [59] offers a concise overview of the last decade of AR language learning projects, noting that while commercially available games can be made to fit pedagogical purposes, so-called ‘serious games’ that are designed for learning are easier to align to learning objectives. The problem is that mechanics and narrative immersion are often compromised in the pursuit of pedagogical targets, resulting in games that feel less enjoyable than commercial titles to play [75]. In addition, the financing available to create serious games is often orders of magnitude less than for big commercial titles, leading to smaller and less experienced development teams.

Commercially available AR games can be used for educational purposes. AR gaming experienced a recent spike in popularity with the viral outbreak of Niantic’s Pokémon GO [76]. Niantic’s first AR title, Ingress (www.ingress.com), is still popular and is scheduled for a major revamp in 2018 [77], and the company is poised to introduce the smartphone AR game Harry Potter: Wizards Unite in 2018. Recently, researchers have examined commercial mixed-reality games for language learning, pointing to various pedagogical applications and outcomes including opportunities for vocabulary learning [78], digital storytelling as a product of play and engagement [79], increased opportunities for L2 practice [59], improved learner confidence [80], and improved willingness to communicate in L2 [81].

AR games that are explicitly designed for language learning have been the subject of increasing scholarly attention. Games like Mentira [73], designed using the open source platform Augmented Reality Interactive Storytelling (ARIS), have demonstrated that place-based mobile games can be powerful tools for situating learning by creating authentic opportunities for collaborative engagement with both language and culture. In Mentira, for example, students interact with a historical version of Los Griegos, New Mexico, in order to solve a murder mystery in Spanish. The game blends location cues, group activities, and over 70 pages of scripted dialog spoken by AR characters to create an immersive language learning experience.

Though ARIS games tend to have favorable learning outcomes documented by design-based research, place-based mobile games are limited by a lack of usability away from the specific place for which they are designed. Their limitation of relying on hard-coded content is being addressed by object recognition-based AR platforms such as WordSense [64], which overlays contextual information on everyday objects in order to create serendipitous learning experiences in mixed reality. Current technological limitations in 3D object recognition will need to be overcome in order to facilitate accurate object recognition-based AR language learning tools.

5. Benefits of Augmented Reality in education

The examples described above demonstrate the potential of Augmented Reality to enhance education and training in diverse settings. Table 1 lists these benefits and provides a brief description and reference for each. It is clear from the table that AR can provide many powerful and unique benefits that support the claims made about its future potential.

AR benefits	Description	Example in research
Visualization	Helping students to visualize and comprehend complex, abstract data	Klopfer and Squire [12]
Contextualization	Connecting new information with the physical world in which it occurs	Ternier et al. [82]
Situation	Blurring the boundaries between inside- and outside-of-classroom activity while encouraging social interaction	Dunleavy et al. [34]
Attention	Drawing and keeping the voluntary focus of students who attend to digital information	Ab Aziz et al. [83]
Integration	Increasing opportunities for ubiquitous interactions with knowledge and incidental learning in everyday life	Vazquez et al. [64]
Motivation	Facilitating gameful design and increasing student engagement in learning tasks	Godwin-Jones [59]

Table 1.

Benefits of Augmented Reality.

6. Discussion

It is worth noting that studies reporting on the use of serious games in education, anywhere on the virtuality spectrum, tend to focus solely on the positive effects rather than examining possible negative outcomes [84]. While indications of improved learning outcomes and motivation are good to see, it is important to consider that the use of technology in the classroom tends to have the typical cycle of hype, research, and disillusionment [85]. It is possible that we may be on the cusp of disillusionment, as AR interfaces become ubiquitous in everyday life and begin to lose some of their novelty for students. It is also important to hedge discussion of positive research outcomes by considering that investigations of novel technology in educational settings may often demonstrate best-case-scenario possibilities [86]. Much of the motivational and attentional findings of AR research could be due to an uncontrolled novelty effect.

The Gartner Group, in its 2017 Hype Cycle for Emerging Technologies chart (Figure 2) reported that AR is moving into the ‘slope of enlightenment’ [87]. They offer an optimistic outlook for AR in the next five to 10 years, as much of the hype and novelty has worn off and the real work of designing useful products and integrating this new technology into the productive fabric of culture is underway. Interestingly, they note that VR is well on its way to becoming a productive tool in the next two to five years. We should view these findings cautiously, since they are simply measures of consumer expectation, corporate investment, and technology industry sentiment. However, while it is likely true that AR is five to 10 years from mass-market adoption, the applicability of these tools to create intelligent interfaces that empower learners is real.

Figure 2.

Years to mainstream adoption of various technologies [87].

For users who can most benefit from this technology, specific learning and accommodation needs should serve to guide researchers and designers who will be responsible for shaping the way that these new technologies will come into being.

It is important that we not only build on the successes of previous design-based educational research but recognize why certain design strategies have failed [88]. It may be beneficial to replicate and extend some of the AR designs in which researchers have invested so much time and effort over the last few years rather than ‘throw the baby out with the bathwater.’

Longer-term studies, and studies that control for time-on-task during learning, are needed in order to determine whether motivational and learning outcome findings persist and can be transferred to real-world contexts. Educational technology researchers tend to design new interventions from the ground up for their unique student-learning contexts. A key limitation of AR studies has been a lack of maintenance probes, as most studies have only measured initial learning [39]. Longer-term effects of AR learning interventions would help support findings of improved learner outcomes.

7. Conclusion

AR should not be looked at as a panacea to fix all of education, but rather a form of human-information interaction that is very effective at serving context-specific and spatially-situated virtual data. The design of each application and its appropriate alignment to learning targets will inevitably determine its efficacy in helping learners to achieve learning targets. AR educational games also rely on thoughtful design and enjoyable mechanics in order to be successful tools for learning and engagement. However, not all AR interfaces are created equal, and therefore it is not always possible to generalize findings. Regardless, there are innate features of AR that allow it to be used as an empowering platform. Taken as a whole, the corpus makes clear that AR has identifiable strengths and will clearly have an impact in the near future.