Applications of Mixed Reality for Smart Aviation Industry: Opportunities and Challenges

Abstract

Nowadays, mixed reality has improved operational efficiency and enhanced passenger experience in the aviation industry. Integrated with advanced machine learning and artificial intelligence techniques, mixed reality can easily deal with tons of aviation data to support decision-making processes in this industry. The chapter presents the state-of-the-art applications of mixed reality in smart aviation industry. Opportunities and challenges of integrating mixed reality with advanced machine learning and artificial intelligence techniques into the aviation industry are introduced. This chapter focuses on how the integrated mixed reality can improve the quality and reliability of maintenance, operation, piloting, training, and product design in smart aerospace engineering. It also describes autonomous, self-service, and data visualization systems in smart airports to enhance passenger experience. Finally, this chapter discusses airline’s digital-based responses to the COVID-19 crisis.

1. Introduction

Along with the contribution of technology, data science (e.g., machine learning and artificial intelligence techniques) and mixed reality have brought much benefit into intelligent industrial revolution [1, 2, 3]. One of their significant impacts is to redefine the business models in intelligent aviation industry. In the aviation industry, mixed reality-based applications such as digital twin, aerospace design, manufacturing, verification, validation, and services have been widely implemented [4]. In addition, applications of smart visualization, mobility, robotics, and analytics were developed and integrated into the passenger journey [5].

Airlines have used mixed reality not only for efficiency improvement of staff operations but also for maximization of passenger satisfaction. For example, airlines are experimenting with mixed reality in smart maintenance in which repair works are transformed from manual to digital assistant to increase the efficiency of maintenance operation [6]. When applying mixed reality for training, virtual instructions are overlaid on physical items, allowing employees to gain hands-on experience [7]. For market, airlines use mixed reality for indoor navigation, advertising recommendations, information notification, and immersive entertainment [8, 9]. Toward an automation industry, there is no doubt that mixed reality can transform the aviation industry by accelerating operator efficiency, developing staff skills, and improving the passenger experience.

The chapter provides a comprehensive review of existing works to understand the integrated mixed reality opportunities and challenges in the intelligent aviation industry. The keywords such as virtual reality (VR), augmented reality (AR), mixed reality (MR), VR/AR/MR aviation, VR/AR/MR design, VR/AR/MR assemble, VR/AR/MR maintenance, VR/AR/MR pilot, VR/AR/MR navigation, VR/AR/MR retail, VR/AR/MR passenger service, and VR/AR/MR COVID-19 were used to capture the most relevant literature published over the last decade from academic databases. Hundreds of papers that applied mixed reality for the aviation industry are then selected to review in the chapter. In addition to the opportunities and challenges of mixed reality in the smart aviation industry, the chapter also discusses a roadmap for intelligent aviation’s future and the role of digital technologies in overcoming COVID-19 challenges. Figure 1 shows that the number of papers published on VR/AR/MR has significantly increased over the last decade. It can be seen that the number of such papers has strongly developed since 2015 when the VR/AR/MR concept gradually received public attention, and its enormous potential for future development is widely acknowledged. Research works that proposed VR/AR/MR solutions to deal with the challenges of COVID-19 have also been concerned since 2020.

Figure 1.

The evolution of VR/AR/MR applications during the period of 2010–2021.

The rest of the chapter is structured as follows: Section 2 reviews MR-based aerospace engineering applications. Section 3 discusses how to apply MR to improve the passenger experience. Section 4 presents digitalization solutions implemented during and after COVID-19. Finally, Section 5 provides conclusions and future works.

2. Mixed reality for smart aerospace engineering

MR plays an integral role in the product design stage of manufacturing. Advanced interfaces and visualization systems are developed for product design, complex assembly, accurate maintenance, assisted piloting, and staff training. Engineers leverage MR to rapidly make improvements and changes before the product goes into manufacturing. Product designers overlay digital features onto the physical product and adjust it accordingly [10]. MR can thus shorten the development cycle and reduce the associated cost. For example, designers create a visualization of product hypotheses and prototypes and overlay virtual embodiment onto the real environment [11]. Engineers can interact with their designs using immersive tools to create a convincing product lifecycle [12]. The PJSC Scientific and Technical Council has recently created two VR sets and two mobile applications that received the highest rating and were recognized by aircraft simulator manufacturers, educational institutions, and aircraft operators [13].

Other application of MR in smart aerospace engineering is to assemble thousands of parts accurately and quickly [14]. Traditionally, manual instructions are painstakingly studied and constantly referred to as product assembly. Since the introduction of MR, detailed 3D instructions for hands-free and voice-controlled operations are provided to support product assembly efficiently [15]. Although installing electrical wires on an aircraft is an impossible task that requires zero error, interactive 3D wiring diagrams have made it possible [16, 17]. Airlines developed AR to provide technicians with hands-free, real-time 3D interactive wiring systems. Engineers’ works have also been altered by MR. 3D manufacturing can improve 90% in first-time quality and reduce 30% in time cost than 2D manufacturing [18, 19, 20]. Gattullo et al. [21] reviewed 1757 papers to study the benefits of industrial augmented reality (IAR) in assembly, to demonstrate how MR can support manipulation, inspection, and positioning, and to build future research directions. Lai et al. [22] created a worker-centric system that uses Faster R-CNN in MR-based mechanical assembly. With CAD models as datasets, the authors successfully deployed the system for detecting simple tools. Hořejší et al. [23] compared MR-based instruction with traditional assembly one on 60 subjects. Staff turnover and proficiency can be improved by converting complex assembly manuals into on-screen virtual instructions. Alves et al. [24] compared the requirements, performance, and user satisfaction of various MR application products (i.e., handheld mobile AR, indirect AR that displays augmented scenes on a monitor, and see-through head-mounted displays) during assembly in a variety of industrial scenarios. Through 30 participants, user preference aspects, the advantages and disadvantages of each method, and the potential benefits of specific use cases were reported.

MR-based maintenance enables faster and more accurate operations [25, 26]. Mechanics have traditionally relied on paper manuals to assist with thorough maintenance. MR overlays instructions on top of the actual product, allowing operators to see machine status in real-time [27]. Quality screenings are essential for product maintenance. Identifying problems early on is beneficial for reducing machine downtime, and associated costs incurred by airlines [28]. On the other hand, human-based defect detection cannot guarantee optimal quality assurance because human errors are unavoidable. MR-based projects overlay over the product, allowing defects and inefficiencies to be identified with high accuracy and speed [29]. Maintenance technical documentation is becoming more visual. The most common maintenance operations in the manuals are integrated into remote maintenance support applications. Scurati et al. [27] created a glossary of graphic symbols to use in mixed reality. Mourtzis et al. [30] designed an AR-based remote maintenance framework that allows technicians and engineers to use real-time feedback by creating appropriate communication channels. The applicability of the developed framework was tested in the laboratory and real-world industrial scenarios with good performance. Verde et al. [31] combined the Internet of Things (IoT) and MR to create a dependable predictive maintenance system for off-site service. With IoT, engineers can share diagnostic information, and experts can monitor and guide maintenance operations remotely via the controller. Heterogeneous sensors integrated with AR visualization can improve operator safety in complex and dangerous plants. Liu et al. [32] proposed a multi-service collaborative machine intelligence predictive maintenance approach for fault prediction that combines CNN and LSTM. MR guides the visualization through the maintenance process, integrating massive amounts of data into the machine. Runji et al. [33] reviewed advanced MR research in manufacturing maintenance, focusing on analyzing user needs in applications via ergonomics, communication, situational awareness, intelligence sources, feedback, safety, motivation, and performance evaluation.

MR is used not only for navigation on the ground but also for assisting pilots [34]. MR creates visuals of terrain, navigation, air traffic, weather, instruments, and airspace information by overlaying information on demand. The MR-based application provides situational awareness in a user-friendly 3D format. Furthermore, a head mount display (HMD) guides pilots during the flight, takeoff, and landing. The MR systems show a corridor overlay to display pilots the proper path, ensuring their safety from takeoff to landing. The air-traffic visualization aids pilots in keeping an eye on the traffic and so increases passenger safety [35, 36]. Extensive training is required for airline personnel before they work in a real-world environment. However, preset manual training is insufficient to meet operators’ growing needs. MR is thus developed to support training operators. It enables them to gain practical experience. When training and execution are combined, the equipment is easier to learn and use [37, 38]. Remote MR assists in lowering training and execution costs. It gives experts a view through the technicians’ eyes, allows remote expert support, and performs inspections without regard for distance. In addition, remote MR makes the support process highly convenient and significantly reduces travel costs [39, 40]. Han et al. [41] conducted a systematic review of MR-based professional training applications to assess their overall effectiveness. Kaplan et al. explored the effectiveness of VR, AR, and MR for training through a meta-analysis and identifies that MR-based training can achieve as similar results as the traditional methods, but with less than time and cost. Moesl et al. [42] used MR for advanced pilot training in turbine engine aircraft and detailed the most challenging elements of the course that can be addressed with AR. A survey of 31 pilots and 22 instructors was conducted to evaluate MR-based pilot training tools thoroughly. To help pilots learn new flight panel layouts for different aircraft, Füchter et al. [43] developed MR as a human-machine interface for training. The system employs Human Centered Design (HCD), integrating a multi-disciplinary process involving flight instructors, students, and pilots, and has been tested in Brazil with positive user feedback. These MR applications show that it has the potential to transform aerospace engineering procedures toward reliant automation in the future.

3. Mixed reality for passenger experience enhancement

At the present, airports have implemented several initiatives to accelerate the development of commercial MR services in the aviation industry. HMD provides passengers with customized audio-visual options as the virtual entertainment market expands. Indoor navigation and sign guides based on MR technology significantly improve passenger travel efficiency. Passengers can see advertisements and targeted recommendations while walking. The virtual tags allow for faster baggage claims and reduce mishandled baggage. Real-time live dashboards display waiting times, flight arrangements, and weather information for passengers. The use of boarding reminders and real-time notification systems reduces the number of missed flight events and makes passengers feel more at ease at airports. Such MR-based initiatives demonstrate that the era of MR applications for improving passenger experience has arrived. Figure 2 illustrates how airports utilize digital and MR to improve passenger experience.

Figure 2.

The application of digital and MR to the entire airport passenger journey.

Traditionally, airports installed beacons in terminals to provide passengers indoor navigation services that help new passengers navigate easily. However, this method is unreliable [44]. Recently, AR wayfinding displays directional prompts by superimposing digital information onto a real-world setting. It is a visual treat that makes life easier for visually impaired passengers by improving turn-by-turn audio prompts. An MR navigation system can locate late-running passengers and send them text reminders [45]. An AR-powered navigation app can even display points of interest along the way. The merchants send promotional messages to nearby passengers, informing them of current ongoing promotions. The app collects passenger information to create “people density” map in different areas that support airport queuing management by directing passengers to a less congested area. Passengers can use MR navigation to get through the airport quickly, accurately find their way, and avoid missing their flight. At the same time, it improves passenger flow management efficiency, promotes consumption, and increases airport and retail store revenue. [46] investigated the application of MR-based indoor positioning technology in airports and train stations. ARBIN was created to help passengers navigate through a user interface and navigation commands. The system was tested on nearly 1800 m² in 35 destinations and points of interest. The system performed very well and proved to be a practical indoor navigation solution for large buildings. Yoon et al. [47] made it easier for visually impaired people to navigate in indoor environments. The app “Clew” provides intuitive guidance along a route by recording the user’s previous route and then using haptics and voice. Ayyanchira et al. [48] developed a collaborative, cross-platform, real-time MR-based navigation system. The system uses visualization and interactive methods to support common tasks in buildings through device sensors and image markers. Gladston and Duraisamy [49] developed an indoor navigation solution based on computer vision and MR to assist navigation for users in any new or unknown environment.

Retail contributes significantly to airport revenue. MR retail boosts in-store and online sales and alters customer purchasing habits. Customers, for example, can make better decisions with virtual try-on for wearable products, eliminating the need to measure and verify dimensions [50]. Nike customers are able to try on different colors and finishes in an immersive environment. It makes shopping more exciting and enjoyable. MR retail increases customer satisfaction while decreasing product returns. AR in retail has effectively converted potential leads by increasing engagement. The use of MR features allows the audience to interact with products. Customers can scan AR markers on print advertisements to access interactive augmented videos rather than plain old print ads. In addition, AR wayfinding can be combined with retail advertisements. Airports, for example, can show passengers the best shops along their route. Airports direct passengers through the airport based on their preferences and previous purchasing habits. Airports, for example, provide information on various product prices and guide customers to the correct location with guided arrows. According to recent research, MR is inspiring new traveler engagement. Riar et al. [51] presented a systematic literature review of MR retailing, analyzing technical features, user behavior, and experience. Tan et al. [52] outlined four broad usages of MR in the retail environment, assessing the potential of such applications and empirically examining their impact on sales. Janssen [53] tested MR-based shopping app as a solution for retailers to engage consumers and enhance their shopping experience. The study investigates consumer acceptance of mobile AR (MAR) shopping apps using the Technology Acceptance Model (TAM) and potentially external factors.

Airport incidents such as lost baggage or missed flights have received much attention by passengers. Implementing MR in airports has not only improved customer experience but also been beneficial in generating a return on investment (ROI). Before passengers boarding, airports use MR systems to notify them of the best boarding time and direct them to the right gate. Gatwick Airport deployed AR to provide personalized flight updates to travelers with a frictionless experience. Despite the importance of baggage in the passenger journey, baggage operations in airports continue to have many flaws [54, 55]. Many airlines are working to improve their baggage handling procedures. Singapore’s Changi Airport uses smart glasses for baggage handling. Staff can instantly scan QR codes on baggage and cargo containers to obtain weight and unit data. MR has reduced the loading time from 60 to 45 minutes. MR has really increased the efficiency of airport operations, delivered more streamlined experiences, and, ultimately, provided more enjoyment for passengers.

4. Opportunities and challenges during and post COVID-19

Digital technologies are critical in assisting airports’ recovery worldwide following the COVID-19 crisis. Most airlines are undergoing a digital transformation, resulting in cost savings, increased productivity, automation, streamlined operations, and reduced risk to smart airports [56]. Figure 3 shows how to fully utilize data to drive the digital transformation of the aviation industry after COVID-19.

Figure 3.

The application of digitalization in the aviation industry.

The length of time passengers spend at airports is critical in reducing the possibility of pandemic infections. Airlines are attempting to achieve seamless, touchless, and frictionless travel [57, 58]. Online ticketing, seat selection, and check-in reduce contact and queuing time. Automated check-in and boarding, self-service systems such as biometric enrollment, facial recognition, and finger scanning are applied. To reduce passenger anxiety, airports deploy notification systems to communicate with passengers and provide real-time traffic information. The airports can use mobile app to optimize queuing management and maintain a safe distance between passengers.

During the COVID-19 crisis, health screening and certificates are special operations [59]. To assist customer service representatives, airports use the web, mobile, social media, kiosk, email, smartphone apps, and chatbots. Each passenger’s health is digitally recorded, and machine learning is used for abnormal data identification, identifying infected suspects in real time. Real-time data analytics ensure that potential patients are not overlooked. In addition, data science manages airport sanitization, ensuring high hygiene standards, which decreased human intervention and increased fidelity, accuracy, and observations.

Although airports will incur short-term losses during COVID-19, renewed digital aviation investments will increase long-term benefits [60]. It is not only necessary to promote airport digitization, but also to prioritize environmental protection [61, 62]. Passengers’ modes of transportation to and from the airport have significantly changed in recent years. Airports need to prepare for an increase in passenger traffic in the future and a faster recovery of the leisure passenger segment [63]. Public airport transportation necessitates substantial support, such as route subsidies [64, 65]. Commercial air taxi services have been expanded at some airports. The Volocopter completed the first human-crewed flight over Singapore. Boeing, Bell, Embraer, Safran, Uber, Fraport, and Groupe ADP, among others, have announced plans for urban air mobility solutions. Airports are working together to reduce their environmental impact.

An innovative travel ecosystem has emerged in the air transport industry. Based on WHO and UNWTO guidelines, Mohanty et al. [66] investigated the possibility of using MR to support the restart of tourism after COVID-19. The study explored the role of MR for social distance and lower mobility norms based on a systematic review of secondary data. Kerdvibulvech and Chen [67] summarized models and tools that use MR in responding to the COVID-19 crisis. For example, MR-based thermal imaging glasses are used to detect virus symptoms and passenger temperatures. Airport personnel is remotely trained by using MR educational applications. Priyan et al. [68] designed a MR-based standard operating procedure (SOP) for detecting and controlling social distance on a large scale. The application aims to help people control physical distance, accelerate social distance, and flatten the increasing case curve.

5. Conclusion

This chapter did a comprehensive review and provided a road map for MR applications in the intelligent aviation industry. Opportunities and challenges for implementing advanced MR in aerospace design, manufacturing, testing, and service are presented. MR technology helps smarter product design, safer assisted piloting, more flexible manufacturing, and more efficient staff training. The ongoing development of MR significantly impacts the passenger experience in the aviation industry, including visualized real-time information, AR wayfinding, virtual baggage tags, and AR retail. Furthermore, this chapter analyzed and discussed opportunities and challenges that the aviation industry can face during and after the COVID-19 pandemic. With the breathtaking array of new data-enhanced technologies, the aviation industry will open a new era of “touchless, seamless, and secure” operations and services in which the importance of machine learning and artificial intelligence techniques are concerned.

Acknowledgments

The authors would like to thank the Editor and the anonymous reviewers for their valuable comments.