Mixed Reality Applications in Business Contexts

Abstract

Mixed reality is becoming increasingly relevant in business. In the corporate environment, such as logistics or maintenance, the use of data glasses allows extensive possibilities for process optimization and quality assurance. In the area of construction, virtual models either as augmentation of reality or mapped in virtual reality offer new approaches to experience ability. The goal of this paper is to show the manifold possibilities of mixed reality in the enterprise environment. For this purpose, selected application scenarios with corresponding realization stages will be shown and analyzed regarding their added value.

1. Introduction

Mixed reality as an umbrella term for augmented reality (AR) and virtual reality (VR) encompasses technologies that have gained significant importance in recent years and are increasingly being used commercially. Even though theoretical principles were developed as early as the beginning of the 1990s, the increased computing power only makes widespread use possible today.

The media often focus on corresponding opportunities; the potential that arises from use in the corporate environment is often still missing from the discussion.

Mixed reality is not a passing hype. The relevance is made clear by studies of various market research institutes and will continue to increase in the discussion around the metaverse.

It is often difficult to translate new technologies into application scenarios. Based on various application scenarios, answers to possible uses and corresponding added values emerge. Decision-makers need to get ideas for applications and business models that they can develop further to achieve added value for their company.

The aim of this article is to present mixed reality, to differentiate augmented reality and virtual reality from each other, to describe technical prerequisites, to show the current state of development, to present suitable application scenarios and realization stages, and to derive opportunities and risks in use to generate competitive advantages.

After the introduction, the basics of mixed reality are presented, followed by realization stages and application scenarios, which are illustrated by applications in the corporate environment. Afterwards, criteria are presented as to when augmented or virtual reality is recommended in which implementation stage. Finally, a conclusion and outlook follow.

2. Basics of mixed reality

Mixed reality (MR) as an umbrella term subsumes augmented reality (AR) and virtual reality (VR) and corresponding gradations. While virtual reality is understood to mean the representation and simultaneous perception of reality and its physical properties in a real-time computer-generated, interactive, virtual environment, and the real environment is consequently switched off, augmented reality aims to enrich the existing real world with additional computer-generated objects. In contrast to virtual reality, no entirely new worlds are created, but the existing reality is supplemented with a virtual reality [1].

There is no uniform definition of MR in the literature [2]; mostly the “reality-virtuality continuum” is referred to. This postulates a steady transition between real and virtual environments (Figure 1) [3].

Figure 1.

Reality-virtuality-continuum.

The left region of the continuum defines environments composed only of real objects and includes all aspects observed when a real scene is viewed by a person or through any medium such as a window, camera, etc. The right-hand domain, on the other hand, defines environments that consist only of virtual objects such as corresponding computer game simulations [2].

Within this framework, mixed reality is defined as an environment in which real and virtual objects are combined in any way in a representation, i.e., lying between the two extreme points of the continuum environments [3]. In Augmented Reality, the real component predominates, whereas in augmented virtuality, the virtual component predominates. A distinction between augmented reality and augmented virtuality is rarely made in practice today. Likewise, the terms augmented reality and mixed reality - more rarely also enhanced reality - are often used synonymously [1].

Mixed reality applications require corresponding devices or displays for their use. The displays used differ depending on the type of mixed reality - augmented reality or virtual reality. Virtual reality can usually only be used with data glasses. In addition to classic devices such as smartphones and tablets, data glasses are also becoming increasingly relevant for augmented reality. Also, pseudo-holographic displays are of interest for innovative augmented reality solutions, which enable new application scenarios but are often less flexible in use than the classic devices [4].

3. Realization stages and application scenarios in mixed reality

3.1 Realization stages

Ref. [5] distinguish according to different realization stages to classify different application purposes. Level 1–3 are presented because of their practical relevance [1].

• Realization stage 1: Visualization.

  With the help of MR, computer-generated objects can be projected into the real or virtual environment. In this stage, the user is a passive observer of an MR scene.

• Realization stage 2: Guidance & Control.

  In this stage, prefabricated scenarios of stage 1 are put into a logical sequence; the sequence is controlled by the change of the real viewing object.

• Realization stage 3: Interaction

  Stage 3 differs from stage 2 in the way that the MR scenario does not run according to a predefined pattern, but the sequence of substeps is controlled by the user himself. This can be done voice- or gesture-controlled. Control via input devices such as mouse, keyboard, or touchscreen, but also via facial expression or gesture control are conceivable and depend on the respective situation.

3.2 Application scenarios

In case of VR, there is no necessity to differ between applications scenarios. According to A. Mehler et. al., [1] the following application scenarios can be distinguished in relation to AR/MR:

• Living Mirror

  In living mirror, a camera detects the viewer’s face and places three-dimensional objects on the face or head in the correct position. The projection is usually done via a large screen or a beamer so that a mirror effect is created.

• Living Print

  This scenario is based on the recognition of a print medium and corresponding augmentation. Here, a distinction is made between different print media, be it collectible or greeting cards (living card), brochures or leaflets (living brochure), or packaging materials (living object). Further possibilities exist in the augmentation of books (living book) or games (living game print-based).

• Living game mobile

  Mobile devices are the basis of living game mobile; e.g., augmented games are brought to bear on smartphones.

• Living architecture

  A typical application in the field of architecture arises when a viewer wants to “experience” an impression of a room or an entire building by determining its representation himself through movements such as turning his head or walking through a real room and further actions such as speech or gestures.

• Living poster

  A living poster is understood to be an advertising message in public space that is enhanced with manipulative information elements using augmented reality.

• Living presentation

  Trade show booths and presentations must become increasingly spectacular and interesting to be perceived at all in times of information overload. AR technology can be used to achieve this goal. In addition, it is possible to display and even interact with real objects that cannot be “presented” live due to their sheer size or complexity.

• Living meeting

  Due to increasing globalization, more and more meetings take place as tele- or videoconferences. Augmented reality can be used to enrich tele- and videoconferences so that they almost seem like real meetings.

• Living environment

  All AR applications that use mobile systems to enhance real environments or facilities with additional information of any kind, such as text, 2D objects, 3D objects, video, and audio sequences, are called Living Environment.

The goal is basically a timely information acquisition (time-to-content) by the user simply by the fact that the camera captures one or more objects and thereby corresponding additional information is provided. In the case of the living environment, a combination of several sensors is possible and often desired.

The list of application scenarios is not necessarily complete, as further application possibilities may arise because of technical developments. The above scenarios are open regarding their intended use [1].

3.3 Combination of realization stages and application scenarios

If we combine AR/MR application scenarios with realization stages, we see that the living environment in particular is suitable for the third realization stage (Table 1). Static elements for generating AR/MR animations such as living print are more suitable for the first two realization levels. living game mobile is excluded because there are no economic opportunities for use here [1].

Application scenario	Realization stage
	Visualization	Guidance & Control	Interaction
Living Mirror	suitable	suitable
Living Print	suitable	suitable
Living Poster	suitable	suitable
Living Architecture	suitable	suitable	suitable
Living Presentation	suitable	suitable
Living Meeting	suitable	suitable	suitable
Living Environment	suitable	suitable	suitable

Table 1.

Combination of application scenarios and realization stages in AR/MR.

4. Application areas of mixed reality in business contexts

4.1 Overview

Mixed reality with a focus on augmented reality can be used in almost all areas of everyday life; many applications have a rather playful character. The use of mixed reality leads to significant advantages in many areas such as production, construction, and logistics. Typical examples are that designers can collaborate with real and virtual colleagues on the same three-dimensional model [1].

Many applications in production, construction, or logistics are also possible in the corporate environment. Possible applications of MR in the industry are mainly:

• Optimization and acceleration of industrial processes

• Immediate display and validation of planning status

• Collision control

• Evaluation of the design of new components on the existing product

• Virtual guidance directly on the real product

• Increased understanding of complex products or processes

• Simplified target/actual comparison

• Improvement of team-internal communication

Communication with customers can also be improved. Conventional advertisements are given an interactive component through the integration of mixed reality, which offers an additional benefit. Product catalogs can be enriched accordingly and offer added value to the customer. More complex solutions are virtual studios, such as those used by television studios. These are virtual sets in which real actors are combined with virtual reality in real-time.

The more the technology develops, the more complex and interesting application scenarios can be realized. In particular, these include electronic devices that exist only virtually but respond to real touch, enabling artificial sensory enhancements such as x-ray vision and open-world computer games. mixed reality supports complex and difficult tasks; these include especially applications in construction or maintenance of machines.

• Maintenance area

  • By displaying important additional information on demand, difficult tasks can be performed more easily, safely, and with higher quality.

  • E.g., based on an exploded view, a maintenance technician can display the machine to be repaired in 3D, move, and rotate it as desired, so that he gets a better impression and can perform the repair more quickly.

  • Another option is to show the mechanic labels for individual parts of the device as well as corresponding work instructions.

• Design

  • MR offers the possibility of simulating constructed objects in space in the design department.

  • Digital design data can be efficiently matched with existing real geometries. This also enables the broad use of digital validation methods when combining digital data with real prototypes or designs.

Further industrial applications with complex tasks have already been partially realized:

• Navigation

  • Navigation supplemented by mixed reality is conceivable in many different application areas:

  • In the maintenance of complex industrial plants

  • For operations in the field of disaster management

  • In the automotive sector, MR is used in cockpit displays that provide the driver with graphical navigation and traffic information in live images

  • In aircraft, MR is used in conjunction with head-mounted or head-up displays to increase safety and efficiency in poor visibility and weather conditions. Use in military aircraft was one of the first industrial applications of MR.

  • As a complement to navigation Google began offering its users the live MR navigation in Google Maps. Pedestrians can use live view instead of the normal 2D map view to see directional arrows as well as street names and additional information [6].

• Visualization

  Although mixed reality is very often used for representation, there are still some application areas whose main goal is to better visualize certain aspects.

  • Display and interactive analysis of maps and terrain features, e.g., for the extraction of mineral resources or in the field of geoinformatics [7].

  • Representation of destroyed historical buildings or future architectural projects [8].

  • Enrichment of museums and exhibitions by displaying virtual objects [9].

• Simulation

  • E.g., to enrich existing flight and driving simulators with mixed reality elements.

• Collaboration

  • E. g., to support virtual teams in their collaboration on simulated 3D models or conferences with real and virtual participants [10].

One example of the use of MR is education/training in the form of edutainment. Learners can view learning material on the computer via webcam and receive information elements on the computer that provide further explanations and information on the topic under consideration. Sound sequences that go beyond this are also possible as part of the animation. This approach gives learners the feeling of tangible proximity as well as individual and interactive help. This helps to reduce or eliminate learning barriers. The aim of integrating 3D animations based on MR is above all to simulate a haptic perception for the learners and to strengthen their interest.

The application examples already realized clearly show that the aspect of communication in the form of visualization is in the foreground. Complex issues are simplified and made easier to understand or experience by visualizing them.

Increasing computing power enables real-time application scenarios in a variety of ways - especially in the areas of user support and entertainment. Operating system interfaces, especially for the consumer sector, can be enriched with MR elements. Program windows and icons can be displayed as virtual elements in real space and operated by facial expressions or gestures. Thinking further, conventional screens or device control panels can be replaced by new types of devices and corresponding multimedia applications that use mixed reality. These include, for example, pseudo-holographic virtual screens, virtual holodecks, or virtual surround cinema, as already known from various science fiction series. Using MR enhancements such as X-ray vision, it is also possible to display occluded targets [11].

In recent years, MR solutions based on natural feature recognition and face tracking are gaining more and more importance:

• Natural feature recognition allows tracking without the use of artificial markers and enables the recognition of completely unknown environments and thus new application possibilities. The further development of this technology will enable mobile devices to recognize natural environments and enrich them with virtual objects.

• Face tracking, for example, opens new forms of communication with viewers. The viewer is recorded - essentially reduced to the face - and age- and gender-specific characteristics are recognized, evaluated, and interpreted, resulting in a corresponding individual response. Possible application scenarios arise in advertising. While posters currently reflect static information, it is possible to develop interactive posters based on mixed reality that react smartly to their counterparts. Based on age- and gender-specific characteristics, the poster reacts individually and offers suitable advertising messages.

Mixed reality has become a strategic product of hardware manufacturers such as Apple and Microsoft. Meta (Facebook) is increasingly investing in MR (VR as well as AR) technology. The so-called Metaverse is intended to enable users to communicate in virtual and augmented spaces. However, this requires glasses that support both VR and AR scenarios and are affordable for end consumers.

The possibility of gesture and voice control will allow further fascinating opportunities in MR, as it supports easy and intuitive handling of MR applications. Meta is already experimenting with a bracelet that measures brain waves to control MR applications.

4.2 Illustration of application examples in business contexts

Based on the three realization stages, concrete examples of MR applications in the corporate context are described below. A distinction is made between AR and VR in order to illustrate the broad range of possibilities. In the case of augmented reality, these examples are usually living environment solutions.

• Realization stage 1: Visualization.

  • Augmented reality

    Displaying technical information during machine maintenance or product visualizations, such as in ref. [12], are typical applications since an integration in real situations is possible.

  • Virtual reality

    Visualized virtual tours through computer-generated buildings, technical facilities, or operation- or process-visualizations, e.g., in ref. [13], are suitable, especially buildings and facilities that are not yet realized or cannot be used for training purposes.

• Realization stage 2: Guidance & control

  • Augmented reality

    Instructions for the maintenance of a machine or picking, as in ref. [14], represent common application areas for this realization level in the enterprise context.

  • Virtual reality

    Virtual tours through buildings or technical facilities under guidance, if necessary, with appropriate explanation or instruction, as well as training of complex activities, as shown in ref. [15], are typical application possibilities.

• Realization stage 3: Interaction

  • Augmented reality

    Interactive instructions for the maintenance of a machine (including the recognition of next steps or errors in the process) (see, e.g., [16]) or collaborative work are typical applications.

  • Virtual reality

    Virtual tours of buildings or facilities that can be changed through interaction, interactive product design, or collaborative work can be realized especially with tools like Mozilla Hubs, Google Spatial, or other tools for generating VR spaces. These tools provide the first approaches to implementing metaverse aspects.

4.3 Criteria for deciding between augmented and virtual reality

AR and VR have many similarities in terms of the levels of realization, yet they differ in the application possibilities. Therefore, the first thing to check is whether you want to implement an AR or VR application. Decision criteria are primarily simulation of situations, their availability as well as their hazard potential, necessity of embedding, effort, and costs. A corresponding classification is given in Table 2.

Criterium	Augmented reality	Virtual reality
Simulation of situations	Everyday situations	Exceptional situations
Availability of situations	High	Exceptions, rarely occur in reality
Hazard potential of situations	None	Low to high
Necessity of embedding in reality	Yes	None
Effort	Low to medium	High
Costs	Low to medium	High

Table 2.

Decision criteria.

If one considers the repair of a plant, then AR lends itself to this. The necessity of embedding, in reality, is given; the functionality, i.e., an everyday situation, is to be achieved. Likewise, the availability of the plant is given and there is no potential danger. The situation is different, e.g., for a flight simulator. In this virtual world, mainly exceptional situations are to be practiced, which rarely occur, but then have a high hazard potential.

Once a decision has been made between AR and VR, it must be clarified which realization stage is required: Is visualization (stage 1) sufficient, is guidance & control (stage 2) needed, or is an application for interaction (stage 3) necessary? The higher the realization stage, the greater the development, training effort, and the associated costs.

Once a decision has been made in favor of AR, a suitable application scenario must be selected depending on the realization stage.

Applications in the AR and VR area stand and fall with the generated added value. Of course, the implementation of an AR or VR application requires corresponding development effort, which should not be underestimated. However, it is much more important that as much creativity as necessary and as much pragmatism as possible flow into the conception of the application so that the application later delivers the necessary added value.

5. Conclusions

MR applications are on the verge of a commercial breakthrough and are becoming increasingly interesting for companies. Since 2015, various providers have been presenting AR as well as VR glasses; in some cases, second-generation glasses are already available and are becoming increasingly suitable for everyday use.

Despite the similarities that AR and VR have in terms of realization stages, there must first be clarity within the company as to which technology is to be pursued. Once this decision has been made, the next steps are to select the realization stage and, if necessary, the application scenario.

Both technologies covered with the umbrella term MR offer a multitude of opportunities, but at the same time have risks and limitations. Two main problems today are the lack of appropriate hardware as well as the lacking integration of corresponding applications into the corporate IT landscape. With the increased appearance of corresponding MR applications, solutions to this will be developed successively.

However, it is important today that companies do not miss the trend and may no longer be marketable or competitive. This will pose major challenges for small and medium-sized companies. The current discussion related to the Metaverse shows the relevance of the topic very impressively.