Mixed Reality in the Presentation of Industrial Heritage Development

Abstract

The chapter ‘Mixed reality in the presentation of industrial heritage development’ is aimed at exploring opportunities for collaboration between theoretical research, monument preservation, virtual reality and architectural practice. It deals with the identified key factors that conditionally affect the quality and efficiency of architectural design process of architects within the cooperation in the conservation process of industrial heritage as well as the opportunities of transfer the research results from futuristic disciplines. For this purpose, the chapter examines the case study ‘the reconstruction of Old Power Plant in city Piešťany’ and describes possible solutions on the basis of the Mixed reality (MR). The opportunity to experience the industrial object with multiple senses (sight, hearing, smell, touch) in MR delivered a unique personalized experience and immersive memories about lost heritage.

1. Introduction

Industrial heritage provides one of the most important records of urban development and progress of human civilization in the last two centuries. Monumental industrial buildings reflect the extraordinary technical and economical development and the progress in science and technology. Even after the termination of their original function, industrial heritage buildings and equipment with their architecture are still significantly participating in the character of each city. A global problem is the decreasing interest of young people in studying natural sciences and engineering, which is a prerequisite for further technological progress and socio-economic development of the life of inhabitants. This lack of interest is justified by the high abstraction and lack of clarity in the scientific and technical fields which are separated from people’s everyday lives. Therefore the current trend nowadays is developing an interactive mixed reality model of presentations—those are able to make more attractive inspirational use of this rich source of knowledge and experiences [1].

The interdisciplinary research team at the Faculty of Architecture STU BA systematically focuses its work on applications of mixed reality by merging different sensorial inputs from real and virtual environment. This chapter aimed to explore opportunities for collaboration between theoretical research, monument preservation, virtual reality and architectural practice. It deals with the identified key factors that conditionally affect the quality and efficiency of architectural design and mixed reality process. For this purpose, the chapter examines case studies and describes possible applications on the basis of the operational research model so-called ‘Educational Polygon’ [2]. This model is used as a tool for identification of industrial heritage potential and it also serves as an effective communication and educational instrument throughout the active development process. Effectiveness of used procedures of the system Educational Polygon (EP) has been verified within the research KEGA and in the main case study reconstruction of Old Power Plant in city Piešťany and in the education and design process in Bratislava.

2. Theoretical scope

Industrial heritage consists of the remains of industrial culture that are of historical, technological, social, architectural or scientific value. These remains consist of buildings and machinery, workshops, mills and factories, mines and sites for processing and refining, warehouses and stores, places where energy is generated, transmitted and used, transport and its entire infrastructure [3]. Industrial heritage represents a considerable qualitative and quantitative economic potential for future development. In this context an architectural profession often finds itself in the role of mediator between investors, government, municipality, scientific community and general public.

This happens during the whole process of industrial heritage restoration, when in the given circumstances architects requires Mixed Reality to present the design changes of industrial heritage to the public.

In order to clarify terms in this article, Mixed Reality is a term used to cover all concepts of reality as shown in the classification of MR technologies in the Figure 1. For the presentation of industrial heritage in the case studies, augmented reality (AR) and virtual reality (VR) were used mainly. Displayed types of realities differ according to degree of reality, on the left side there is real reality with the highest degree of reality. On the other side of the scope is placed VR, which could be understood as complete absence of real world.

Figure 1.

Order of reality concepts ranging from reality to virtuality (Schnabel et al., 2008) [4].

The crucial difference between AR and VR is that AR in contrast to VR does not abstract completely from the physical world; virtual objects interact with the physical world and are placed into the context of real world. Furthermore, AR represents a less invasive concept as it is based on real physical laws, which does not have to be the case with VR. Technological progress erases the borders between reality and virtual reality. Perception of the world can be manipulated through the technology. Various illusions can be fabricated in real world through the physical installations or in mixed reality. New mixed reality devices are coming to the commercial market and enable more dynamic and realistic perception of the computer-designed world. The possibility to create photo-realistic scenes in game engines plays important advance in design of projects and applications in virtual and mixed reality in the presentation of industrial heritage [5].

This research is based on Steed’s revisiting of virtual continuum by extending the notions of virtual and real environment, building on Milgram’s and Kishino’s diagram. Steed explained that even within a ‘standard’ VR, there are links to the real world, and what one sees in the virtual environment might reflect some aspects of the current state of the real world. This situation could be observed by using body avatars or real object’s representations in the virtual environment.

This blend between real and virtual has long been objective of studies by the real-time graphics communities. In 1994, Milgram and Kishino created a diagram, which has framed these concepts and provided the description for virtuality continuum (Figure 2). ‘Milgram and Kishino have placed real environments and virtual environments at the opposite ends of a spectrum that includes various levels of “mixing” of realities, hence the generic term mixed reality (MR). This is a rough description that shows that one can add virtual elements to a real scene to create an “augmented reality” (AR), or real elements to a virtual environment to create an “augmented virtuality” (AV). Some authors just use the term AR, without using the term augmented virtuality’ [6].

Figure 2.

Virtuality continuum diagram by Milgram and Kishino [6].

The taxonomy of Milgram and Kishino provides a way of contrasting different types of mixed reality. Complementing their taxonomy, Steed introduces two further considerations that distinguish between different systems: primary environment and immediacy of representation. The primary environment is always one of three things: a pure virtual environment, the local real environment or a remote real environment. Immediacy of representation is a simple concept, which refers to the age of the represented content in the mixed reality and thus its veracity [6]. Steed is describing those terms mainly on visual situations and examples. Besides mixing of visual elements from real and virtual, the experimental work described in this chapter is focused also on various fusions of other different sensorial inputs from real world such as touch, smell and hearing with virtual environment.

Didactic theory confirms that the senses are portals of information. One learns by hearing, sight or by activity (Figure 3). We all use these methods and each of us prefers a different way of teaching. The use of the senses and their combinations is typical for mixing learning styles.

Figure 3.

Graph of sensory reception and picture of Senzulor (Scheme: Ganobjak and Hain, 2017) [8].

With every sense, we receive a different percentage of information and we remember it differently. A distinction needs to be made between receiving and remembering information. We receive most information visually and less by hearing. We remember 20% of what we hear, 30% of what we see in visual form and 90% of what we are actively doing [7].

Mixed reality also actively uses the first two senses through which we receive the most information, sight and hearing. Kinesthetic style uses activity and engages all (other) senses without preference. It is proven that the greatest learning effectiveness is the way of learning through a combination of learning styles. In this way, one can remember up to 80–90% of what one hears, sees and does at once. Although the representation of other senses versus sight and hearing is negligible in receiving information, it appears that combinations of activating multiple senses are highly effective. The sensory overlap with which the information was captured, creates stronger links between them for remembering, which is absent in the case of selective perceptions.

Among us, there are several cases of people with visual, hearing or other disabilities that should not be forgotten. In such a situation, one or more of the senses are lacking and are therefore replaced, represented and compensated by another. Each is unique and different, it would be appropriate to pay special attention to each person with regard to its properties. However, it is not possible to set up a specific exposure for each, either spatially or financially. Here, universal design rules are offered, as if they are the opposite of barrier-free design. It is not a design for a narrowly specified group, but rather for the widest possible range of users. All you have to do is create one quality exposure that is universal for everyone. One solution to achieve such a balanced state is to create an exposure and at the same time every single exhibit perceived by multiple senses simultaneously. Thus, anyone will be provided with full information. Moreover, such a Mixed Reality exposition allows the situation to be better, more clearly, imagined, understood and remembered not only for children but also for people with limited abilities.

When we create an exposition, it is necessary to focus not only on the collection of objects that can be seen, but also to make the exhibits available in a non-visual way for universal design. Such an exhibition focused on other senses than sight, will bring a new experience, allowing visitors to get to know nature often from another side. An inseparable part is also a fun factor. The fun factor is always pleasant refreshment in the amount of testimonial information that seeks our attention.

The sensor image (Figure 4) shows the reach of our senses. At the same time, it shows the radius of information that we are able to take in that sense. While our eyes catch most of the information, we are also saturated with visual information, so it is possible to use the way of inverse engagement of the senses. Not many educational exhibitions are conceived in taste, tact, smell or acoustically. Just as we perceive the stimulus closer to the body, it may leave a larger memory footprint.

Figure 4.

Inverse sensory orientation of exposure. Combinations of sensory perception affect the overall impression (scheme: Ganobjak and Hain, 2017) [8].

One subconsciously favours those impulses and stimuli from the environment that act closer to the surface of the body. This proximity gives rise to an approximatively defined sequence of its sensory zones from the tactile zone to the thermal zone, the olfactory zone, the acoustical zone, to the human dominant visual zone. The irritation of our receptors affects the perception of the surroundings, the orientation and behaviour in space and the overall relation to the environment. By the centre of gravity are activated sensory organs determine to the size and nature of individual spatial frames of human zones. This dependency is expressed by the sensor.

The sensory organs provide the brain with information about the specificities of the external environment. Different organized sensory organs, with different sensitivity and complexity, can either receive one and the same information or multiple information at the same time. These combinations of sensory perception affect the overall impression, feeling or condition of a person in a variety of situations. These phenomena are positively or negatively manifested especially in the perception of the exhibition and therefore it is important to address them also when designing Mixed Reality.

Based on the results of the FA STU research and the KEGA grant [8], an industrial heritage visitor will best understand the information with a logical structure where the individual themes and exhibits are linked to one another. Therefore, when designing Mixed Reality, it is crucial to organize the exposure with the exhibits into a system with a logical (semantic) structure that clearly implies what is primary, principal, essential, and secondary, complementary, which are the main and secondary elements of the exposure and what are the relationships between them.

The average person remembers approximately:

• 10% of what he/she reads,

• 20% of what he/she hears,

• 30% of what he/she sees in visual form,

• 70% of what he/she sees and hears at the same time,

• 80% of what he/she sees, hears and speaks at the same time,

• 90% of what he/she is actively doing.

It follows from the above that it is important to include and combine the perception of multiple senses during exposure, to change the senses several times during the exhibit and to repeat the new stimuli [8].

The fluency/continuity of the exposure is achieved by its spatial and thematic continuity. This can be done by linearly designing the exposure, by its loop, or by multiple looping. Linearity means: ‘A loop allowing the linearity and sequence of exposure to be maintained, giving the possibility of returning to the previous point where the rest zone can be situated’. Combining exposure helps achieve spatial compactness. On the smallest scale, a single room can also be a loop. It is not advisable to create blind offshoots of exposure with longer and deeper spaces, after which visitors must go back in the same way as in virtual reality (Figure 5).

Figure 5.

Rounding the exposure helps to achieve spatial compactness and prevent muscular fatigue syndrome (scheme: Ganobjak and Hain, 2017) [8].

This augmentation of virtual by senses is related to the understanding of virtuality and its perception. As Calleja indicates, media can create the phenomenon described by the notions of presence and immersion in them [9]. This phenomenon could be seen as degree of ‘realness’ of medium. In the heritage presentations and architectural design, we can presume that building plans have different degree of immersion as the physical and digital models. The immersion rises with quantity and nature of information received by the user. By involving various sensory stimuli, the information stream is widened and thus it is easier to compare user experience to the real situation [10], which is closer to our innate learning by experience, and to gather relevant data about the users. These data are used as feedback from the users, which could improve future designs of the next presentations and environments.

For this purpose, there are several techniques of how to process spatial information. It is possible to 3D scan the space or measure it with routine techniques and to compare it with suitable historical documentation. Then to model it accordingly in form of a digital mesh model, individual characteristic surfaces need to be photographed to create textures with appropriate qualities such as colour, reflections, structure etc. For unpreserved surfaces, it is possible to use retouched techniques or replace them with equivalent textures from similar spaces.

3. Methodology

The choice of theme, exhibits, choice of methods of mixed reality presentation, organizational forms and material means should be guided by didactic principles.

3.1 Didactic principles

1. The principle of creating optimal conditions for the observation and educational process of exposure

2. The principle of adequacy of exposure to target groups and individual treatment of visitors

3. Principle of science

4. The principle of connection of scientific exhibits with life, theory and practice

5. Principle of illustration

6. The principle of motivation, awareness and activity

7. Principle of continuity and sequence of teaching

8. Principle of durability and operability of the educational process of exposure

3.2 Modern type of exposure in the form of mixed reality

Its essence is that the knowledge and skills of users should be the result of their own thinking. All modern concepts agree that the visitor should be motivated and active in getting to know museums or expositions of industrial heritage. To do this, a clear and comprehensive specification of exposure and education objectives is needed, and the following principles serve to create a mixed reality design.

Mixed reality is an interesting option for representation of objects within heritage conservation. Objects are exhibited in augmented or virtual reality and aspect of interactivity produces greater immersion for users. Representation of objects within heritage conservation through mixed reality creates an opportunity to rediscover history in new and exciting way. However, it is a complex scheme of organized design process (Figure 6) with key educational elements of Educational Polygon, which is divided into several phases:

1. Defining the target user of mixed reality: for the needs of the Educational Polygon, we can basically divide all participants of the process into five main groups of stakeholders: 1. architect, 2. investor, 3. municipality, 4. professional community and 5. general public—NGOs, people living in the neighbourhood, former employees and important stakeholders in local development—potential users (Figure 6).

2. Key elements for definition of optimization problem of mixed reality:

   1. accessibility: 1. personal, 2. local, 3. regional, 4. continental and 5. worldwide.

   2. aspects that represent creation of the mixed reality model—6 limits (6E) which represent legitimate requests for creation of the mixed reality model: economic, 2. ecological, 3. ethical, 4. effective, 5. aesthetic and 6. educational.

   3. the target we want to optimize by mixed reality—this objective must be measurable (max/min): maximize the potential of the industrial heritage presentation; and minimize the loss in value of industrial heritage.

   4. the period for which is designated the result of presentation: 1. past, 2. present and 3. future (short term, medium term and long term).

Figure 6.

Scheme of organized design process with key educational elements and interdisciplinary cooperation (scheme: Hain, 2014) [2].

The scheme includes criteria and aspects generating ‘matrix of externalities’ [11].

The matrix of externalities reflects a combination of all possible decisions. By the interaction of all these elements, an educational benefit for all subjects could be received.

Using the principles of Educational Polygon ensures a certain flexibility, cross-checking feedback as well as analysis of the results (Figure 7), which is a prerequisite for setting qualitative conversion process of industrial heritage [12].

Figure 7.

Educational Polygon-managing team in dynamic model in the process of industrial heritage maintenance and presentation. Various relations emerge between the subjects by the presentation of the different time periods of the project [13] (scheme and photo of Educational Polygon: Hain).

The case study ‘the reconstruction of Old Power Plant in city Piešťany’ is an example of how to organize work in interdisciplinary partnership in order to integrate and implement Educational Polygon into practice within the existing structure of the restoration process. In addition, the study shows how it is possible to learn and discover new values and possibilities for designing architectures through the Operational research [14] and Mixed mediated reality [15] (Figure 8).

Figure 8.

Educational Polygon and Operations research in practice—case study ‘Reconstruction of Old Power Plant Piešťany 2014’ (scheme: Hain [2]).

4. Case studies

4.1 Presentation by mixed reality in Old Power Plant Piešťany

The first case study described in this chapter is representative by implementation of mentioned methodology from the previous sections. Additionally the research is focused on the use of Mixed reality as an analytical tool of design. This way, the exploration of the new simulation techniques and educational qualities of industrial spaces is connected to the gathering information about users.

The power plant for heavy oil burning in Piešťany was built in 1906 as one of the first of its kind in the former Austro-Hungarian Empire. Later, the plant only provided distribution and energy transformation till the 1990s. The original engine equipment was sold off and the main hall became empty [16] (Figure 9).

Figure 9.

Picture of the virtual machinery hall with machine equipment—at the first stage of the power plant in 1906 and the machinery hall in 2014 (Archival images: Hain).

After conversion, the building is now used as a technical science museum, which interactively educates about the energy and electricity sector. The machinery hall, which originally had six diesel engines and generators, is now a multifunctional room for exhibitions, scientific devices and social events. Retained documents about the original state of the machinery hall allowed the exact appearance to be replicated through VR (Figure 10).

Figure 10.

Typological power plant and archival documents of the building, changes from the National Archive in Trnava from 1906 to 1938 (Archival images: Hain).

In the Mixed reality, the part of real world represents the old industrial building and the virtual part digital objects of original engine equipment that are already gone. Thanks to that, the building itself can be used for multifunctional cultural purposes and at the same time the visitors could find out a lot of interesting additional information about the history of electricity. The exhibition is a hybrid of augmented reality, virtual reality, 3D models and physical industrial artefacts and creates ‘mediated reality’ about industrial heritage. The presentation of a hypothetical reconstruction by VR can serve to bring the history, culture and technology closer to the public.

After creating a spatial scheme of exposure and optimizing the distribution of individual exhibits according to the above-mentioned didactic principles, it was decided within the interdisciplinary team of experts in what form of Mixed Reality the individual parts will be presented. A realistic 3D model was then created for VR (Figure 11).

Figure 11.

The spatial scheme and final results of reconstructed building with realistic virtual presentation—output of Unreal Engine 4 by O. Virág (Archival images: Hain) [16].

Model solutions are defined according to the restoration value of the monument [17]. The materials, proportions and details have been derived from preserved and functional historic diesel engines from the Technical Museum in Vienna through 3D scanning. Photogrammetric processes took 3 days. A 3D remodel of the historic 1906 engine was then created. Based on the interdisciplinary cooperation of STU experts and the analysis of historical documents, the historic appearance and hypothetical scene of the power plant machinery hall was hypothesized, presented via VR and later fully animated.

The movie was accompanied by sound taken from similar diesel engines recorded at the Technical Museum in Vienna (permission granted 2014). The sound was recorded using a camera Canon Eos 20D and Nikon D7000 with microphone (after permission was granted in 2014) and then optimized and purified via Adobe Premiere Pro and Agisoft.

This model serves as a 1:1 reference from which it was possible to analogically capture the proportions of the details (Figure 12) and draw them in new precise 3D model. Based on the archival research and the measurements in situ, we sought to find out whether the initial building was built according to plan in 1906. The next research identified all periods of the building’s construction additions and removals and various stages of the finished look (1920–1945). For this case study, it was decided to visualize the first and oldest period from 1906 [16].

Figure 12.

Historical diesel engine from Vienna Technical Museum and Photogrammetry via software Reality Capture and Agisoft by O. Virág. (Archival images: Hain) [16].

The digital 3D model of the building was created in accordance with the current measurements and compared with historical plans and identified construction phases. Some standard components of the models (Industry Props Pack, Handyman Tool Pack) are from UE marketplace & Turbosquid (screws, watering-can), and graphic works have been carried out with texturing, UV mapping (UV Layout), animation and programming (Textured: Quixel NDO, DDO, Substance Painter & Designer).

The final application runs via the Unreal Engine (Figure 13). Initially the scene was tested with Oculus Rift, which had delays in the synchronization of head movements and caused dizziness of VR users. Finally the new more developed version is compatible with HTC Vive as well [16].

Figure 13.

Final VR 3D model of the virtual presentation was presented by VR headset Oculus Rift in the Power Plant Piešťany, where it was possible to compare the current and historical status on-site—an overlay of physical and virtual reality (Archival images: Hain [16]).

At this point, a user can see an atmosphere of characteristic historical design of space in the original, photo-realistic quality, along with animations and sounds in real time. The 3D model and VR objects were prepared in Unreal Engine 4, which provides photo-realistic images with high-quality textures and lighting. Outcomes are suitable for all these chosen devices: Oculus Rift, HTC Vive, Cyberith, etc. [1].

The VR scene for the Old Power Plant created in 1906 (Figure 13) is designed for the visual communication of technical information, but it also ties in with the diversity of the educational and multisensory exhibition, which is more universal (e.g., for people with disabilities). The target audience represents all the visitors to the hands-on science centre EP (Elektrárňa Piešťany—Power Plant Piešťany), who can not only be entertained but also educated by an exhibition created in this way. The project target group consists of professionals and the general public. Primary school pupils can gain additional educational support from the exhibition. Animators, tutors, lecturers, heritage methodologists, curators, artists and culture administrators can present new findings from the interactive history in practice, in addition to mediating facts from the world of science and technology history.

The created VR 3D model of the machinery hall seeks to eliminate the extreme situations of negative emotions of the space; it is ‘phobia free’. VR respects the senses and aims to eliminate negative emotions, thereby becoming universally appropriate. VR evokes feelings from this environment supplemented by authentic sounds of diesel engines that invoke an industrial atmosphere. At the event that took place on Friday 13 May 2016, the virtual reality project was presented for the first time in the Old Power Plant Piešťany through Oculus glasses (https://www.youtube.com/watch?v=Pk-8gCx03WM&feature=youtu.be).

The presentation in animated Virtual Reality with the possibility of synchronized movement in space is interactive and creates a subjective experience. It uses an audiovisual design, and in the original Old Power Plant hall it is sensually complemented by the historically present smell of black oil (unrefined diesel). This affects the imagination of the observer. It allows him to better immerse, so-called ‘deep-rooted’ and the potential for long-term information storage. At the same time, the presentation of the premises through the VR is a more interesting form for a wider audience of different ages and for people with some forms of disability.

The VR is able to appeal to an age-wide and professional audience, thus ensuring the transmission of the legacy of the non-preserved cultural values of the buildings of the past. Virtual reality has proven to be a suitable tool for commemorating the extinct heritage and reinterpreting its significance for the present (Figure 14).

Figure 14.

VR application testing at the Researchers’ Night 2019 in Bratislava and testing by students of the University of the Third Age of the Faculty of Architecture in Bratislava (photo: Hain).

The virtual machinery hall was tested at the European Researchers’ Night in Bratislava, where it was explored by tracked visitors. The motivation, which induced natural behaviour was taking photos of subjectively interesting motives. Supportive reward system with the impact on the real was publishing of their photos and motions on the second screen. Additionally, the users’ photos were valuable feedback information describing the most attractive exhibition places and motives. After the visit, users were asked to fill short questionnaires about the exhibition’s quality and feelings in VR [18].

Mixed reality a presentation is suitable for people with various disabilities—the possibility of virtual movement without physical movement for people in wheelchairs, for the deaf a visual scene, for the visually impaired an intensive contrast of colours and brightness, and for the blind, a sound experience.

The exhibition itself allows arbitrary graphical design, expression dynamics while saving space and adaptability. Authentic unavailable spaces shall be made available to the public and the diagrams shall explain the operation of the cooling water and fuel pipes to the generators in the engine room. The original equipment is complemented by LCD touch panels with educational presentation schemes in different languages explaining their function and operation during operation, as well as other options for generating electricity. Complemented reality is utilized on an example of an interactive timeline of electricity milestones. The individual points of the axis are traceable via tablets bound to a specific power plant background (wall or floor) by visitors independently. The exhibition space is complemented by an impressive Tesla coil, which is suspended on steel ropes and throws lightning over the heads of visitors. At the bottom of the turret room is a Van der Graaf generator in the form of a stainless-steel ball that bristles the hair of the person who touches it (Figure 15).

Figure 15.

Mixed reality exhibition in the Old Power Plant Piešťany with augmented reality, virtual reality and of original engine equipment (mixed reality design: Hain, Ganobjak).

The absence of a virtual avatar body in the VR as reported by visitors was a strange experience with feelings of disorientation and confusion, although it is disputable if the presence of an avatar body in VR would have avoided those feelings. Augmented reality, accompanied with the use of physical reality as an anchor for position and navigation, appears to be a further tool for effective education, with the brain effectively distinguishing the essence of a variety of information at a real place. Virtual reality has also shown in this case study to be useful for presentations at several events outside the industrial heritage site.

The Mixed reality visit of the industrial space teleports the viewer into a virtual scene where it is also still possible to look around in a traditional manner. Virtual reality allows the handicapped to perform virtual movements without physical effort to places/through place where it would otherwise be impossible to go.

In this case, Oculus was more useful than HTC Vive (depending on the mobility of physically impaired persons). The same virtual scene is perceptible from the perspective of a pedestrian. The perception of users and feeling of size could be changed (the visitor is like a giant and the scene is only a scaled model, or vice versa).

The opportunity to experience a future, fictional world, to take a walk in the past or virtually teleport to other points of interest is opened up through VR presentations. Visual perception is supported with realistic materials and textures. Experience in a VR scene installed in the original Machinery Hall is supported by the real in situ scent of heavy oil that is still possible to smell in the existing premises.

Virtual reality with synchronized movement enables the visitors to view the exhibition from anywhere, even from outside Piešťany, it is possible to walk in the historic yet nonexisting interior of the Machinery Hall of 1906. Synchronized movement in virtual and physical reality is compelling and confirms the meaningful use of Mixed reality as a vehicle for presenting the defunct cultural (industrial) heritage against the backdrop of a direct comparison of the contemporary and the original state [18].

4.2 Presenting in situ Bratislava: sense of sight, smell and hearing

The further studies show other attempts to present the revitalization of heritage and future architectural designs by mixed reality. The subsequent study has compared mixed reality by combination of virtual environment, sound and smell of real exterior environment. The presentation was an outcome of the interdisciplinary cooperation of FA STU, Pixel Federation and Eurosense. For that study, three students’ projects have been prepared for different types of virtual reality: Oculus Rift, Google Cardboard and HTC Vive in Unity Engine. The presented projects included the proposals for revitalization of Danube River bank and old industrial bridge, near the forested site in Bratislava. In the study, the involvement of the visitors, their willingness to discuss and their ability to link the projects with the real site were observed.

The visitors participated when the projects were presented in situ, near the forest and in the university building, away from the real site. When presenting in situ, the primary environment was the local real environment, but when the presentation took place in the university, the primary environment was the remote real environment (Figure 16). The projects were presented in situ with Oculus Rift and Google Cardboard. Away from site, the projects were presented with included sounds from site by these technologies and with HTC Vive, which allowed users to move more naturally.

Figure 16.

Left—Presenting the projects in situ. Right—Presenting in the university (photo: Hajtmanek, 2016).

The three projects were similar in the means of orientation; the main dominants (main building, old industrial bridge and Bratislava Castle) were on the same places with the same visual and road connections, but the projects differentiated in the way of displaying and architectural form. One of the projects was displayed monoscopically (both eyes had the same image), the other two projects were displayed stereoscopically. All the projects were included in one exhibition application used on all mentioned VR technologies, so every user could visit all projects successively. One of the stereoscopically displayed projects had fluid architectural form, without recognizable architectural elements as columns, windows and doors. The other projects had more usual form with recognizable architectural elements. The summarized projects with their properties are shown in Figure 17.

Figure 17.

Presented projects. Left—Project with recognizable architectural elements, presented monoscopically. Middle and right—Projects presented stereoscopically, the project on the right was the one with fluid form (Archival images: Hajtmanek).

The study showed that users did not notice that the one of the projects was presented monoscopically; but in this project and in the project without recognizable architectural elements, the users had problems with orientation. On the other hand, the problems with the orientation were lesser while presenting in situ. The comparison of the users’ ability to orient in the projects is summarized in the Table 1.

	Presented in situ	Presented in the remote environment
Project with familiar architectural elements, monoscopically presented	Lesser problems with orientation	Problems with orientation
Project with familiar architectural elements, stereoscopically presented	No problems with orientation	Lesser problems with orientation
Project without familiar architectural elements, stereoscopically presented	Lesser problems with orientation	Problems with orientation

Table 1.

Comparison of the projects with different ways of presentation by users’ ability to orient in them (author: Hajtmanek).

The ability to see, smell and hear the sounds from the local real environment helped to better blend and understand the proposals with the reality. The visitors of in situ presentation were also more attracted and open to discussion by the proposals as they could directly compare it to the real situation.

Used different technologies of VR for such a presentation showed that they did not have the effect on the orientation, but they influenced natural behaviour of the users. The HTC Vive was shown to be most suitable tool for similar presentations, because it allowed the users to move more freely in real and virtual space simultaneously.

4.3 Augmented physical model: sense of sight and touch

The Mixed connection between virtual and real was shown to be a proper tool to present the future proposals of new use of other historical heritage. The subsequent study examined the presentation in scaled physical model by augmenting it with the virtual layer, thus combining touch and visual senses. The combination of visual senses from real and virtual to improve the perception of scale and proportions on physical model of the designed environment has a long history. One of the first attempts to achieve the correct perception of scale of the physical model was filming it by special camera simulating the first-person movement [19]. The setup for this filming is shown in Figure 18.

Figure 18.

Laboratory for model simulation FA STU in Bratislava (Kardoš, 1999) [19].

Today, it is possible to merge the virtual and real by using the tools for augmented reality and augmented virtuality. In cooperation with Studio Hani Rashid, University of Applied Arts in Vienna the augmented model of speculative proposal for Museum of Futures on Heldenplatz was made and presented on Vienna Speculative Futures Exhibition. The virtual layer was mapped on the physical model by tablet and Vuforia for Unity. The system precisely showed kinetic and programmatic capabilities of the building (Figure 19).

Figure 19.

Augmented model of Museum of Future on Heldenplatz. Left—Setup of the exhibition with the marker. Right—Running of the application on tablet in the exhibition without any markers (Archival images: Hajtmanek).

This way of presenting showed its weaknesses, when it was compared to the mixing of the visual senses from virtual reality with local real surrounding environment. The interaction with the augumented reality model can be observed and interacted by more users together. However, the scale was not understood completely in the comparison to the visiting the model through the augmented virtuality. On the other hand, visitors comprehended the bigger picture of the building’s program and its surroundings. Nevertheless, this way of presentation attracted the attention of visitors and showed potential for detailly presenting the smaller objects in product design and building context with its surroundings in architectural design.

5. Spatial evaluation and predictions of users in situ

Inviting the observers to visit the old and future spaces in augmented virtuality proved that they behaved very similar to the real situations, because they intuitively related the virtual environment to the real one. This relation between the real and virtual was further explored to use the user’s behaviour as feedback for designing the new presentations and future spaces. In one of the studies of Old Power Plant Piešťany, the users’ movements and gaze were recorded to predict their behaviour in new exhibitions by machine learning model [18].

Similar approach was applied to evaluation of co-working offices of Hub Hub in Bratislava. The digital representation of offices was modelled, prepared for VR and presented in the real offices. It was visited and evaluated by the local co-workers, who had the best knowledge of the real space, which they were using (Figure 20).

Figure 20.

Left—Virtual model of the offices. Right—Evaluating of the space in situ (3D model and photo: Hajtmanek, 2019) [20].

The users had the task to choose the best place in the spaces and subjectively evaluate its openness, height, contact with exterior and illumination. The evaluation was recorded via gradient spots, creating heat maps in the plan of the building. The evaluation was expressed by white colour—nevertheless the spot was, the evaluation was more positive. In the virtual model, the same evaluated properties of space (openness, height contact with exterior and illumination) were precisely measured in every position from the grid in module of 60 by 60 cm. This information was noted via RGB channels to small textures—samples of size 16 by 16 pixels (Figure 21).

Figure 21.

Left—Users’ evaluation of the space. Right—Measured parameters of the space noted via small textures (Hajtmanek, 2019) [20].

This way, every position in the grid had the values of subjective evaluation and measured properties of the space. On these data, the artificial neural network (ANN) was trained by supervised learning. ANN learned on data from small offices and terrace and then it was tested on the space in the middle. The comparison between the original and the predicted evaluation on the testing space proved that the final model could predict the evaluation of the users from new given spatial parameters. The evaluation of the openness was most effectively predicted with accuracy of 90, 25% (Figure 22).

Figure 22.

Comparison of original and predicted evaluation of the openness of the tested space. Evaluations are coloured (blue—positive evaluation, black—negative evaluation) and blurred to blend sampling (Hajtmanek, 2019) [20].

The study showed that the relation in between the simultaneous perception of virtual and real by all senses is possible to learn by machine learning model and use it as evaluation tool in architectural design of future spaces, which are similar to the evaluated one in the study. This feedback loop between the designer and users could bring the more effective and better suited future environment.

6. Discussion

In the multiple studies, the relation between virtual and real was explored by combining different sensorial stimuli. Combination of the senses of smelling, hearing and touching the real environment with the visual sense of virtual environment showed that viewers behaved more usually, because they easily related the virtual environment to the real one.

In such presentations, they realized and perceived the scale and proportions of the presented objects more properly as seeing on the plans, scaled physical models or screens. On the other hand, combining of touch and visual stimuli from real environment and visual stimulus from virtual environment on the scaled physical model showed that perception of scale was not trivial.

To provide the better idea of scale in this combination of stimuli from real and virtual, it would be better to see the physical model, choose the position and then visit it from the first-person view in virtual reality or by the camera, which implies that in these studies, the use of augmented virtuality could be suited better than application of augmented reality.

The research raises questions about VR’s usefulness, relevance, controversy and entertaining applications. Numerous psychologists also suggest that inappropriately applied VR may constitute a risk: being cut-off from the real world and creating a brain fallacy by optical illusion is unnatural and in the long-term risky. In this case study, VR as a practical tool enables the public to learn about by-gone heritage. Even with the numerous controversial VR uses, this example of VR could be considered meaningful and beneficial in practice [21].

7. Conclusion

Mixed reality (MR) in the presentation of industrial heritage requires thorough knowledge and evaluation of the subject, causality—with a strong theoretical background and a target-oriented assessment perspective of the presentation and education level.

The principle of interdisciplinary cooperation is not only synergistic element in a complex scheme of organized design process, but also a key educational element in mixed reality.

The case study through MR has reinterpreted the history of the cultural industrial heritage, which was not possible to recover in physical reality, and has brought it to a contemporary audience. Through this practical interactive tool, the general public can learn about lost heritage. Interactive virtual parts can be embedded in conventional channels and animations controlled by focusing on specific objects.

User tracking and the whole principle of interdisciplinary cooperation is not only a synergistic element in a complex organized design process, but also a key educational element in the protection of the local industrial heritage for involved participants.

However, each case of heritage management requires a specific and detailed study of the subject. Therefore, the study aims to serve as an initial model for further studies on the application of Mixed reality in the preservation and educational management of industrial and cultural heritage.

Acknowledgments

This project has been supported with public funds provided by the Slovak Arts Council FPU 16-362-03415, by the subsidy project Supportive Program for Young Researchers SUPNVN and project KEGA 038STU-4/2017.