Augmented reality in BIM SWOT analysis.
Building Information Modeling is an increasingly common process for managing the entire lifecycle of a building - from design and planning, through the construction phase, to operation and maintenance. The result of this process is a building information model with all the generated data and information about the construction process that can be used in a variety of different end-user scenarios. One such use of the model is in a number of different augmented reality applications. Augmented reality technology is being used to bridge the gap between the digital and real worlds and is rapidly becoming an essential part of modern building data modeling design workflows. The chapter provides an overview of building data modeling and the current state of the art in the use of augmented reality in various user scenarios of building data modeling and explores various challenges that need to be addressed for the adoption of augmented reality technology in architecture, engineering, and construction in general.
- augmented reality
- computer integrated engineering
- civil engineering
- project documentation
- building information modeling
According to , Building Information Modeling (BIM)  is a set of technologies, processes and policies enabling multiple stakeholders to collaboratively design, construct and operate a Facility in virtual space. The result of a BIM process is a building information model that a shared digital representation of a built asset to facilitate design, construction and operation processes to form a reliable basis for decisions. The term BIM continues to evolve over the years and is thus best understood as an ‘expression of digital innovation’ across the construction industry and the overall built environment and is an increasingly common process for managing the entire lifecycle of a building - from design and planning, through the construction phase, to operation and maintenance. As noted by D. Richard and C. Harty  highly structured and semantically rich 3D geometry information is in practice used on physical 2D medium, e.g. paper or digital displays. So at the end, basic concepts remain the same even though we now have access to much richer information with BIM models.
As not much has changed on the conceptual model of translating virtual 3D plans into real world structures, engineers still have to rely on their knowledge, experience and spatial awareness to map these virtual plans, that may include 2D drawings (plans, elevations and sections) or 3D models (viewed on a 2D display medium) into real environment (see also Figure 1). This challenge has been recognized by different researchers, including P. S. Duston and H. Shin  that proposed this mapping process be facilitated by mixing virtual information with the real environment. A technology that enables mixing digital information with the real world environment objects and spaces is called augmented reality (AR) .
AR Systems generally consist of three main phases: Data Phase, Computation, and Presentation. The data phase is primarily concerned with the creation, curation, and formatting of data. In the context of BIM related workflows, this means designing a BIM model so that it can be used as a source for augmenting reality. Merging virtual data (3D geometry, specific element information, etc.) and real environments is a computationally intensive phase and can take place on the mobile device or on a remote server. Finally, the display of the mixed visualizations can be done on handheld mobile devices such as smartphones and tablet computers and (2) head mounted devices. Several AR systems already exist; however, they are mainly used to display proprietary models prepared using specialized custom software, including the software that was used for this research .
The chapter provides an overview of building data modeling and the current state of the art in the use of augmented reality in various user scenarios of building data modeling. Specifically, it describes various challenges that need to be addressed as well as a spectrum of different end-user scenarios and use-cases for use of AR technology in architecture, engineering, and construction (AEC). As part of the conclusions, the SWOT analysis of using augmented reality system in to context of BIM is also provided.
2. Building information modeling
The roots of Building Information Modeling (BIM) can be traced back to the first ideas on how to use the concept of product models using various media in architectural designs , as reported by Russell and Elger  regarding the introduction of BIM into the AEC market. And following this evolution, the beginning of the Industry Alliance for Interoperability (IAI) in 1995, and from experiences established a standard for describing buildings, which would allow the exchange of information about buildings without the loss of their semantic information . This working format is called Industry Foundation Classes (IFC) , having been published for the first time in 1997.
Following the definition of BIM by the US National Building Information Modeling Standard (NIBS 2012), BIM digitally represents the physical and functional characteristics of a building, enabling the collaboration of different stakeholders throughout the building’s lifecycle to enter, update or modify information in the BIM model. BIM is always evolving, seeking to optimize technology according to the complexity of the processes applied in civil construction and the construction community is looking to innovate with BIM through specific workflow tools such as VR and AR that are being applied directly to solve real-world problems as installation verification and estimating .
The essence of building information modeling is the ability to add useful information  using BIM models that are more than geometric representations; therefore, they can be viewed in various dimensions, from 3D (design planning), 4D (scheduling), 5D (costing), 6D (life cycle information) and 7D (facility management)  (Figure 2) and, the data models used differ from each other according to the schema used both to organize the data and the schema language to transport the data.
The IFC data model it is an open and platform-neutral file format, facilitating interoperability in the architecture, engineering and construction industries, being used through specialized BIM programs and therefore with its own platforms such as Revit, ArchiCAD, Navisworks, Bentley, among many others. Since IFC is an open standard it enabled development of different translators from one data format to another, for example from IFC to XML .
3. Augmented reality
Current virtual reality technologies are based on ideas constructed and reported from the 1960s and possibly earlier, such as the iconic Ivan Sutherland who in 1968 created the first head-mounted display that rendered simple wireframe models for pose change of the viewer . It was through the foundations of this invention and with technological innovations and other evolutions that we now call Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR) (Figure 3) .
With the concept of VR conceived more than fifty years ago, when the first immersive human-computer interaction (HCI) mock-up called “Human-Machine Graphic Communication System” was invented  the formal term of VR was placed in 1989 on the RV continuum based on Milgram’s taxonomy where VR represents the creation of a virtual reality or. environment in which the user can enter this scene with the feeling of being in the “real” world, with limited level of “realism” such as visual and sound effects .
According to this definition, AR is an environment where additional data generated by the computer is fed into the user’s view of a real scene . With AR, users can access, visualize and interact with complex information in the context of the real environment, or in other words, computer-generated elements are added to the seen reality. Since the entry requirements for a AR capable device are relatively low, many of today’s smart devices (phones, tablets) are suitable for AR use.
With the growth and maturation of technologies in AR, applications end up becoming more viable and popular for both the education, design, manufacturing, construction and entertainment sectors, becoming potential in helping to improve existing technologies and, with that, can promote a better quality of life mainly for people with physical and/or mobility limitations.
4. Augmented reality in BIM
In the context of augmented reality and BIM we can define augmented reality can as system in which BIM model is used to augment real environment. An AR enhanced BIM modeler could also be envisioned where BIM model information is edited in an AR environment. AR is currently being considered by scholars and practitioners in the area of knowledge, as a “New Age” of information through advanced technology due to the positive effects of its technological potential that its correct use offers to the construction industry, especially in the construction phase, providing significantly affects the efficiency of projects, quality, health and safety , and consequently the project cost and duration, positively .
In Section 2. we already established BIM as a type of a “central hub” to all data and information related to a specific projects – this of course includes 3D models necessary for AR applications as well as non-geometrical information that could also be linked and accessed in AR applications.
But combining real world objects with virtual ones can be challenging. Researchers Bajura M and Neumann U  identified four main challenges to be addressed: (1) the origin of the tracking system is not aligned with the world coordinate system; (2) the transformation from source to object is not accurate; (3) the position of the virtual camera is not correct – usually related to inertial and motion-based sensors errors; and (4) virtual camera-to-image mapping does not accurately model the real camera.
To organize the data processing steps for the final BIM visualization in AR, Williams et al.  describes a generalized three step workflow (similar architecture is also proposed by Meža et al. ): (1) First, it is necessary to generate geospatial properties for each BIM object where mobile AR application uses geolocation to identify the user’s position and, consequently, be able to produce information related to the viewed object; (2) Second, from the moment that the information displayed in mobile AR refers to a user’s position, several points surveyed within BIM also need to be identified. These points will represent where users can stay in a physical location and perform mobile AR tasks with BIM data sets related to that location; and (3) Third, for BIM to be usable in a general mobile AR environment, the geometry and property data set needs to be separated into two exchange formats.
5. Use-case scenarios
In the context of BIM workflows, the use of augmented reality is usually associated with three basic use-case scenarios : (1) design phase - review of proposed solution, (2) construction phase - monitoring of construction progress  and (3) operation phase – building maintenance . But as identified by C. Woodward and M. Hakkarainen  many other relevant use-case scenarios should be considered, including layout optimization, excavation, positioning, inspection, coordination, supervision, commenting, etc.
In the following subsections, three different representative use cases are described. Two of them are based on infrastructure projects (railway/tunnel, bridge) and the last one is based on an office building. In all presented use cases, a BIM specific mobile application  has been used to access BIM models as well as to use the integrated AR solution. The mobile apps use a BIM shared data environment solution that provides a central hub for all person-to-person communication, data sharing, etc.
5.1 Use-case 1: infrastructure (design and planning phase)
The project used for this use-case is a construction of the second railway line from Divača to Koper that includes construction of more than 17.4 km of access roads with various structures serving as service lines for the construction of tunnels, retaining walls and a bridge approximately 35 m long. It is one of the largest infrastructure project in Slovenia.
The first use-case clearly falls into the design and planning phase of the full life cycle using AR for visualization of the construction site organization as well as visualization of the completed structure in real environment. One of the main goals for the use-case was also to examine how AR technology and its implementation in the selected mobile application can be used for an infrastructure projects that are defined primarily with long distances.
We used a tablet computer with augmented reality application to test the use-case in two different locations that are a few kilometers apart. However, both are georeferenced in a 2D model on the tablet (Figure 4) and correctly link the required information in the 3D model.
Its integration through AR technology to visualize interactive 3D models on site [24, 25] (Figure 5). As seen in the images bellow, it was possible, for example, to check the tunnel portal and MEP installations, railway track to be build, as well as to get the general scale and complexity of the construction site.
Key takeaways: (1) it is absolutely essential that all BIM models are correctly georeferenced for use by AR system, (2) use of tablets/phones can be challenging in bright environments, (3) difficult to orient yourself in the field and correctly position and scale BIM model, (4) can be used for general understanding of future construction site, and (5) determining geolocation can be challenging and it is depended on geographical characteristics of the construction area.
5.2 Use-case 2: infrastructure (operation and maintenance phase)
The project used for the second use case is an already completed infrastructure project - a new bridge (Figure 6) over the Savinja River in Marija Gradec, near Laško. The bridge is 123 m long and up to 12 m high and designed as an anchored containment structure. The bridge construction started in mid-July 2019 and was opened for traffic in February 2021.
The use case clearly addresses the use of AR in the operation and maintenance phase (7D, see also Figure 2) of the life cycle [26, 27]. BIM model (3D geometry only) was available and correctly georeferenced via the mobile platform used . This made it possible to obtain real-time architectural visual interaction and communication. Another important point for the choice of this site as a case study is precisely the counterpoint that apart from the 3D model BIM in IFC format as basic information, no other accessible database is available, which imposes major limitations for its use and visualization in augmented reality. Overall, it is possible to check the 3D model BIM and compare the same coordinates on Google Maps as shown in Figure 7.
Key takeaways: (1) it is essential for the of AR system, that BIM model includes additional information to basic 3D geometry, (2) positioning/scaling of BIM model in AR system possible due to available reference points, (3) difficult to use tablets/smart phones in bright sunlight.
5.3 Use-case 3: office building
The construction of the industrial complex “Industrial and commercial building Iskra Mehanizmi Brnik” (Figure 8) is structurally divided into 3 main blocks: the commercial part, which consists of a monolithic reinforced concrete structure and two production and storage units made of prefabricated reinforced concrete.
The use case focuses on the Iskra Mehanizmi office building (Figure 9). It is a complex structure that is in the final stages of construction. The example can be used to demonstrate the use of AR technology at different lifecycle stages (see also Figure 2). A complete common data environment was available for the project, which contained various 3D models BIM with all associated information, allowing the use of AR.
The model BIM supports geometric and non-geometric design information such as location through geo-referencing (3D) and technical specifications (4D) from door and window manufacturers as well as maintenance information (5D) (Figure 10a). The convergence between the 3D model BIM and the AR (Figure 10b) is practically done through three main components: the 3D model BIM itself, the whole readout and the transformation of the data for interpretation in augmented reality through an appropriate application on a mobile device, in this case a tablet was used. To enable visualization on the AR platform, the 3D model BIM sends and receives information BIM, such as installation adjustment requirements and work plan review, allowing the user to interact with the 3D model BIM and other members of the project team in real time (Figure 10c). BIM models are created based on 2D design drawings and with information included in the design specifications according to the client’s requirements. In this case, there is a marker (orange rectangle) that refers to the requested information (Figure 10d). After clicking on the orange rectangle, a new window immediately opens with the details of the information in a 2D drawing with comments about the location, the type of installations performed, and what needs to be done to complete the jobs (Figure 10e). The levels of construction details included in the same information can be structural, architectural, MEP installations or others required to complete the project (Figure 10f).
One of the main objectives for the use case was also to test the use of AR inside a building, where it is usually impossible to collect GPS signals for geolocation of the stand position. The mobile solution used in the example, AR , uses a manual geolocation method where the user (1) selects an approximate position in the 2D plane and (2) rotates and scales the 3D model based on reference points (windows, doors, columns, etc.).
Key takeaways: (1) determining geolocation within a building can be challenging and different AR may take different approaches to solve this, (2) positioning and scaling a BIM model in the AR system is possible due to the available reference points, (3) BIM models are usually more complete for buildings compared to infrastructure objects (bridges, tunnels, etc.), therefore the AR system can make better use of the available information.
This paper mainly focuses on the major application scenarios mentioned by various researchers and practitioners in the field of architecture, engineering and construction. The advances in the digitization of all processes and workflows in the AEC industry in general and beyond with the wide adoption of BIM as a common methodology for managing large construction projects enables the use of advanced information communication technology (including, augmented reality) in AEC workflows. It is not emphasized that the approach of AR is better and more effective compared to others, but it is a helpful complement to other technologies for managing a building life cycle.
Through AR it is possible to obtain a special feature, namely “instant visualization”, which facilitates communication and decision making between the parties involved in the project. According to Wang J et al. , AR inherently involves human interaction with real and virtual information sources. Within BIM technology, the use of AR allows designers to place the virtual construction schematic in a real physical environment; it grants owners an engaging and interactive experience and suppliers the ability to communicate effectively with both clients and the technical team.
The case studies are different, the first being a road infrastructure project, more specifically a tunnel (already completed) and the second an architectural project for a commercial/service building (under construction), both with 3D models from BIM, which allowed visualization in AR. Table 1 shows the SWOT analysis for use of augmented reality in BIM based on the above presented use-cases and key takeaways from those examples.
Future research should generally focus on three main themes: (1) improving the understanding of information transfer processes, (2) enhancing software solutions, and (3) better understanding the required information contained in BIM models.
By improving the understanding of information transfer processes, we mean determining the tasks that could benefit most from AR technology . It would also be interesting to explore an alternative information flow where building information models could be generated or updated in a real 3D space, on mobile devices, rather than on 2D computer screens in offices. It is expected that building information modeling software vendors will improve their products in this direction, which would also be based on research such as that conducted or reviewed in this paper.
Computer market leaders say augmented reality will change everything. But right now AR still has many challenges to overcome (improving occlusion, the process of creating 3D content and asset quality, connectivity, computing power of devices, miniaturization of hardware components …) and if AR on mobile phones is the first step and most AR experience for a few years, AR on smart glasses will completely change the user experience. With smart glasses, users will have access to hands-free experiences, with content displayed right in front of their eyes and a full augmented view of the real world.
The presented research was supported by Erasmus Mundus Joint Master Degree (EMJMD) - BIM A+ European Master in Building Information Modeling (https://bimaplus.org). Their support is gratefully acknowledged.
Conflict of interest
The authors declare no conflict of interest.