Cultural Heritage in Marker-Less Augmented Reality: A Survey

Abstract

Augmented reality (AR) is considered as one of the most significant technologies in the field of computer graphics and is utilised in many applications. In this chapter, we have presented a brief comprehensive survey of cultural heritage using augmented reality systems. This survey describes the main objectives and characteristics of marker-less augmented reality systems through presenting up-to-date research results in this area. We describe the marker-less technologies in the area of AR, indoor marker-less AR, outdoor marker-less AR, real-time solutions to the tracking problem, real-time registration, cultural heritage in AR, 3D remonstration techniques, as well as presenting the problems in each research.

1. Introduction

Augmented reality is a technology in which user’s view of a real scene is augmented with extra virtual information. Augmented reality (AR) registration of virtual objects is required having an accurate tracking or camera pose estimation, but tracking is one of the key technical challenges of AR system [1]. AR has many conceivable applications in a wide range of fields such as education, construction, public health, manufacturing and entertainment. With the increased computational speed and advancement of particular computer technology, augmented reality applications become possible in multidisciplinary fields, for example, education, simulation, entertainment, medical and games. Researches related to augmented reality (AR) and virtual reality (VR) have shown significant growth with the development of interactive computer technology and sophisticated 3D modelling packages [2]. Virtual heritage is considered one of the important fields in the computer-based interactive technologies in virtual reality [3]. It created visual representation of monument, artefacts, building and culture to present openly to global audiences. However, virtual heritage becomes as a platform for promoting the education process, motivating and understanding particular events and historical elements for the use of students and researchers. Augmented reality techniques can be classified into two main categories: vision-based AR and location-based AR [4, 5]. Location-based AR uses the capability of a specific device to determine its position in the world, for example, GPS and then the retrieval relevant information to that location. Then, this information is superimposed into the output of their device’s camera to permit a more natural data presentation compared only using the map alone. Vision-based AR particularly depends on processing the data that is extracted from the images or video frames that have been taken by the device. This kind of AR includes a number of techniques that lend significantly from computer vision to the range, where research progress in AR relies on the progress of the latter [6]. Lately, augmented reality technology has become an accepted technology among scientific community and even public, which is used for merging of real and virtual objects, and mixed it into the real-world environment. However, this technology is used in virtual heritage to improve the visitor experience of a cultural heritage site, as well as, the possibility to present the ancient-ruined building without any damage. In this chapter, we have presented a survey of marker-less AR. This survey is based on the state-of-the-art related to marker-less AR such as indoor marker-less AR, outdoor marker-less AR, real-time solutions to the tracking problem, real-time registration and cultural heritage in AR. Section 3 introduces the marker-based AR and Section 4 is allocated to marker-less AR, while Section 5 presents the researches related to cultural heritage in augmented reality.

Section 6 reveals the issues with virtual heritage in augmented reality and Section 7 presents 3D reconstruction techniques for cultural heritage. Section 8 is all about location in AR. The work is concluded by section 9 which is the conclusion.

2. Augmented reality

Augmented reality is considered as one of the modern technologies that blends virtual objects into the real world. Augmented reality (AR) can be simply defined as a live and integrate direct or indirect view of a physical, real-world environment. It is real-time data that elements are augmented by computer-generated virtual content, for example, sound, video, graphics, or GPS data [7, 8]. Augmented reality is considered as an area in which 3D virtual objects are completely integrated into a 3D real environment in real time. An AR environment supplements the real-world with virtual objects, which are generated by using computer that appear to coexist in the same space as the real world [9]. In another words, augmented reality can be defined as the interactivity of humans with virtual objects that is located in the real environment in order to help the user in executing a task in a physical setting. AR is one of the significant forms of mixed reality (MR), in which real and virtual objects are mixed and showed in a single display in the same time and location as shown in Figure 1. Augmented reality seems like fiction because it creates interactive interfaces that specify the delusion that physical and virtual worlds are connected together and that users can physically cross from one to the other [10].

Figure 1.

Reality-virtuality continuums [23].

AR does not replace reality, as the virtual reality (VR); it complements real environment with digital information, virtual and computer-generated graphics and/or virtual objects as shown in Figure 2. However, users navigate in VR by using a computer simulated or imaginary environment called a virtual environment, preventing the real environment. In this environment, all users’ senses are controlled using a computer and immersed in a simulated environment [11] as shown in Figure 3.

Figure 2.

A real environment view is augmented with digital information.

Figure 3.

A user navigating in a virtual environment of VR.

3. Marker-based augmented reality

Marker-based AR uses markers (A two-dimensional pre-defined screen) that are placed in the scene and within the field of vision of the camera in order to help guide the camera pose estimation process [8]. The markers are frequently indicating to as fiducial markers because their position and orientation relative to the scenery are steady. The markers are always planar makers and commonly have powerful feature, for example, long edges, as well as, corners among black and white regions. In this technique, AR puts a powerful emphasis on the design of the marker. One of the most common kinds of marker design is square because the feature of square will allow for accurate localisation of the markers by using its four corner points [12]. Marker-based AR uses computer vision techniques in order to calculate the position and orientation of the camera relative to the marker. The virtual 3D objects can be overlaid accurately on the markers as shown in Figure 4. It has a primary operational principle: capture the video input from the camera, add 3D graphics to the scene and show the augmented frames as a video stream [13].

Figure 4.

Marker-based AR [14].

4. Marker-less-based augmented reality

Marker-less AR is completely different from marker-based AR because it does not depend on the artificial markers in order to reveal outstanding features in the scene. Marker-less AR systems work to integrate virtual objects into a 3D real environment in real-time, promote user’s perception of and interaction with the real world [15]. Marker-less AR works by revealing features that are easily available from the natural objects in the scene, as well as, try to create a model or map from the scenery in order to represent the world as it is displayed by the camera.

4.1 Camera pose estimation

There are two main methods to camera pose estimation techniques called relative orientation and planar homography. Relative orientation is an approach that used to calculate the position and orientation of a camera relative to another from correspondences between five or more ray pairs. A ray pair can be defined as the vectors that arise from a fixed and visible point in the scenery to the camera centre positions [16]. Use the aspect of AR process that recruit computer vision algorithms, called feature detection and tracking, and then suggest a method to improve the subsequent process to output a best camera pose estimate [16]. A localised feature descriptor is used for the matching of salient feature points belonging to the present camera frame with those extracted from the reference frames. Camera pose can be estimated relative to it, however, the calculated 3D pose parameters can be used in order to render virtual objects into the real world [17]. Frikha et al. are proposed real-time monocular piece wise planar SLAM method using the planar scene assumption. Planar structures have used for mapping process in order to allow rendering virtual objects in a meaningful way, as well as improving the camera pose resolution in addition to the quality of 3-D reconstruction of the environment by adding restrictions on 3-D points, and settings in the optimisation process [18]. An energy function based on epipolar geometry has been developed in order to estimate intrinsic camera parameters during camera zooming [19]. Intrinsic camera parameters at each zoom value are calibrated, in order to obtain an accurate camera parameter estimation. The intrinsic camera parameters changes depending on the zoom value that are modelled [19].

4.2 Outdoor marker-less-based augmented reality

Augmented reality is the real-time incorporation of the virtual and physical worlds into a new environment, where digital information is registered with real-world elements in a coherent method. One of the big challenges when working in outdoor AR is the registration of the virtual elements in the real-world environment, where it is not realistic to prepare every building with visual markers. This issue is certainly much more accurate when dealing with outdoor augmented reality. Most augmented reality applications are taking the benefit of backpack systems with head-worn displays [20] or handheld devices [21] in order to compose real-worlds’ views with digital information. Sophisticated hardware contains tracking devices, for example, GPS and gyroscope, which can be used to determine the position in the physical world.

Bateau Ivre [22] project have presented on the Seine River in order to make a considerable audience conscious of the possible developments of augmented reality through an artistic installation in an outdoor environment. The installation can be seen from a ship by a huge number of audience without specified equipment, through night-time video-projection on the River banks. The augmentation of the physical world is implemented using real-time image processing for live special effects, for example, contouring, particles and non-realistic rendering. The technical objective of the project was to immerge the audience into a non-realistic view of the River banks that would be different from traditional tours that highlight the main features of Paris’ classical architecture. The implemented software is used in standard algorithms for particular effects to a live video stream and then re-projected these effects on the captured scenes to merge the real world with its modified image [23].

However, Sato et al. [5] have developed a novel marker-less AR system that uses local feature-based image registration and structure from motion (SfM) technology. The proposed system has some advantages, such as it supports free movement, less limitations, less efforts, as well as lower cost for outdoor AR applications. For the verification of the developed system, it has been applied to a renovation design project. One of the main advantages of the system is that it does not require particular equipment, for example, sensors for geometric registration between augmentations and the real world because the system uses sensor-based registration. Furthermore, the system does not need artificial markers, which reduce user’s flexibility [5]. The accuracy of system’s registration and tracking for this research is not enough for AR.

A development of a 3D map oriented handheld AR system has been presented by Chen et al. [24]. The system achieves geometric consistency by using a 3D map in order to obtain position data instead of using GPS, which provides low position information accuracy, especially in urban areas. In addition, the system features a gyroscope sensor to obtain posture data, as well as a video camera that used to capture live video of the present surroundings. All these components are installed in a smartphone and can be used to assess urban landscape. The authors have used the evaluation of registration accuracy in order to simulate an urban landscape from a short- to a long-range scale. The proposed AR system allows users to simulate a landscape from multiple viewpoints in addition to long-distance simultaneously, as well as walking around the viewpoint fields using just a smartphone [24]. The proposed system has the optical integrity and occlusion problem of the 3D-AR system when simulating urban landscape.

In addition, Chen et al. [25] presented tracking natural features in an agricultural scene. The main objective of the system is to perform marker-less AR techniques in order to assist in the visualisation of robotic helicopter-related tasks. By creating a virtual marker under a known initial configuration of the robotic helicopter, camera and the ground plane, the system is able to continuously track the camera pose using the natural features of the image sequence to execute augmentation of virtual objects. A simulation using a mock-up model of an agriculture farm have developed to evaluate the performance of the marker-less AR system. The experiment results showed that there are a number of improvements, which need to be taken in consideration before distributing the system in actual flight. The intermittent movement of the virtual marker vertices must be reduced in order to obtain better camera pose estimation. A feature recovery algorithm is one of the most important techniques for scaling the marker-less AR system to operate outdoors on the robotic helicopter [25]. This technique is trembling in the virtual marker vertices. Therefore, camera pose estimation accuracy is low.

5. Cultural heritage in augmented reality

Virtual heritage in AR can be defined as an interactive computer-based technology, which can be used to achieve visual reconstruction, assist scholars and educators of traditional entities, for example, buildings, artefacts and culture [26]. This technology is used to maintain delicate historical buildings from natural disasters and sabotage [27]. In order to create a virtual heritage, there are seven main design principles, which must be taken into account such as high geometric accuracy, high level of automation capture for all details, low cost, photorealism, flexibility, portability and model size efficiency [28]. Cultural heritage layers are proposed to visualise historic media such as drawings, paintings and photographs of buildings and historic scenes seamlessly superimposed on real environment through video see through using X3D [29]. The registration of the virtual objects in the video images is done by using a robust 6DOF tracking framework depending on two technologies that work simultaneously: randomised trees are used for initialization step and a frame-to-frame tracking phase based on KLT. This technique achieved simple, cheap and sustainable development augmented reality applications in the area of the cultural heritage depending on industry standards [30]. The main idea of this research is to use current historic media from archives and superimpose them seamlessly on reality at the suitable place. These local layers are context sensitively telling the location’s history and give the impression of a virtual time trip. The results of the application showed in the area of cultural heritage, where the system runs on an Ultra Mobile PC (Sony Vaio UX) with 15 frames/sec. Only the reality filters and the 2D overlays can be selected by the application developer or online by the user [29]. This application is very simple and presented just 2D overlays, as well as the detection of the filter is done manually. Augmented reality for historical tourism using mobile devices has been proposed by Bres et al. [30]. The core of the proposed system is related to a marker-less outdoor augmented reality solution. This technique is based on scale invariant feature transform (SIFT) features for localisation and integration of 3D models into video. These features are used to project a digital model of the building facades of the square in order to get 3D coordinates for each feature point. The algorithms executed are responsible to calculate the camera pose for frame of a video from 3D-2D point correspondences among features that extracted from the current video frame and points in the reference dataset. The algorithms were successfully evaluated on video films of city squares. Despite they do not yet work in real-time, they are able to correct pose estimation and projection of artificial data into the scene. The algorithms automatically recover any loss of track. The research showed that the possibility of SIFT features are purely used for image-based marker-less outdoor augmented reality applications [30]. This research presented a simple mobile application that used to augment a small 3D image. HeladivaAR [31] proposed to reconstruct the historical and cultural heritage of Sri Lanka. HeladivaAR is a mobile phone application that used to show a reconstructed 3D model of these ancient ruins as they were in their initial state. In addition to use of AR technology, the application has used the mobile phone camera to determine and track the remaining ruins of the historical place and reconstructs the 3D model on it and then displays on the application interface. This application used different aspects to reconstruct the cultural heritage building such as image processing, 3D modelling, tracker identification using Android platform, historical books and reconstruct ruined sites. By using of AR, the real scene is enhanced by interactive multimedia information in order to increase the experience of the user, who can recover this information by a user-easy interface through their mobile phone. In education field, virtual heritage becomes a platform of learning, motivating and understanding of particular events and historical elements for students and researchers. This research provides a better understanding of Sri Lankan cultural heritage and allows users to gain interactive knowledge on archaeological facts of ancient kingdoms [31]. However, this research has several limitations. The first one is the application can apply only to android-based augmented reality devices; it cannot apply for the ISO-based operating system devices. The second limitation is the quality of the application that is based on the mobile device because it is not a desktop application. The last limitation is the application developed for Android 3.0 or above. The versions below may encounter rendering problems when running.

Indrawan developed marker-less augmented reality utilising gyroscope in order to demonstrate the position of Dewata Nawa Sanga [32]. This application is designed to learn, understand and recognise the properties of Dewata Nao Sanga by using a gyroscope. The sensor works to achieve the object of the deities in the coordinates to be identified, as well as, it is worked to provide information about Dewata Nawa Sanga along alongside and informative 3D animation. This research evaluates the usefulness, functionality of the application, in addition to the impact of the AR Dewata Nawa Sanga application that can motivate its users. The result of usability and satisfaction questionnaire value showed that the percentage average is 84.8%. It illustrates that the application is very useful for the participants to have knowledge about Dewata Nawa Sanga as well as very satisfied to use [32].

Kolivand and El Rhalibi presented a new technique to augment a realistic virtual building in real environments to be observed live through an AR camera [23]. There are some outdoor components when augmented a realistic building, for example, the sun position, shadows, sky illumination and virtual traditional animated characters. It is augmented in real environments at the place of real historical buildings, or desirable locations, at different times of the day and different days of the year [2]. The authors have presented some new ideas in the case of virtual heritage. First of all is modelling the 3D model of Portuguese Malacca. A structured real-time system is provided to trace the sun position, by using Julian dating, and Perez sky model is used for modelling sky colour, have presented in order to create outdoor illumination. A semi-soft shadow algorithm has been implemented to support the realism of outdoor augmented reality systems. A simple camera setup system has used to present Marker-less AR. The final system can be installed on head mounted display (HMD) or in the proposed device called ReVitAge to show the realistic reconstructed virtual heritage buildings, taking into account the main components of outdoor illumination [23].

6. Issues with virtual heritage in augmented reality issues

There are four main issues related to the virtual heritage in augmented reality. These issues are registration, reconstruction orientation, tracking and location.

6.1 Registration

Registration is one of the most significant issues in virtual heritage AR systems and currently subtracts some restrictions to different AR applications. Registration indicates the accurate compatibility of augmented objects with real environments [23]. Any AR system without an accurate registration leads to unsuccessful mixed environments because of the results of the defect wrong. The registration process is the overlay of virtual objects onto a real scene by using information that have extracted from the scene. Especially, this information is the feature points that extracted from the real scene using some tracking techniques. There are two categories of registration techniques, sensor-based and computer vision-based techniques. In sensor-based technique, there is a need to calibrate the external sensors, but the available sensors equipment’s are either huge or expensive, or lack satisfactory levels of accuracy. Computer vision-based methods techniques work to avert calibration of external sensors, as well as offer the possibility for accurate tracking without huge and costly equipment. It can be categorised as two main types depending on camera calibration requirements [33]. The first kind does not require any calibration of camera parameters in advance, which includes the use of a known 3D calibration object. However, the second type is assuming that the intrinsic camera parameters are pre-calibrated. This is a common assumption in most of the existing AR systems.

There are several researches that work to develop the registration of the virtual elements in the real-world environments. These researches will be explained in the following section.

Gao et al. [6] introduced a new technique to improve the stabilisation and the accuracy of marker-less registration in AR. Based on three-dimensional map information generated by visual simultaneous localisation-SLAM. The proposed technique allows tracking and registration of virtual objects in order to ensure a stable in addition of real-time performance of marker-less AR applications. The stability of the system can be performed by integrating the Hough voting algorithm with the repeated Closest Points (ICP) technique. The proposed technique is faster than the standard methods. In addition, it is able to achieve more accurate registration results when compared with the previous techniques. The experimental results showed that the proposed technique can efficiently repress the virtual object jittering, as well as a higher tracking accuracy with good performance [6]. This technique can identify only one object for each recognition. Kanade-Lucas-Tomasi (KLT) natural feature tracker and the reconstruction technology is presented by Pang et al. [33]. KLT tracker technique is used to track the identical feature points in two control images. The authors presented three key stages in the proposed technique. The first stage is the affine reconstruction. In this stage, two control images from the video sequence are chosen and the KLT tracker is used for the extraction of the natural feature points. After that, the Affine Coordinate System (ACS) is defined by using these natural feature points. The user is responsible to select four planar points for setting the Euclidean WCS in two control images, respectively, and then the affine coordinates of the specific points are reconstructed by using the affine reconstruction method. While, the second stage is re-projection. Compute the corresponding affine re-projection matrix in the live video frame by using the natural feature points that have been tracked by the KLT technique. The image projections of the selected points are predestined in the live video sequence by using the affine re-projection matrix. However, the third stage is the camera extrinsic parameters such as camera pose, which are predestined in terms of the four selected points achieved in the second stage. Eventually, the virtual objects can be rendered on the real scene by using the graphics pipeline techniques such as OpenGL. The experiment results showed some improvement compared to the previous work [33]. The main limitation of this research is that the user has to manually determine the four points in the initialization stage, as well as, the authors do not consider tracking the feature points.

6.2 Reconstruction

Reconstruction is one of the basic processes in the AR. It refers to the construction of virtual objects in a similar way to replicate the original building [23]. Many cultural heritage applications require to reconstruct of real-world objects and scenes. Reconstruction process becomes increasingly common to use for modelling purpose of cultural heritage. This is fundamentally because of rapid development in laser-scanning techniques, 3D modelling, image-based modelling techniques, the power of the computer and virtual reality. The default objects appear on an appropriate model that covers the details of accurate enough is essential [23]. Objects must be exactly identical to the original ones which visitors can see clearly at the background of live videos. In addition, interest in objects’ shadows is an essential part of the reconstruction process. Real-time shadows are created in relation to the sun position in a specified location, date and time. Eventually, the influence of the sky lighting on the virtual building during the daytime is the last part of creating the realistic virtual heritage in AR systems. Most virtual reconstruction techniques are based mainly on 3D scanning techniques, in order to get the objects faithfully [34]. Figure 5 shows the reconstruction of the building and place it in the real environment.

Figure 5.

Realistic reconstruction of cultural heritage.

6.3 Tracking

Tracking is a substantial subject in a real-time augmented reality context. The key requirements for tracking are the high level of accuracy and low level of latency at a sensible cost. Objects’ tracking in the scene is defined as the amount of the pose between the camera and the objects. Virtual objects can be displayed into the scene using the pose. A local moving edges tracker have been used to provide real-time tracking of points normal to the object contours [27].

A new method for conception of vision-based augmented reality systems is presented by considering either 3D model-based tracking techniques or 3D model-free tracking approaches [1]. The method depends on decreasing the cost function expressed in the image and this decreasing is achieved via a visual serving control law. The main feature of a model-based method is that the information about the scene allows improvement of robustness and system’ performance by the ability for predicting hidden movement of the object and acts in order to reduce the effects of outlier data introduced in the tracking process [35]. It is occasionally necessary to achieve the pose computation with minimal constraining information on the viewed scene because 3D information is not readily available in certain circumstances. The algorithm has been tested on different image sequences and for diverse applications, which illustrate a real usability of this approach [1]. This research has several limitations. The first limitation is the system that relies on a course manual initialization on the very first image. The second limitation is the system that does not take spatiotemporal aspect of the tracking process in depth consideration. Robustness can also be treated from one time-step to another. A novel marker-less camera tracking system and user interaction methodology for augmented reality (AR) on unprepared table-top environments is presented by Lee et al. [36]. A real-time system architecture is presented to merge two kinds of feature tracking. Marker-less tracking method is initialised by a simple hand gesture using the Handy AR system that used to estimate a camera pose from a user’s outstretched hand. Detecting distinctive image features of the scene and tracking frame-to-frame by computing optical flow. The proposed system used distinctive image features for recognising the scene and to correct for accumulated tracking errors. For achieving real-time performance, multiple operations are processed in a synchronised multithreaded method: capturing a video frame, tracking features using optical flow, detecting distinctive invariant features and rendering an output frame. The speed and accuracy of hybrid feature tracking system have been evaluate and demonstrate a proof-of-concept application to enable AR in unprepared table-top environments, by using bare hands for interaction [36]. One of the significant limitation of this research is the system applied on 2D scene.

Novel interactive techniques for outdoor augmentation have presented to use a mobile device [37]. The system can be executed and perform real time on simple mini PC equipment. Feature tracking have been used for estimating camera motion when user turns the mobile device and examines the augmented scene. The authors have considered two scenarios. The first scenario is constantly applicable with any 3D model for ad hoc use without prior information or calibration process. The second scenario uses GPS for realising the viewing location and Google Earth KML files for locating the augmented object and its placement. This method, 3D object placed on Google Earth can be viewed on site without any addition data transformation steps. The systems have been tested with potential end users. The authors believe that the system is useful in diverse current real-life applications [35].

A model-based hybrid tracking system is proposed for outdoor AR applied for urban environments that allows accurate, real-time overlays for a handheld device [38]. The system merges different well-known techniques in order to provide a powerful experience that surpasses each of the individual components alone: an edge-based tracker that used for accurate localisation, gyroscope measurements to cope with fast motions, gravity measurements and magnetic field to avert drift and a rear store of reference frames with online frame chosen used to re-initialise automatically after dynamic occlusions or failures. A novel edge-based tracker distributes with the traditional edge model, and uses instead of a coarse, but textured, 3D model. This technique has several features [39]. The first feature is automatically disposing from scale-based detail, appearance-based edge signatures can be used to improve conformity and the models required are more usually available. The second feature is the system’s accuracy and robustness is pretending with comparisons to map-based ground truth data. The tracking system have the possibility to apply to other types of display such as head mounted displays using video see-through overlays, while optical see-through displays would demand further calibration of the HMD’s virtual camera with take in account of the video camera [38]. This system has the resulting asymmetry in the information display capabilities of the two environments (virtual and real-time environments). An integral natural feature-based tracking system is proposed to support the creation of AR applications that concentrated on the automotive sector [40]. An AR application was construct on top of the system to refer to the location of 3D coordinates in a specific environment. It can be applied to many various applications in cars, for example, a maintenance assistant, an intelligent manual and many more. The system is evaluated during the Volkswagen/ISMAR Tracking Challenge 2014, which designed to test state-of-the-art tracking technique based on requirements encountered in automotive industrial settings. Evaluation results illustrate that the system allowed users to correctly determine tasks points that involved tracking a revolving vehicle, tracking data on an integral vehicle and tracking with high accuracy. The evaluation of the system is allowed to understand the applicability boundaries of texture-based technique in the texture less automotive environment, a problem not addressed considerably in the literature [40]. This research has several limitations. The first limitation is low frame rate when the number of 3D key-points in the model is large. The second limitation is error accumulation when the entire vehicle is reconstructed in a single take. The third limitation is lack of temporal continuity, which may result for shivering; sensibility to extreme illumination conditions. The fourth limitation is accidental failures when cope with scenes that have minimum of texture information.

7. 3D reconstruction techniques for cultural heritage

AR technologies have become increasingly popular. These techniques are not just practical for developers of AR system, but also to the scientific community. The standard approach to create a 3D model is to build it from scratch using tools such as the unity 3D programme, which provides building blocks in the form of primitive 3D shapes. Many new technologies aim to increase the level of automation and realism by beginning with the real images of the object or converting it to direct digitisation using a laser scanner [28].

7.1 Image-based modelling

This technology includes vastly available devices, so the same system can handle a wide range of objects and scenes. These systems have the ability to create a realistic model, and those rely on photogrammetry have high geometric accuracy. This technique is usually used for geometric surfaces of architecture objects or for modelling precise terrain. It uses a mathematical model to capture 3D object information from 2D image dimensions or obtain 3D data using methods such as shading, texture, theory, contour and 2D edge gradient [41]. Deriving 3D measurements from images naturally requires that interest points be appearance in the image. Often, this is not potential, either because the area is hidden or covered behind an object or surface or because there is no mark, edge or visible feature to extract [28]. The main goal of image-based reconstruction is the ability to represent arbitrary geometry. For modelling complete geometric structures, it is usually necessary to remove the labour-intensive task through this approach [41]. The mechanism can also deal with the real-world effects that images take, but difficult to reproduce with the customary graphics techniques.

7.2 Range-based modelling

3D geometry information for an object can be captured directly by this technique [28]. The 3D measurement of images requires that interest points or edges be visible in the image, which is not constantly possible. Illumination or ambient light problems can impact the extraction process of such points and edges. Active sensors, for example, laser scanners have the ability to avert these restrictions by creating features on the surface using controlled light projection [41]. Many range sensors are produced organised points, in the form of an array or range image, appropriate for automatic modelling. However, texture information or colour can be attached from the scanner using colour channel or from separate digital camera [42, 43]. High-resolution colour textures that obtained from separate digital camera help to create of realistic 3D models. Generally, a single range image is insufficient to cover any object or structure [42]. The amount of necessary images rely on the shape of the object, the amount of self-locking and obstacles and the size of the object compared to the sensor range [41]. In order to wrap each aspect of the object, it is mostly required to perform multiple scans from various locations, which is commensurate to the size and shape of the object and occlusions. The alignment and groups of the various scans can affect the final accuracy of the 3D model, where each scanner has different range of resolution [28]. In addition, this technique can provide accurate and complete details with a high degree of automation for small and medium size objects, which reach the human size [42]. There are two major kinds of range sensors: triangular based and based on the principle of flight time [41]. Triangulation-based sensors are working dependent on project light in a known direction from a known position, as well as measure the direction of the returning light through its detected position. The measurement of accuracy depends on the triangle base relative to its height. Sensors based on the principle of flight time measured the delay between emitting and detecting reflected light on the surface, thus, accuracy does not quickly deteriorate as the range increases [44]. Time-of-flight sensors have the possibility to provide measurements in the kilometre range.

7.3 Image-based rendering

Image-based rendering used images as modelling and rendering primitives [28]. Image-based rendering uses images directly for creating new views for rendering without explicit geometrical representation. This technique is a significant mechanism for generating of virtual view, where certain objects and under particular camera motions and scene conditions. From the image input, this technique creates a new view of the 3D environment [41]. This technique has the feature of creating realistic virtual environments at speeds independent of scene complexity [42]. Image-based rendering depends on accurately knowing the camera positions to use automatic stereo matching, where the absence of geometry data, requires a major number of carefully spaced images to succeed [42]. Most of image-based rendering correspond to hybrids image-geometry, using means of the equal amount of geometry ranging from per-pixel depth to hundreds of polygons [45].

8. Augmented reality location

Each of indoor and outdoor sites offered many of similar challenges that must be processed to successfully implement AR systems, such as content acquisition [11], content storage and categorisation [46], tracking and calibration [47], marker placement, usability [48] and ergonomic issues [49]. Hence, there are various issues that must be taken into account in order to overcome by special internal or external sites.

8.1 Augmented reality for indoor heritage sites

AR’s previous applications for indoor cultural heritage sites have frequently taken the form of “virtual museums”. The visitors use AR technology to display objects that may not be accessible to them. This is because the great value or fragility of such objects, or the lack of space inside the museum or the physical object is existing in another museum [36]. One of the main issues that affect the design of AR systems for indoor sites are those of marker placement if using marker-based tracking, as well as ensuring the optimal use of the systems for all age groups and levels of computer literacy. In addition, it is substantial to make sure that hardware used is strong enough in order to support AR applications, and it is structurally robust if being lent to the public.

8.2 Augmented reality for outdoor heritage sites

It can be said that the development of AR systems for outdoor applications is more difficult than indoor applications. Realistic historical buildings in outdoor rendering AR systems require advanced effects such as shadows, lighting and the ability to detect the impact of sky dome illumination on virtual in addition to the real objects [23]. The environment and resources, such as lighting conditions and electrical energy, cannot be as tightly controlled, as well as hardware cannot normally be left outdoors. The use of mobile computer systems in outdoor AR generates several problems such as it is uncomfortable and heavy to wear, and it is very expensive if it is a wearable system combined with an HMD [11] Outdoor AR is a technology of executing augmented reality using outdoor GPS, compass, gyroscope sensor based on augmented reality technology. Unlike to indoor AR, outdoor AR is not subject to spatial restrictions. Indoor AR used a marker to ensure suitable synthesis of virtual object because it happens in relative narrow space, while outdoor AR used location information; it does not use any marker like in indoor system because it happens in relatively wide area [50]. Often the lack of ideal conditions means that marker-based tracking systems cannot be used, leading to rely on other techniques, for example, GPS and inertial sensors, which can be inexact.

One of the key problems that faced to design AR systems for outdoor sites are effectively tracking without using of markers in an environment that may be devoid of features in order to use for tracking. In addition, ensuring that any device used is weather-resistant and vandal-resistant. Furthermore, all the hard-wires that are used must be powerful enough to support AR applications, as with indoor sites. Figure 6 shows the AR for outdoor cultural heritageTable 1 shows a comparison between investigated works in AR systems, concentrated on outdoor, indoor, reconstruction and realism.

Figure 6.

Augmented reality for outdoor heritage.

Researcher	Area	Indoor	Outdoor	3-D reconstruction	Realistic VR
Sudirman and El-Rhalibi [16]	Camera pose estimation	X
Yuan [17]	3D camera pose	X		X
Frikha et al. [18]	Camera pose estimation	X		X	X
HoK llerer et al. [20]	mobile augmented reality system	X	X	X
Newman et al. [21]	AR wide-area sentient	X
Jacquemin et al. [22]	Mobile AR		X		X
Fukuda et al. [24]	AR registration		X	X	X
Chen et al. [25]	AR tracking	X			X
Lee and Höllerer [36]	AR tracking	X		X	X
Honkamaa et al. [37]	AR tracking		X	X	X
Reitmayr and Drummond [38]	AR tracking		X	X	X
Lima et al. [40]	AR tracking	X		X	X
Pang et al. [33]	AR registration	X		X
Hanisch et al. [26]	VR reconstruction			X
Andrés et al. [27]	VR generation		X	X	X
El-Hakim and Beraldin [28]	VR reconstruction		X	X	X
Zoellner et al. [29]	Cultural heritage layers		X	X	X
Bres and Tellez [30]	Mobile applications in culture heritage		X		X
Galmangoda et al. [31]	VR reconstruction		X	X	X
Purnami and Putri [32]	VR reconstruction	X		X
Kolivand and El Rhalibi [23]	Realistic virtual heritage		X	X	X

Table 1.

A full comparison of different techniques for marker-less augmented reality.

9. Conclusion

This chapter has presented the survey of marker-less augmented reality system. In this chapter, we have discussed different techniques related to the augmented reality. An overview on each of them was introduced, identifying the major features and highlighting the main characteristic of each technique. In addition, we have explained in detail the main issues with virtual heritage in augmented reality. We have introduced the key techniques for 3D reconstruction that used in cultural heritage. We have focussed on the main issues of augmented reality for cultural heritage such as indoor marker-less AR, outdoor marker-less AR, real-time solutions to the tracking problem, real-time registration and cultural heritage in AR. We have presented the research related to these areas and highlighted the main problem of each research.

Acknowledgments

The financial support of this research is provided by KE & I fund from Liverpool John Moores University.