Application of a Photogrammetric System for Monitoring Civil Engineering Structures

Several traditional measuring apparatus are used to check the stability of civil engineering structures and maintenance of them. The measured results are applied for the deformation and stability analysis of civil engineering structures. Currently, precision and micro measuring instruments are used for stability evaluations of civil engineering structures. Furthermore, the measuring apparatus have been changed from manual systems to automatic systems. For example, total station, one of the traditional and manual measuring methods, is transferred to digital photogrammetry with high technology development. Especially, the movement of target points is able to be measured in real-time automatically because it can be obtained 3-dimensional coordinates by digital photogrammetry. The use of automatic measuring methods has been researched in several different industries (Hannah, 1989; Lee et al., 2006). The applications of digital photogrammetry are increased in various civil engineering structures (Han et al., 2001; Han & Song, 2003; Han et al., 2007, 2008). It shows that the automatic and high-tech measuring system likeVisual Monitoring System (VMS) based on digital photogrammetry is able to apply to the stability evaluations of large civil engineering structures.


Introduction
Several traditional measuring apparatus are used to check the stability of civil engineering structures and maintenance of them.The measured results are applied for the deformation and stability analysis of civil engineering structures.Currently, precision and micro measuring instruments are used for stability evaluations of civil engineering structures.Furthermore, the measuring apparatus have been changed from manual systems to automatic systems.For example, total station, one of the traditional and manual measuring methods, is transferred to digital photogrammetry with high technology development.Especially, the movement of target points is able to be measured in real-time automatically because it can be obtained 3-dimensional coordinates by digital photogrammetry.The use of automatic measuring methods has been researched in several different industries (Hannah, 1989;Lee et al., 2006).The applications of digital photogrammetry are increased in various civil engineering structures (Han et al., 2001;Han & Song, 2003;Han et al., 2007Han et al., , 2008)).It shows that the automatic and high-tech measuring system likeVisual Monitoring System (VMS) based on digital photogrammetry is able to apply to the stability evaluations of large civil engineering structures.
Most of recent measurement methods are based on image process method even though in some case, Global Positioning System (GPS) is used like the measurement of the deformation on the surface of large civil engineering structures (Stewart & Tsakiri, 2001).Also, automatic measuring method begins to use for stability evaluations of civil engineering structures; for examples, there are slope failure prediction (Han et al., 2001;Han & Song, 2003), displacement measurement (Bae, 2000;Kang et. al, 1995), monitoring of dams (Park et. al 2001;Han et. al, 2005).However, the current measuring systems like total station based on manual measuring are almost not possible to measure the movements of civil engineering structures in real-time based due to various field conditions even though most of measuring systems have a high accuracy.For example, the manual measuring method requires a lot of analyzing time for measured data because it has several procedures like photographing, drawing and analysis manually.
In the late 21th century, the image acquisition apparatus with high technology has been developed continuously and rapidly.If real-time automatic measuring system based on high technology is developed, the measuring system technique can be developed with digital image technique rapidly as well.It means that stability and maintenance technique for civil engineering structure can be more epochal improved in the future.
Therefore, this chapter introduces a real-time measuring system, VMS which is based on digital photogrammetry.The measuring results of VMS are compared with those of manual apparatus in various tests and then the accuracy of VMS is evaluated.At first, background of digital photogrammetry and VMS process are explained.Secondly, laboratory and field tests using VMS and cultural assets restoration based on application of digital photogrammetry are described.Finnally, conclusions about various tests and case study are summarized.

Historical background
The past photogrammetry requires several steps to obtain the adequate results which are divided into photographing, drawing, analysis, 3-dimensinal coordinates acquisition and conversion, because it uses films that have sensitive emulsion.Therefore, the past photogrammetry has problems in both the application of real-time measuring and distortion and storage of data.
In the late 20th century, CCD camera was introduced as image acquisition apparatus because CCD camera could not only be economically purchased but also, easily process and save the image data.The image acquisition apparatus has been developed in high resolution and wide analysis range based on the development of electric and electron.Also, the accuracy of camera lens has been improved by correction technique of geometric distortion and establishment of analysis algorithm (Fraser, 1997).
In the early 1990s, the image analysis techniques with film scan started to apply to various fields.In the mid-1990s, several researchers have studied the improvement of accuracy in image analysis techniques using establishment of analysis algorithm and commercialization of real-time measuring (El-Hakim & Wong, 1990).Furthermore, the 3-dimensional position measuring has been researched to apply the field of photogrammetry (Fraser, 1993;Shortis & Fraser, 1998).The correction of multi-focus lens has been studied to apply in close range photogrammetry (Munjy, 1985;1986).The lens correction and image analysis improvement of CCD camera have been studied by using research results (Shortis et al., 1994) of close range photogrammetry and focus lens (Lichti & Chapman, 1995).CCD camera has been used in image analysis of robot vision and position decision of targets (Gruen, 1992).
The study of combination programs has been a trend, so separate image processing programs could be combined by using visual analysis and connection analysis based on existing researches.Especially, the photogrammetry that uses automatic technique of 3dimensional data acquisition has been studied in the application of object movement and automatic system of industrial (Hannah, 1989;Lee et al., 2006).Also, the user interface design using digital photogrammetry technique was able to apply in high accuracy measuring system and vision system.
The 3-dimensional measuring system based on existing photogrammetry is used in film with sensitive emulsion.However, the technique takes a time to respond to the risk in spite of great accuracy.Therefore, real-time measuring system, which improved measuring system based on existing photogrammetry, was requested for civil engineering structure that specially requires both economical, rapid reaction and high accuracy.

Short-overview of digital photogrammetry
Photogrammetry provides the 3-dimensinal coordinates of unknown points from acquired multiple images, which are images of the same scene on different viewpoints.Photogrammetry procedure has object decision, image acquisition, image processing and analysis.The image acquisition of the past photogrammetry is used in film with sensitive emulsion.Nowadays, those are used in CCD and CMOS cameras.The 3-dimensinal coordinate precision of images depends on various factors like object condition, camera calibration and image orientation.That is, the scale and pixel size of image based on CCD and CMOS cameras is important factor in the image acquisition of photogrammetry.This means that camera calibration needs to enhance image accuracy (Kraus, 1993;1997;2007).
In present, the digital images are produced by electric sensor of metric cameras based on CCD or CMOS, as above mentioned.The mathematical-geometric parameters of digital image based on principal distance, coordinates and principal point are defined as the factors of interior orientation.Even though geometric parameters represent the photogrammetry analysis model, it does not correspond to real object due to distortion of camera lens.Therefore, the camera calibration must be conducted to achieve the highest accuracy of model (Kraus, 1993;1997).
In order to create 3-dimensinal coordinates of image, the image plane is produced by electric sensor in focal plane of cameras.The orientation parameters based on image plane can be calculated using collinearity equations (see Kraus, 1993) and then the 3-dimensinal coordinates are produced (Kraus, 1993;1997;2007;Ohnishi et al., 2006).

Visual Monitoring System (VMS)
VMS is monitoring system using digital photogrammetry and it is made to analyze deformation of object based on real-time visualization.This section introduces composition, principle and process of VMS.

Composition of VMS
The past photogrammetry using films makes a 3-dimensinal image by image acquisition, drawing, analysis and 3-dimensinal coordinate creation.A coordinate conversion of film image is conducted by coordinate formation program.It results in conversion process and visualization program and then it forms 3-dimensinal coordinates.The past photogrammetry has a problem for an immediate maintenance about deformation and displacement of structure in field because it requires a lot of processing time.If disaster risk happened, it would require a lot of time to establish countermeasure.
The VMS photogrammetry analysis software which combine with 3-dimensional coordinate extraction, drawing and analysis, can analyze structural deformation as shown in Fig. 1.The VMS as a unified program is real-time measuring system for civil engineering structure with using digital photogrammetry.VMS can evaluate the structural stability by comparing the continuous obtained information to the existing information of structure.Also, it is real-time monitoring system for maintenance of structure.

Process of VMS
The procedure of VMS is shown in Fig. 2. First of all, the image obtained from digital camera is conducted by image processing.Secondly, the modified image compares with the selected model image.Then it searches the model in the image.Finally, the measuring result is obtained by the searched result using image analysis module.
VMS is developed windows based program by Delphi and it consists of five tools as followed; 1. Project tool has camera information to photogrammetry -Input focal length, pixel size and position (3-dimensinal coordinates) of cameras, and information of reference and unknown points 2. Second tool select analysis model -Select size, range and area center of model for photogrammetry analysis, then modify the selected model using image processing wok 3. Third and fourth tools find metric image and model position in image, respectively -Search the modified model of mertic image using geometric model finder 4. Fifth tool performs photogrammetry analysis and sees analysis results.
-Analyze measuring data using the combined photogrammetry analysis software

Image noise removal
If the image has noise, there is a clear difference between noise depth and its surroundings in the image.The noise removal using noise characteristic is defined as smoothing method.Also, the noise removal can improve image quality.

Model finding
The geometric model finder module is used in photogrammetry procedures because the model finding needs to search an unknown points corresponding to reference point.The function of geometric model finder module only extracts the required data from pattern recognition system based on the modified image by image processing.The characteristics of extracted data are distinguished by pre-processing.Then its pattern is recognized and processed by system.That is, VMS requires a pattern recognition procedure because it is automatic and complex system.
The procedure of pattern recognition is shown in Fig. 4. The metric image is pre-processed to pull out the required data because it includes unnecessary information.Next, the extracted data is divided into smaller units for facilitation of pattern recognition.Small units are idealized in both size and length.Finally, the characteristics are selected to obtain the important factors of pattern recognition, and then it is compared to the identification of patterns.
The pattern preprocessing includes preprocessing, partition and normalization.It is simple and does not require a lot of time in process.It does not change the required characteristics in pattern recognition.
The characteristic extraction is the selecting process of the stabilized physical property for data transformation.Furthermore, the process includes both removal of redundancy property without loss of data information by reduction of pattern dimension and minimization of the processing time and storage space to pattern recognition.It means that the obtained data by measuring are converted into the required characteristics for pattern recognition.The methods of pattern matching, which is to identify the range of random pattern in the image, are theoretical analysis and structural analysis.Fig. 5 shows example in flow of pattern recognition system.

Experimental investigation using VMS
Digital photogrammetry is applied in various fields.VMS has an important role for the deformation measuring of civil engineering structures as well as historic preservation and restoration of cultural assets.
The VMS results of tensile strength tests for welded members in civil structure, friction tests for concrete masonry block (CMU) retaining wall in civil engineering structure, and the measured deformations of cultural assets are presented.

Deformation measuring of welded members
The VMS and a test instrument (MTS-810) are used for the deformation measuring of the tensile strength tests of welded members.The accuracy and applicability was verified with comparing the deformation values of VMS to those of a test instrument.The characteristics of the behaviour in a welded joint were investigated by using VMS.

Loading test
In order to measure the local and overall deformations, measuring points are marked at a welded member as shown in both Fig. 6 and Fig. 7. U0 and L0 are measuring points to observe the overall deformation of the test member.U1~U10 and L1~L10 are measuring points to observe the local deformation near a welded joint at upper and lower part.The loading tests are conducted using MTS-810, which has loading capacity of 250kN.The loading increasement is controlled by tensile displacement at a 0.01 mm/sec with the assumption of static deformation.Also, the deformation is measured by two sets of digital camera, which are located at distance of 2.1m.VMS measures the deformation of measuring points at each 50 second.Fig. 7 shows a welded member marked measuring points and tensile test rig (Han et al., 2008).

Comparison of results and analysis
To confirm the accuracy and the applicability of VMS, strain (Syt) values of VMS are compared with those of a test instrument (MTS-810) as shown in Fig. 8. Syt values measured by VMS are good agreement with a stress-strain curve measured by MTS at the elastic and at the plastic regions.And it is seen that the developed VMS can accurately measure an overall deformation of a test member.It is concluded that the developed VMS has the high accuracy of 90% and the applicability of deformation measurement for the welded members.Strain values near a welded joint at upper part are shown Fig. 9. Strain values (Sxu1~Sxu4) in the x-direction show compressive (-) up to -0.02 in the elastic region.In the plastic region, strain values constantly keep compressive.In the y-direction strain values (Syu1~Sxu5) are tensile (+) about +0.01 in the elastic region.In the plastic region, strain values are rapidly larger up to about +0.06.
It is shown from VMS measuring observations for a welded joint that strain values in the xdirection keep compressive up to -0.02.Strain values in the y-direction are tensile and rapidly increased up to +0.06 in the plastic region.

Displacement measuring of reinforced concrete masonry unit (CMU) retaining wall
The manual measuring apparatuses like displacement transducer, settlement meter and total station are applied to perform both stability evaluation and maintenance of the reinforced CMU retaining wall.However, it is difficult to measure the overall deformation and crack of the reinforced CMU retaining wall because most manual measuring system can only measure the partial deformations.Also, the manual measuring systems have some disadvantages in durability and maintenance of apparatuses due to malfunctions.The malfunctions cause low accuracy for immediate reaction.Therefore, VMS is one of the good solutions for reasonable measuring of the reinforced CMU retaining wall because it can solve the disadvantages of the existing measuring systems.The forced displacement laboratory test of the reinforced CMU retaining wall is conducted to evaluate both the behaviour of reinforced CMU retaining wall and real-time measuring system at a long distance.

Forced displacement laboratory test of reinforced CMU wall facing units
Measuring both local displacement and overall deformation of CMU retaining wall facing units in real scale is performed.The test procedure is shown in Fig. 10 and follows as; 1.The rubber mats (like soft ground) is located to lead settlement of the wall facing units when vertical load is applied to the wall facing units 2. The wall facing units with three tiers is located on rubber mats of laboratory test apparatus 3. Obtain initial 3-dimensional coordinates of the image using VMS and total station 4. Obtain 3-dimensional coordinates from cameras using total station 5. Apply loads on the wall facing units 6.Obtain the deformed image coordinates of left and right due to the applied loads The six tests are conducted.There are three measurments in vertical direction, two in horizontal measuring, and one simultanuous measuring in both vertical and horizontal directions.The test procedure is repeated at each case.The reference target points on steel frame of laboratory test apparatus and the unknown target points on the renforced CMU retaining wall face are located respectively as shown in Fig. 11.The movements of the target points are monitored by both total station and VMS (refer Fig. 12).The vertical loads apply from 30kN to 50kN and the vertical displacements are measured.The horizontal displacements occur due to the forced movement of rubber mats (Han et al., 2006).

Comparison of results and analysis
The initial coordinate for six reference target points and nine unknown target points is obtained by both VMS and total station.The displacements of wall facing units by vertical load and forced displacement are measured.The test results are shown in Table 1.However, the measuring results of target points 13, 15 and 18 are eliminated because of dropping the target points off wall facing units during tests.The measuring difference between total station and VMS is from 0.1% to 16.7%.Therefore, the measuring results of unknown points are only presented the maximum and minimum differences as shown in Table 1.The average difference of total results is less than 3.6% as shown in Table 2.
The test results confirm the applicability of VMS as a good measuring system to the reinforced CMU retaining wall.Furthermore, the analysis results show that VMS can be real-time measuring systems while the targets are moved.Therefore, it is suggested that VMS can be a reasonable measuring method for reinforced CMU retaining wall systems.The field test of VMS is conducted for the concrete wall (refer Fig. 13), which is made of concrete, because of the limitation of field application on large structure.The simulation idea comes from large civil engineering structures made from concrete materials (Han et al., 2007).The field test is conducted to evaluate VMS application to concrete structure at long distance.The results of VMS compares with those of total station.The targets points are located on the concrete bleacher at the distance around 76m to 97m from CCD cameras and the distance between cameras is 87.9m as show in Fig. 14.The reference target points (numbered as 1, 3, 4, 6, 7 and 9) and unknown target points (numbered as 2, 5 and 8) are located on the face of concrete bleacher, respectively (refer Fig. 13 and Fig. 14).The deformation of the targets points are measured by VMS system.As there was not a actual structural displacement for measuring, the unknown points have a simulated movement from initial position by the forced displacement as shown in Fig. 15.
The forced displacement of targets points assumes 5cm at top and bottom and 9cm at left and right.Therefore, the each unknown point is measured in total of four steps.The simulation procedure is followed as; 1. Set up the CCD cameras and prepare VMS system (refer Fig. 16) 2. Fix reference point and unknown point targets points on object and then 3. Obtain initial 3-dimensional coordinates of the image using both VMS and total station 4. Obtain 3-dimensional coordinates of cameras using total station 5. Obtain images coordinates of left and right by forced displacement 6. Repeat (5) process according to the each step

Comparison of results and analysis
The measuring results based on initial and forced displacement are analyzed as shown in Table 3 and 4. The initial coordinates of the image using both total station and VMS are almost identical as shown in Table 3.The initial displacements of total station and VMS are zero.Also, the displacement according to the forced displacement is measured by both total station and The results are shown in Table 4.The comparison of measuring results shows that the difference is from 0.325mm to 10.436mm irregularly.However, the accuracy of VMS is acceptable because the calculated difference range based on camera pixel and measuring distance has maximum 10.436mm.
It means that VMS can apply to measure displacement, deformation and crack of civil engineering structures.In addition, the accuracy of VMS will be improved continuously because the error rate in camera capacity will be decreased by the industrial development related to image acquisition apparatuses.Recently, there have been many attempts to restore cultural assets in South Korea.However, both maintenance and restoration of cultural assets are very difficult due to several factors like site conditions, natural deteriorations, damages and restoration methods.Therefore, the photogrammetry methods based on 3-dimensonal analysis was introduced as non-contact methods without damage of cultural assets (Kim et al., 2003).
The case study presents the application of a digital photogrammetry to seven-story stone tower in Sinsedong, Gyeongsangbuk-do, Korea for the maintenance data establishment of cultural assets as shown in Fig. 17 (Sin & Kim, 2006;Yoon, 2006).(Sin & Kim, 2006;Yoon, 2006) The stability of the seven-story stone tower is evaluated by the results of photogrammetry analysis.The stability evaluation is as followed; 1. Extend and determine to vertical axis for center of the seven-story stone tower 2. Evaluate incline of the seven-story stone tower based on the determined vertical axis The evaluation results are shown in Fig. 20.In front view, the incline and the maximum eccentricity are 1.3° and 0.336m on the left, respectively.In back view, the incline and the maximum eccentricity are 0.78° and 0.251m on the right, respectively.Therefore, the digital image information based on the results of photogrammetry analysis can be applied to space analysis like cross section and inclination.It means that digital photogrammetry can be used for stability evaluation, maintenance and restoration of cultural assets.(Sin & Kim, 2006;Yoon, 2006)

Conclusion
This chapter introduces research cases related to photogrammetry and a real-time measuring system (called as visual monitoring system, VMS) which is based on digital photogrammetry.It confirms the applicability of digital photogrammetry in civil engineering structures and cultural assets.The research conclusions are followed as; 1.In deformation measuring of welded members, the accuracy and applicability of VMS was verified by comparing the deformation values of VMS with those of the loading test instrument (MTS-810).It is seen that the developed VMS has the high precision comparison with test results using MTS-810.2. VMS is applied to displacement measuring of reinforced CMU retaining wall.The average difference of VMS is less than 3.6% in total test cases.This means that the measuring result was verified reasonably in all target points except the dropped target.3. The accuracy of VMS based on simulated field test is satisfied in theoritical difference range.Therefore, the its applicability was verified reasonably in civil engineering structure 4. It introduces the applicability of digital photogrammetry in maintenance and restoration of cultural assets.The result confirmed that the stability of cultural assets is evaluated by digital photogrammetry.Also, it can be applied as maintenance technique.
Digital photogrammetry is applied in various fields and it is developed rapidly with development of image processing techniqe and image acquisition apparatus.Therefore, if digital photogrammetry is applied continuously in civil engineering structures, micro measuring instruments, and cultural assets, it is a very useful tool as disaster prevention monitoring system due to the quantitative digital image.However, image acquisition apparatus should be improved in high precision.Techniques of camera capacity should be developed to decrease an error rate of image acquisition apparatus.With high techniques, real-time monitoring system will be used more widely in various fields.

Fig. 3 .
Fig. 3. Step of Digital image processing: a) original image, b) binary image applied in threshold; c) line extraction image

Fig. 11 .
Fig. 11.Front view of the fixed targets on steel frame and CMU

Fig. 13 .
Fig. 13.Simulation structure view for 3-dimensional image acquisition: a) left image, b) right image

Table 1 .
Comparison between total station and VMS in measuring results