Tilt Monitoring of Tower Based on Video-photogrammetry

The tilt of a tower is an important indicator in its health assessment. In particular, for an ancient tower, continuous tilt monitoring is necessary because of the long history of earthquakes and other factors. We propose a continuous tilt monitoring method for a tower based on videophotogrammetry. This method applies video-photogrammetric technology to obtain 3D coordinates of monitoring points, and then the inclination of the tower is estimated from the deflection of the space vector or space plane, which is fitted by suitable monitoring points. This inclination calculation method, which is a simple and convenient way to obtain the tilt of a tower during its monitoring, only uses the coordinate information of a limited number of monitoring points in images. To verify the effectiveness of the tilt monitoring method, an indoor experiment on a tower model and an outdoor experiment on Kuiguang Pagoda were carried out. In the indoor experiment, the tilting of a tower was simulated, and the whole process was photographed through a video-photogrammetric system. The tilt of the model tower was calculated to gradually increase from 0 to 2.5°, which was consistent with the actual tilt. The outdoor experiment involved the monitoring of the tilt of Kuiguang Pagoda in Dujiangyan City for about six months. It was found that Kuiguang Pagoda had no obvious tilt and was basically in a stable state during the monitoring period. The experimental results demonstrate the effectiveness of the method for tower tilt monitoring and show that video-photogrammetry is an alternative technique for tower tilt monitoring.


Introduction
Ancient towers are an important part of the architectural cultural heritage and of great value for the study of ancient architectural technology, history, and culture. However, most ancient towers are tilted as a result of the long-term effects of ground movement, (1) earthquakes, (2) wind, (3) and other geological factors. Serious tilting will directly lead to the collapse of a tower, resulting in the irreversible loss of architectural heritage. Therefore, it is necessary to monitor the inclination of ancient towers to ensure safety and to protect them.
Great effort has been made to monitor the inclination of ancient towers and other towerlike buildings. The traditional methods of tilt monitoring require direct contact with the monitored buildings and mainly include the use of various transducers such as inclinometers, accelerometers, strain gauges, the Global Positioning System (GPS), total stations, digital levels, and seismographs. Other monitoring methods do not require direct contact with the building, such as those using terrestrial laser scanners (TLSs), ground-based synthetic aperture radar interferometry (GBInSAR), and photogrammetry.
Traditional methods of tilt monitoring can be applied to ancient buildings and modern highrise buildings. A sensor network with an inclinometer, GPS, and a total station as the main sensors is applied to monitor the inclination and displacement of supertall structures such as Canton Tower and Shanghai Tower. (4,5) These methods have also been used for the deformation monitoring of historical buildings. In those studies, sensors were mainly used to obtain the information of monitoring points, including levelness, horizontal displacement, vertical settlement, and strain, and to monitor building tilt and other deformation conditions. Although the measurement results of sensors are accurate, sensors are typically cumbersome to deploy and can only provide local and relative deformation information. (6,7) Noncontact measurement is widely used to monitor building deformation. TLSs have the advantages of noninvasivity, high precision, and high spatial resolution, which make them widely used. TLSs are used to obtain high-accuracy geospatial data and build 3D models of buildings. Through the geometric measurement of a 3D model, deformation data such as the displacement and inclination of buildings can be obtained. Moreover, the 3D model also provides a scientific reference for deformation monitoring in the future. (8)(9)(10) Geometrical surveys with TLSs have great potential for the deformation monitoring of historic buildings, but there are still some limitations and problems, especially occlusion during scanning and the long processing time of a large amount of point cloud data. (8) GBInSAR can also be used to monitor the deformation of buildings. However, although this technology has advantages in monitoring the precise displacement of building surfaces, it is not suitable for monitoring building inclination. (11) Because of the limitation of traditional contact measurement methods of requiring installation and the limited time resolution of noninvasive measurement methods such as TLS and GBInSAR, (12) these methods are not suitable for the long-term dynamic measurement of the deformation of ancient buildings. In this respect, photogrammetry is worthy of consideration owing to its advantages of flexibility and high precision. Photogrammetry technology not only can be used in indoor dynamic simulation experiments, (13) but also has a wide range of applications in civil engineering, such as the dynamic monitoring of structures including tunnels and dams. (14,15) Photogrammetry has also been used in building inclination surveys. (16) In existing studies, characteristic points of the monitored building itself were used, or marker points were added manually to calculate the 3D coordinates of these points through photogrammetry technology, so as to analyze the displacement and inclination of the building. Baj and Bozzolato (17) performed a systematic survey of the Leaning Tower of Pisa by photogrammetry. In their experimental results, the maximum difference between the coordinate values of points in 3D space in the Y direction was 2 cm, and the difference in the X and Z directions was a few mm. Martinez et al. (18) applied close-range photogrammetry to analyze the inclination of towers in Spain. In this case, the maximum absolute error was 10 mm. To obtain more detailed information of long-term building deformation, we can record the position and state of a building in the form of an image sequence by increasing the shooting frequency and then obtain accurate 3D coordinates of the monitoring points by analytical photogrammetry processing. This method is also called video-photogrammetry, which is a branch of photogrammetry. Video-photogrammetry is often used to measure rapidly changing objects in a short time. (19,20) Regarding the monitoring of building deformation, video-photogrammetry has often been used to monitor the deformation of bridges. (21)(22)(23)(24) Rodriguez et al. (25) applied a videophotogrammetric system to monitor the condition of a wind turbine. However, there have been few studies on the application of video-photogrammetry technology to the long-term tilt monitoring of a tower. Therefore, we attempted to apply video-photogrammetry technology to the long-term monitoring of a tower and proposed a simple and convenient tower tilt calculation method for our monitoring scheme.
In this study, we first used a tower model to simulate the tilt of a tower, then applied a videophotogrammetric system to shoot the entire tilting process at a high frame rate. Through the calculation and analysis of the displacement of the monitoring points and the inclination of the tower, the effectiveness of the method for tower tilt monitoring was verified. Then, the same video-photogrammetric system was applied to Kuiguang Pagoda to monitor its inclination over about six months.

Methods
In this study, binocular video-photogrammetry was used to monitor the tilt of a tower. The specific process is to install artificial marker points on the tower as monitoring points and take photos of the monitoring points regularly with dual cameras to obtain images. After the sequence of images is obtained, the 3D coordinates of the monitoring points are obtained by videophotogrammetry. These 3D coordinates are used to analyze the spatial displacement of the monitoring points and the inclination of the building. The layout of the video-photogrammetric system for monitoring Kuiguang Pagoda is shown in Fig. 1.

Detection and tracking of monitoring points
Since the cameras used in the experiment were nonmeasuring industrial cameras, they were calibrated before the calculation to obtain their internal parameters and distortion. In this study, Zhang's high-accuracy calibration method was applied. (26) To improve the accuracy of deformation monitoring, marker points were manually selected instead of the geometric characteristics of the tower itself. After the video-photogrammetric system was set up and images were acquired, image processing was carried out. In the experiment in this study, the artificial markers were elliptical. The target monitoring points in an image were recognized and located through the steps of image segmentation, morphological edge monitoring, ellipse contour extraction, ellipse center point fitting, and target point matching based on windows. Then, the target points were tracked through the sequence of images. The matching strategy was from coarse to fine. First, the search area in the image was roughly matched by the normalized correlation coefficient measure, and then the area with the largest correlation coefficient was precisely matched by the least squares method. Finally, the pixel coordinates of the target monitoring points in the final image of the sequence were obtained.

Calculation of 3D coordinates of monitoring points
Through the above calculation, the pixel coordinates of all elliptical tracking marks in the image sequence were obtained. The next task was to accurately obtain the 3D coordinates of each tracking point. In this study, the integrated bundle adjustment method was used. In the bundle adjustment method, (27) the pixel coordinates of the tracking points and camera parameters are adjusted as a whole beam to obtain accurate 3D spatial coordinates. However, the conventional bundle adjustment method cannot process a whole image sequence or the 3D coordinates of all the tracking points in a certain image sequence simultaneously, which limits the accuracy of the 3D coordinates of the target point. We used the integrated bundle adjustment algorithm, which incorporates all the tracking points into a single collinearity condition equation, to accurately calculate the corresponding 3D spatial coordinates of the tracking points for the image sequence.
The basic model of integrated bundle adjustment comprises the collinear condition equations expressed as where (X P , Y P , Z P ) are the 3D coordinates of the target point, ( including tangential distortion and radial distortion, f is the focal length of the camera, and a i , b i , and c i ( [1,3] i ∈ ) are the coefficients of the rotation matrix composed of three angular elements (ω, φ, κ).
The integrated bundle adjustment method takes the control point as the true value and the 3D coordinates of the tracking points and the camera's external orientation parameters as unknown values, and calculates the object spatial coordinates of the tracking points and the camera's external orientation parameters jointly.

Calculation of inclination
The displacement of each monitoring point in the X, Y, and Z directions was calculated from the 3D coordinates of the monitoring points obtained by the above-mentioned process. To obtain the tower inclination, we used a limited number of monitoring points to fit the space vector or space plane, and used the deflection of the space vector or space plane to approximate the tower inclination. In the indoor tower model experiment, the tilt direction was known and there were few monitoring points, so the deflection of the fitting space vector, which is perpendicular to the tilt direction, was used to better approximate the tower tilt. Two monitoring points located on the platform and on the tower model were selected to form such a space vector in this experiment. In the health monitoring experiment of Kuiguang Pagoda, the tilt direction of the pagoda was unknown. Considering that there were about 4-6 monitoring points on a plane on each pagoda floor, the inclination of the pagoda floor was approximated by the deflection angle of the fitting space plane. In the Kuiguang Pagoda experiment, the monitoring points were located on floors 7 to 9 of the pagoda, and the 9th floor had six monitoring points and the 7th and 8th floors had four monitoring points. As the monitoring points of each floor were roughly located on the same level, we selected more than three monitoring points from each floor that were clear in the image and used these points to fit the plane. Each plane has a normal vector perpendicular to it. From the offset angle between the normal vectors calculated at different times, the inclination of the floor in the time period can be approximately estimated (Fig. 2). The angle between the two planes was calculated as Fig. 2. Schematic diagram for calculation of inclination angle of each floor of Kuiguang Pagoda. L 1 is the plane fitted by the monitoring points on the same floor at the initial time, and n 1 is the normal vector corresponding to the plane. L 2 is the plane fitted by the same monitoring points on the same floor at a certain time, and n 2 is the normal vector corresponding to this plane. θ is the angle between n 1 and n 2 . In this paper, θ is used to replace the inclination of one floor of the pagoda.
where (a 1 , b 1 , c 1 ) and (a 2 , b 2 , c 2 ) are the normal vectors of the two planes determined by the same monitoring points on the same floor at two different times, and θ is the angle between the two vectors. This angle can also be approximately regarded as the inclination of the building at this floor.

Experiment
To verify the effectiveness of the tilt monitoring method, we carried out an indoor experiment on a tower model and an outdoor experiment on Kuiguang Pagoda in Dujiangyan City. In the indoor experiment, a tower model placed on a rotatable platform was used to simulate a tower with a tilt, and the video-photogrammetric system was used to photograph the entire tilt and reset process. The outdoor tilt monitoring experiment was carried out over a period of about six months. A video-photogrammetric system was used to take images of the pagoda at a frequency of one image per day. In this section, the two experiments are described in detail.

Overview of experiment
The video-photogrammetric system used in the experiment included two industrial cameras; a control system with an asynchronous controller, main controller, router, and other equipment; a storage and transmission system; and lighting equipment. On one side of the experimental platform where the tower model was placed, seven circular artificial marker points were attached as control points. Two circular marker points were respectively set on the inclined worktable and the tower model as monitoring points. The circular monitoring points become elliptical when viewed through the lens. It is faster, more accurate, and more reliable to identify and locate the artificial monitoring points with high contrast than to use the feature points of the building itself. The distribution of the marker points is shown in Fig. 3. At the same time, a high-precision total station was used to locate the control points, and the obtained 3D coordinates were used for the subsequent bundle adjustment. During the experiment, the whole process of tilting and resetting the tower model was captured at a high frame rate, and a total of 200 image pairs were obtained.

Location and tracking of monitoring points based on sequence image
Checkerboard calibration was used to calibrate the camera. When shooting the calibration images, the calibration boards in different directions were made to appear in the camera's field of view. Some of the calibration images of the left camera are shown in Fig. 4. Zhang's calibration method was used to calculate the internal parameters of the camera. Then, it was necessary to match the same monitoring points in the left and right images of the first frame, and track the monitoring points in the image sequence. Finally, a forward intersection was performed to obtain the 3D coordinates of the four monitoring points on the tower model.

Calculation of coordinates and inclination
After obtaining the 3D coordinates of the four monitoring points of the tower model, the displacement of each point and the inclination of the tower were analyzed. The displacements of the four monitoring points in the X, Y, and Z directions are shown in Fig. 5. We can see that the monitoring points on the rotatable platform and the monitoring points on the tower model had corresponding displacement changes in the X, Y, and Z directions during the simulated tilting process. The tilting process mainly caused the changes in the coordinates of the monitoring points in the Y and Z directions. Since the tilt of the simulated tower was recovered after the tilting, the change in coordinate value presents a convex triangle.
Regarding the tilt, we used the offset of the space vector composed of the monitoring points on the platform and the monitoring points on the tower to approximate the tilt of the tower. The selected space vector should be as perpendicular to the tilt direction as possible to ensure the reliability of the approximation. The offset angle change of the space vector in the tilting is shown in Fig. 6. We used the deflection of space vector 1-4, composed of monitoring points No. 1 and No. 4, to approximate the inclination of the tower. It can be seen from Fig. 6 that the maximum tilt of the tower body was 2.5°. This is similar to the inclination of the table in the actual experiment. Therefore, it is feasible to fit the space or plane vector through the monitoring points on the tower and use the deflection of the space or plane normal vector to approximate the inclination of the tower. Indeed, in each experimental process, it is necessary to fit a suitable space vector or space plane in accordance with the characteristics of the monitoring point to better approximate the actual tilt of the tower body.

Overview of experiment
In this experiment, the inclination of the specific floors of an ancient pagoda was measured in a noncontact way using a video-photogrammetric system, so as to monitor the pagoda over six months.
The target of the experiment was Kuiguang Pagoda, which is located in Kuiguangta Park, Dujiangyan City, Sichuan Province. The pagoda was built in the Ming Dynasty and rebuilt in 1831. It is 52.67 m high and weighs about 3460 tons. It is a 17-story hexahedral double-cylinder brick pagoda. The pagoda was severely damaged in the 2008 Wenchuan earthquake and was subsequently strengthened several times.
The deformation monitoring of Kuiguang Pagoda began on May 1, 2018, and the first frame of the monitoring process was obtained on that day. As mentioned earlier, the videophotogrammetric system took synchronous shots every morning and evening, then the group with the better image quality was selected as the monitoring images for the day. The monitoring experiment continued until October 17, 2018. After discarding blurred or overexposed images owing to the weather and images that were severely occluded by leaves, a total of 166 valid image pairs were obtained in the experiment.

Monitoring scheme
The basic video-photogrammetry setup consisted of two industrial cameras with a resolution of 2592 × 1944. To avoid factors such as image occlusion due to trees and accidental interference with the cameras, the two cameras were set up at the east side of the pagoda, 16 m from the bottom of the pagoda. The two cameras were 6 m apart and fixed on poles. Since the maximum elevation angle of the camera housing itself was only about 45°, the pagoda body portion that met the monitoring height requirements could not be photographed. To enable the camera to capture floors 7-9 of Kuiguang Pagoda, the monitoring area of the experiment, a camera bracket was installed in the camera housing to increase the camera elevation angle to the required 60°. Each industrial camera was fixed in its camera housing and connected to an ARM motherboard. Using the embedded image processing system, the industrial camera was turned on at two fixed times each morning and afternoon to automatically take pictures. The captured images were stored in a memory card and then wirelessly transmitted to the designated server for storage.
The distributions of the four, four, and six monitoring points on floors 7-9, respectively, were described in Sect. 2.3. Images taken by the left and right cameras and the distributions of artificial marker points are shown in Fig. 7.
Establishing a control network is an important step in the early stage of deformation monitoring of the pagoda. A position about 45 m from the pagoda was selected as the total station monitoring point. A Leica TS-30 high-precision total station was used to locate the four ground control points around the pagoda and the monitoring points on the pagoda. The total station located the monitoring points on the pagoda with an accuracy of 2 mm, obtained the 3D coordinates of these points, and provided the initial state and the basis for the comparison with subsequent results.

Location of monitoring points based on sequence image
For camera calibration, since the camera was about 30 m from the monitoring point on the pagoda, 40 calibration images were taken by the two cameras at a distance of 30 m. The calibration images taken by one of the cameras are shown in Fig. 8. The calibration results are shown in Table 1. The average errors of the left and right cameras in the calibration were 0.248784 and 0.244838, respectively. Then, 166 pairs of sequence images were processed, including the matching of the same monitoring points of the first image pair and the recognition and tracking of monitoring points in the sequence images. In this step, the continuous 2D pixel coordinates of each monitoring point in the image sequence were obtained. Finally, the 3D coordinates of each monitoring point were obtained through the bundle adjustment described in Sect. 2.2.  We used the absolute accuracy to evaluate the spatial coordinate accuracy of the monitoring points obtained by video-photogrammetry. The absolute accuracy was defined as the coordinate difference between the total station and the video-photogrammetric measurement. The root mean square values in the X, Y, and Z directions were 10.51, 2.92, and 14.52 mm, respectively.

Calculation of coordinates and inclination
Through video-photogrammetry, the 3D coordinate changes of monitoring points 1-14 in the X, Y, and Z directions during the monitoring period were calculated and are shown in Fig. 9. It can be seen that the monitoring points showed no obvious displacement in the three directions and that the positions of the monitoring points were stable, indicating that floors 7-9 of Kuiguang Pagoda were stable during this period. However, because of the serious leaf occlusion problem of point 6, there was a large error in the coordinate calculation of this point, which was manifested in the large fluctuation of displacement in the X and Z directions.
Through the plane fitting method described in Sect. 2.3, three monitoring points were selected on each floor of the pagoda and used to fit the spatial plane of the floor. Then, the deflection angle of the normal vector of the plane compared with the initial data was used to approximate the inclination of each floor of the pagoda. The deflection angle of the normal vector of the space plane fitted with floors 7-9 is shown in Fig. 10. We can see that the higher the floor number, the greater the fluctuation. The angle of inclination of the 9th floor was larger than that of the other two floors. Among the floors, the inclinations of the 7th and 8th floors showed little change, and the range of change was about 0.2°. The inclination of the 9th floor was greater than those of the 7th and 8th floors, and the variation was within 0.4°. In general, the inclination of each floor was small and each floor was basically stable.

Conclusions
We proposed a continuous monitoring method for tower tilt based on video-photogrammetry. After obtaining accurate 3D coordinates of the monitoring points using video-photogrammetry technology, appropriate monitoring points were selected to fit a space vector or space plane, the deflection of which was used to approximate the inclination of the tower. This is a simple and convenient way to calculate the inclination of a tower during the monitoring period when there is no or little information of the key structural points of the tower or when there is a limited number of monitoring points in the image sequence. In our tower model experiment, the tower inclination calculated by the proposed method was consistent with the actual inclination. In the Kuiguang Pagoda monitoring experiment, it was found that the monitored pagoda layer did not significantly incline and was in a stable state. The experimental results show that the proposed continuous tilt monitoring method based on video-photogrammetry is an effective alternative method for monitoring the tilt of ancient towers and other historic buildings.