Optical Crackmeter for Retaining Wall in a Landslide Area Using Computer Vision Technology

An innovative 3D optical crackmeter employing computer vision technology is used for displacement monitoring in a crack of a retaining wall automatically and remotely. The 3D optical crackmeter is composed of a Raspberry Pi device and a digital camera in a box, and a fixed chessboard on the two sides of a crack. A network with LoRa wireless communication can be connected as an IoT system to provide automatic remote functions. The OpenCV library is employed to analyze changes in chessboard imaging so that relative displacements of the crack in the retaining wall can be measured in a landslide area. Through laboratory and field testing, the resolution and accuracy of the 3D optical crackmeter were determined as 0.04 and 0.1 mm, respectively. Using the crackmeter, we observed significant displacements in the xand z-directions of the crack in a retaining wall of 0.067 and 0.060 cm, respectively, in the Jhongsinlun landslide area of Taiwan over three months. Overall, the 3D optical crackmeter with computer vision technology can accurately measure the 3D displacement of cracks in a retaining wall. Moreover, the IoT-based 3D optical crackmeter is more cost-effective than traditional crackmeters used in landslide areas.


Introduction
Measuring the relative displacement on two sides of a crack in a structure is a common monitoring method. Conventionally, crack displacement has been detected using devices with a manual reading such as a mechanical crackmeter, which measures crack movement in bricks and cement or an electrical crackmeter, which is used for the continuous measurement of structural cracks. (1,2) Nevertheless, these conventional crackmeters are manually employed and only installed in the main direction of a crack. Manually employed crackmeters are not suitable for alpine landslide monitoring because manual operations are costly and timeconsuming in landslide areas. The more recent electronic 3D crackmeters with three vibratingwire transducers can automatically monitor three-way displacement across cracks in concrete, rock, soil, and structures; however, they are still costly and their operation is complicated. (3)(4)(5) Thus, the aim of the study is to explore a new type of sensor for the measurement of crack displacement using computer vision technology and an Internet of Things (IoT) system in order to automatically and economically detect the magnitude of 3D displacement of a crack in a retaining wall. Vision metrology of digital cameras has been used for accurately measuring tunnel deformation. (6) Digital image processing techniques are feasible for measuring the dynamic displacement of bridges with cameras. (7,8) In this study, we employed computer vision technology and the OpenCV library to calculate changes in image pixels of a chessboard relative to a camera embedded on a Raspberry Pi device. The Raspberry Pi can be easily applied for computer vision technology to analyze changes in images with the OpenCV library. The camera connected to the Raspberry Pi can take photos of a 3 × 3 chessboard target in which there are nine coordinate points using computer vision technology. (9) Thus, we employed these devices to analyze changes in the image pixels of a chessboard. The OpenCV library, such as 'cvCalibateCamera', 'cvtColor', 'Threshold', and 'Corner Detection', is used for camera calibration, referring to the chessboard corner point, and the image pixels of a chessboard target are tracked. (10)(11)(12)(13) Computer vision technology has been applied to the detection of construction deformation and movement using data from site images and videos as an alternative to time-consuming and manual traditional practices. (14)(15)(16)(17)(18) Thus, the relative displacements between the camera and the chessboard in the x-, y-, and z-directions can be measured using computer vision technology. This computervision-based crackmeter, called the 3D optical crackmeter in this study, can be employed for measuring 3D displacements of cracks. Furthermore, a wireless sensor network was applied to landslide prevention using the Long Range (LoRa) radio transmission technology, which is a radio frequency (RF) platform for long-range communication with low consumption. (19,20) As a result, the 3D optical crackmeter, the LoRa wireless communication, and a solar energy harvester are combined as an IoT system for crack monitoring in a landslide area.
Two steps of this research were performed to determine the feasibility and practical use of the 3D optical crackmeter. The first step was laboratory testing to evaluate the accuracy of measuring a chessboard target using the computer-vision-based device; the second was field testing of the 3D optical crackmeter installed on a retaining wall in a landslide area.

Laboratory testing
Laboratory tests were undertaken to measure displacement using the 3D optical crackmeter using a STAGE manual fine adjustment platform with a chessboard of 3 × 3 squares and a recording device composed of a Raspberry Pi microcomputer and a digital camera [see Fig. 1(a)]. Nine internal corner points within the 3 × 3 chessboard were interpreted as the reference coordinates. (21) The 3D optical crackmeter consists of the Raspberry Pi 3 Model B with 32G microSD [ Fig. 1(b)] and a digital camera with 640 × 480 resolution [ Fig. 1(c)]. Selfprogramming was performed with Python to calculate the image pixel values of the corner points in the chessboard, which was recognized as an image with a green rectangle for the perimeter and a red circle for the center point, as shown in Fig. 1(d).
For example, the x-and y-coordinates of the center point in Fig. 1(d) were 342.05 and 274.51 pixels, respectively. Then, the distance between the chessboard and the camera in the laboratory was calculated to be 0.1 m (10 cm) through self-programming that converted pixel units to meters. The images at the four corners of the chessboard in Fig. 1(d) had a side of 154.13 pixels when one side of the physical perimeter of the 2 × 2-cm chessboard was 2 cm, so one pixel equaled 0.01 cm for the 10-cm-distance test in the laboratory. In other words, if the image of the chessboard is detected to move by one pixel in the x-or y-direction in the 10-cmdistance test, it means that the chessboard is detected to have moved by 0.01 cm using the method of computer vision technology.
To explain the principle of the 3D optical crackmeter using computer vision technology, Fig. 2 shows the chessboard and camera installed on two sides of a crack. Image changes in the chessboard can be automatically recognized to calculate the displacements of the chessboard relative to the camera. For the x-and y-axis displacements of the center point on the chessboard in Fig. 2, changes in the x-and y-axes of the chessboard mean x-and y-direction displacements relative to the camera, respectively. The initial image of the chessboard was detected using the OpenCV library so that a chessboard image could be moved from x 0 to x 1 or from y 0 to y 2 in the x-or y-direction, respectively.
For the z-axis distance between the chessboard and the camera in Fig. 2, a change in the chessboard image means a displacement in the z-direction. This principle was used to calculate the distance between the camera and the chessboard using a pinhole camera model. (9,22,23) The focal-length function of the digital image in the pinhole camera is −x = f × X / Z, where Z is the distance from the camera to the chessboard, f is the focal length of the camera, X is the physical length of the chessboard, and x is the pixel value of the chessboard corner points on the imaging plane. (9,(24)(25)(26) Thus, changes in the chessboard image can be detected and calculated as physical changes in the distance between the chessboard and the camera, namely, the z-axis displacement in Fig. 2.
Two kinds of laboratory tests were performed using the STAGE manual fine adjustment platform in Fig. 1. The first was distance testing in the z-direction; the other was simulation of the x-and y-axis displacements of the chessboard. For the former, two distances between the camera and the chessboard of 10 and 20 cm were set to measure the accuracy of the 3D optical crackmeter in the laboratory. Accuracy histograms of the center point of the chessboard are shown in Fig. 3. For instance, the x-coordinate of the center point was 332.16 pixels about 420 out of 1000 times in the repeated testing in Fig. 3 The physical displacement was calculated by converting the pixel unit to centimeters using the focal-length equation for the z-axis displacement in Fig. 2, for example, the equation of −x = f × X / Z, where Z is 10 cm for the distance between the camera and the chessboard, X is 8 cm for the physical perimeter of the chessboard, and x is 490.37 pixels for the perimeter of the chessboard with the corner points on the imaging plane. Thus, the value of f, which is the fixed focal length of the camera for the distance of 10 cm, was found to be 612.96 pixels by substituting the values of x, X, and Z into the equation. However, x is a change value on the imaging plane because of the system error in the camera resolution, which is similar to the changes in the x-and y-coordinates (see Fig. 3). Thus, the Z value changed with x, as shown by the accuracy histograms for the 10-cm-distance tests in Fig. 4(a), where the total number of cells was 1000 in the calculation of the distances using the perimeter pixels of the chessboard corner points. As can been seen, the distance of 10 cm appeared 771 out of 1000 times in the repeated testing. Meanwhile, the range of the distance in the testing was between 9.992 and 10.023 cm. Then, the standard deviation of the distance testing was calculated as 0.003 cm. Also, the results of the distance test were analyzed to determine the standard deviation of the crackmeter for the 20 cm distance [ Fig. 4(b)]. The distance of 20 cm appeared 801 out of 1000 times and the standard deviation was calculated as 0.006 cm.
In further laboratory testing, we set different displacements of the x-and y-axes of the chessboard of 1 to 10 mm. Figure 5 shows the measurement errors for displacements from 1 to 10 mm in the x-and y-directions for the center point of the chessboard in the 10-cm-distance testing. As can be seen, the average errors of the x-and y-displacements in five repeated tests were between 0.007 and −0.006 cm for the testing of the ten different displacements. The average absolute measurement errors for the displacement testing were 0.002 and 0.001 cm in the x-and y-directions, respectively.
Finally, an instrument similar to that used in the laboratory testing was installed at a retaining wall in a landslide area in order to monitor the displacement of an existing crack in the structure to evaluate the potential of the instrument for long-term monitoring. Through laboratory and field tests, the resolution and accuracy of the 3D optical crackmeter were determined, as discussed later.

Field testing
Our field test was performed in the Jhongsinlun landslide area in central Taiwan, as shown in Fig. 6, where the area is 32.62 ha and there are two smaller sliding blocks M1 and M2 within a larger sliding block L. The altitude of the area is approximately 468 m. The geological structure of the slope mass in M1 and M2 is mainly composed of mudstone and sandstone; however, the geological structure of the upper slope in L1 mainly consists of mudstone and sandstone with gravel.
The instrument was installed at a retaining wall in Jhongsinlun landslide area to monitor a crack in the wall as shown in Fig. 7, which shows the devices of the instrument, comprising a black box with the 3D optical crackmeter and wireless communication, a solar power system, and a power storage device. The retaining wall was located in a landslide area and previous landslides led to the formation of a crack in the retaining wall [ Figs. 7(a)]. To measure the    potential displacement of the crack, the 3D optical crackmeter was installed at the top of the wall, as shown in Fig. 7(b), along with a linear crackmeter as a manual monitoring device for comparison. Figure 7(c) shows the inside of the box comprising the Raspberry Pi with a digital camera and an LED light, the chessboard, and a wireless communication module. We used IL-LoRa 1272 wireless communication. The LoRa communication device, composed of an RFIC SX1272 chip and a microcontroller unit, has the advantages of long-distance transmission, low cost, and low power consumption. (27,28) The distance between the chessboard and the camera in the field was 12 cm because of the limited box size. The chessboard was attached on the side of the crack in the retaining wall, and the camera embedded in the Raspberry Pi was fixed on the other side of the crack, as shown in Fig. 7(d), so that the amount of crack displacement could be detected using the computer-vision-based crackmeter. In contrast to the laboratory testing, the direction of the camera shot was from top to bottom rather than in the horizontal direction. This was to both detect the displacement of the chessboard and take an image of the chessboard in the crack, as shown in Fig. 7(e). Finally, self-programming with Python was used to calculate image pixel values of the chessboard, which was recognized as the image in the green rectangle in Fig. 7(f) with a red circle for the center point.
The image values of the center point in the chessboard were 310.44 and 156.55 pixels in the x-and y-directions, respectively, as shown in Fig. 7(f). Furthermore, the image value of the perimeter in the chessboard was 139.75 pixels, which corresponds to the initial physical distance between the chessboard and the camera of 0.12 cm when the chessboard was installed on 13/12/2019. If the chessboard is moved or the wall undergoes subsidence, changes in the image pixel value will be automatically detected using the OpenCV library with Python programming in the 3D optical crackmeter.
All field monitoring data from the optical crackmeter were obtained using an image resolution of 640 × 480 pixels captured at a rate of one frame per hour. The Python programming was designed to automatically interpret pixel changes, record data, and transmit data to an FTP cloud system through the LoRa communication as an IoT monitoring system in real time. Graphs of the monitoring data from the optical crackmeter over three months showed pixel changes in the x-, y-, and z-directions every hour (Fig. 8). The relative displacement in the x-direction between the chessboard and the camera in the optical crackmeter is shown in Fig. 8(a), where the left vertical axis indicates image value changes of the center point in the x-direction (unit: pixel).
For example, the initial image value of the x-coordinate of the center point of the chessboard was 310.44 pixels on 13/12/2019 in Fig. 8(a). Then, the image value of the x-coordinate changed to 315.75 pixels on 08/03/2020 so the change in the image value was 5.31 pixels in the period. Because one pixel equals 0.01 cm, as clarified in the laboratory test, the corresponding physical change in the x-direction was 0.0531 cm in the period. Also, the image value of the y-coordinate of the center point of the chessboard changed from 156.55 pixels on 13/12/2019 to 153.41 pixels on 08/03/2020, as shown in Fig. 8(b), so the corresponding physical change in the y-direction was 0.0314 cm. For the change in the z-axis in Fig. 8(c), the initial image value of the perimeter on the chessboard was 559.52 pixels on 13/12/2019, corresponding to the initial distance of 12 cm between the camera and the chessboard in the field. Thus, the x value of 559.52 was substituted into the focal-length function of −x = f × X / Z, where Z was 12 cm and X was 8 cm (the physical perimeter of the chessboard), from which the fixed focal length f was found to be 839.28 pixels. Then, the perimeter image value of the z-axis changed to 560.00 pixels on 08/03/2020, from which the Z value for the distance was calculated as 11.99 cm using the above function, where x was 560.00 pixels, f was 839.28 pixels, and X was 8 cm. Finally, the change in the distance for the z-direction was 0.01 cm, corresponding to the change in the perimeter image value from 559.52 to 560.00 pixels in the period.
According to the monitoring data in Fig. 8, there were regular fluctuations in the coordinates every day. Generally, optical devices are affected by the environmental temperature. (29,30) Thus, other environmental monitoring data in the area such as rainfall and temperature are shown in Fig. 8(d) for comparison with the results of crack displacement monitoring. There was a clear relation between the temperature and the pixel changes as shown in Fig. 9, which presents the monitoring data over a week. Figure 9 illustrates that the pixel changes are greatest when the temperature is highest at noon. By contrast, the pixel changes become small when the temperature drops near sunset, especially at night, as well as on rainy days. To observe the pixel changes more clearly, the one-day moving average of the field monitoring data was used to smooth the time series (31,32) and show the trend of pixel changes without the influence of the temperature on the optical device. The optical crackmeter was originally recorded every hour so the 24-point moving average corresponds to one point of monitoring data per day, as shown in Figs. 8(a)-8(c). The analysis of the moving average of the pixel changes is discussed later.

Results and Discussion
In the laboratory tests, several displacements of the STAGE platform of 0.01, 0.015, 0.02, 0.025, 0.03, 0.05, 0.1, 0.3, 0.5, and 1.0 cm were manually set, and the z-axis distance and the xand y-axis displacements of the 3D optical crackmeter were recorded with a distance of 10 cm between the chessboard and the camera. As mentioned above, the relation between an image pixel and the physical length was that one pixel was equal to 0.01 cm for the 10-cm-distance test in the laboratory. Table 1 shows the measurement errors and actual values for the z-axis distance and the x-and y-axis displacements. In the laboratory testing, the standard deviations were approximately 0.0031, 0.0069, and 0.0020 cm for the z-, x-, and y-axes of the 3D optical crackmeter, respectively. Thus, the resolution of the 3D optical crackmeter was determined as 0.004 cm from the average standard deviation in the laboratory. The maximum errors of the laboratory testing were from 0.0001 to 0.0091 cm, so we considered the accuracy of the 3D optical crackmeter to be 0.01 cm. Furthermore, errors of the field testing must be considered and compared with the laboratory testing as follows.
In the field testing, the monitoring data from the 3D optical crackmeter were recorded in real-time by the IoT technology, where the data recorded were the changes in the image pixels of the x-, y-and z-axis coordinates between 12/2019 and 03/2020. As shown in Fig. 8(a), there was a slight increase in the x-axis coordinate of the chessboard from approximately 310 to 319 pixels. As shown in Fig. 8(b), the y-axis coordinate fluctuated between 155 and 160 pixels during the period. Also, there was a fluctuation from 556 to 564 pixels for the z-axis coordinate up to 15/02/2020 as shown in Fig. 8(c). As mentioned above, the relative displacement in the optical crackmeter was measured using the moving-average method to eliminate the temperature effects in Fig. 9. There was a slight increase in the moving-average x-coordinate of the chessboard from approximately 310.68 to 317.41 pixels. This means that the chessboard on the retaining wall was moved by 6.73 pixels in the x-direction relative to the camera in the optical crackmeter in the period. The corresponding physical x-axis displacement was 0.067 cm from the fact that one pixel equals 0.01 cm in the laboratory test. The moving-average y-coordinate fluctuated between 156.42 and 157.38 pixels in the same period corresponding to a change of 0.94 pixels. The moving-average z-coordinate fluctuated from 558.86 to 559.28 pixels up to 15/02/2020. After that, it significantly changed, dropping by 3.07 pixels from 559.28 to 556.21 pixels from 02/2020 to 03/2020, corresponding to a change of 0.060 cm using the focallength function.
Consequently, cumulative changes in the displacement of the optical crackmeter and their errors were determined from the monitoring data, as shown in Table 2. The errors of the field testing ranged from 0.57 to 1.09 pixels and the average error of the changes was 0.008 cm. Thus, the average error of the displacement amount was determined as 0.01 cm, which can be regarded as the accuracy of the optical crackmeter in the field. Then, the one-day moving average of field monitoring data was used to show the trend of displacement changes in the optical crackmeter. Cumulative changes in the x-, y-, and z-axis displacements for the optical crackmeter are presented in Fig. 10, which showed the amount of relative crack displacement in three months obtained using the moving-average method. As can be seen, the maximum amounts of relative crack displacement in the x-and z-directions increased by 0.08 and 0.12 cm, respectively, in the period. However, the relative crack displacement in the y-direction was only approximately 0.03 cm in the three months.
There was a linear crackmeter installed at the side of the optical crackmeter to measure the crack deformations as shown in Fig. 7(b). The linear crackmeter detected a displacement of approximately 0.1 cm in the three months, while its resolution was only 0.1 cm. It showed the same deforming trend as the 3D optical crackmeter. Table 3 shows that the cost of 3D   optical crackmeter is about half that of traditional monitoring devices and that its resolution is better than that of some mechanical crackmeters, such as linear crackmeters. In addition, the optical crackmeter is part of an IoT system with computer vision technology and LoRa wireless communication.
However, there are some limitations of the optical crackmeter. (a) It needs light, so we used an LED light within the black box in the field. (b) The optical crackmeter is often affected by the environmental temperature. (c) The optical crackmeter in the field is a sensitive sensor for any external forces, such as retaining wall movement or man-made touch. That is why we used a black box to cover the optical device and to protect it from changing ambient conditions as much as possible. Overall, the optical crackmeter using computer vision technology can be effectively and automatically employed for crack monitoring in a retaining wall.

Conclusions
Our proposed 3D optical crackmeter was composed of a Raspberry Pi device, a digital camera, and a chessboard target. Computer vision technology was applied to recognize the chessboard in an image that is made of pixels. The image was used to calculate the relative movement between the camera and the chessboard. A network with LoRa wireless communication was connected as an IoT system to provide remote monitoring functions automatically. The resolution of the optical crackmeter was determined as 0.04 mm in the laboratory, while its accuracy was 0.1 mm in the field. Changes in the image pixels of the chessboard of the optical crackmeter were influenced by the temperature in the field, so we employed a 24-point moving average to analyze the monitoring data. There was a significant displacement in the x-axis direction and a significant change in the z-axis distance in the crack of the retaining wall in the Jhongsinlun landslide area of 0.067 and 0.060 cm, respectively, during three months. There was a similar displacement of approximately 0.1 cm in the same period according to measurements made with a linear crackmeter installed next to the optical crackmeter. The 3D optical crackmeter with the IoT system is more cost-effective than traditional crack meters. Overall, the 3D optical crackmeter with the computer vision technology can be used for 3D displacement monitoring of a crack in a structure and for realtime and long-term monitoring in a landslide area.