AI-aided Hammering Test System to Automatically Generate Anomaly Maps

range the of the integrate this information into the hammering echo analysis results to automatically generate an anomaly map. We performed hammering echo collection experiments using the AI-aided hammering test system and evaluated its performance. In the experiments, we inspected seven actual bridges in which internal defects (float) were detected by a detailed manual hammering test and compared the results with those obtained using our system. No defects were missed in a coarse block unit, and the accuracy for each hammering echo was determined to be 96.3% at maximum and 90.4% on average.


Introduction
In recent years, with the aging of social infrastructure, a maintenance management method from the viewpoint of preventing third-party damage has become essential. The overall inspection of the bridges and roads in Japan is stipulated by the Ministry of Land, Infrastructure, and Transport, and the demand for checking infrastructure is increasing rapidly. However, the number of skilled inspectors has tended to decrease as the population ages and the labor force shrinks. In many cases, it is difficult to hire inspectors. Therefore, in this research, we aim to establish a hammering echo inspection technology that can detect defects accurately irrespective of the skill of the inspector.
In a general hammering test, an inspector strikes a concrete structure using an inspection hammer, listens to the hammering echo carefully, and assesses the state of the structure. This test method depends on the experience, opinions, and skill of the inspector, and the inspector may reach incorrect conclusions concerning the inspection results.
The impact-echo method, which is a method of objectively analyzing hammering echoes, was proposed in the 1980s as a method to eliminate variability among inspection results and has been used as a nondestructive inspection method for concrete structures. (1,2) As a general description of the impact-echo method, a hammer or a similar object is used to strike the surface of a concrete structure, as shown in Fig. 1. Shock waves caused by the impact propagate inside the structure and are measured by sensors installed near the impact point. The state of the concrete is evaluated on the basis of the measured signal in the frequency domain obtained by Fourier transformation. In this method, the specific frequency (dominant frequency) at which the spectral intensity is maximum is commonly used as the index (feature value) of the concrete state evaluation. (3) However, the use of a single feature value makes the signal susceptible to noise, and other features are discarded. In this study, we aim for more accurate defect detection in a concrete structure by adopting an approach of analyzing hammering echo data acquisition and the machine learning technique, which is a key technology of artificial intelligence (AI). To achieve the objective, here, we propose and develop an AI-aided hammering test system that automatically detects the anomalous points and the degree of anomaly of a structure using the machine learning of the difference in the hammering echoes generated by the inspection hammer. In this system, the hitting position of the hand-held hammer is determined using a laser range sensor; by integrating this information with the hammering echo analysis results, the system automatically generates an anomaly map. Consequently, the number of manhours associated with the laborious recording of inspection results after the hammering test is reduced. The hammering echo analysis method developed for this system is described in Sect. 2, where the concept and configuration of the system are explained. In later sections, we compare the hammering echo analysis performance with that of the conventional impact-echo method and discuss the results of a performance evaluation test using the hammering echo data collected for actual bridges.

Developed Hammering Echo Analysis Method
In this section, we describe the hammering echo analysis method developed for the AI-aided hammering test system.
As described in Sect. 1, the impact-echo method, which is a method of automatically analyzing hammering echoes, has been used as a method for the nondestructive inspection of concrete structures to eliminate variability among inspection results obtained by different inspectors. However, because this method uses only a specific frequency (dominant frequency) at which the spectral intensity is maximal, it is susceptible to noise and other features are discarded. Therefore, the objective of this work was to achieve more accurate defect detection by the hammering analysis method through the application of machine learning technology (4) to the collected hammering data of concrete structures.
In attempts to distinguish between a normal hammering sound and the sound of a defect on the basis of machine learning, supervised learning (4) is commonly used. Supervised learning is a method of gathering numerous hammering echo data for a normal object and an object with defects, where the actual state of the object (i.e., normal or defective) is known beforehand; the classifier then learns from statistical information embedded in the data. However, compared with data for normal objects, the data for defective objects is more difficult to acquire, which introduces a problem regarding the acquisition of sufficient data for defect detection. Therefore, in this method, by assuming most of the acquired hammering echo data to represent a normal object, a method of unsupervised learning, which enables the statistical learning of a normal pattern for each examined object, is adopted. Unsupervised learning enables the presence or absence of a defect to be judged even if various hammering sound data have not been gathered for training. It is a general-purpose method that is independent of both the object being investigated and the inspection apparatus.
In this research, anomaly detection based on the subspace method (4) was performed as an unsupervised learning method. In this method, most of the normal patterns are represented by a lower dimension in accordance with the subspace, and the distance between the hammering echo samples to be determined and the subspace is calculated as the degree of anomaly. By expressing the subspace with lower dimensions, the method becomes robust against disturbances such as noise and measurement errors. Figure 2 shows the process flow of this system. After hammering echoes and hitting positions are acquired using the sensor unit, the learning phase proceeds. In this phase, as preprocessing for hammering echoes, the hammering echoes are converted into a spectrogram, split into each echo, and normalized with respect to the intensity. Subsequently, unsupervised learning via the subspace method is performed, and the subspace of normal hammering echoes is generated as a standard for judging hammering echoes. Next, in the detection phase, the preprocessing and feature extraction of hammering echoes are executed, and the system calculates the distance from the subspace of the normal hammering echo, thereby obtaining the degree of anomaly of each hammering echo. Then, the degree of anomaly of each hammering echo is combined with the impact position, and an anomaly map is generated.
Here, we present the detailed hammering echo analysis procedure using the subspace method. (5) set of hammering echo spectra acquired by inspection, where n denotes the timeframe index and M is the dimension of the spectra. To extract the echo signal subspace, we calculate eigenvalues Λ = diag(λ 1 , …, λ M ) and , is the transpose of x i . We sort eigenvectors by eigenvalues in decreasing order. These eigenvalues denote the significance of the corresponding eigenvectors for expressing the spectrogram. The cumulative contribution ratio is defined as All eigenvectors of U = [u 1 , …, u M ] (M × M) span the subspace characterizing the spatiotemporal distribution of the echo signal. In particular, U M is ranked by the eigenvalues; in other words, the importance of expressing the echo spectrograms and dominant features lies in the first several basis vectors. As mentioned previously, there is the assumption that most of the acquired hammering echoes are normal; thus, if we collect the eigenvectors of greater importance in this method, we can establish the feature of a normal hammering sound. For this reason, we set a constant cumulative contribution ratio η K (approximately greater than 0.8) and select K eigenvectors corresponding to this ratio as U K = [u 1 , …, u K ] (1 < K < M). By this procedure, it is possible to concentrate on predominant patterns in the echo signal and remove the noise influence simultaneously. Also, in this method, we regard the vectors of hammering echo spectra projected on the subspace as feature values extracted from hammering echoes. In contrast to the feature value of the conventional impact-echo method, which has only one dominant frequency, the feature values of our method have an advantage that much information is compactly summarized from the frequency components of a hammering echo.
The subspace obtained in this manner represents the feature of normal hammering echoes. Using the subspace, feature extraction and the calculation of the degree of anomaly are performed by the following calculation concerning the hammering echo to be determined.
The projector U K for the subspace of the normal hammering echo is represented as P = U K U K ', and that for the orthogonal complement space is P o = I M − P. When the spectrum of the hammering echo to be determined is x, the feature extraction is performed by the projection In this method, we use this d o as the degree of anomaly of the hammering echo. An advantage of the subspace method is that it requires less computational complexity than other nonlinear methods. Although we carried out hammering analysis using another unsupervised learning method such as spectral clustering, (6) which is a nonlinear clustering method, the computational complexity increased. For this reason, in the system, we adopt the subspace method with a relatively small computational load to lower the throughput requirement to the PC that executes the hammering echo analysis process, making the realization of a practical system easier.

System overview
We have developed an AI-aided hammering test system to assist hammering tests of infrastructure with AI and automatically generate an anomaly map. Figure 3 shows the outline of the system. This machine learns the difference in hammering echoes generated by the inspection hammer and automatically detects the anomalous parts of the structure and the degree of anomaly. Then, it automatically creates an anomaly map by integrating the detection results with the hammering position information of the inspection hammer.

System configuration
The AI-aided hammering test system developed in this work consists of a sensor unit that measures the hammering echo and position, and a tablet PC for the control, recording, and analysis of hammering echoes using AI (Fig. 3). The system is convenient in that it can be used with the sensor unit simply leaned against a wall or a similar flat surface of a structure (planar structure), whereas the inspector uses a commercially available inspection hammer, as shown in Fig. 4.
This system automatically generates an anomaly map in response to hammering sound inspection using a general inspection hammer and presents the map to the inspector immediately upon the completion of the series of hammering echo inspections. When the hammering test is completed, the anomaly map is automatically generated by integrating the acquired hammering position and the degree of anomaly of the hammering echoes.

Explanation of system components
As described in Sect. 3.2, the AI-aided hammering test system consists of a sensor unit and a tablet PC. Each component is described below.
A laser range finder (LRF) was integrated into the sensor unit to measure the position of the inspection tool, such as a hammer, in contact with the surface under inspection. In this study, an LRF that can acquire distance information on an object intersecting the measurement target plane was used. The plane is fan-shaped centering on the apparatus (Fig. 5). We attempted to acquire the hammering position by setting this measurement plane as parallel as possible to the surface to be inspected such that the distance between the wall surface and the measurement surface was as short as possible. From the obtained LRF data, it is necessary to recognize the hammer as an object. For this purpose, the hammer was identified by applying the moving-body recognition algorithm reported in Ref. 7 and then the hammering position was specified. Additionally, a contact-type acoustic sensor was used to detect the hammering echo of infrastructure. This approach enabled the hammering echo to be detected with low noise even in places with high noise, such as road tunnels. A contact-type acoustic sensor collects hammering echoes by detecting vibrations induced at the surface of a structure by hammering echoes propagating in the infrastructure. Additionally, a condenser microphone was mounted as an acoustic sensor to assist the contact-type acoustic sensor in detecting the hammering echo of air propagation in a supplementary manner. With this design, the detection of the hammering echo of air propagation complements the detection of the hammering echo by the contact-type acoustic sensor. In this work, we concatenated the spectra of the two types of hammering echoes and used the result as input for analysis.
The tablet PC stores the hitting position acquired by the sensor unit and the hammering echo. While the hitting sound is being acquired, the hitting position is displayed in real time. Using these stored data, the anomaly map, determined by the degrees of anomaly calculated using Eq. (3), is automatically generated. The hammering echo analysis was implemented by creating a hammering echo analysis program using the subspace method with a programming language, MATLAB. Figure 6 shows an example of an anomaly map acquired for an actual structure (the wall surface of a tunnel). In the map, the anomalous part (upper central part) of the wall surface is shown and the degree of anomaly is quantitatively shown. The degree of anomaly, which conventionally depends on the skill and experience of the inspector, was quantitatively analyzed from the learned normal hammering echo and visualized in color. Because it is mapped immediately after the examination, hitting omission can be confirmed onsite and additional tests can be performed. Furthermore, the repair/reinforcement design of the abnormal part previously included a work step of creating a detailed damage diagram; however, with this system, the number of steps can be reduced.

Performance Comparison with Impact-echo Method
As a basic study, we compared the results obtained by the impact-echo method, (1) which is the conventional method of automated hammering echo analysis, with the hammering echo judgment result obtained with the system developed in the present work.
In the impact-echo method, because the dominant frequency of the hammering echo is used to judge whether the hammering echo is normal or abnormal, it is susceptible to the influence of differences in sound owing to the hammering angle of the hammer, the influence of the unevenness on the hammering face, and noise, among other factors. However, because the subspace method used in this system is less susceptible to these effects, better hitting analysis results are expected to be obtained.
To confirm the improvement in the hitting analysis results, we recorded hammering echoes in an actual tunnel with automobile driving sounds and compared the hammering echo analysis results obtained by the impact-echo method with those obtained using the method proposed in this work. The hammering echo was recorded at a sampling frequency of 44.1 kHz. In the impact-echo method, as shown in Fig. 7, excitation was cut out in units of one hammering echo and spectra were obtained by Fourier transformation to determine the dominant frequency. Figure 8 shows a graph of the dominant frequency in the impact-echo method and the anomaly graph resulting from the method developed in this work. The horizontal axis of this graph shows the hammering echo index. Comparing these graphs, we cannot distinguish the anomaly points from the dominant frequency. In contrast, with our method, we find that the degree of anomaly is high at the anomaly points and that the anomaly points can be correctly detected by setting an appropriate threshold value. In the impact-echo method, we used only the hammering echo recorded with a contact-type microphone, which tends to reject noise. However, we speculate that vibrations created by moving vehicles generated noise in the hammering echo. As described previously, numerous sources of noise were present where the actual hammering echo tests were performed; thus, we deduced that the method proposed in this work, which is not susceptible to noise, is more effective when used for practical tests in the field.
Next, we investigate the subspace obtained in this analysis. Table 1 shows the relationship between the cumulative contribution ratio [Eq. (2)] of the analysis result shown in Fig. 8(b) and the corresponding number of eigenvectors. From this table, we observe that the cumulative contribution ratio exceeds 0.95 at the top 5 of M = 105 eigenvector numbers in this experiment. Because the threshold value of the cumulative contribution ratio in this experiment was 0.85, a subspace was composed of the first two eigenvectors. Because high cumulative contribution ratios are obtained with a few eigenvectors, we find that most features of the hammering echoes (features of normal hammering echoes in this analysis) were successfully extracted by the subspace method.

Performance Evaluation on an Actual Structure
We conducted evaluation experiments of the AI-aided hammering test system targeting real structures (seven bridges). In the evaluation experiments, we performed manual detailed hammering tests, inspected structures with internal defects (floats) using the system, and compared the results. As a method of calculating the accuracy, we first confirmed that there was no oversight in rough blocks (a series of hitting points in an anomalous area). Consequently, we confirmed that there was no oversight in any of the results. We also calculated the accuracy as the rate at which parts judged normal by one echo were actually normal and parts judged anomalous by one echo were actually anomalous. An example of an anomaly map obtained via analysis using our method is shown on the right side of Fig. 9. In the anomaly map, yellow indicates that a hammering echo has a high degree of anomaly and blue indicates regions where the hammering echo is a normal sound. In the photograph on the left side of Fig. 9, the range where the detailed hammering test was conducted by an inspector (in the square frame) and the range in which the damage was recognized (in the dotted line frame, 0.6 × 1.2 m 2 ) are shown. A comparison of this photograph with the anomaly map reveals that there were no oversights of defect blocks. Moreover, the accuracy was as high as 96.3%. Table 2 shows the accuracy in each evaluated location in unsupervised learning. Good results of 96.3% at maximum and 90.4% on average were obtained. From this result, it can be considered that unsupervised learning makes the hammering echo analysis applicable to the hammering test for a real structure. Evaluation execution site Nos. 3 and 6 are excluded from the calculation target of the accuracy for the following reasons. At site No. 3, since most of the examination area was the damaged area, the assumption for the unsupervised learning technique was not satisfied, and the defect hammering echo could not be correctly detected. Evaluation site No. 6 was a place where no defect area was noted, and although a matching rate of 96.3% was obtained, we regarded it as an exceptional measurement target.

Discussion
In this system, by automatically generating an anomaly map, the number of work manhours previously spent recording can be reduced, and inspection work can be performed without overlooking anomalous regions of a structure, even by unskilled inspectors. Therefore, the AI-aided hammering test system is considered to be useful as a system for infrastructure inspection, the need for which is expected to rapidly increase in the future, including in regions where skilled inspectors are scarce. The unsupervised learning used in this analysis has an advantage applicable also in the case where it is difficult to collect data for learning in advance, such as defect sounds in the hammering echo test, which is the target of the present research. However, when most of the examination area is a defect area or when there is no defect hammering echo, such as in evaluation sites Nos. 3 and 6 described in the previous section, judgment by supervised learning is needed. As mentioned in the introduction, however, gathering data related to anomalous hammering echoes is difficult. Therefore, a technique called transfer learning would be a key technology for streamlining learning such that anomalous echoes can be recognized with a small amount of data diverting the learning results obtained in other environments without acquiring a large amount of data in the new environment. We have already demonstrated the effectiveness of transfer learning in defect detection using test concrete blocks, (8) and transfer learning could reduce the costs associated with acquiring learning data.
Because the machine learning technique is a statistical technique based on data, a certain recognition error is unavoidable. As a countermeasure, AI technology that cooperates with people, such as error correction by workers and additional learning that improves instrument accuracy in working environments, becomes important. In this case, an interface/interaction technology related to presenting information to the operator and the visualization of the machine learning results to convey to the operator the criteria judged by machine learning will be necessary.
At present, automatic inspection technology for social infrastructures is attracting attention in Japan, and various such technologies are being developed for practical use. (9) However, developing only one excellent technology for automatic inspection is not enough. Excellent automatic inspection performance will be possible only when several types of technologies complement each other. As compared with other technologies, a major advantage of practically  implementing our technology is that the inspection hammer used by the inspector can be used as it is, indicating that our method can be easily introduced in practice without significantly changing the existing work procedures. Therefore, it is expected that this technology will be used in infrastructure inspection work across Japan, which is expected to increase rapidly, including work undertaken by local governments, wherein it is difficult to hire skilled inspectors.

Conclusions
We proposed and developed an AI-aided hammering test system with the aim of realizing a hammering test technology capable of accurately detecting damage irrespective of the skill of the inspector. This system conveniently obtains the hitting position of the hammer by manual operation with the laser range sensor and integrates the hitting-position information with the hammering echo analysis results to automatically generate an anomaly map on site. In the performance evaluation of this system, no defects were missed at any location in rough block units of seven actual bridges. Additionally, the correct answer rate for each hammering echo was determined to be 96.3% at maximum and 90.4% on average. Future research opportunities include introducing a technique that makes hammering echo judgments based on hammering data learned beforehand by transfer learning. Furthermore, our system successfully notified the inspector of the analysis results in real time by learning hammering echo data obtained online. (10) We expect that this real-time anomaly detection system will be useful in the future. Moreover, the unsupervised learning method used in this paper may be applicable to anomaly detection using various other sensor signals.