Automatic Measurement Algorithm for Brinell Indentations Based on Convolutional Neural Network

To avoid the interference of a material’s surface factors in Brinell indentation images, which adversely affect measurement accuracy, an automatic measurement algorithm for Brinell indentations based on a convolutional neural network (CNN) is proposed. To eliminate the influence of factors such as scratches and collapses of the material surface on the measurement accuracy, the Brinell indentation image as the foreground is divided by the proposed algorithm and an indentation bounding box calculation is carried out after obtaining the binarized pixel mask of the indentation area. The measurement accuracy of the Brinell indentation image under the interference of some material background factors is thus improved. Our experimental results show that compared with the traditional automatic measurement method for Brinell indentations, Brinell indentation images with a complicated background environment can be measured more accurately by the proposed method, with the maximum relative error reduced by 20%. Moreover, the proposed method has strong applicability and high robustness for different material surfaces under different illumination conditions.


Introduction
The Brinell hardness measurement method is widely used to measure the hardness of materials. The Brinell hardness of a material is expressed by ISO (1) as where F is the selected test force, D is the diameter of the tungsten carbide alloy ball indenter, g is the local acceleration due to gravity, and d is the arithmetic mean of the indentation diameters measured in two perpendicular directions. According to Eq. (1), the accuracy of Brinell hardness measurement is directly determined by the indentation diameter.
With the development of material hardness measurement methods, many researchers have begun to study automatic measurement for indentation images. (2) Leta et al. explored a computer vision method to replace manual measurement operations for indentation measurement. (3) However, in practice, some samples used in Brinell hardness measurement lack surface polishing treatment, so there are obvious scratches, dents, and other disturbances on the indentation image. When Brinell indentations are observed under microscopic magnification, the measurement accuracy is reduced as the high-magnification objective lens approaches the material surface because the shorter the distance, the less light enters the field of view. Similarly, the top ring illuminator and parallel fill light commonly used with hardness-testing instruments cast different spots on the surface of different materials. Since neural network methods can achieve superior results to other algorithms in image processing, Tanaka et al. proposed the use of convolutional neural networks (CNNs) to extract the features of indentation images, where the feature vectors of the indentation edge locations are obtained by the fully connected output layer. (4)(5)(6) This means that a neural network can achieve excellent measurement accuracy on complex material surfaces. The resolution of an image captured by a high-resolution industrial camera is reduced by partially cropping or scaling the image. Long et al. proposed a fully convolutional network (FCN) (7) method for processing high-resolution indentation images. A convolutional structure was used to replace the fully connected layer, which was originally attached to the end of the CNN, and the pixel accuracy of image segmentation was improved. Ronneberger et al. proposed a modification based on the FCN; they proposed a U-Net (8) network to splice feature maps. This network allowed the network architecture to achieve better accuracy when trained on a dataset with a small number of samples. However, the above methods only obtained diagonal length values of the indentation without considering its overall shape. To improve the automatic measurement accuracy of Brinell indentations, in this paper, we propose a CNN to segment Brinell indentation images. By obtaining a pixel mask of the indentation area, scratches, collapses, and other factors on the surface of the tested sample are eliminated by the subsequent algorithm. The proposed algorithm can also measure the surface of most materials with most hardness grades. The biggest advantage of this method is that there is no limitation on the size of the input image. In addition, our method is much less affected by the surface roughness of the material. (9) In this study, the system flow chart and the structure of the CNN were first designed. Then, a dataset of Brinell indentation images containing manually labeled indentation regions was constructed, which was used to train a CNN. Finally, experiments were conducted using blocks of standard hardness to verify the reliability and robustness of the proposed algorithm.

Design of Automatic Measurement Method
In the course of production, different materials reflect light differently, and factors such as lighting and scratches or etching on the surface of the material affect the accuracy of indentation measurements. To avoid the interference caused by the background in Brinell indentation images used for indentation extraction, a Brinell indentation segmentation network (BSN) is designed for Brinell indentation images. The main function of the BSN is to operate on the input Brinell indentation image and obtain a binarized masked image of the indentation region. Because no fully connected layer is used in the network structure, a Brinell indentation image of any resolution and size can be accepted by the BSN as an input. Because the size of the image output from the network is the same as the input image, the predicted indentation shape and its location can be mapped directly onto the input image, and the diameter of the circular Brinell indentation is obtained. The BSN used in the proposed method converts the original indentation image, for which it is relatively difficult to identify the edges, into a binarized image that is convenient for a computer program to make accurate measurements. The processing flow of the method is shown in Fig. 1.

Pretreatment procedure
To determine the approximate location of the indentation in the image, thereby reducing the number of subsequent CNN operations and the interference of complex backgrounds, a preprocessing operation is performed on the input raw image. The preprocessing operation searches the input Brinell indentation image to find an approximate range where the indentation is located, so that the region of interest can be truncated and sent to the BSN for subsequent operations. The main flow of the preprocessing operation is shown in Algorithm 1.
The input image is processed by contrast-limited adaptive histogram equalization (CLAHE) (10,11) to improve the image contrast. Compared with adaptive histogram equalization (AHE), this method avoids the overamplification of noise and can achieve better results. Subsequently, mean filtering is used to eliminate the noise interference, and the image is  Image optimization (2) Image binarization (3) Maximum internal connected field search (4) Calculation of enclosing frame (5) Expansion of rectangles and cropping thresholded using the Otsu (12) algorithm. After thresholding, the indentations and surface scratches on the image tend to be connected by fine pathways. By using morphological expansion and erosion processing algorithms, the fine pathways at the edges of the indentations and scratches are disconnected to help find the maximum internal connectivity domain. Then, its enclosing frame is calculated. Because the search for the maximum internal connectivity domain does not necessarily yield the whole Brinell indentation region, the enclosing frame obtained from the previous search must be expanded to avoid cropping of the indentation part. The dimensions of the original input image are (w, h), where w is the width and h is the height of the image. The coordinates of the upper left corner and the lower right corner of the searched enclosing frame are (x 1 , y 1 ) and (x 2 , y 2 ), respectively. The coordinates of the upper left corner and the lower right corner of the expanded enclosing frame are (x e1 , y e1 ) and (x e2 , y e2 ), respectively, which can be obtained by the following steps.
Firstly, find the scaling factor of the enclosing frame with dimensions of d x by d y , , , where p is the scaling factor (p = 1.5). Secondly, calculate the end coordinates of the expanded enclosing frame as follows.

Design of Brinell indentation segmentation network
The Brinell indentation image within the expanded enclosing frame is cropped and fed into the BSN, and a binarized indentation mask image output is computed. To adapt to different resolutions of the Brinell indentation input image and to improve the fineness of the image segmentation, a design of the BSN based on U-Net is proposed.
In the BSN, the basic arithmetic unit is the convolutional block (CB), which is shown in Fig. 2. The CB unit is mainly composed of the residual operation, (13) batch normalization, and rectified linear units. To extract the details of the indentation image, both the padding size and step size of the convolutional operation are set to 1. The CB unit as a whole has different numbers of input and output channels. In the BSN, an upsampling convolutional block (UCB) is used to decode the features extracted by the downsampling convolutional block (DCB). In the feature extraction stage, to reduce the size of the feature map extracted by the convolution operation, a maximum pooling operation with a filter size of 2 × 2 is added to the front of the CB unit to obtain the DCB unit. Similarly, an upsampling operation is added to the front of the CB unit to obtain the UCB unit.
As shown in Fig. 3, the input image is a single-channel grayscale image. To improve the spatial scale information, (14) three images of different sizes are obtained through upsampling and downsampling of the input image. A DCB unit for convolutional feature extraction is used in the original input image, upsampling image, and downsampling image. Then, the feature map extracted on the downsampling feature channel is upsampled, and the feature map extracted on the upsampling feature channel is downsampled, so that the two feature maps fit the feature map extracted from the original input image channel. The three feature maps are connected by the channel splicing method. If the sizes of two of the feature maps are not the same, they are expanded or cropped using zero-filling or pooling operations. The feature map obtained by stitching uses the UCB unit to perform an upsampling convolutional operation. Finally, an image with the same size as the original input image but a different number of channels is obtained. To obtain the output of the binarized mask image, the number of channels is adjusted through a convolutional operation with a filter size of 1 × 1, then the pixel mask output is obtained. The number of channels in the feature map at each stage is labeled with a number below each feature map as in Fig. 3.

Dataset preparation
The BSN requires a Brinell indentation image dataset with marked indentation areas for training. An HMAS series Brinell hardness tester produced by Shanghai Yanrun Company is used in the experiments, as shown in Fig. 4. The top of the platform is equipped with a miniature industrial camera, which outputs single-channel grayscale images with a resolution of 1280 × 1024 by default. The equipped optical lens has a small field of view and an aspheric lens inside, and can eliminate the distortion of the Brinell spherical indentation image. The front of the instrument is equipped with a micrometer eyepiece for manual measurement. The micrometer eyepiece and the industrial camera on the top can be used to observe the surface of the material at the same time through the optical path branch in the instrument, which is convenient for the collection and measurement of indentation images.
To improve the ability of the BSN to discriminate Brinell indentations on a variety of materials and the robustness of the proposed method, various materials are used for testing, such as standard-hardness blocks, chrome vanadium steel, and carbon steel. In accordance with the material characteristics, various test forces are applied to each sample material. Some research  results show that the target location distribution in the training sample dataset has a major impact on the robustness of the trained neural network. (15) A problem is that the location distribution of Brinell indentations in the training sample data is too concentrated around a certain point, resulting in a low recognition rate for the trained BSN for samples with large deviations from this point. To solve this problem, the test platform is electrically controlled to translate the pressed test sample while collecting the Brinell indentation image data. The Brinell indentations in the dataset are thus distributed as evenly as possible within the field of view of the industrial camera.
After collecting the indentation images, the indentation areas must be manually marked for reference during the subsequent BSN training. Note that a Brinell indentation is sometimes not a standard circle in this study. When using marking software, directly using a circular area as a simple mark would often result in part of the actual indentation area being missed or for part of the area outside the indentation to be marked. Therefore, a dot field labeling method is used for manual indentation area labeling. Part of the labeling results are shown in Fig. 5.
For a small amount of training data, such as the annotated Brinell indentation area dataset constructed in this study, it is necessary to perform a data enhancement operation on the training samples. (16,17) After the data enhancement operation is performed, the number of samples used for neural network training is significantly increased, and the robustness of the trained neural network can be further improved by artificially introducing interference. The Augmentor (16) data enhancement tool is used in the training process of this method. Rotation, scaling, translation, and deformation operations are added on the basis of the existing artificially marked image data. The probability of each transformation operation is shown in Table 2.
Various transformation operations are synchronized with the original indentation image and the corresponding marked image. When performing rotation, scaling, and translation, the specific rotation angle, zoom ratio, and translation distance are obtained randomly and the range is also limited. Two thousand sets of image data that can be used for neural network training are obtained after data enhancement.

Optimization of training phase and design of loss function
The PyTorch (18) neural network framework is used to build and train the BSN, which includes two steps. The first step is the training process. In this process, the Brinell indentation dataset is randomly divided into three parts: a training set, a verification set, and a test set, with a size ratio of the three sets of 8:1:1. To limit the resources used during training, the input image is scaled to 50% of the original size for each training. The whole training process is iterated five times on the training set, and after every 100 images are trained, the verification set is used for verification. In each iteration, the data used for training are randomly selected from the training set. To verify the consistency of the results each time, the data in the verification set are extracted and verified in a predetermined order. The second step is the evaluation of the effectiveness of the training, in which the image obtained by BSN prediction is compared with the manually labeled reference image pixel by pixel, and the similarity between them is calculated. The results obtained on the verification set each time are used to adjust the learning rate of the neural network. When the correct rate of the verification results has been within a certain interval several times, the learning rate is reduced and further training is carried out.
Using an appropriate loss residual function to measure the difference between the BSN prediction and the expected result can make the neural network converge quickly. The difference between the output of the CNN and the artificially labeled reference image in the subsequent learning stage can be gradually reduced. In the proposed method, the results of the BSN and manual labeling are both binarized pixel mask images. For this type of image segmentation problem, some research results have shown that the two-category, cross-entropy algorithm can better reflect differences between the BSN and manual labeling results. (19) The two-category, cross-entropy result after applying the sigmoid function is passed to the framework as the loss value for evaluation and optimization. The loss function is calculated as  where z n and y n represent the values predicted by the neural network and the artificially labeled value for the nth training sample, respectively, and δ represents the sigmoid function.

Comparison between prediction and reference image on test set
After the BSN is trained, the proposed method is tested with the images in the test set to verify the effectiveness of pixel mask extraction of the BSN on Brinell indentation images and also verify the difference between the final bounding boxes obtained by the proposed method and manual measurement. The images in the test set do not overlap with those in the training set and validation set. Part of the experimental results are shown in Fig. 6.
We compare the indentation area predicted by the BSN with the manually marked indentation area using the mean of the differential image. The result of the BSN prediction is in red and that of manual marking is in green. The differential image shows that the prediction result of the BSN is very close to the result of manual marking. The prediction results of the BSN are still relatively satisfactory for the image with blurry edges even if there are many cracks around the indentation due to the nature of the material. To reduce the misjudgment when solving the bounding box for the BSN prediction results and reduce the influence of the deviation when extracting the edge in the diameter measurement of the Brinell circular indentation, some small islands are regarded as noise and discarded in the binary mask image predicted by the BSN. The largest connected domain is selected as the range of the indentation, and the circumscribed rectangle is found for this domain. Because a spherical indenter is used in the Brinell hardness experiment and the indentation is approximately circular, it is usually only necessary to find a bounding box whose edges are parallel to the axes of the image coordinate system. The length d 1 and width d 2 are used in the Brinell hardness calculation formula. In practical applications, OpenCV is used to calculate the bounding box of the artificially annotated image and the BSN image.

Testing on other materials
To test the effectiveness of indentation extraction by the proposed method on other materials, several different samples from those in the training dataset are selected for the Brinell hardness test. The robustness of the proposed method is tested by comparing the indentation bounding boxes obtained by the proposed method and by manual marking. In the experimental process, it is found that although some of the materials in the training dataset are metals, such as titanium alloys, due to the different properties of the materials, the interior of the indentations is markedly different from those of the samples in the dataset under the annular light on the top of the test platform. Some relatively soft materials clearly show phenomena such as indentation collapse. For such difficult samples, the indentation prediction and the extraction cannot be performed well by the BSN trained on the dataset with an insufficient number of sample types. The DCB unit is used as an encoding network in the BSN. Its main function is to abstract the image data and extract the high-level semantic features so that the UCB unit can generate pixel mask images based on the features. Therefore, for the different samples, the DCB unit is still the main object that requires training. The UCB unit weight of the BSN that has been trained on the original dataset is fixed, and the PyTorch neural network training framework is only used to perform incremental training on the weight of the DCB unit part using the newly collected indentation image samples. Then, the trained network is used to test new materials. Part of the experimental results on different materials obtained using the proposed method are shown in Fig. 7.
The left column in Fig. 7 shows the original images, the middle column shows the indentation areas predicted by the BSN, and the right column shows images of the bounding boxes on the original images obtained by manual annotation and BSN prediction. The green boxes show the manual labeling results and the red boxes show the measurement results obtained with the proposed method. As shown, the indentations are clearly visible in image (a) due to the hard material; images (b) and (c) are less homogeneous due to the softer material, resulting in collapses and breaks around the indentations. However, they tend to interfere with the segmentation of the indentation algorithm. As can be seen from the middle segmentation result graph, our method is more robust to collapse and rupture around the indentation, and still does a good job of segmenting the indentation in this case. It can be seen that the proposed method can also obtain measurement results that are close to the manual measurements on materials other than those in the training dataset. This shows that the proposed method is applicable to not only the Brinell indentation measurement of the materials covered by the training dataset, but also the Brinell indentation measurement of most materials, indicating its high robustness to the input material of the sample.
To compare the measurement accuracy of the proposed method with that of the traditional method, the maximum internally connected domain (MICD) method is introduced in the experiment, which consists of the first four steps in the preprocessing algorithm. The range of the indentation is obtained by calculating the largest internally connected domain binarized in the Brinell indentation image. For the same Brinell indentation image, the MICD method and the proposed method are used to measure the Brinell indentation, and the results are compared with the manual measurement. The experimental measurement data are shown in Table 3.
As shown in Table 3, each material is tested three times under each test force. The diameter of the Brinell circular indentation and the calculated Brinell hardness are recorded as the average of the three test results. Compared with the MICD method, the final result of the Brinell hardness calculation is closer to the manual measurement, which means that the proposed method gives a result closer to manual measurement than the MICD method. Figure 8 shows the minimum, maximum, and average relative error between the MICD method and the proposed method for different materials, such as stainless steel, titanium dioxide, and copper alloy. As shown in Fig. 8, the error between the proposed method and the manual measurement is generally lower than that of the MICD method, indicating that the proposed method has higher measurement accuracy. Comparison of the data between the proposed method and manual measurement showed that the maximum relative error between them is 1.05%. The minimum relative error is 0.02%, indicating that the error between the proposed method and manual measurement is small. This means that, in some cases, the proposed method can replace the automatic measurement of Brinell indentation operated manually.

Conclusion
To improve the accuracy and robustness of the automatic measurement of Brinell indentations, a CNN method was proposed. The indentation foreground in the Brinell indentation image is extracted by the CNN and converted into a binarized pixel mask image. The bounding box of the pixel mask image is solved to obtain the average diameter of the Brinell circular indentation, so as to complete the automatic Brinell indentation hardness calculation. Compared with solving the bounding box of the indentation area directly on the original image, the proposed method of using the CNN for indentation image segmentation reduces the difficulty of the subsequent process to solve the bounding box. To verify the measurement accuracy and robustness of the proposed method, a Brinell indentation image dataset was constructed and an experiment was carried out on samples different from those in the training dataset. The experimental results show that the proposed method has high robustness to Brinell indentation images of different materials. Hardness-testing instruments are gradually developing in the directions of miniaturization and multifunctionality. Our next work is to quantify the neural network in the automatic measurement algorithm and deploy it on a small embedded board.