Analysis of Hemostasis Procedures through Machine Learning of Endoscopic Images towards Automatic Surgery

Laparoscopic surgery reduces patient invasiveness; however, the burden on the surgeons is high because such surgery requires them to have skills higher than those for open procedures. In particular, improving the working environment of surgeons involves reducing the amount of human resources required and providing high-level medical services. The cooperation between robots and surgeons has been effective in the medical field; therefore, we focus on the automation of hemostasis procedures. An important factor in automation is target detection and the decision on the completion of the procedures. In this study, we analyzed hemostasis procedures by region detection through machine learning and developed a method of defining the termination conditions of the procedures. In hemostasis procedures, the bleeding region is coagulated by an energy device, the area of the hemostasis region increases, and the surgical procedure is continued. The method could detect the end of the procedures by monitoring the variations in the sizes of the bleeding and hemostasis regions.


Introduction
In the medical field, minimally invasive surgery (MIS), which is performed by inserting miniaturized tools and an endoscope into a body cavity, is widely used to improve the quality of life (QoL) of patients. (1) However, the burden on surgeons tends to increase because MIS requires them to have skills higher than those required for conventional surgery. In general, the introduction of automated systems and robotics technology has been effective in reducing the burden on surgeons. (2) Some studies have proposed the cooperation between robotics and surgeons for MIS. (3,4) Improving the QoL of patients and reducing the burden on surgeons are major issues in medical engineering. Therefore, we focus on a new surgical procedure, water-filled laparoendoscopic surgery (WaFLES), (5,6) which uses a liquid instead of a gas conventionally used to secure the space inside the body cavity. Surgery in a liquid is commonly performed on narrow lumen organs such as the transurethral resection. (7) The WaFLES technique can be used to perform surgery in a large space by irrigating the liquid. By suppressing the cooling effect of the liquid and drying the organ, organ inflammation is suppressed. (8) This considerably contributes to improving the QoL of patients. In addition, the vision of the endoscope in the liquid under irrigation is better than that when using gas. (6) Furthermore, by washing the organ surface, the treatment site and bleeding point can be constantly observed; therefore, sensing can be achieved through endoscopic images.
In robotics, systems that detect targets and measure state transitions, converting them into digital data, are important technical requirements. In a previous study, we proposed and developed a system that can automatically detect the organ, bleeding, and hemostasis regions in real time to assist hemostatic procedures. (9) Hemostases are frequently performed in surgical procedures (10,11) such as tumor ablation and tissue dissection; therefore, robot assistance or automation can considerably reduce the burden on the surgeons. In addition, hemostasis induces the thermal degeneration of organs using energy devices. (12,13) Robots capable of highly accurate maneuvers can be expected to more effectively suppress invasion than humans, because the energy device can be guided to the correct place, and coagulation errors can be reduced. The method developed by us in our previous study (9) contributes to the automation of hemostatic procedures by detecting changes associated with the protein degeneration of organ surfaces induced by using energy devices from the color features of endoscopic images through machine learning. In this method, we used a support vector machine (SVM), which is a machine learning method. Machine learning can reduce the computational cost of image processing and provide high accuracy; therefore, it is widely used in the detection of bleeding regions for endoscopy. (14)(15)(16)(17)(18)(19)(20)(21) The automation of hemostasis procedures is being studied because it is difficult to detect the hemostasis region; therefore, we have proposed a novel detection system (9) for hemostasis regions. In the present study, we analyzed the bleeding and hemostasis regions measured from endoscopic images using our detection method to clarify the termination conditions of our procedure in an automated hemostasis procedure.

Proposed surgical system
Bleeding is frequent in surgical procedures because of the ablation of organs and tumors. The mechanism of hemostasis begins with the detection of the bleeding region. (22) The bleeding region tends to have various forms and textures in endoscopic images. Therefore, we proposed a detection method based on machine learning that focused on a color feature descriptor. (14) In our system, high-accuracy, real-time detection (9) was targeted because the robot needed to induce coagulation while tracking the detected bleeding region. In addition, the heat-treated bleeding region was discolored owing to protein degeneration. We considered that the end of hemostasis can be determined by comparing the hemostasis region with the bleeding region, because the discolored region can be detected as a hemostasis region. Figure 1 shows an automated hemostasis robotic system that uses the WaFLES technique.
In this study, we extended the machine learning dataset in our previous study (9) to enable a more robust detection of the organ, bleeding, and hemostasis regions. Furthermore, the end of a hemostasis procedure was analyzed by monitoring the variations in the sizes of the bleeding and hemostasis regions, and we tried to quantify the termination condition of this procedure. We established a method of constructing the sensing system for our surgical robot system by integrating the detection and analysis techniques.

Linear SVM method and multiclass classification
The SVM method (23) is a classification method that uses prior learning based on a training dataset. The SVM method can obtain high classification accuracy using a small number of descriptors, and the computational cost is lower than that of other machine learning methods. The learning of SVM classifiers involves solving the following optimization problem for the hyperplane: where w is the gradient of the hyperplane, w 0 is the intercept, t n is the label corresponding to data point x n , C is the parameter that determines the tradeoff between the penalty and the margin, (23) ξ n is the slack variable (23) given as and φ(•) is the kernel function. In the linear SVM, the kernel function is applied to Eq. (3). In general, Lagrange's undetermined multiplier method is used to solve the minimization problem of Eq. (1). In this study, we used the sequential minimal optimization (SMO) (24) algorithm.
When a new x n is given, y is obtained as follows by solving the optimization problem: In this study, we attempted multiclass classification. We employed a multiclass model (25) that performs binary classifications as the necessary number of classifications. We used MATLAB ® R2018a (Mathworks, Inc., Natick, MA, USA) to develop the SVM classifier and classify the regions.

Color feature descriptors
According to our previous study, endoscopic images in a liquid have a higher contrast than those when a gas was used, because the liquid provides a washing effect and soft lighting that moderately diffuses the forward light in the surgical area. Therefore, in addition to the color feature factor in the conventional (21) RGB model, we proposed color feature descriptors based on saturation (S) and lightness (V) in the HSV model, (26) which is the hexcone model. (26) This model provides a higher accuracy than the conventional method. (21) We used three color feature descriptors suitable for real-time processing, which are given by the following equations: where R (i) , G (i) , and B (i) are the red, green, and blue values in the RGB model, and S (i) and V (i) are the saturation and lightness at the ith pixel, respectively.

Training dataset and SVM classifier verification
A training dataset is required to develop the SVM classifier. The objects were endoscopic images obtained in animal experiments carried out with the approval of the local ethics committee for animal experiments using four specific pathogen-free (SFP) pigs that weighed approximately 30 kg. Laparoscopic non-ischemic partial nephrectomy (27) was performed using the WaFLES technique by two urologists. The non-ischemic partial nephrectomy does not cause ischemia; therefore, although damage to the normal tissue is suppressed, (27) hemostasis is important because bleeding is frequent. From the obtained endoscopic images, the organ region pixels (206606 pixels), bleeding region pixels (147456 pixels), and hemostasis region pixels (147456 pixels), a total of 501518 pixels, were randomly extracted (Fig. 2). The organ region showed various color characteristics; therefore, more samples were obtained for it than for the other regions.
By comparing the SVM classifier using our descriptors with the conventional descriptors, (21) more robust classification was guaranteed.

Analysis of the termination conditions of hemostasis
We performed the time series analysis of hemostasis to obtain the termination conditions. The objects were the endoscopic images of the bleeding and hemostasis regions obtained in the four-animal experiments, and ten hemostasis procedures were extracted. For the endoscopic images of hemostasis, the region of interest (400 × 400 pixels) was manually defined. Our SVM-based region detection technique was used for each pixel. In one hemostasis procedure, five endoscopic images were extracted from the endoscopic videos, from the bleeding to the hemostasis procedure, because the endoscopic images were not fixed. Endoscopic images from the same viewpoint were suitable for our analysis. Figure 3 shows the endoscopic images of the ten hemostatic techniques targeted in this study. Moreover, these procedures were confirmed by a urologist to have resulted in complete hemostasis.
In accordance with the mechanism of the hemostasis procedure, we first observed a large area of bleeding and found that the hemostasis area increased as the hemostasis progressed. Therefore, to visualize and quantify the state transition of the hemostasis procedure, the variations in the sizes of the bleeding and hemostasis regions were calculated as where S b is the bleeding region size, S h is the hemostasis region size, and S c is the ratio of the sizes of the bleeding and hemostasis regions. The termination condition based on S c was applied after our SVM-based region detection.

Results
The five-part cross-validation for the linear SVM classifier using our descriptors revealed that the classification accuracy was 96.7%, which was higher than that obtained using the conventional descriptors (92.1%). Figure 4 shows the confusion matrices and the TP and FN rates of each region in the classifier using the proposed and conventional descriptors.  For the termination condition analysis, the changes in the sizes of the bleeding and hemostasis regions in the hemostasis procedure in a time series labeled 0 to 5 are shown in Fig.  5. In addition, the labels confirmed to be hemostasis labels by the urologist are indicated using blue circles.   Furthermore, Fig. 6 shows the change in the ratio of the bleeding and hemostasis region sizes obtained using Eq. (7). In the figure, the vertical axis has an s logarithmic scale.
Moreover, we compared the values of S c of the label before and at the endpoint of the hemostasis procedure in Fig. 7 and obtained p = 0.04 using the paired t-test.

Detection using proposed descriptors based on an extended dataset
We extended the dataset to enable a more robust detection of the organ, bleeding, and hemostasis regions. The sample data increase from 319488 to 501518 pixels, focusing on the organ regions considered to be misclassified. The results of the five-part cross-validation showed that the proposed color feature descriptor was more robust than the conventional color feature descriptor. The results in Fig. 4 showed that our SVM classifier was more able to reduce the misclassification of the organ and hemostasis regions than the conventional SVM classifier; thus, we obtained satisfactory results with high TP and low FN.
In our system, the lighting environment, the color of the organ, and the white balance of the complementary-metal-oxide-semiconductor camera were considered to affect the detection accuracy, because our SVM classifier used color information of the endoscopic images. Establishing a dataset with more cases helped improve the robustness of the SVM classifier. We also demonstrated that the proposed robust detection technique was effective when applied to various hemostasis procedures, thereby indicating the versatility of the method.

Analysis of hemostatic procedures
The urologist confirmed that for all ten procedures shown in Fig. 3, the hemostasis conducted using an energy device from the state when bleeding occurred was complete. Moreover, as shown in Fig. 5, we were able to visualize the transition between the bleeding and hemostasis regions using the proposed detection method. Many hemostasis procedures initially detect a large bleeding region. The size of the bleeding region detected in Procedure 6 was smaller than that in the other procedures because Procedure 6 was carried out at a stage where the amount of bleeding was very small. Furthermore, as the procedure progressed, there was a tendency for a larger hemostasis region to be detected. The hemostasis procedure in the WaFLES technique involves coagulation using the energy device and blood washing. In addition, this procedure can suppress bleeding rapidly after a certain period of time owing to the applied water pressure. (5,6) This effect is an advantage of the WaFLES technique.
In Fig. 5, it can be observed that the hemostasis region was larger than the bleeding region. For many procedures, it has been complete when the hemostasis region was larger than the bleeding region. In Procedure 9, even when the hemostasis region was larger than the bleeding region, hemostasis did not immediately come to an end, because in this procedure, the bleeding point could not be efficiently coagulated by the energy device. If coagulation cannot be performed efficiently, the size of the hemostasis region will increase but the bleeding will not stop; thus, it is considered important to coagulate the bleeding point properly. The results also suggest that our system can perform the hemostasis procedures efficiently and that it can be used during surgery.

Termination conditions of hemostasis procedures
The proposed method using Eq. (7) adopted an algorithm with a low computational cost, because the final goal of our study is to assist and automate the hemostasis procedures based on a real-time analysis. The results showed that monitoring the hemostatic procedures using the proposed method based on Eq. (7) was effective in assisting and enabling the automation of the hemostasis procedures. The results in Fig. 6 showed that the ratio (S c ) of the sizes of the bleeding and hemostasis regions tended to converge to 1 or less, because as the hemostasis procedures progress, the size of the bleeding region decreases and the size of the hemostasis region increases. Additionally, from Fig. 7, it is considered that by using our method, sufficient hemostasis can be achieved when the ratio (S c ) of the sizes of the bleeding and hemostasis regions is ≤ 1. In Fig. 7, the S c for the endoscopic images confirmed by the urologist to show a complete hemostasis and the S c for the endoscopic images of the immediately preceding state were compared. The paired t-test results confirmed a significant difference with a high value of the dominant level of 5% or less. These results confirmed that calculating the ratio of the bleeding and hemostasis region sizes and using it as a threshold for the termination conditions can be effective in automating the hemostasis procedures. In this study, hemostasis was considered complete when S c was ≤ 1; however, as the absolute value may vary, it is necessary to carry out further verification experiments. The accuracy of a threshold-based method using the termination conditions was 90% when S c ≤ 1. The case that failed was Procedure 9. The hemostasis procedures were not completed until a while after S c became 1 or less in Procedure 9. As described in Sect. 4.2, in Procedure 9, effective hemostasis could not be performed; therefore, it is considered necessary to add another condition if the decrease in the size of the bleeding region is small regardless of the increase in the size of the hemostasis region.
Moreover, although it was determined that hemostasis was complete when S c was ≤ 1 in Procedures 2 and 3, the detected bleeding region was larger than those observed in the other procedures. This may be caused by the false detection of the bleeding region and small regions of remaining bleeding. The erroneous detection should be corrected by expanding the dataset. The surgeons finished the hemostasis procedures because the remaining small amount of bleeding was expected to stop naturally over time. The continuation of the hemostasis procedures increased coagulation, which caused inflammation and damaged normal tissues; (13,28) this was undesirable from an invasive point of view. In other words, even if S c is sufficiently small, caution is still required when the bleeding region is still large; the size of the bleeding region is considered to be part of the termination conditions of the hemostasis procedures. In this regard, we consider that further verification experiments are necessary.

Limitations of the current study
In this study, we determined the termination conditions of hemostasis procedures based on the detected and measured bleeding and hemostasis regions with the aim of providing assistance for and enabling the automation of such procedures. During detection, a region of interest is defined, and an analysis is performed for a procedure in which there is one bleeding region. Therefore, we consider that there are limitations when multiple bleeding regions occur. However, when multiple bleeding regions occur, it is considered that the amount of bleeding will be very large, and in this case, it is highly likely that a direct manual hemostasis procedure by a surgeon will be required. It is considered necessary to select a target bleeding region even when the proposed system is used to assist and automate the hemostasis procedures.
Furthermore, because it is necessary to coagulate the bleeding point efficiently, a surgical robot that tracks moving organs by pulsation can be integrated with the proposed system. Methods using a three-dimensional position measurement device as well as methods using the optical flow (29) calculated from feature points on endoscopic images have been studied as tracking methods. Therefore, our future research will focus on the development of an organtracking hemostasis support robot system incorporating the proposed detection method and termination conditions of the procedures.

Conclusions
We conducted a time-series analysis of endoscopic images obtained during hemostasis procedures and examined some of the procedure termination conditions required to develop a robot system that enables automation and provides assistance. The organ, bleeding, and hemostasis regions were detected from the endoscopic images using a linear SVM based on an extended dataset. Our SVM classifier was more robust than the conventional method as shown by a comparison of conventional feature descriptors and the proposed feature descriptor. Moreover, an analysis of ten hemostasis procedures suggested that the size ratio of the bleeding and hemostasis regions can be used as part of the termination conditions of hemostasis procedures. In addition, an analysis of a hemostasis procedure is expected to contribute to its automation and an increase in its efficiency. Further research will be conducted to integrate the proposed detection method with surgical robots.