Laser-beam Projection Mapping with Compensation for Soft-tissue Deformation

The compensation for tissue deformation is a significant concern in computer-assisted navigation for soft organ surgeries. In brain surgeries, brain shift depends on surgical conditions, but might be almost non-negligible to realize precise navigation. Since organ deformations can cause large navigation error, most applications are still in orthopedic surgeries. Another problem in navigation is intuitive visualization of surgical plans. A number of techniques such as augmented reality and projection mapping have been proposed for it. However, most systems do not support any compensation for organ deformation. Therefore, we propose a method of PM surgical navigation that supports the compensation for brain-shift deformation. The proposed system reduced error in brain deformation tracking to around one second and showed the feasibility of PM in brain surgeries.


Introduction
Surgical navigation systems became major systems used for guiding surgical geometries in the mid-1980s. They have been applied to every part of the body such as the brain, abdomen, arms, and legs. They are used to prepare surgical plans preoperatively, register the plans to the patient's body geometrically through a registration process, track the motions of surgical tools and the patient's body, and then visualize the plans with geometrical correspondence of the patient's body. Although one of the first surgical navigators, NeuroNavigator™, was developed for neurosurgeries, most of their applications are still in orthopedics because the bone retains its shape. Tissue deformation such as brain shift sometimes leads to non-negligible geometrical errors between the surgical plan geometry and the patient's organ especially in softtissue surgeries. Nauta first measured brain shift quantitatively and reported that the shift was around 5 mm during surgery. (1) Later, a preliminary research carried out by Hill et al. in 1998 showed brain shifts of 25.6 ± 28.0 mm in five patients. They increased the patient population to 21 and noted that the mean amount of brain shift was 3 mm with a maximum of 8 mm. (2) The compensation for geometrical errors is a hurdle that should be overcome to apply surgical navigation systems to soft-tissue surgeries.
Many tracking techniques for organ deformation have been proposed. Registration seeks informational correspondence in geometry between pre-and postdeformation and provides organ motion and deformation. The iterative closest point (ICP) algorithm (3) is the most commonly used method of registration. It finds point correspondences by seeking the closest point pairs between two surfaces. It works well in many cases of bone surgeries under the assumption that objects are rigid and do not deform. However, the ICP algorithm does not support organ deformation and thus will not work well for surgical navigation for soft tissues. A nonrigid ICP algorithm (4) focuses on this target but does not work well because it impairs sliding along the organ surface although most soft organ surfaces are smooth. Some researchers addressed biomechanical modeling compensation. (5,6) They, however, obtained significant errors because of the variation of brain mechanical profiles for each patient. Additional information on intraoperative measures is most likely needed to deal with such patient variations. Mountney et al. compensated for 3D tissue deformation during laparoscopy or endoscopy. (7) They employed a numerical model of nonrigid deformation to interpolate dimensional loss by 2D imaging, but the error and stability were slightly below the acceptable level for clinical applications. Paul et al. found the correspondence of texture intensities between the two views of a binocular and reconstructed brain surface stereoscopically. (8) Their method was based on the intensity correspondence of texture. Although there was no constraint for selecting features, the points of extracted features appeared mainly on the vessels. Ding et al. registered brain images at pre-and post-tumor resections with a range finder. (9) They extracted the vessel texture of the brain surface using a Frangi filter (10) and estimated the brain shape deformation by thin-plate spline (TPS) interpolation. Their method, however, suffered from the serious constraint of deformation.
Another concern in surgical navigation is the intuitiveness of surgical information visualization. Most navigation systems visualize surgical information with computer graphics drawn on a cross-sectional image of CT image volumes or a 2D X-ray image. They require surgeons to understand the geometrical relationship between the computer visualization geometry and the real-world field. Introducing XR techniques, such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and projection mapping (PM), is one of the promising ways to intuitively visualize surgical information in surgical fields. Systems employing the AR technique have been proposed since the 1990s (11,12) and tablet-PC AR visualization was proposed in the previous decade. (13) Since AR and MR navigations require to insert a device into the signal field to display surgical information, PM might be preferred for convenience of clinical use. However, previously proposed PM navigation systems did not support accurate nonrigid registration or the compensation for tissue deformation. Therefore, we propose a PM visualization system with brain-shift compensation. Figure 1 shows the processing flow of our PM. The device consisted of a laser-beam projector (SHOWWX, Microvision Inc., US) and a high-speed CCD camera (IDP Express R2000, Photron Inc., Japan). They were fixed geometrically with a plastic frame made by an additive manufacturing printer. The offset and angle between them were 70 mm and 15°, respectively. The overall size was 100 × 80 × 95 mm 3 . The device was located around 400 mm above the surgical field.

Method
In the procedure, textured surfaces of the brain are first acquired with the projector and the camera. The projector projects sinusoidal patterns with a green laser beam and the high-speed camera captures them. Stereographic geometry develops the patterns and forms 3D shapes of brain surfaces. Then, graph-cut segmentation extracts the brain region from the entire area that the device captured. After that, the system provides textured surfaces, as shown in the figure. The image processing, which will be described in detail below, highlights vessel structures of the brain surface from the original images captured by the device. Then, registration among the textured surfaces provides the deformation of the brain. The picture at the bottom center in Fig.  1 shows the brain before and after deformation. Finally, PM using the projector is carried out with the geometrical correction of surgical plans.

Acquisition of textured surface of brain
The projector draws sinusoidal intensity patterns on the organ with time-sequential shifting of the curve phase. Let I bias and I oscillation be the bias and amplitude of laser beam intensity, respectively. The intensity of the ith projecting image at pixel (x laser , y laser ) is expressed with an arbitrary phase offset ϕ offset as laser i laser laser bias oscillation offset laser laser Here, n is the sampling number in a wave cycle. The camera then captures laser-beam light reflected at the organ surface. The phase shift method gives ϕ, which indicates phases describing the projected pattern positions, at each point on the camera image for each projection with the equation where I i (x, y) is the intensity at the point (x, y) and N is the number of projected images. The procedures of pattern projection and image capturing are carried out for each of the vertical and horizontal patterns. The point geometry (x estimated , y estimated ) on the laser projection image is given with ϕ for both horizontal and vertical directions as where L x and L y are the wavelengths of the projected pattern. Since Eq. (2) gives the point correspondence between the laser projection and the camera capturing, 3D surface points can be obtained by stereoscopic geometry calculation. Using the phase-shift point correspondence eliminates the error caused by the variation of laser reflection coefficients at each brain surface point and results in the stability of correspondence computation. After a surface shape is obtained, the camera captures the brain texture as a 2D image. The graph-cut algorithm (14) extracts the surgical area in the captured image and eliminates the areas outside of the surgical field. The corrected brain texture is then mapped onto the surface to provide a textured surface of the surgical area.

Registration among textured surfaces
Nonrigid registration is performed among the textured surfaces. First, texture images of the captured surfaces are processed. The vessel structure is highlighted on the brain texture using a Frangi 2D image filter. (9) The filter is described by the following equation: where λ 1 and λ 2 are the eigenvalues of a local Hessian matrix. R B = |λ 2 /λ 1 | is a blobness measure.
where α and β are constraint weights, and ( ) where T registration is a set of 3D registration matrices.

Laser-beam PM
The surgical plan is initially registered to the first captured brain surface with point pair registration. First, feature point pairs are given manually on both the surgical plan and the first captured brain surface. Next, the ICP algorithm is conducted to compensate for the surface difference with the constraint of point-pair geometrical correspondence maintained. Then, the surgical plan is deformed nonlinearly following T registration to correspond to the present brain surface as deformed registration where p and p deformed are the point sets of each surgical plan before and after deformity, respectively. They are vectors in a homogeneous coordinate system. After registration, the surgical plan is visualized onto the patient's brain directly using the laser beam emitted from the projector by the following transformation. A point geometry 1 in the homogeneous coordinate system of the projector is given by is a 3 × 4 projection matrix by which p deformed is projected into the projector coordinate system and s is a scale that is commonly equal to the point depth of T projector T registration p from the camera. R and t denote rotational and translational components, respectively. Then, u 1 is transformed to 2 where k 1 , k 2 , k 3 , k 4 , k 5 , and k 6 are the coefficients of radial distortion, and p 1 and p 2 are the coefficients of tangential distortion. on the projector image is given as is a homography matrix that projects points onto the projection image plane. Because a surgical plan image is expressed using a 3D point cloud {p}, it can be transformed to {u} of projector image points and then rendered.

Accuracy test of brain deformation tracking
The system performance was tested on tissue deformation tracking and PM. Five porcine brains were used for the experiments. They were compressed and stretched using a custombuilt setup with stepping motors, as shown in Fig. 2. Compression displacement of 5, 10, and 15 mm was applied horizontally. Stretching of 5, 10, 15, and 20 mm was applied by elevating a 20 × 20 mm 2 stage behind the brain from the floor. Fifteen feature points were selected at vessel bifurcations on each brain surface. The bifurcation points were localized with an optical sensor (Optotrak Certus TM , Northern Digital Inc., Canada) and a stylus before and after a given deformity. The device was located 300-500 mm above the brain.

Component analysis of the tracking error
We randomly obtained deformation patterns that combined brain deformities induced by compression and stretching, which were carried out 25 times. The vessel bifurcations were localized by the same method as described above. The errors were decomposed into parallel and perpendicular components of the brain surface.

PM visualization
We tested errors in PM with plastic phantom and porcine brains. Brain sulcus bifurcations were manually specified on the 2D captured images as feature points and then projected onto the surface of the phantom or porcine brain. In the test using the phantom brain, geometrical errors caused in 3D surface measurement and projection were examined, but the test showed no nonrigid brain deformity. In the porcine brain test, feature points were manually specified on the first captured brain surface to be treated as surgical plans. The brain deformity was identified manually. The porcine brain test evaluated total error including the error caused in tracking nonrigid deformity of the brain. The projected points were localized with the stylus three-dimensionally. The locations of real bifurcations were also localized with the stylus. The trials were performed five times for each brain and condition. By comparing the real and projected points, we obtained the RMS error in the PM. Figure 3 shows the tracking error evaluated by comparing the proposed method with the rigid ICP and nonrigid ICP registration methods. The errors in the rigid ICP registration method increased linearly from 0.9 to 1.1 mm for compression deformity and from 0.7 to 1.9 mm for stretching. The error in the nonrigid ICP registration method showed a dependence on deformity conditions and remained over 1 mm for the stretch deformation. On the other hand, our proposed method produced smaller errors than the rigid ICP and nonrigid ICP registration methods. The errors were less than 0.8 mm and the proposed method showed stable performance that was independent of deformity conditions. Figure 4 shows parallel and perpendicular error components remaining on the brain surface. The displacement refers to the stage-induced displacement of the brain leading to its deformity. It is described in detail in Sect. 3.1. The error in the proposed method increased with the displacement but was moderate compared with that in the nonrigid ICP registration method. The largest difference between the two methods was in the parallel component, which slid on the brain surface.

PM visualization
We tested the accuracy and feasibility of the PM visualization for brain surgeries. Figures  5(a) and 5(b) show pictures of brain shape acquisition and plan visualization, respectively. The measurement and tracking computation were conducted in 1.7 and 1.5 s, respectively. The error was 0.1 mm in the projection onto the rigid phantom. The RMS of total errors was 1.2 mm in the porcine brain test. The error was caused mainly in the deformation tracking.

Discussion
The proposed method showed its feasibility for compensating for brain-shift deformation in our experiments. It compensated for the error perpendicular to brain surfaces with surface information and for surficial error with texture information. It was stable against brain deformation compared with the registration methods of rigid ICP and nonrigid ICP, which are conventionally used in the present surgical navigation.
PM visualization worked well for projecting surgical plans intuitively onto the brain. The time required for tracking data acquisition and computation was not short but was acceptable for clinical applications. Handling the topological changes of the brain induced by, for example, tissue resection remains a problem to be solved.

Conclusion
We tested the performance of our proposed method in tissue-deformation tracking and PM for brain surgical navigation. Our method showed its feasibility in guiding surgical tools intuitively with acceptable accuracy of brain deformity compensation.