Improvement of Fiducial Planar Marker Tracking by Integration with Gyroscope

A fiducial planar marker is generally used for positional tracking by a single camera. The technology is widely applied to entertainment systems and to medical assistance such as surgical navigation systems. However, the tracking accuracy is not sufficient for precise navigation; therefore, fiducial marker tracking has not replaced position sensors such as infrared light tracking systems. In particular, attitude accuracy is important for tracking because a marker is attached to a convenient position of a tracked tool far from the tool’s tip. Therefore, the attitude error generates a large positional error at the tip. To address this issue, we propose a method of improving fiducial planar marker tracking accuracy by sensor fusion. The proposed tracking system consists of a fiducial marker and a gyroscope, which provides angular velocity. To integrate the sensors, a sensor fusion filter based on a Kalman filter was designed. The feasibility and performance of the filter were validated experimentally by using a three-axis motorized rotational stage with potentiometers. The results showed that the root-mean-square error of attitude measurement was reduced by the proposed integration method. We confirmed that sensor fusion with a gyroscope is feasible for 3D tracking of a fiducial planar marker.


Introduction
Positional tracking is needed for augmented or mixed reality applications such as annotation of information, directional guidance, and other entertainment purposes. One of its practical applications is navigation, and we have focused on its use in surgical navigation.
Among the navigation systems for medical purposes, image guidance, 3D virtual reality, and augmented reality (AR) systems have been proposed. (1,2) In image guidance, which is the most basic and general approach, surgical plans are shown on intraoperative images such as X-ray and echography images. Surgeons can monitor an actual tool position shown in the image with the drawn plan. Typical applications are needle insertion for catheter insertion, tumor ablation, and bone fracture reduction. In 3D virtual reality, surgical plans, tools, and a 3D model of target organs obtained by preoperative CT/MRI are visualized as 3D graphics. This technology is used for orthopedic surgery, brain surgery, abdominal surgery, etc. AR overlays a plan, a tool, and a target tissue/organ on a video frame. The advantage of AR is that the information is directly drawn on the surgical view. (3) AR is applied to surgeries such as abdominal surgery and brain surgery. These clinical navigation systems require a position sensor to track both the patient and surgical tool positions in order to integrate virtual and real spaces.
For positional tracking, professional tracking systems such as infrared reflective marker and magnetic field tracking systems are widely used owing to their high accuracy and reliability. However, the tracking systems require a large investment, preventing their widespread use. On the other hand, a fiducial planar marker can be tracked by low-cost camera systems such as a Web camera, but the accuracy is not sufficient for precise navigation.
One of the solutions is to improve the accuracy of fiducial planar marker tracking. The accuracy of marker pose estimation depends on the camera resolution, camera calibration errors, and image processing errors for corner detection. However, the accuracy is limited using a single-camera system. (4) Thus, another approach is needed to significantly improve the accuracy of fiducial planar marker tracking.
Inertial measurement sensors (INSs) are widely used for absolute attitude estimation by a sensor fusion technique that integrates an accelerometer, a gyroscope, and a magnetometer using a Kalman filter. (5) The fused system is called an attitude and heading reference system (AHRS). Several studies have reported improved vision tracking accuracy for AR. (6,7) The studies validated the superposition accuracy by sensor fusion; however, 3D tracking was not a target of the studies. A surgical application of the AHRS to track the attitude of a handheld device has also been reported. (8) It was shown that the AHRS can estimate attitude more accurately than a conventional electromagnetic tracking system. However, the AHRS can only measure the attitude of an attached device itself and cannot provide the relative position and attitude among multiple objects to be tracked. As another related work, sensor fusion of a highly accurate optical tracking system and INSs has been proposed for the compensation of marker occlusion and the augmentation of frequency by using an unscented Kalman filter. (9) According to the above-mentioned studies, the integration of an INS and a fiducial planar maker can potentially improve the 3D tracking accuracy of conventional marker tracking without increasing the cost. In this study, the feasibility of 3D attitude tracking by integrating a fiducial planar marker and a gyroscope was investigated. The measurement accuracy was experimentally validated.

Sensor Integration Algorithm
The strategy of the proposed method is compensation of the x-axis and y-axis directions of a fiducial planar marker by using a Kalman filter technique. Figure 1 shows the proposed fused marker and a block diagram of the sensor integration algorithm. The algorithm consists of three steps: prediction, error estimation, and correction. Then, the filter gives the maximum likelihood attitude and gyroscope bias. Details of the steps are described below.

Prediction
A three-axis gyroscope provides angular velocity ω k = (ω x , ω y , ω z ) including bias and noise, , , , where ω True is the true angular velocity, b is the gyroscope bias, and v Y is the gyroscope noise. The bias can be modeled by a random walk, 1 , .
Since the actual bias b k is not provided, let the predicted and estimated biases be ˆk − b and 1k + − b , respectively. The predicted bias is equivalent to the estimated bias in the previous step: Using the previous-step bias, the angular velocity ˆk − ω is predicted as A small rotation in the sampling period Δt can be expressed as a rotational vector: is the predicted rotation angle and k n − is the predicted rotation axis. Then, the predicted quaternion of the attitude change in the kth iteration is given as  sin Finally, the kth-step predicted attitude ˆk q − is given by applying the predicted small rotation ˆk q − ∆ to the previous estimated attitude 1k q + − :

Error estimation
Let the state vector be the 9 × 1 error vector x ε,k with components , , , where the 3 × 1 vectors q xε,k and q vε,k are the vector components of the quaternions relating the x-and y-axis angular errors between a fiducial marker and a gyroscope prediction given by Eq. (7), respectively, the 3 × 1 vector b ε,k models the error of the gyroscope offset bias, and w k is the process noise. The error is caused by the sensor noise from the previous step and depends on the previous error. The 6 × 1 measurement error vector z k is defined as where q zxε and q zvε are the vector components of the quaternions relating the measurement errors of the x-and y-axes, respectively. The measurement error vector z k is modeled as being related to the error state vector x k through the 6 × 9 measurement matrix C k and measurement noise v F,k : Then, the measured x-axis tilt error q zxε , which is derived from the true tilt error q xε , gyroscope bias noise ω b,k , gyroscope sensor noise v Y,k , and fiducial marker measurement error v Fx,k , is written as By applying a similar argument for the y-axis tilt error, the measurement matrix C k can be written as 3 3 3 Here, we employed an indirect Kalman filter, in which the previous state vector , where Q w,k is the covariance matrix of the process noise vector w k , Q v,k is the covariance matrix of the measurement noise vector v k , and are the vector components of the tilt error quaternions in the x-and y-axis directions, respectively. The process error covariance matrix Q w,k and the measurement error covariance matrix Q v,k , which are obtained from the IMU sensor fusion algorithm, (10) are defined as The estimator gives the maximum likelihood errors of the x-and y-axis directions and the gyroscope bias.

Correction of attitude and bias
The indirect Kalman filter computes an a posteriori estimate, ,k ε + x . By using the x-and y-axis tilt errors, the predicted axes are respectively corrected as follows: where i x and i v are the x-and y-axes, respectively, and ( ) * q is the conjugate of the quaternion. The gyroscope bias vector k + b is also updated by adding the estimated bias error as follows: After the estimated axes are orthogonalized, the rotation matrix R + can be obtained, and then the estimated quaternion ˆk q + is calculated from the rotation matrix R + .

Experimental Validation
To investigate the performance of the algorithm, measurement experiments were performed.

Experimental setup
The combined marker consisted of a 20 × 20 mm 2 square fiducial marker and a three-axis gyroscope (BMX055, Bosch), which were assembled together with their axes aligned. For the generation and detection of the fiducial marker, we employed the well-known AruCo library. (11,12) The combined marker was attached to a three-axis rotational stage with three potentiometers as the ground truth. The gyroscope and potentiometers had I2C interfaces with data transfer speeds of 400 and 100 kbps, respectively. They were respectively connected with a PC via I2C-USB converters (FT232H, FTDI). A USB camera with 1280 × 920 resolution was used for the fiducial marker measurement. The experimental setup is shown in Fig. 2. The intrinsic parameters of the camera were prepared by using a camera calibration algorithm provided by OpenCV. The components were connected to a Windows workstation via a USB interface.
The rotational stage was moved by pivot motion, as shown in Fig. 3, with angles of 30° roll (α) and 10° pitch (β) and periods of 3.0, 6.0, and 9.0 s. During the motion, all the sensor data with timestamps were recorded by using in-house software written in C++. Then, the attitudes of the rotational stage, the fiducial marker, and the marker fused by the proposed method were compared off-line. For the implementation of the Kalman filter, the sensor noise and marker noise in Eqs. (1), (2), and (10) were manually tuned (v Y = 0.03, w b = 1.0 × 10 −6 , v F = 6.4 × 10 −3 ).
To calibrate the coordinate systems of the rotational stage and the USB camera, which is the origin of the fiducial marker, the AX = ZB hand-eye-camera calibration algorithm was used. (13)

Results
As highlighted data, the direction vectors of measured and estimated axes in the 9.0 s pivot motion are shown in Fig. 4. The attitudes of the rotational stage (dotted line), the fiducial marker (dashed line), and the fused marker (solid line) are shown. The result shows that the measurement noise of the fiducial marker was reduced by the marker integration.    To validate the performance of the proposed method, we statistically analyzed the rotation angle accuracy. Figure 5 shows the results of linear regression analysis of the rotation angles for the measured and ground-truth data in the rotation periods of 3.0, 6.0, and 9.0 s. The regression lines and the coefficients of determination R 2 were computed. The R 2 values were 0.95 for the fiducial marker and 0.98-0.99 for the fused marker. The results show that the fused marker can measure the rotation angle with better linearity than the raw fiducial marker. Figure 6 shows the results of statistical analysis of the rotation angle measurement in all the periods. The numbers of samples n marker and n fused were 25570 and 25472, respectively. The rootmean-square errors of the raw and fused rotational angles, indicated as stars, were 3.3 and 2.7º, respectively. The errors include three errors: the alignment error between the fiducial marker and the gyroscope, the calibration error between the camera and the rotational stage, and the mechanical error of the stage. The results show that the rotation angle errors were reduced (p < 0.001).

Discussion
The experimental results show that the proposed method measured the attitude of an object with higher accuracy. The designed filter compensates the detected x-and y-axes of a fiducial planar marker by using a gyroscope. The performance of Kalman filters depends on the modeling parameters, especially the process and measurement error covariances. In the present implementation, the measurement error covariance has been defined as a constant value; however, the measurement error covariance in a fiducial marker will change with the attitude and the distance from a camera. Therefore, a dynamically tuned error covariance would provide a better estimation in the future.  6. Comparison of the rotational angle error between raw fiducial marker data and fused data. The stars indicate the root-mean-square error of each data set. Each box shows the first quartile, the third quartile, the median, and whiskers corresponding to 1.5 interquartile ranges (IQRs).

Conclusions
In this study, we proposed an algorithm for integrating fiducial planar marker tracking and a gyroscope to improve tracking accuracy without increasing the cost. We confirmed that sensor fusion with a gyroscope is feasible for the 3D attitude tracking of a fiducial planar marker.