Projectile Flight Trajectory and Position Estimation System Based on Stereo Vision

Projectile trajectory estimation systems, such as missiles, rockets, intelligent robots, and other parabolic trajectory prediction systems, are often used in our daily life. In this paper, we propose a projectile trajectory and position estimation system based on stereo vision. When a human captures a flying projectile, he quickly looks at it attentively to estimate its flight path and then catches it. To simulate the human eyes sensing dynamic images, two parallel network cameras capture the dynamic images of projectiles. The projectile target images are found from a complex background using the image process. The target’s three-dimensional positions are calculated using the sum of absolute difference (SAD) algorithm and stereo vision, and a Kalman filter (KF) or unscented Kalman filter (UKF) algorithm is used to estimate the trajectory and landing position of a projectile. Experiments were carried out using different illumination backgrounds and different weights of projectiles. Experiment results showed that the proposed projectile trajectory and position estimation system based on stereo vision can estimate a projectile’s three-dimensional trajectories, which correspond closely to actual trajectories, and UKF has higher performance than KF. Our proposed estimation system can be widely applied in daily life and industry.


Introduction
When a human captures a flying projectile, such as a baseball player, first, whose eyes look attentively at the ball, his brain analyzes the projected flight direction, speed, and height, and then judges the landing position on the ground, finally he runs to capture it. When a robot simulates a human receiving a projectile, the visual system and image processing are important technologies for it.
In this paper, we use two web cameras as human eyes, and a computer that simulates a human brain processes the stereo vision as computer vision. Computer vision is also popularly applied in, for example, gesture identification, (1) image monitoring, a fingerprint identification system, (2) a lane offset system, (3) and a stereoscopic vision system.
The images caught by cameras are dynamic continuous motion images. The optical flow is applied to the detection of moving objects from dynamic images, which utilizes the instantaneous changes in pixel intensity to obtain moving object information. (4) Stereo vision and the sum of absolute difference (SAD) are applied to calculate target point coordinates in three-dimensional space using target points in different visual angle images. (5) In Ref. 6, the authors imitate the baseball outfielder that catches the ball with a single camera setup on a mobile robot on the basis of motion control laws. A monocular robotic ball catching system uses the view of an eye-in-hand camera with a visual servoing control to catch a thrown ball. (7) The projectile trajectory is assumed to be a 5th-order time polynomial, and the extended Kalman filter (KF) algorithm is used to estimate the landing position of the projectile. (8) In this paper, we propose a projectile flight trajectory and position estimation system based on stereo vision. We use two web cameras that sense the projectile dynamic images as human eyes sensing the images, and the target is detected by image processing from complex background images. The target center point coordinates in the three-dimensional space are calculated using SAD and stereo vision. Finally, KF and an unscented Kalman filter (UKF) are used to estimate the projectile flight trajectory and landing position.

System Architecture
The flowchart of our projectile fight trajectory and landing position estimation system is shown in Fig. 1. Two web cameras capture dynamic images; the target is detected by image processing involving color recognition, binarization, morphology, and the center of gravity calculation. The target center position is a two-dimensional position with X-and Y-axis coordinates, which is determined by image processing. The Z-axis coordinate of the target is calculated using the SAD and stereo vision algorithm, then the target tree-dimensional coordinates are calculated. Finally, KF and UKF are used to estimate the projectile flight trajectory and landing position.

Target Detection
Two web cameras capture dynamic images; the projectile object is detected from a complex environment by image processing involving: color recognition, binarization, morphology, and the center of gravity calculation.
The color image model of the camera capturing images is the red, green, and blue (RGB) color model. We convert the RGB to YUV color space using Eq. (1), where Y means the luminance, and U and V components represent chrominance.
After color space conversion, the source image color space is converted to the grayscale components src(x, y), followed by binarization using Eq. (2). dst(x, y) is the binarization result of src(x, y). If dst(x, y) is equal to 255, we define it as logic 1, otherwise, as logic 0.
After binarization, the target binarization image still has some noise and voids. The erosion and dilation of the morphology are carried out to delete the noise and voids. The erosion step eliminates noise and small protruding points of the target image, and the expansion step fills the voids in the target image. To obtain a clear image, we first adopt the erosion step and then the expansion step to remove the image noise.
After morphology, the gravity center coordinates ( ) , x y of the target object are defined using Eq. (3). (x, y) are the logic 1 point coordinates of the target binarization image, and m and n are the total numbers of logic 1 points of the X-and Y-coordinates, respectively.

Stereoscopic algorithm
The stereoscopic vision uses different cameras to obtain different visual angle images of a three-dimensional object. (5) In Fig. 2, we use two cameras with different visual angles to catch a target P, as the characters of human eyes. O l and O r are the center points of the left and right camera lenses, respectively, and P(x i , y i , z i ) are the target coordinates. Moreover, P l (x l , y l , z l ) and P r (x r , y r , z r ) are the target image coordinates of the left and right cameras on the dotted line x l x r , respectively, x l is the distance between P l and O l , x r is the distance between P r and O r in the X-axis direction, f is the camera's focal length, and B is the distance of the base line between O l and O r . The two cameras center points O l and O r have the same Y-axis coordinate.
The stereoscopic geometric relationship diagram of stereo vision is shown in Fig. 2. Equation (4) can be deduced from the relationship of the two similar triangles of Δx l Px r and ΔO l PO r using the X-and Z-axis coordinates in Fig. 2, where Z is the distance between the target P and the base line O l O r . The distance between x l and x r is defined as the parallax d. Equation (4) can be rewritten as Eq. (5).

Regional alignment algorithm
As the characteristics of stereo vision shown in Fig. 2, a target P has a visual angle image P l with the visual angle of O l and a visual angle image P r with the visual angle of O r . Equation (6) shows the SAD in the region alignment, where m and n are the row and column pixel values of an image, respectively. We select a reference point (i, j) in the left visual angle image and use a 5 × 5 block to calculate the gray value W L (i, j). We find the target point (i, j) in the right visual angle image and calculate the gray value W R (i, j) using a 5 × 5 block. We move the target point to find the minimum V SAD point using Eq. (6). The distance between the target point and the minimum V SAD point is d, and d is the parallax between the left and right visual angle images.

KF algorithm
KF is a highly efficient recursive filter that estimates the state equation and measurement equation of a dynamic system from incomplete and noise-containing states. (9) The major KF applications are filtering and estimation, which have been widely applied in radar, aviation, computer vision, satellite navigation, and other aspects. The KF state equation and measurement equation are Eqs. (7) and (8), respectively.
Here, x k and x k−1 are the state matrices at times k and k − 1, respectively. A is the state transition matrix, u k−1 is the input at time k − 1, B is the input matrix, ε k is the state noise at time k, y k is the observation matrix at time k, C is the output matrix, ω k is the observation noise, ε k and ω k are Gaussian white noises with the mean value 0, and Q k and R k are covariance matrices. There are two major steps of KF estimation, namely, state estimation and measurement update.

State estimation
The KF state estimation equation and estimation covariance matrix are shown in Eqs. (9) and (10), respectively. Here, 1k − x is the state estimation vector at time k − 1, | 1k k− x is the state estimation vector at time k using the previous time k − 1 state is the estimated covariance matrix, and Q k is the process noise covariance matrix.

Measurement update
The Kalman gain K k and the system optimal estimation state vector ˆk x are shown in Eqs. (11) and (12), respectively.

( )
Here, R k is the measured noise covariance matrix and is the residual of measurement. Finally, the current time estimated covariance matrix is updated using Eq. 13. The flow of KF recursive calculation is shown in Fig. 3. The system initial state vector is 0 x , the initial covariance matrix is P 0 , the state estimation uses Eqs. (9) and (10), and the measurement update uses Eqs. (11)-(13).

UKF algorithm
The basic KF algorithm is developed on the basis of the linear system state equation and linear measurement equation. However, the state equations or observation equations of most practical systems are nonlinear in daily life. To solve the nonlinear conditions of the estimation problem, Julier et al. proposed the UKF algorithm (10) to solve the nonlinear estimation problem. The discrete system of the UKF algorithm is as follows: ( ) Here, X k is the state vector of the system, f(·) is the nonlinear system state function, and w k is the system process noise, which is assumed as zero mean Gaussian white noise. z k is the measurement vector, h(·) is a nonlinear observation function, and k υ is the system measurement noise. Similarly, UKF has KF with the state estimation and measurement update steps. The nonlinear system using the UKF process is as follows: The UKF gain matrix is K k . The flow of UKF recursive calculation is similar to that of KF recursive calculation as shown in Fig. 3.

Experimental Results
In our experiment, two web cameras catch the projectile dynamic images. The image process step detects the target from a complex background and calculates the target center point of the X-and Y-axis coordinates. The Z-axis coordinate of the target center point is calculated using the SAD and stereo vision algorithm. Finally, the KF and UKF steps estimate the projectile trajectory and landing position, respectively. The state equation vector x of the KF and UKF state equations indicates the target center point position coordinates (p x , p y , p z ) and speed (v x , v y , v z ). The gravity acceleration g is the input u. The current time k state is estimated using the previous time k − 1 state shown in Sects. 5.2 and 5.3.
In our experiments, the projectile dynamic images are captured using two web cameras, and we select 21 image frames of the projectile dynamic images to observe the actual projectile trajectory. The flight duration of each experiment trajectory is about 0.8 to 1 s. The weight of the green ball is 32.5 g and that of the blue ball is 157 g. Our experiments are carried out under different daylight illuminations. The actual trajectory of each ball is observed using the dynamic images from the two cameras. Under different daylight illuminations, the trajectories are estimated using KF and UKF. Figure 4(a) shows the three-dimensional coordinates and Fig. 4(b) shows the time function of the blue ball trajectory under 515 lux illumination. The height of the projectile trajectory is about 1 m. The "KF trajectory estimation" is the projectile trajectory estimation using KF. The "UKF trajectory estimation" is the projectile trajectory estimation using UKF. The "actual trajectory" is the actual projectile trajectory, which is observed using the dynamic images from the two cameras.
The projectile trajectories of UKF and KF, and the actual trajectory are very similar in our experiment results. The estimation errors of the blue and green balls under different illuminations are shown in Tables 1 and 2, respectively. The maximum error is less than 0.05 m. The average error is controlled at 0.03 m. The blue ball UKF errors are less than the KF errors under different illuminations. The green ball has the same result. The projectile trajectory  estimations using UKF correspond more closely to the actual trajectory than those using KF. The color of the test ball does not affect the projectile trajectory estimations. When a projectile lands on the floor, the Y-axis coordinate of the projectile trajectory is zero. We estimate the landing position using UKF and KF, and the position estimation errors of the blue and green balls under different illuminations are shown in Tables 3 and 4, respectively. The maximum error is less 0.05 m and the average error is about 0.03 m for all samples. The results of experiments on projectile landing position estimation using KF and UKF correspond closely to the actual landing position. The projectile landing position estimations using UKF correspond more closely to the actual position than those using KF.

Conclusion
A projectile three-dimensional trajectory and position estimation system based on stereo vision is implemented in our research. After cameras sense the projectile, target images are identified from complex backgrounds, and then the three-dimensional coordinates of the projectile target are calculated using the stereoscopic visual algorithm. After obtaining the target image information, we use KF or UKF to estimate the trajectory and landing position of the projectile. It is found that UKF can estimate the projectile and landing position more accurately than KF. Our system can intelligently track the projectile trajectory and landing position, and can be widely applied in daily life and industrial intelligent robots. The robot senses a target, actively tracks the target trajectory, estimates the landing position, and judges where to capture the target or beat the target. Our system can be widely applied in intelligent robot industries, such as intelligent trash cans, badminton robots, and rehabilitation robots.

About the Authors
Hsiang-Yueh Lai received her Ph.D. degree from the Department of Engineering and System Science, National Tsing Hua University, Hsinchu, Taiwan, in 2010. She is currently an associate professor in the Department of Electrical Engineering, National Chin-Yi University of Technology, Taichung, Taiwan. Her current research interests include intelligent systems, embedded systems, and solid-state electronic devices.
Hao-Yuan Ke received his M.E. degree from the Department of Electrical Engineering, National Chin-Yi University of Technology, Taichung, Taiwan, in 2017. He is currently an engineer in the Department of Automation, HIWIN Company, Taichung, Taiwan. His current research interests include factory automation, industrial robots, and vision applications.