Realization of 3D Aqua Hologram Augmented Reality of Robot Fish

This paper is about the realization of robot fish motion along with 3D hologram fish in an aquarium, with the augmented reality of 3D aqua holograms to attract viewers. In this new proposed concept, the animated fish video with 3D hologram is projected into a robot fish aquarium. However, this process of projecting 3D hologram fish into a robot fish aquarium may cause the overlap of the robot fish with the animated fish. To avoid this problem of overlapping, sensors play an important role. Image sensors track the robot fish movement. The proposed concept can be realized by boundary detection and optical flow algorithms, which are used to detect the positions of the robot fish. The performance of this detection algorithm was evaluated and satisfactory results were obtained. This 3D hologram with augmented reality has been successfully installed at Busan National Science Museum, Busan, South Korea.


Introduction
Holograms and their applications will have tremendous impact on everyday life. Gabor studied the basic principles of holograms to improve the transmission efficiency of electron microscopes. (1) The recent development in hologram is the use of bio-inspired robots, which were developed on the basis of nature. (2,3) For instance, holograms have also been used to create 3D pictures and movies. Many researchers have been working with 3D hologram displays. (4) Humans have always been impressed by the incredible swimming ability of fish, and much research attention has focused on the high efficiency and maneuverability of their propulsion. Robot fish are tested in the aquarium for collision avoidance, maneuverability, and control performance, posture maintenance, path design, and data communication. In particular, robot fish are part of the field of fabricating robots that are inspired by biological systems from nature. (5) Many researchers developed 3D holograms and applied them in creating movies, IT, and communication and packing sectors. The 3D holograms that have been developed thus far were projected on various sources or media such as screens, but in this project, we developed a new 3D hologram technology that projects images into water to mimic the motion of robot fish as an example of augmented reality. To mimic robot fish in an aquarium, we designed animated swimming fish with 4D cinema technology. Two image sensors are placed at the top of an aquarium to recognize the color of fish using the OpenCV program. A projector is placed at the bottom of the aquarium and controlled by the position control algorithm. This projector projects the hologram into the aquarium.
The proposed aquarium for displaying 3D holograms of robot fish consists of robot fish, a camera, a radio frequency (RF) modem, a drawing table, a scanner, a beam projector, 3D hologram fish, and a personal computer (PC) as shown in Fig. 1. Sensors are important components in this study. The data recorded by various sensors such as infrared distance, pressure, and acceleration sensors are collected in the data acquisition system.

Methodology of Proposed Aquarium Robot Fish 3D Hologram
The two processes involved in the proposed aquarium for displaying 3D holograms of robot fish are the position detection and boundary detection of the robot fish. Position detection was carried out by an optical flow algorithm. The Canny boundary detection algorithm was used to detect boundaries. In the next process, we created a 3D model with 4D cinema technology. These processes are explained in detail in the subsequent sections.

Boundary detection algorithm
To detect the boundary, we follow the canny edge detection algorithm. The Canny operator is an edge detection operator that can be used to detect a wide range of edges in images. The canny edge detector should have the following essential characteristics: (1) good signal-to-noise ratio, (2) good locality, that is, the edge should be detected where it actually is, and (3) a small number of false alarms, that is, the maxima of the filter response should be mainly due to the presence of true edges in the image rather than due to noise. (6)  The Canny operator showed that a filter function has a maximum signal-to-noise ratio, represented by S in Eq. (1), if it is chosen in such a way that it maximizes the quantity Here, n 0 is the standard deviation of additive Gaussian noise. It has been shown that the output of the convolution of the signal in Eq. (2) with a filter will contain the minimum number of false responses if the function f(x) is chosen in such a way that its first and second derivatives maximize the quantity Therefore, we attempt to design an optimal edge-enhancing filter to maximize S and C. The performance measure P used in this study was the product of S and C, i.e., In this manner, the performance of the boundary condition is obtained.

Optical flow detection algorithm
An optical flow is used to analyze the motion of moving objects and estimate the direction and speed of objects from one frame to another frame in the same video or different videos. Its operation is entirely based on the video frames of moving objects. (7,8) In our proposed aquarium for displaying 3D holograms of robot fish, to make the captured video smoother, we used the optical flow algorithm shown in Fig. 2. The Horn-Schunck method of estimating the optical flow was selected in this study. By assuming that the optical flow is smooth over the entire image, this method computes an approximate velocity.
The optical flow algorithm for the proposed aquarium for displaying 3D holograms of robot fish is explained as follows: (1) Live video device: The output of this block is an image that is given as the input to the second block, that is, RGB to intensity. (9,10) (2) RGB to intensity: This block takes the color data and converts it into a high-intensity color by using color codes from 000 to 255.

3D hologram modeling and animation with 4D cinema
In this study, 3D animated fish need to be created. Because these animated fish moving in the aquarium will follow a path that avoids overlapping with the robot fish, there are five steps in creating a 3D hologram, as shown in Fig. 3 and explained below, (11,12) namely, modeling, coloring and texturing, rigging, animating, and rendering.
The velocity of the animated fish is set equal to that of the robot fish. The main idea of a hologram is to reconstruct a real object in a place where it does not exist, analogous to the image generated when we look at ourselves in the mirror. (13-15) Holograms can be used to create animations and motion graphics, to model 3D objects, for rendering, and to generate common features that can be found in 3D modeling applications.

Experimental Setup
The proposed augmented reality system consists of two main blocks, namely, 3D hologram and robot fish control blocks, as shown in Fig. 4. The 3D hologram block consists of position command, 3D hologram formation, 3D fish coordinate data, and augmented reality sub-blocks. As shown in Fig. 4(b), children draw fish on the drawing table and scan them through a scanner. The boundary of the scanned fish can be extracted and the drawn fish can be converted into a 3D fish hologram. This converted 3D fish will mimic the robot fish movements. To achieve this mimic process, the user must give a position command. The robot fish control block consists of a camera, the OpenCV library, a color segment algorithm, a color tracking algorithm, position data, and robot fish sub-blocks. Here, we used the OpenCV library along with the Python programming language. When the camera interfaces with the PC through the OpenCV library, it detects the color of the object (robot fish). After detecting the color of the robot fish, the position of the corresponding robot fish will be identified and, by using this position data, we can track the robot fish using color marks. (16)(17)(18)(19)(20)(21) The robot fish stop swimming when they encounter a color mark and continue swimming when the color mark is removed. The coordinates of 3D fish were recorded by sensors, and these coordinates were used to identify the position of fish.
The purpose of this experiment is to mimic the robot fish motion with 3D hologram fish. To achieve this initially, we identify the position of the robot fish in the aquarium. After detecting the position of the robot fish, the obtained position data are given as the input through OpenCV. As shown in Fig. 4(c), the created 3D hologram fish are projected into the aquarium. This projection of 3D hologram fish motion into the aquarium forms augmented reality by following the position data given as the input. To achieve this, the experimental setup shown in Fig. 5(a) consists of two image sensors placed on top of the aquarium. These sensors recognize the motion of the robot fish and give feedback to the computer. The position detection algorithm is used to detect the position of the robot fish, whose output is the coordinates of the robot fish.
The proposed aquarium for displaying 3D holograms of robot fish consists of the aquarium, robot fish, top-view camera, RF modem, 3D hologram fish, and PC as shown in Fig. 5(a). Figure 5(b) shows the setup for the projector reflector used to reflect the 3D hologram video into the water. The proposed aquarium for displaying 3D holograms of robot fish is tested while applying the position detection and boundary detection algorithms. The coordinates of each robot fish are detected with position detection sensors. These coordinates are used to identify the position of the robot fish. After identifying the position of the robot fish, 3D hologram fish will mimic the movements of the robot fish. Robot fish following the movements of 3D hologram fish is called augmented reality. The results will be explained in the section below with figures.

Experimental Results
After reflecting the projector video into the water, the 3D hologram fish are used to give an effect of augmented reality in the aquarium. These fish swim together with the robot fish in the aquarium. For this purpose, 3D hologram fish were designed using a 3D animation maker. Then, we displayed the 3D hologram fish by reflection into the aquarium. This type of hologram produces very high quality images. Practically, we showed the fish 3D hologram video using the mini beam projector and then reflected it into the aquarium. The result of the hologram fish mimicking the robot fish motion is shown in Fig. 6. Figure 6(a) shows the reflection of the 3D hologram fish into the aquarium, Fig. 6(b) shows the positions of the robot and 3D hologram fish in the aquarium, and Fig. 6(c) shows the result of the 3D hologram fish mimicking the robot fish, indicating that the experimental results of this algorithm have been satisfied.

Conclusion
This research shows the realization of robot fish, which leads to the development of 3D hologram fish, which is known as augmented reality, and the establishment of robot fish tracking control techniques. It also shows the realization of the proposed aquarium for displaying the 3D hologram fish that mimic the robotic fish in the aquarium. The image of the swimming robot fish is projected into the aquarium for augmented reality. In this study, position detection sensors played a crucial role in finding the position of the robot fish, and image sensors were used to track the color of the robot fish in the aquarium. The boundary and object detection algorithms were used in this study. The position was detected by using OpenCV. The performance of these detection algorithms was found to be satisfactory.
Jeong Bea Lee is a professor in the Department of Electronics & Robotics of BUFS. He received his Ph.D. degree in real-time communication from Hanyang University in 1995, and his master's and bachelor's degrees from the Department of Electronics of Kyung-Buk National University, Korea, in 1983 and 1981, respectively. He worked as a researcher in the Electronics and Telecommunications Research Institute (ETRI) from 1982 to 1991. His research is on embedded systems, IoT systems, smart fish farms, and robotics.