Development of Smoke Detection System with Omnidirectional Telepresence

In this study, we developed a telepresence smoke detection system, with which personnel in a control center can learn about the relative positions of firefighters and fire sources at a fire scene. Using an omnidirectional projecting system, such personnel can communicate with firefighters and assist them in shortening the time required for fire suppression. Conventional smoke detection systems rely heavily on multiple surveillance cameras and smoke detectors. However, conventional cameras in such systems are unable to take 360° images unless multiple cameras are installed to monitor the entire fire scene. In addition, detectors are easily damaged at high temperatures. In this study, we employed an omnidirectional camera to take images of a fire scene elsewhere and sent these images back to the control center through WiFi transmission. An omnidirectional projecting system was then used to present the obtained realtime fire scene images on a 360° cylindrical projection screen. Finally, the Lucas–Kanade optical flow algorithm was used to accurately mark the fire source location. The proposed telepresence smoke detection system can present a complete fire scene and determine the accurate location of a fire source, allowing the emergency response center to provide precise information for on-site firefighters to shorten the time needed for fire suppression and increase the escape time.


Introduction
In this study, we applied a telepresence smoke detection system using image sensing and processing techniques as a distant sensor to present real-time fire scenes. This system can display fire scenes on a 360° cylindrical projection screen, a head-mounted display, and even on a smartphone app. Image processing technologies for fire alarm systems equipped with devices such as stereoscopic cameras and thermal cameras have been applied in an increasing number of studies. This type of fire alarm system is not subject to distance constraints because stereoscopic or thermal cameras can take photographs from afar. This enables the direct capture of images of a local environment to determine whether a fire exists and thereby apply image processing to set off timely fire alarms. Thermal cameras can accurately determine fire sources and fire temperatures at a fire scene. However, these cameras are expensive and have numerous limitations such as their small angle of coverage. Therefore, in this study we used image processing technologies with an omnidirectional camera to reduce the cost of the proposed system and increase user convenience by employing only a single camera. Surveillance cameras can be used to examine the status of a regular fire scene, but they are incapable of accurately showing the relative positions of the flames and personnel on site. We employed an omnidirectional projecting system and a distance estimation algorithm to localize fire sources and firefighters as well as find their relative distance. The omnidirectional projector can project remote fire scenes to a 360° cylindrical projection screen in the control center. In this manner, control center personnel can determine the real-time relative position of the flames and onsite firefighters and provide relevant information to the firefighters, helping them reduce the time required to identify flame locations. In addition, this system can send users relevant information through an app and present real-time fire scenes on the smartphones of firefighters rushing to the scene. With such information, firefighters can receive instantaneous notifications to plan their response before they arrive, thereby managing the crisis immediately upon arrival. Compared with other commercially available products, the proposed system features enhanced functions that enable real-time communication. (1)(2)(3)(4) The colors and characteristics of flames or smoke have been analyzed in numerous studies. For example, Ojo and Oladosu (5) used RGB color information and the hue, saturation, and intensity (HSI) color model to analyze flame characteristics at high and low temperatures. Yuan et al. (6) marked the location of smoke in an image on the basis of the characteristics of smoke such as brightness consistency, motion accumulation, and spread features. In recent years, researchers have applied wavelet transforms to convert images to a spatial domain for analysis. For example, Deldjoo et al. (7) applied a wavelet transform to divide the edge of smoke into horizontal and vertical parts to further analyze and capture the location of smoke. To effectively differentiate whether a moving object in an image was smoke or another object, Zhu et al. (8) used a one-dimensional discrete wavelet transform (1-D DWT) to compare the energy gradients of smoke and nonsmoke data. The results revealed that the energy gradient of a nonsmoke moving object was greater than that of smoke. This method was used to identify whether a given moving object was smoke. Allison et al. (9) used a level 3 DWT to differentiate smoke from other objects using the following six characteristics: arithmetic mean, geometric mean, standard deviation, skewness, kurtosis, and entropy. Additionally, to increase the image processing speed, Yuan et al. (10) employed accumulative motion orientation to reduce the time required to estimate the direction of smoke movement. Although this method did not enable the precise identification of smoke, the image processing speed was increased. They also modified the eigenvalue detection method and used the hue, saturation, and value (HSV) color model to differentiate flames and smoke. Then, the optical flow method was adopted to rapidly and accurately identify flame and smoke flow. The aforementioned studies were conducted under sufficient and stable light sources. However, the performance of regular smoke detection systems can be affected by changes in background light sources. In particular, under insufficient light at night, the brightness of background light often generates noise during the process of image subtraction. Verstockt et al. (11) used the long-wavelength video detection method with night vision goggles to reduce the luminance noise of the background at night. They enhanced this function by employing an 868 nm invisible near-infrared light source to rapidly establish a three-dimensional environment image in the absence of ambient light. This enabled fire source locations to be identified by observing positions where near-infrared light is absorbed. Their proposed method helps mitigate various environmental limitations.
Conventional methods for training and establishing smoke detection databases include back-propagation neural networks, which were adopted by Prema et al. (12) to train a smoke characteristic database. Unconventional training approaches have been employed in other studies. Matlani and Shrivastava (13) and Han et al. (14) used wavelet transforms to convert images and calculate energy. Specifically, they captured colors and vector features of the images to perform classification using Bayesian classifiers. This approach can identify whether a moving object is smoke or flame but it is time-consuming. In addition, Ye et al. (15) captured three characteristics of smoke in open areas, namely, contrast, inverse difference moment, and difference entropy. Subsequently, the Mahalanobis distance of these three feature vectors was calculated to determine whether each moving object was smoke or another object. Recently, support vector machines (SVMs) have also been employed. (8,16,17) An SVM classifier transforms image data into high-frequency space, which is then used to calculate the optimal solution of the classifier and discriminate linear and nonlinear data. This approach can be adopted to distinguish smoke from other objects. Because of the translucent nature of smoke and its tendency to be influenced by the wind or environmental interferences, the shape of smoke can fluctuate greatly. After capturing moving objects from an image, Ho (18) applied fuzzy logic and level crossing rate (LCR) principles to scan through pixels that might be adjacent to smoke or flames, thereby marking the locations of smoke or flames in the image. In this study, we employed an SVM classifier and a clustering algorithm to train the identification capability of the system, in addition to enhancing its identification speed.

Omnidirectional image
We used a hyperbolic mirror and an infrared camera to fabricate an omnidirectional camera, which was then combined with another hyperbolic mirror and a projector to fabricate an omnidirectional projector. Determining the parameters of the hyperbolic mirrors was one of the aims of this study. The relevant parameters are shown in Figs. 1 and 2. Analyses of such parameters are presented as follows: (19)(20)(21)(22) (1) According to the general equation of a hyperbolic mirror, (2) c and k were substituted into h = c/k to determine h.
(3) Let r = 0. We use z = (h 2 − r 2 )/2h to determine the actual distance between the camera lens and hyperbolic mirror. Furthermore, this distance reveals the position in which the camera is placed.
Here, r denotes the radius of the hyperbolic mirror. When z = 0, r = h. When r = 0, the distance between any point on the hyperbolic mirror and the viewpoint is h/2 (Fig. 3). Moreover, r = h and h/2 can be used to determine the parameters of the hyperbolic mirror. Figure 4 displays the resulting omnidirectional image.

Smoke clustering algorithm
We employed a smoke clustering algorithm to classify smoke clusters. In addition, an SVM was used for identification and training to isolate the most vigorous fire sources so that firefighters may rapidly locate the key spots and impede the spread of fire. The smoke clustering algorithm sorts all data through machine learning techniques. Similar data are classified into the same group, and each datum can only belong to a single group or cluster. The  fundamental principles of the smoke clustering algorithm can be divided into two types: the first principle entails repeatedly sorting similar items into the same cluster; the other involves constantly splitting each cluster (Fig. 5). The smoke clustering algorithm involves the following steps.
(1) Define the number of smoke clusters as K and randomly distribute K points, each of which serves as the center of a cluster (existing points are often employed). (2) Determine the distance from each point to the nearest cluster center (linear distance is often employed). (3) Recalculate the mean value of each cluster center (mean value is often employed).
Repeat steps (2) and (3) until all clusters and cluster centers remain fixed. This yields the Voronoi diagram of the clusters. Erosion and dilation were adopted in the analysis of the image data obtained using the HSV color model and thresholding. By adjusting the thresholding value, we can then eliminate the unnecessary features (Fig. 6).
Here, M is a thresholding value, f is an input image, n is the number of all pixel items, and f(x, y) is gray-scale value of pixel coordinates.
Smoke clustering algorithm: K-means clustering (Fig. 7) was applied to the target object extracted using depth images. The sample number was assumed to be {x 1 ∈ . K cluster centers were then randomly selected as {μ 1 , μ 2 , μ 3 , ..., μ k }, n j R µ ∈ . Each cluster center was calculated using each sample i, C i = 1 − K.
Here, C i is classification of sample i closest to cluster K, K is the number of clusters, μ j is estimated cluster center, and j is centroid of each cluster center.
In this study, we distinguished flames from smoke to assist firefighters in rapidly putting out the fire. Additionally, the classification of smoke clusters enabled the locations to be identified, which helps improve the safety of firefighters during rescue missions. This technique can be adopted to inform trapped personnel regarding which directions to avoid and thereby effectively reducing casualties at a fire scene.

Lucas-Kanade optical flow
Lucas-Kanade optical flow algorithm: To determine the direction of flow of smoke and flames, we employed the image differencing method and applied Taylor's formula to the image constraint equation. The following equation is obtained.     I  I  I  I  I x x y y z z t t I x y z t H.O.T. refers to higher-order terms, which can be overlooked when the movement is sufficiently small. Accordingly, the following equation is obtained: Rearranging the equation yields the following: V x , V y , and V z are respectively the components of x, y, and z in the optical flow vector of I(x, y, z, t).
, , , are the differences in the corresponding direction at (x, y, z, t). Therefore, can be rewritten as Equation (11) comprises three unknowns that cannot be solved and is associated with the so-called aperture problem in optical flow. Another set of solutions is required to determine the optical flow vector. The Lucas-Kanade optical flow algorithm is a nonrepetitive operation algorithm.
Assuming the optical flow vector (V x , V y , V z ) is a constant in a window size of m × m × m (m > 1), the pixels 1, ..., n, n = m³ can be used to generate the following system of equations.
This system of equations has three unknowns but more than three equations; hence, it is an overdetermined system of equations, meaning that it contains redundant equations. This system of equations can be expressed as follows:  I  I  I  I  V  I  I  I  I which is rewritten as To solve this overdetermined problem, we adopted the generalized least-squares technique, and obtained  I  I I  I I  V  I I  V  I I  I  I I  I I   V  I I  I I  I I We adopted the Lucas-Kanade optical flow algorithm and the aforementioned techniques to quickly identify the locations of flames and smoke. This technique can also be used to rapidly determine the direction of flame spread as well as the direction, distance, and rate of smoke spread. In the meantime, real-time information can be sent to the emergency response center for decision support. Accordingly, instructions can be immediately sent to on-site firefighters and help them reduce the flame spread and break up the smoke in advance to achieve efficient rescue and firefighting.
Images from the two-dimensional image-warping omnidirectional projector are projected onto a screen using a geometrical projection method. Figure 8 illustrates this process.
A warped image is converted into an omnidirectional image and transformed into a cylindrical shape. This allows the image to be projected onto a cylindrical omnidirectional projection screen. In Fig. 8, the two foci of the hyperbolic mirror are located at c = (0, c) and c′ = (0, −c). Equation (16) is used to determine these locations: Next, steps 1-15 in Table 1 are used to determine the geometric relationship between the hyperbolic mirror and the projector. These steps are described as follows: Step 1: ϕ is the angle between the incident light and the normal line; α is the angle between the reflective light and the horizontal axis, and β is the angle between the incident light and the horizontal axis. Accordingly, the angle between the incident light and the tangent is determined as γ.  Table 1 Steps used to derive the geometrical equations describing the relationship between the hyperbolic mirror and the projector.
Step Equation Step Equation   Step 2: δ = π/2 − β is determined as its relationship with the projector's optical axis.
Step 3: Determine cp and c p ′ for each point p.
Step 4: When δ is relatively small, cc′ is then determined.
Step 6: An equation is determined using Δcpc′ and the law of sines.
Step 7: An equation is determined using Δcpc′ and the law of cosines.
Steps 8-14: The relationship between α and β is determined.
Step 15: Let (u, v) represent pixels on the projected image and f the focal length. Accordingly, the relationship between β and the projected image's pixels is determined. Smoke detection: If a background is obstructed by smoke and becomes unclear, the unclear image of the background is subtracted from the clear background image to obtain a complete smoke image, as demonstrated in Figs. 9 and 10. When a background is obstructed by a solid object (that is, not smoke), observing the background with the naked eye is impossible. Nevertheless, subtracting the obstructed image of the background from a clear image yields a complete image of the solid object, as shown in Fig. 11. Summarizing these figures indicates that the size of the area of smoke ranges between those of the background and the solid object and is noncontinuous (Fig. 12).

Smoke and flame identification
We not only developed a telepresence smoke detection system using image sensing and processing techniques as a distant sensor but also applied smoke and flame detection system algorithms to omnidirectional images. Cluster classification was carried out to determine flame and smoke locations as well as the types and spread direction of smoke. In addition, the locations of core fire sources were identified. Following SVM training, and the system required a reaction time of only approximately 0.285 s (Fig. 13) to determine environmental conditions and detect smoke. The Lucas-Kanade optical flow and flame algorithms were also applied to accurately identify the location, distance, and moving direction of flames and smoke. The omnidirectional projecting system was used to mark the flame in red and the moving direction of smoke in yellow and with an arrow of the same color (Fig. 14). Figure 15 presents the results of comparing variations in 12 smoke diffusion measurements with and without classification. The results indicated that the smoke detection error of nonclustered data was higher than that of clustered data (blue line). In practice, clustered data can be immediately sent to the emergency response center, which instantly responds with strategies for firefighters to mitigate damage and casualties.
We applied the optical flow and target clustering methods (i.e., K-means clustering) to identify the distance and location of a fire source and the direction and distance of smoke spread. Such information enables the emergency response center and firefighters to make highly accurate estimates. Figure 16 presents the experimental data. Without target clustering, the distance identification error resulting from optical flow smoke detection was approximately 12%. This fluctuation in accuracy might cause judgment errors by users. Figure 17 presents the experimental data for optical flow smoke detection applied indoors. Without target clustering, the distance identification error was approximately 10% and the accuracy fluctuated, which may cause judgment errors by users.

Omnidirectional projection and remote transmission
We developed an omnidirectional projector by using an omnidirectional camera and a hyperbolic mirror with a projector. (19)(20)(21)(22) An image obtained with this device is presented in Fig. 18.    As shown in Fig. 19, the calibrated dewlap image obtained using the omnidirectional camera must be projected onto a 360° cylindrical projection screen. (18) This approach provides images with color and size similar to those of the original objects (Fig. 20). To fit such images, we employed a surface correction technique to modify the images so that they fitted the curve of the screen. Additionally, a holographic mirror and a projector were used to project a fire scene   onto the 360° cylindrical projection screen for personnel in the control center to observe the relative positions at the fire scene.
An actual test was conducted by sending data of images taken by the omnidirectional camera to the back-end server host through the Transmission Control Protocol/Internet Protocol (TCP/IP) and WiFi. Then, the images were projected onto the 360° cylindrical projection screen or head-mounted display for personnel in the emergency response center for further confirmation and determination. Additionally, firefighters can use a smartphone app to monitor the scene remotely and familiarize themselves with the on-site situation (Fig. 21). Real-  time images captured from a fire scene help the emergency response center to make judgements and allocate the most appropriate personnel, thereby optimizing the use of personnel and mitigating damage.

Conclusions
We developed a telepresence smoke detection system using image sensing and processing techniques as a distant sensor to instantaneously present a fire scene and immediately confirm the fire source and smoke flow. Fire scene images can be displayed using a 360° cylindrical projection screen, a head-mounted display, or through a smartphone app. Surveillance cameras can display the situation of a regular fire scene but are incapable of accurately showing the relative positions of flames and personnel on site. Therefore, we employed an omnidirectional projecting system and smoke detection technology to determine the relative position, distance, and spread direction of flames and smoke, as well as the location of the most vigorous fire source. Images from a fire scene can be sent through the Internet and projected on a 360° cylindrical projection screen in the control center using an omnidirectional projector. The proposed smoke and flame clustering techniques, which were verified to exhibit an accuracy of 96.1%, can be used to identify the fire source and the locations where smoke is thickest. Finally, the optical flow and flame algorithms can be applied to confirm the direction and distance of smoke and flame spread. Personnel in the emergency response center can use such information to immediately command firefighters to perform appropriate actions and therefore speed up the rescue process. They can also instantly provide response strategies to on-site firefighters and disseminate real-time information on the location of the fire source, the situation of smoke spread, and the relative positions of on-site personnel. This helps shorten the time required to locate the fire source. Accordingly, evacuation can be planned on the basis of smoke spread data to minimize the risk of smoke inhalation and improve the safety of firefighters.
We also developed an app for viewing on smart devices. The app can send notifications to users and display real-time fire scenes on their smartphones. This allows firefighters to make immediate judgments and plan crisis management before arriving at a fire scene, as opposed to conventional missions in which relevant decisions are made after arriving at the scene. This improvement may greatly enhance firefighters' response efficiency and expedite fire rescue missions.