Room Occupancy Determination Using Multimodal Sensor Fusion

In home/office automation applications, pyroelectric infrared (PIR) sensors have been widely used for human presence detection. However, PIR sensors suffer from false-on and false-off problems. In this study, we used multimodal sensors to complement each other in order to improve the detection performance. In addition, we proposed a lowcomputational-complexity sensor fusion algorithm to infer the status of room occupancy, which is very suitable for manipulation using the sensor nodes of wireless sensor networks. By combining spatial and temporal data through a sensor fusion mechanism, the proposed method can address the missing sensing values problem of PIR sensors, thus improving the accuracy of room occupancy determination. The inference algorithm of sensor fusion was evaluated for the sensor detection accuracy and compared with multisensor fusion using dynamic Bayesian networks (DBNs). The experimental results showed that the detection accuracy of room occupancy was greater than 99%, which was better than that of the DBN-based sensor fusion method.


Introduction
Cameras and vision algorithms can be used for human occupancy detection, but these systems suffer from cost and privacy issues. Pyroelectric infrared (PIR) sensors can detect the presence of humans without the need to carry any device. (1) Thus, PIR sensors are widely used in home/office automation systems. Generally, sensor sensing values are uncertain because internal and external sources may add noise to the values or cause a malfunction of the sensor. The detection by PIR sensors is based on people movement. Moreover, PIR detection requires a direct line of sight. For room occupancy detection, PIR sensors suffer from problems of false-offs because of missing sensing values. Some false-ons of PIR sensors are caused by unexpected noise. (2,3) Multisensor fusion can provide significant advantages over single-source data. The use of multiple types of sensor may increase the accuracy with which a quantity can be observed and characterized. In addition, multiple sensors can provide diverse, complementary as well as redundant information. (4)(5)(6) This redundancy is an important contributor to the fault tolerance of sensor networks. Xiong and Svensson proposed that information from different sensors can be combined using data fusion algorithms to obtain synergistic observation effects. (7) Guo et al. (3) and Hsiao et al. (8) used acoustic sensors (microphones) to complement PIR sensors, so-called dual-technology sensors, in order to enhance accuracy. A number of methods are available for sensor fusion, such as fuzzy logic and neural networks. (9) However, these methods lack the sufficient expressive power to handle the uncertainties, dependences, and dynamics exhibited by sensory data in many applications. Bayesian networks (BNs) are primarily used for uncertainty representation, and have shown great promise in performing multisensory data fusion. dynamic Bayesian networks (DBNs) extend BNs for modeling dynamic events. It is therefore natural to consider a DBN as a basis of general spatiotemporal sensor fusion. (10,11) However, the accuracy of DBN inference depends on accurate sensing data. Moreover, it is not suitable for operation at sensor nodes owing to limited power resources and computational capacity.
Wireless sensor networks are dense with redundant and correlated sensing values for coverage and connectivity purposes. Spatial and temporal correlations between the sensing values at sensor nodes exist in a wireless sensor network. In ref. 6, the authors define the spatial relationships between spatially adjacent sensor nodes and the temporal relationships between history sensing values of the same node as contextual information of the network. This therefore enables the sensors to locally predict their current sensing values knowing both their own past sensing values and the current sensing values of their neighbors. (6,12) In this study, we propose a low-computational-complexity data fusion algorithm, which uses spatial and temporal correlations to solve the missing sensing values problem of PIR sensors and improve information accuracy.

Techniques of Multisensory Fusion
To avoid false-offs and false-ons for the devices, multimodal sensor technology was adopted. Also, two types of sensor fusion algorithm, namely, DBN-based multisensor fusion and the proposed multisensor fusion algorithm, were developed and compared.

DBN-based multisensory fusion for occupancy determination
A DBN is considered an effective approach for sensory data fusion. The BN is based on Bayes' theorem. BNs encode conditional dependences among a set of random variables in the form of a graph. An arc between two nodes denotes a conditional dependence relationship, which is parameterized by a conditional probability model. The structure of the graph encodes domain knowledge, such as the relationship between sensor outputs and hidden states, while the parameters of the conditional probability models can be learned from the data. Another advantage of BN models is that they can be easily extended to handle time series data, by means of the DBN framework. (13) A DBN-based multisensor fusion for occupancy determination is presented, as shown in Fig. 1, in which three sensors, PIR, acoustic, and reed switch, were used. From observations using the three sensors, the occupancy probability was determined, which follows eq. (1): where S P , S A , and S R are three random variables that denote the state of the PIR, acoustic, and reed switch sensors, respectively. The prior probability P(Ocp), the likelihood P(S P , S A , S R | Ocp), and the joint probability of the three variables P(S P , S A , S R ) were obtained from previous training data. Moreover, the current state depended on the previous state according to a state transition probability, and the current state probability of P(Ocp t ) follows eq. (2): The occupancy state (Ocp t ) contains four (n = 4) possible states: X 0 : Empty; occupancy state is false. X 1 : Door is opened; occupancy state is true. X 2 : Occupant stays in room and moves; occupancy state is true. X 3 : Occupant stays in room and is still; occupancy state is true.

Proposed multisensory fusion algorithm
We modified the dual technology sensor, presented in ref. 3, by adding a reed switch, which was installed at the entrance door. Also, the multisensor fusion inference mechanism was developed, as shown in Fig. 2. The system is initialized at the "Unoccupied" state; the reed switch is triggered and the "Occupied" state is entered if someone enters the room. In the "Occupied" state, the "Decision" state is then entered if someone leaves. In the "Decision" state, the next state depends on whether the room is still occupied. If some people remain, it stays in the "Occupied" state. Otherwise, it enters the "Unoccupied" state if no PIR sensor has detected a presence.
The decision as to whether the room is occupied relies on the acoustic sensors and PIR sensors. However, these two types of sensor cannot detect a signal if the occupant remains still in the room. To solve the problem of missing sensing values, we present a decision algorithm that utilizes the spatial-temporal correlations between the sensing values of PIR sensors. The decision algorithm is shown in Fig. 3, and the parameters were defined as follows: PIR n : Serial number of the PIR sensor. d n : Distance from PIR n to entrance. speed: Speed of human walking, which is assumed to be from 0.75 to 1.25 m/s. Therefore, d n /speed denotes the time needed for people to walk from the PIR n -sensorcovered area to the entrance. The response time of the PIR sensor may have to be added in order to obtain a more precise time value. Then, the precise time for people to walk from the PIR n -sensor-covered area to the entrance can be calculated individually. All of the PIR sensors are synchronized and read every second. When someone leaves the room and the door is closed, we examine the previous record of each PIR sensor node according to the corresponding time value at which the occupants walk to the entrance individually. The room is determined to be in the state of "some people remain" if any one of the PIR sensors has a detected record in its corresponding time period. As people that walk from a specific PIR-sensor-covered area to the entrance need to take a specific time, it is impossible to detect people at the entrance within the given amount of time. Therefore, some people remain in the room. Otherwise, if no PIR sensor has detected movement, the algorithm moves on to the next decision in order to ensure the accuracy of determination. In the next decision, we examine whether any PIR sensor has detected people in the coming t s. In practice, we set the t value to 3 s. It is determined to be in the state of "some people remain" if any PIR has detected a presence. Otherwise, it is determined to be in the state of "no one detected".

Experimental Results
The experiments were conducted in three different rooms of our department building, where room_1 and room_2 are laboratories and room_3 is a personal office. A reed switch was installed at the door. PIR and acoustic sensors were installed at the ceiling to ensure full sensing coverage. For the DBN-based multisensor fusion algorithm, we set up the initial conditional probability of P(Ocp t | Ocp t−1 ) and the P(S P , S A , S R | Ocp). These data were collected and updated daily. After three weeks of training, the values in the conditional probability table are steady and become more practical than the initial values. The final values in the conditional probability table depend on the room purpose, the habits of the occupants, and environmental factors.
We compared the accuracy of room occupancy determination between the DBNbased sensor fusion algorithm and our proposed fusion algorithm. The detailed experimental results, as shown in Table 1, showed that the average accuracies of the proposed multisensory fusion algorithm and the DBN-based sensor fusion algorithm were 99.66 and 97.81%, respectively. It is apparent that our proposed multisensory fusion algorithm is better than the DBN-based sensor fusion algorithm.

Conclusions
The multimodal sensor fusion method has been proven to be very effective for improving the accuracy of sensing data. In this paper, we present a more effective multimodal sensor fusion mechanism for room occupancy determination, in which the sensing data comes from the contextual information of the network. The contextual information consists of the spatial and temporal relationships between sensor nodes that are used to solve the problem of missing sensing data. The proposed multimodal sensor fusion mechanism was evaluated in three different testbeds for an extended period of time. The results showed that the accuracy of room occupancy determination exceeded 99%, which was better than that of the DBN-based sensor fusion method.