Development of Smart Home Gesture-based Control System

In recent years, smart speakers have been widely used to control related smart home appliances. For people with speech disabilities, smart speakers cannot be used to perform related voice control tasks. To help people with speech difficulties and effectively control related smart facilities, we use the Internet of Things to develop a smart home gesture-based control system. The prototype of the system is established by means of sensors and a control panel system. When a sensor detects a control gesture command, it can send related control signals to a controlled smart device. The controlled devices used in this study include smart home appliances such as doors and windows, air conditioners, curtains, dehumidifiers, and air purifiers. The system functions include the opening/closing of doors, windows, and curtains; the activation/ deactivation of heating and cooling systems, dehumidifiers, and air purifiers; temperature and humidity adjustment; and fan operation. With the development of this system, it is possible for people with language disabilities and elderly people to control the operation of various smart devices with gestures. It can also solve the inconvenience of needing to look for remote controls for various facilities. Moreover, it can save precious time and provide a safe, convenient, and smart home living environment for people with speech disabilities.


Research background
The population in Taiwan has been aging rapidly because of subreplacement fertility and medical advances. According to statistics from National Statistics Bulletin No. 037, Chief Accounting Office, Executive Yuan, the elderly population in Taiwan reached 3.804 million by the end of January 2021, accounting for 16.2% of the total population (Fig. 1). The aging index (that is, the number of elderly people per 100 young people) also reached 128.7 (Fig. 2). (1) Taiwan has become an aged society, and how to use smart facilities to assist in the care of elderly people is a problem that needs to be faced. According to Taiwan's Ministry of the Interior, older adults with a severe disability typically require professional care or residence in a nursing home. Because the majority of older adults prefer to stay at home and look after themselves, (2) improving the safety and convenience of daily life is crucial for older adults. Therefore, smart control facilities have increasingly been introduced into the home environment to increase the safety and convenience of daily life for older adults. For example, smart speakers have become increasingly popular. Users can issue voice commands to control various home appliances. Physical functions in older adults tend to deteriorate with age, (3) such as eyesight, (4) hearing, speaking, and physical ability. (5,6) Improving the quality of life for older adults through sensors and smart control facilities has become a critical task. (7) We have developed a prototype smart home gesture-based control system. With this system, older adults can gesture to the sensor to give instructions, allowing them to conveniently operate smart appliances in the home.

Purpose of research
The smart home gesture-based control system creates a barrier-free smart control environment for elderly people in Taiwan and allows users with difficulties in speaking to enjoy the convenience afforded by smart speakers developed from modern artificial intelligence (AI) technology. The objectives of this study were as follows: (a) Investigate the current status of elderly care in Taiwan, the degree of comfort and difficulties inherent to the living environment of older adults, and problems to be solved. (b) Investigate the smart facilities developed to provide comprehensive smart home care by analyzing online data and identifying aspects that cause inconvenience and require improvement. (c) Use Internet of Things (IoT)-related components and sensors to develop a prototype smart home gesture-based control facility that can control various home appliances, including air conditioners (ACs) (including switches, wind direction, and fan speed), air purifiers, dehumidifiers, windows, and curtains, that allows the use of various gestures to control home appliances and includes an interface that displays the type of home appliance being controlled and its operational function.

Smart home care
The various products used in smart home care can be divided into three categories: those providing health care, a safe environment, and living support. These products allow family members and professionals to remotely monitor the living conditions of older adults at any time through the Internet and provide various at-home services, thereby affording older adults a sense of privacy and independence without a feeling of loneliness. In recent years, smart home systems have aroused great interest owing to their capacity to make our lives easier and more comfortable. A smart home system allows people to control the home environment in an efficient and comfortable way. The advancement of science and technology in recent years has produced a large number of intelligent and functionally complex systems. Hasan et al. have provided a complete analysis of various smart and smart home automation systems. (8) It includes a clear comparison and analysis of the advantages and disadvantages of existing systems, a study of the working principles of different types of smart home systems, and comments on their different characteristics.

Health care
For the health care category, physiological data (9) on care recipients can be collected to help recipients understand their health status. The data can also be sent to professional medical teams as reference for further consultation or care services. (10,11) When uploaded to cloud storage, the data can immediately be viewed and distributed to medical care units for long-term follow up and analysis. This mechanism facilitates practices that can improve the health of care recipients, such as the implementation of smart pill boxes to remind recipients to take their medicine, the use of gait analysis and evaluation software to perform data-based physical analyses, and the conduction of two-way video meetings or virtual reality environments to help rehabilitation. This mechanism can be used for healthy older adults, those with suboptimal health who hope to improve their health, and patients with a disability or chronic disease who require long-term follow up.

Safe environment
A diverse range of smart home products have been developed to provide a safe environment. The data acquired by sensors can be used for activity positioning, fall monitoring, out-of-bed detection, and access control. (12) Safe environments prioritize safety and provide automatic emergency notifications in the event that care recipients have accidents and require assistance. In addition to detecting humans, sensors can activate a warning system upon detecting a danger in the environment, such as flooding, and can be configured to remotely turn off devices such as stoves. Furthermore, temperature, humidity, indoor air quality, and lighting can be automatically adjusted and controlled through physical environment control technology, thereby allowing a comfortable environment to be maintained.

Living support
Living support refers to the provision of services necessary to sustain the daily activities of care recipients, including those associated with diet, transportation, and chores. These services use smart technology as a bridge between users' requirements and service units. In addition to addressing basic needs, smart technology can acquire data on user preferences over a long period of time to provide customized services. Psychological support has received considerable attention in the field of elderly care. For older adults with limited mobility, social media and virtual reality devices can facilitate interpersonal connection without the limits of time or location. Although AI robots have not been widely used, their mobility and composite patterns make them appropriate for companionship and media-based entertainment services.

IoT based for smart appliances control
The IoT has become one of the industry standards for automation. Embedded devices with edge computing capabilities are connected to the Internet and can be remotely monitored and controlled using the IoT. It is very common to apply this cutting-edge technology to household appliances to monitor and control them. Kolanur et al. used multiple strategies to control each device to empower the end user without being restricted to a remote control system that relies on a single strategy. (13)

Operation through gestures
Touchscreen technology has been successfully introduced to smartphones and tablet computers. However, for certain smart engineering devices, touchscreens are not appropriate human-machine interfaces. Gesture detection represents an alternative. Gesture is a convenient and natural way to control a smart home device. (14) Gesture-based control and touchscreen technology are generally not in conflict and can even complement each other in their applications. For example, when using a smart TV, users maintain a certain distance from the TV and cannot touch the screen with their fingers. Operation by touch is also not appropriate for augmented and virtual reality. Furthermore, the screens of smart watches are too small and are ill-suited for touch control. Future IoT devices may not incorporate display interfaces. These problems can be solved through gesture-based control.
Different situations involve requirements that can be satisfied with different smart devices. Therefore, products must incorporate a range of techniques in gesture-based control. (15) For example, the control of certain products may only require simple and specific gestures, whereas other products may require machine vision to construct 3D images and recognize gestures and body movements. Han et al. have developed a system that can recognize nine different gestures. (16) Using this platform, they designed a smart home system based on gesture recognition. It contained intelligent electronic photo frames, audio and video entertainment modules, intelligent security monitoring, ZigBee wireless communication environment detection, and light control.
Since 2010, gesture recognition and 3D visual recognition based on optical principles and visual recognition algorithms have been applied to the game console market. The humanmachine interface for gesture-based control previously used in controllers has gradually been replaced with freehand technology in a new generation of devices, such as those based on timeof-flight, (17) stereo vision, (18) and structured light. (19) In 2015, Intel Corporation commercialized the RealSense module and introduced it to notebook computers. (20) Researchers have conjectured that integrating structured light technology into Apple TVs can impact the human-machine interface of smart TVs, thereby increasing the appeal of Apple TVs and the popularity of smart TVs.
Rahman et al. have proposed a gesture interaction system to control smart home IoT health devices to support elderly people and those with special needs. (21) Their system uses smart home IoT health devices and gestures to control the operation of these devices. Figure 3 displays the framework of the smart home gesture-based control system developed in this study. This framework comprises an input gesture sensor, a central processing controller board, an editing software package, an output display, a relay controlled by the system, an IR emitter, an automatic window opener, a dehumidifier, an AC, and an automatic curtain machine.

Arduino MEGA 2560
Arduino MEGA 2560 (Fig. 4) is a type of microcontroller board based on ATmega 2560. (22) It contains the components required to sustain the microcontroller and can be started by connection to a computer through a USB or by using an AC-DC adapter or a battery.

ADPS-9960 RGB gesture sensor
APDS-9960 RGB (Fig. 5) is a small sensor with features of ambient light detection, color detection, proximity detection, and nontouch gesture detection. (23) With this gesture sensor, computers, microcontrollers, and robots can easily be controlled through gestures. Its detection range is 10 to 20 cm.

Relay
A relay (Fig. 6) (24) is a type of electronic device used for control. It comprises a control system (i.e., input circuits) and a controlled system (i.e., output circuits). A relay is typically applied to automatic control circuits. It is a type of automatic switch that uses small currents to control large currents and has functions of automatic regulation, safety protection, and circuit conversion.

IR emitter
An IR controller (Fig. 7) (25) uses an IR LED to emit invisible IR light with a wavelength of 940 nm to send signals. The controller system comprises two parts: the emitter and receiver. The emitter sends out IR control signals through the LED. The signals are then received by the IR receiving module, which decodes the control signals and outputs the signals to perform the function of remote control.

Liquid crystal display (LCD)
An LCD (Fig. 8) can display two lines of words, each of which comprises 16 bytes. A standard 16 × 2 LCD with white words on a blue background can display English letters, Greek letters, punctuation marks, and mathematical symbols and has scrolling (leftward or rightward), cursor display, and LED backlight functions.

Electric window opener
An electric window opener (Fig. 9) opens windows by using an orbital electric puller based on the structure of horizontal sliding windows. Electric window openers use special hardware to connect to window sashes and stably move window sashes by rotating screws. Electric window openers should be installed at the lower edge of a window frame and can be customized to fit the width and length of the window frame.

Prototype of the system
We used the above sensors and a controller board to connect to the prototype system (Fig. 10). The Arduino control board is installed in the middle of the prototype module. It uses cables connected to relays to control the opening and closing of windows and curtains. At the top left is an LCD to show which smart devices are currently being controlled. At the bottom of the screen  is a gesture sensor to sense the direction of the gesture. At the bottom right of the Arduino control board is an IR transmitter, which is used to emit IR to control the functioning of the air conditioner.

Study results
When users move their hands or arms up and down, the smart home gesture-based control system cycles through numerous control modules in a circular display pattern: Window, In Curtain (inner curtain), Out Curtain (outer curtain), All Curtain (all curtains), Air Cond (AC switch), AC Temp UD (AC temperature adjustment), AC SWING (AC blade direction setting), AC FAN (AC fan speed adjustment), and back to Window (Fig. 11). A new control module can be added as necessary, provided that the control board has pins available for it. In Fig. 11, all operation modes display "N/A", indicating that the module has not begun operating. Figure 12 displays the names and operation statuses of the control modules in the smart home gesture-based control system: Window, In Curtain, Out Curtain, All Curtain, Air Cond, AC Temp UD, AC SWING, and AC FAN. The operations of the various modules are controlled by gestures. A leftward hand movement instructs the device to begin operating, and a rightward hand movement instructs the device to stop operating.
When the Window module is selected and the device receives an instruction to open (i.e., a leftward hand movement), the system opens the window (Fig. 13). When a curtain module (e.g., In Curtain, Out Curtain, or All Curtain) is selected and the device receives an instruction, the  Figure 15 displays photos of the temperature and fan speed being adjusted when the AC is on and a photo taken when the AC is off. Figure 16 displays photos taken when the AC blades are motionless, when they move leftward, and when they move rightward.

Novelty of this study
Hasan et al. (8) have made a complete comparison and commented on the various novel technologies, functions, advantages, and disadvantages of a current smart home system. However, they did not report on its actual application. The prototype developed in our study is connected to smart home appliances such as curtains, windows, and an air conditioner to verify the function of the smart gesture system. The system proposed by Han and Cui (16) can only control at most nine types of facility units. In contrast, by making upward and downward gestures, we can theoretically control an unlimited number of smart appliances. Rahman et al. (21) used gestures to control smart home health monitoring devices, whereas our prototype system can connect and interact with various smart home appliances to construct a comfortable and smart home living environment. The research of Kolanur et al. (13) can only be used in motordriven facilities, such as one-side sliding curtain movement, two-side sliding curtain movement, roller blind curtain movement, vertical blind curtain movement, and theatre curtain movement, and cannot be used in other smart appliances. Our system does not have this limitation and can connect with and control all types of smart appliances, windows, curtains, and other facilities.

Discussion
The smart home gesture-based control system developed in this study increases the convenience of living at home. Various smart home appliances can be selected and their functions controlled through nontouch gestures, thereby allowing the integration of a range of facilities and functions into a single interface. The advantages of the smart home gesture-based control system are as follows: (a) Intuitiveness: the control interface requires only a gesture in one of the four directions (up, down, left, and right) to control the smart facilities. (b) Simplicity: the system uses gestures in one of the four directions to activate and operate integrated home appliances and consists of a single interface for display and control rather than the conventional multidisplay control method. (c) Convenience: the system controls the operation of various facilities, thereby eliminating the need for manual operation. (d) Safety: the operation of various facilities is controlled through gestures, thereby eliminating the need to touch objects and preventing infection. In addition, the human error caused by an incorrect operation can be avoided. (e) Effective assistance: older adults and those with a disability making them unable to use their voice to control facilities can easily use this system to create a comfortable home environment.

Conclusion
The smart home gesture-based control system developed in this study can effectively and conveniently control the operation of smart home appliances and is easily operable by those with a disability, those with difficulties hearing or speaking, and older adults. The contributions of this study are as follows: (a) A description of the application of IoT components to smart buildings and the use of IoT components, sensors, and control boards to develop a smart home gesture-based control system. The development of a system that does not require the simultaneous or repeated operation of multiple controllers. In this system, up or down hand movement to a gesture sensor changes the control modes and left or right hand movement to the gesture sensor turns appliances on or off. (b) The creation of a system as efficient as smart speakers that allows older adults and those with speaking difficulties to use gestures to control smart home appliances.
In the system developed in this study, a single control board and simple gestures to a sensor are used to control smart home appliances. In the future, smart speakers can be integrated into the system to control smart home appliances through modes and tools required for certain situations.