Haptic Feedback to Detect Obstacles in Multiple Regions for Visually Impaired and Blind People

vibration isolation for a prototype.


Introduction
The increasing number of visually impaired and blind people (VIBs) has been highlighted by international organizations including the World Health Organization (WHO). (1,2) It was estimated that there were more than 200 million people with various levels of visual impairment in 2020, and it was projected that by 2050, the number of people with moderate to severe visual impairment will reach 550 million people. (3) Those affected by visual impairment experience a drop in the quality of life due to limitations in activities they can do and the locations they are able to go to. (4) Hence, assistive technologies have been developed by researchers and industry to alleviate the struggles of VIBs. Currently, there are numerous solutions that try to tackle this problem, from traditional white canes used like an extended arm (5) to wearable ultrasonic sensors that detect obstacle proximity (6) and implants that stimulate the visual cortex to invoke "sight" in VIBs. (7) However, most recently proposed assistive devices for the mobility of VIBs require multiple sensors, and their accuracy is greatly influenced by external conditions, the limited interface with the user, the inability to detect obstacles, especially at head level, and insufficient portability. (8)(9)(10)(11) In this paper, we used the robotic operating system (ROS) framework to develop a wearable assistive device for VIBs, which provides localized obstacle detection with a haptic-based interface to address the major gaps existing in assistive devices for the mobility of VIBs.

Materials and Methods
We implemented the obstacle detection and localization system to enable safe mobility for VIBs through the early detection of obstacles in their paths. A traditional device used by VIBs in their travels is the white cane, (12)(13)(14) which can only detect obstacles on the ground and cannot detect obstacles such as hanging branches and other overhead obstacles. Although a sensor can be attached to a white cane to detect head-level obstructions, only binary data, i.e., an obstacle existing/not existing around the sensor, can be obtained without any location data. (15) Hence, to overcome the limitations of existing devices, we implemented a region-based single sensor for obstacle detection, which provides more information to the user. A haptic feedback mechanism was developed for the sensor that provided one motor for each of the different regions of interest around the user.

Proposed system architecture
The ROS framework was used to develop the proposed system architecture. The ROS framework is a common environment in the field of robotics that is used for the development of scalable applications. It enables different programs to run simultaneously even when using different programming languages. It is also convenient for integrating these programs into a single architecture because it uses a message-passing type of communication, wherein a node publishes a language-neutral variable on a topic for use in any other node within the architecture. This message-passing type of communication allows the ROS framework to be a multilanguage architecture. (16,17) The proposed architecture using the ROS framework was run on a Raspberry Pi 4 (RPi4) embedded system. The researchers made use of Ubiquity Robot OS, which runs on Ubuntu 16.04-based images and has an ROS pre-installed. Most of the nodes were developed using Python language and made use of both the ROS and Python 2.7 dependencies. Two ROS nodes were utilized for the system to implement the system workflow, namely, the depth image acquisition node and the obstacle localization node.

Depth image acquisition
The system made use of an Intel RealSense D435i depth camera (Intel Corporation, California, U.S.A.) for its visual inputs. This camera enables the RGB and depth streams to be used in the different modules of the device. The depth information was obtained by using a dot projector and two infrared cameras at different positions, and the concept of triangulation was applied to calculate the depth. (18) The value obtained by the depth sensor is the depth value, not the range. Hence, the depth pixel value is a measurement from the parallel plane of the viewer and not the absolute range. For the proposed system, we made use of the librealsense package developed by the ROS organization for rapid integration of the depth camera with the ROS environment.

Obstacle localization
The depth image captured by the sensor is pre-processed by applying a median filter to eliminate salt-and-pepper noise in the raw depth image stream. Then it is divided into five regions, namely, the left ground, right ground, left torso, right torso, and head levels, to cover the regions in which the user may encounter obstacles. The ground levels are covered by the white cane, while the upper levels correspond to regions where there may be suspended obstacles to the side of the user. (15) The left and right regions for the torso and ground levels allow the user to react to obstacles on one side by moving in the opposite direction. The region setup within the depth frame is shown in Fig. 1. The obstacle detection algorithm detects not only the presence of obstacles in each region but also the distance between the user and obstacles. This is done by extracting the fifth percentile of the depth values for each pixel in a region.

Haptic interface
The implementation of the haptic interface is portrayed in Fig. 2, wherein each region is represented by a specific haptic motor embedded on a haptic lace. Martinez et al. found that a haptic interface for mobility information is more efficient than using an auditory system, since the former can also be used as other interfaces such as for communications, textual notifications, etc. (19) The GPIO pins of the RPi4 were used to trigger the controls for the haptic motors based on the output of the localization algorithm using pulse-width modulation (PWM) signals. A stronger vibration is felt whenever an obstacle is closer, and this is achieved by changing the operating voltage of each motor through the PWM signal.

System component evaluation
The accuracy of the Intel RealSense D435i depth readings was tested for different lighting conditions and distances to determine their reliability and whether a recalibration was required. The developed obstacle localization algorithm was evaluated on many distinct types of obstacles. Figure 3 shows a sample test input and the result of the algorithm, wherein Fig. 3(a) is a pseudo-colored plot of a depth frame in a region with a 1 m obstacle present in the frame and Fig. 3(b) shows a histogram of the input depth image, where the red line indicates the result of the algorithm. Subsequently, the results showed accurate distance measurement of the obstacle present for each region. For each scenario, 30 depth readings were accumulated per combination of distance and identified lighting condition, with the average shown in Table 1. Data on the different scenarios (distances and lighting conditions) showed results very close to the ground truth, demonstrating that the sensor is sufficiently accurate to be integrated with the proposed system without any recalibration.
A prototype of the developed system was developed, as shown in Fig. 4(a), with the location of the motor marked with a cross, and it was evaluated to verify the vibration isolation of each motor from the other parts of the prototype. The Eulerian video magnification technique developed by Wu et al. was utilized to visually detect the location and spread of vibration of each haptic motor in the prototype. The technique basically enhances the small vibrations in a    video stream, which are originally indistinguishable by the naked eye, through the application of spatial decomposition, temporal filtering, and amplification of the frames in the video stream. (20) A 20 s video of the prototype with the right ground motor turned on was recorded at 240 fps to serve as input for the Eulerian video magnification method with the magnification set to 50. A per-frame comparison of the resulting magnified and original videos was done through calculation of the per-pixel grey level value differences of each frame. Figure 4(b) shows a pseudo-colored representation of the accumulated differences of the magnified and original videos and Fig. 4(c) shows a graph of the magnitude of the accumulated differences. According to these results, the prototype was clearly able to isolate the motor vibration relatively near to the location of the motor. This will allow the effective transfer of information about the obstacle detected by the system to the VIB user.

Preliminary user testing
An initial user evaluation of the system was conducted with 10 volunteers, five males and five females, with ages of 19-28. They were all normally sighted individuals who were blindfolded to conduct the test. It is important to note that we conducted minimal training of the test participants on the basic user instructions of the prototype prior to the evaluation. The evaluation has two main parts: identification of every pattern and combination of the haptic motors, and navigation in an obstacle course with different types of obstacles as shown in Fig. 5(a). The haptic implementation patterns and combinations are detailed in Table 2 and  Right chest and left chest 4, 5 Full ground obstruction the researchers performed a 30-item test on the blindfolded volunteers on vibration pattern comprehension. For the second experiment, as seen in Fig. 5(b), the black ellipse is the starting point and the pale ellipse is the end point. Although there were several obstacles in the path, it was expected that the participant would be able to navigate around the obstacles and reach the intended destination. The participant moved forward by following the sound of the beacon at the finish point. For this experiment, the participant wore the prototype of the haptic lace and attempted to follow navigational instructions communicated through vibrations, moving away from obstacles perceived through vibration patterns by stepping to the left or right to navigate around them or through touching an obstacle in the head region to safely go under it. Table 3 shows the results of the recorded 30-item test on vibration pattern comprehension. It shows that the users were able to perceive the correct location of the obstacle with an average accuracy of 85% and a reaction time of 2.28 s. Vibration 3, which activates the vibration motor in the user's nape, showed the highest accuracy and had the quickest response time. This may be because of the sensitivity of the nape area, which can easily be distinguished from the other locations of the body where the vibration motors are placed. Vibration 9, which activates vibration motors located on the right and left chest areas, had the slowest reaction time, and this may be due to the participants being extra careful in determining whether the vibration was from just one of the chest areas or both. On average, the vibration patterns involving the activation of multiple vibration motors (Vibrations 6, 7, 8, and 9) were less accurate than those involving the activation of a single vibration motor. This is because all the participants had minimal training on using the prototype prior to the experimentation; thus, most of the participants tended to focus on the region where they felt the strongest vibration and could not always perceive other vibrating motors. However, as the participants got used to the device, it was observed that their comprehension of each vibration pattern combination improved, which is supported by the results in the subsequent experiment involving obstacle course navigation. In the obstacle course navigation, the system was able to accurately detect the obstacles in the course and effectively communicate them to the test participants through the haptic lace. Given the overall satisfactory response time of all participants in the first test, they found it easy to react to any obstacle by immediately stopping, followed by comprehension of the vibration patterns and necessary adjustments to navigate around all types of obstacles.

Conclusions
Existing assistive devices to improve the safety and mobility of VIBs still have limitations. The requirement of multiple sensors, the degree of accuracy, the limited user interface, the detection of types of obstacles, especially head-level obstacles, and portability are several aspects in which related assistive technologies are still unsatisfactory. Thus, there is still a need for improvements and innovations in these areas through the integration of state-of-theart technologies. We developed a wearable assistive device that addresses the above-mentioned limitations and presents a viable solution through the application of modern technologies. This was achieved through implementing an ROS-based system architecture on a wearable assistive device that can localize different types of obstacles through region-based obstacle detection with an intuitive haptic interface. The reliability of the proposed system was evaluated by conducting multiple system component assessments. The depth camera sensor showed high distancereading accuracy, not only for different distances but also for different lighting conditions. The localization algorithm was also proven to be robust through testing with different types of obstacles. The haptic interface of the prototype was also assessed to determine the degree of motor vibration isolation using the Eulerian video magnification method, with results showing very well isolated vibration. After obtaining these promising results for the system, initial user testing was conducted on 10 blindfolded participants. An average accuracy of 85% was obtained for the perception of different vibration patterns and combinations with a mean response time of 2.28 s, and all participants were easily able to complete an obstacle course navigation task relatively quickly. The results indicate not only the reliability of the system but also its comprehensiveness, making it easy to use. Further evaluation was planned with actual VIB users to further improve the system, which has been delayed owing to the current COVID-19 pandemic.