High-Speed Detection of Down Staircase for Visually Impaired People Using Illuminance of Point Pattern

Visually impaired people are limited in movement both indoors and outdoors. To solve this problem, existing studies propose a navigation system that leads visually impaired people on the route to their destination by detecting obstacles on the route and tracking their position using information and communication technology. Visually impaired people recognize their position and the route to their destination by combining these positioning methods. On other hand, existing studies propose detection of obstacles by image processing, but this method take a long time. Hence, we have proposed the high-speed detection of a down staircase for visually impaired people. The proposed method quickly detects the boundary line between the down staircase and floors using the illuminance of a point pattern. Moreover, the proposed method recognizes the down staircase quickly because it removes images that are clearly different from the down staircase and applies image processing to the pedestrian route only. Experimental results have shown that the proposed method can recognize a down staircase and border line with more than 90% accuracy. As a result, visually impaired people can safely go down a flight of stairs using this method.


Introduction
Visually impaired people are limited in their movement indoors and outdoors because they cannot see obstacles before them or the route from a starting point to a destination. Under current condition, visually impaired people follow a route with a guide dog and braille guidance blocks. However, a guide dog is very expensive, and braille guidance blocks cannot indicate the current position and the direction for reaching a destination to a visually impaired person.
To solve this problem, existing research (1)(2)(3)(4)(5)(6)(7)(8)(9) has proposed a navigation system that leads visually impaired people on the route to their destination by detecting obstacles on the route and tracking their position using information and communication technology.
Methods for tracking a person's position are mainly Global Positioning System (GPS), (1) WiFi positioning, (2) and Bluetooth Low Energy (BLE) positioning. (3) GPS can measure a person's position outdoors with an error of 0.5 m. WiFi positioning and BLE positioning can measure a person's position indoors with error between 1 and 10 m. Visually impaired people recognize their position and the route to their destination by combining these positioning methods. On the other hand, existing studies (4)(5)(6)(7)(8)(9) have been able to detect obstacles such as a bumpy road and a flight of stairs. Sadi et al. (4) detected an obstacle with head-mounted ultrasonic sensors. Mahmud et al. (5) detected a obstacle with the tip of a white walking stick mounted with ultrasonic sensors. Mustapha et al. (6) detected an obstacle with the tip of shoes mounted with ultrasonic sensors and infrared sensors. Valaminck et al. (7) detected a door and a flight of stairs with two cameras and stereo matching. Aladren et al. (8) and Murakami et al. (9) detected a flight of stairs with RGB-D sensors such as Kinect. To lead visually impaired people to a safe route, the navigation system must recognize the shape and size of an obstacle. Moreover, a navigation system should be composed of a sensor that is small and lightweight to minimize the burden to users.
Existing systems using ultrasonic sensors cannot determine the size and shape of obstacles. A navigation system using a camera or RGB-D sensors can recognize the shape and size of an obstacle. The camera is small and lightweight, but RGB-D sensors are heavier than a camera.
For these reasons, we have proposed the high-speed detection of a down staircase for visually impaired people. Visually impaired people cannot safely go down a flight of stairs by the stereo matching method because they have to prepare to go down the stairs immediately before placing a foot on the down staircase owing to a long detection time. In addition, it is difficult for the stereo matching method to match images of a floor and a down staircase that are painted a single color or a repeating pattern. Hence, the proposed method detects quickly the boundary line between a down staircase and floor using the illuminance of a point pattern. Moreover, the proposed method recognizes quickly the down staircase by matching part of an image after removing images that are different from the down staircase image by the Hough transform. The stereo matching method is a phase-only correlation (POC). The proposed method can recognize a floor and a down staircase that are painted a single color or have a repeating pattern because the floor and the down staircase are marked with a specific illumination pattern with a laser pointer.
We explain the proposed method in § 2. In § 3, experimental results are shown. In § 4, we discuss the experimental results. In § 5, the conclusion and suggestion for future work are given.

High-Speed Detection of Down Staircase Using Illuminance of Point Pattern
To support the movement of visually impaired people, existing studies detect obstacles by the stereo matching method. The stereo matching method can estimate the distance between a camera and target objects because it applies triangular surveying to target objects in images obtained by two cameras. However, if some objects have similar color and shape, the stereo matching method frequently mismatches target objects in images. A floor and a down staircase may be painted a single color or have the same repeating pattern. Hence, stereo matching frequently mismatches the floor and the stairs. In addition, stairs detection methods have a long detection time because they use complex image processing such as the Hough transform and stereo matching. The proposed method recognizes the down staircase images with a single color and repeating pattern by marking a point pattern on the floor with a laser pointer. Moreover, the proposed method recognizes the down staircase quickly by multistep detection. Figure 1 shows the processing flowchart of the proposed method. First, the proposed method detects the boundary line between the down staircase and floor quickly using the illuminance of a point pattern with a laser pointer. Visually impaired people are warned when the proposed method detects a boundary line. The walking of visually impaired people becomes slow after a warning.
Hence, these methods provide enough time to recognize the down staircase exactly. Second, the proposed method recognizes the down staircase quickly by removing images different from the down staircase image using the Hough transform. Third, the proposed method recognizes the down staircase exactly by stereo matching. The proposed method recognizes the down staircase quickly because it removes images that are different from the down staircase and applies image processing to pedestrian routes only.

Detection of the boundary line by the illuminance of point pattern
To enable a person to go down a staircase safely, the system should detect the boundary line between the down staircase and floor from images that are filmed at a place far from the down staircase. In this case, the camera cannot photograph the steps of stairs such as Fig. 2 because the steps are concealed by the floor. Hence, the down staircase cannot be detected by the Hough transform and stereo matching. From the inverse square law, illumination intensity is inversely proportional to square of distance. Thus, when visually impaired people radiate visible rays at the down staircase, illumination intensity of the middle floor becomes lower than the illumination intensity of the floor around the feet. For this property, the proposed method quickly detects the boundary line between the down staircase and the floor using the illuminance of the point pattern. Figure 3 shows the outline of the down staircase detection method using the illuminance of the point pattern. First, the system removes both ends of an image as an excepted area. That width is 1/5 of the width of the image, and the height is the same as the height of the image. Second, the proposed method divides the image into l x × l y blocks. Third, the point pattern is extracted using the evaluation function on the basis of the color of the visible rays, as where p g (x,y) is the green component of the picture elements, p r (x,y) is the red component of the picture elements, and w is the adjustment factor. The proposed method should extract the green point pattern, because we used a laser pointer. The term E 1 (x,y) extracts the green point pattern, and E 2 (x,y) evaluates the ratio of the green to red. Natural light such as evening sun irradiates yellow visible rays. Yellow is composed of green and red. If system extracts the green point pattern using E 1 (x,y) only, the system extracts the green point pattern and the yellow visible ray. The yellow visible ray disturbs the proper extraction of the green point pattern. The system should remove the effects of the yellow visible ray. Thus, the proposed system extracts the green point pattern using E 1 (x,y) and E 2 (x,y). When E 1 (x,y) and E 2 (x,y) exceed the threshold, the point pattern elements are extracted. The thresholds T e (k)(e = 1, 2, k = 1, 2,…, 5) for E 1 (x,y) and E 2 (x,y) are set on the basis of the number of point pattern elements as 500, 1000, 1500, 2000, and 2500. Fourth, the number of the point pattern elements in the each block is counted. From the number of point pattern elements in each block, point pattern feature is calculated as where f e (k,i x ,i y ) is the point pattern feature, I x is the number of the block on the x axis, I y is the number of the block on the y axis, and b e (k,i x ,i y ) is the number of point pattern elements in each block. Fifth, the Euclidean distance u e (k,q) between f e (k,i x ,i y ) and the point pattern feature of the template image as radiating visible rays to various flooring materials is calculated as  where f(q,i x ,i y ) is the point pattern feature of the template image radiating visible rays to flooring materials. The term f(q,i x ,i y ) is calculated by the same procedure as Eq. (3). The thresholds for E 1 (x,y), and E 2 (x,y) are set on the basis of the number of point pattern elements in the template image being 1500. The number of point pattern elements differs depending on the type of floor material. The images of the radiated visible ray on each floor material in a building are learned as template images. A robust feature to changing illuminance owing to changing floor materials is obtained by learning the point pattern feature of the template image as radiating visible rays from various flooring materials. When the threshold processing extracts exactly the point pattern elements, u e (k,q) becomes a minimum value. Hence, the threshold for extracting point pattern elements is defined to regard T e (k), when u e (k,q) becomes minimum value, as T E . The point pattern feature f b (i x ,i y ) is extracted using T E . The robust feature for changing illuminance is extracted by changing T E adaptively on the basis of the number of point pattern elements.
Finally, the boundary line between the down staircase and floor is detected on the basis of f b (i x ,i y ). If there is a down staircase in front of visually impaired people, the proposed system obtains an image with middle floor in upper half of the image. Six blocks in the upper half of the image are defined as the detection area in Fig. 3. From the inverse square law, E 1 (x,y) and E 2 (x,y) of the point pattern in the middle floor become smaller than E 1 (x,y) and E 2 (x,y) of the point pattern in floor around the feet. In this case, many E 1 (x,y) and E 2 (x,y) values in detection area become less than T E . The number of point patterns in a block is defined as f b (i x ,i y ). If there is a down staircase in front of visually impaired people, f b (i x ,i y ) in the detection area become less than the other f b (i x ,i y ). Thus, f b (i x ,i y ) in the detection area counts. The number of f b (i x ,i y ) in the detection area is defined as C B . Based on this property, if C B is more than the threshold T P , the input image is detected as the image including the boundary line between the down staircase and floor.

Selection of image using the Hough transform
The step of a down staircase has many straight lines parallel to the horizontal line in the image. On the other hand, an image different from the down staircase image does not have straight lines parallel to the horizontal line. The images that are different from the down staircase images can be distinguished by counting the straight lines parallel to the horizontal line in the image. The straight lines in the image are detected using the Hough transform. The processing time for the Hough transform is less than that of POC. The proposed method recognizes the down staircase quickly because the proposed method removes images that are different from the down staircase and applies image processing to a pedestrian route only. Figure 4 shows the detection of the straight lines parallel to the horizontal line in the image using the Hough transform. First, the straight lines in the image are detected using the Hough transform. Second, when the angle θ h between the horizontal line and the straight line is θ h > T θ or θ h < T θ , that line is not considered. Moreover, when the length of the straight line is less than 0.1 times the width of the image, that line is also not considered. Third, the straight line is connected to the nearest straight line. The distance between the two straight lines is calculated as u RL (n l , n r ) = x R (n r ) − x L (n l ) 2 + y R (n r ) − y L (n ) 2 where u RL (n l ,n r ) is the Euclidean distance between two straight lines, x R (n r ) and y R (n r ) are the rightend coordinates of the n r -th straight line, and x L (n l ) and y L (n l ) are the left-end coordinates of n l -th straight line. The term θ near is defined as the angle between line n l and line n r . The lines n l and n r are selected when u RL (n l ,n r ) becomes a minimum. When θ near is more than 0.1 radian, the two straight lines are connected. If the number of connections is less than T RL , the system identifies an image that is not a down staircase. That image does not apply stereo matching method. Figure 5 shows that the proposed method recognizes the down staircase from a selected image in § 2.2 by stereo matching. First, the stereo matching calculates the distance between cameras and target objects from images with two cameras installed in parallel as

Down staircase recognition by stereo matching
where x L is the x-coordinate of a target object in image g L with a camera on the left side, x R is the x-coordinate of a target object in image g R with a camera on the right side, L is the focal distance in a camera, B is a distance between the left and right cameras, and d is the pixel pitch in the camera. The 1/5 parts from both ends of g L are not considered. The value x L -x R is calculated using block matching algorithms. (12) The block matching algorithm divides g L and g R into N x × N y blocks and matches the block in g L and the block in g R . The term x L (n x ,n y ) is defined as the block of the n x -th in the x-axis and n y -th in the y-axis. The term x R (n x ,n y ) is defined as the block in g R with the highest similarity to x L (n x ,n y ). The degree of similarity is calculated by POC. POC calculates the degree of similarity from the phase information of the frequency, which is obtained from g L and g R by a one-dimensional Fourier transform. The block in g R with the highest similarity to x L (n x ,n y ) is identified by a coarse-to-fine search (13) . Second, the down staircase is recognized from x Lx R . The down staircase is composed of steps of constant height. From Eq. (5), x L -x R is inversely proportional to the square of the distance between the camera and each step. Hence, when x Lx R changes regularly, the image is recognized as the down staircase. The degree of regularity is calculated as c(x, y) = where u c is the Manhattan distance, and T d and T u are thresholds. When c(x,y) is greater than three, the image is recognized as the down staircase. Moreover, the proposed method can reproduce the down staircase from the straight line and the distance between the camera and the down staircase.

Experimental methods
We evaluated the borderline detection method using the image including the boundary line between the down staircase and floor (DSF) and other images (NDSF). The detection of the down staircase using the Hough transform and stereo matching was evaluated using the down staircase images (DS) and other images (NDS). The photography was done at Kanazawa University. Stairs size conforms to enforcement ordinance of construction standard law. The numbers of DSF, NDSF, DS, and NDS are 103, 105, 100, and 100, respectively. The image size was 512 × 384 pixels. The DSF and NDSF were photographed 2 m away from the borderline between the down staircase and the floor. The DS and NDS were photographed 1 m away from the down staircase. Table 1 shows the PC specifications. The DSF, NDSF, DS, and NDS were photographed with a camera (FinePix REAL 3D). The value of T θ is set as 0.7; T d as 3; T u as 9; N x as 2; and N y as 8. Table 2 shows the results for DSF and NDSF. From the table, the mean of the f-measure was 0.92. Table 3 shows recognition results for the DS and NDS. From the table, the mean of the f-measure was 0.96.  Figure 6 shows examples of images that were correctly detected using the proposed method in § 2.1. From the figure, the proposed method could correctly detect the images of a floor and down staircase that are painted a single color or in a repeating pattern. Figure 7 shows examples of images that were incorrectly detected using the proposed method. From the figure, the proposed method detected images incorrectly because the illumination intensity of the point pattern changed for each floor material or shadow of a wall. The detection time was 0.17 s. The proposed method could detect a down staircase about 2 m away. The person's walking speed was 1.4 m/s. Hence, by the proposed method, visually impaired people could stop walking before the down staircase is reached. Figure 8 shows examples of images that were incorrectly recognized using the Hough transform and stereo matching. From the figure, the proposed method recognized incorrectly the NDS images that have many straight lines parallel to the horizontal line in the image. The proposed method in § 2.3 recognized the images incorrectly because the illumination intensity of the point pattern changed for each floor material. In addition, misrecognized images include the down staircase and other objects. Figure 9 shows a reproduction of the down staircase from the straight line and the distance between the camera and the down staircase. From the figure, the proposed method can reproduce the down staircase using the Hough transform and stereo matching. We evaluated the border line detection method using the image including a boundary line between DSF and NDSF. The detection of the down staircase using the Hough transform and stereo matching was evaluated on the basis of DS and NDS. In contrast, the Hough transform can recognize an up staircase because straight lines are parallel to the horizontal line in an up staircase. Also the point pattern and stereo matching method can recognize an up staircase because illumination intensity of the middle floor is lower than the illumination intensity of the floor around the feet. The proposed method recognizes with certainty the down staircase even if the illuminance of external light changes because it normalizes illumination intensity in images using Eq. (3). However, if illumination intensity in an image changes locally, the proposed method decreases an identification accuracy. To solve this problem, we need to correct the illumination intensity in images using the measured illuminance of external light.

Conclusion and Future Work
We proposed a detection method for the boundary line using the illuminance of a point pattern. In addition, we proposed the high-speed detection of a down staircase using the Hough transform and stereo matching. Evaluation experiments show that the proposed method can recognize the down staircase and borderline with more than 0.9 identification accuracy. However, we must improve its accuracy more for safe walking. In future work, we would like to improve identification accuracy. For that purpose, we will correct the illumination intensity in images using the illuminance of external light measured using the illuminance sensor within a smartphone. In addition, we will reduce the effect of noise, such as from a handrail, using the direction and state of visually impaired people measured using an accelerometer, an angular velocity sensor, and a magneto metric sensor within a smartphone.