Development of VR Tactile Educational Tool for Visually Impaired Children: Adaptation of Optical Motion Capture as a Tracker

1Advanced Manufacturing Technology Institute, Kanazawa University, Kakuma-machi, Kanazawa City, Ishikawa Prefecture 920-1192, Japan 2Department of Art, Kinjo College, 1200 Kasama-machi, Hakusan City, Ishikawa Prefecture 924-8511, Japan 3Faculty of Social Work, Kinjo University, 1200 Kasama-machi, Hakusan City, Ishikawa Prefecture 924-8511, Japan 4Faculty of Mechanical Engineering, Kanazawa University, Kakuma-machi, Kanazawa City, Ishikawa Prefecture 920-1192, Japan


Tactile teaching materials for visually impaired children
Educational resources for visually impaired persons are a very important issue since at least 2.2 billion people around the world have a vision impairment. (1) Generally, visually impaired children recognize educational materials using tactile confirmation. Tactile educational materials are understood by touching and are important means of recognition for visually impaired children. In recent years, although tactile educational materials made using a 3D printer have become available, (2,3) they have many issues such as the selection of the most suitable materials, colors, and finishing paints.
Traditional tactile educational materials are tactile graphics and 3D educational materials. There are three types of tactile graphics.
• Braille graphics: These are created with a Braille graphics editor and output with a Braille printer. The DotView device (4) shown in Fig. 1 can also show displays in real time. • 3D copies [PIAF (5) ]: Drawing written by a black pen on a special sheet swells after processed by the device. • Raised-line drawings: (6) These are drawings made with a ballpoint pen while applying strong pressure on the elastic side of the raised-line drawing paper to swell the drawn part. Formerly, 3D educational materials, such as those shown in Fig. 2, (7) were made by teachers and manufacturers in clay and resin. However, with the spread of 3D scanners and printers, 3D educational materials are now being made with 3D printers. However, the following are some objects that are difficult to portray: • objects in motion: rolling balls, fluttering curtains, flying insects, and so forth.
• objects that grow and change: plants, animals, organisms, and so forth.

General virtual reality (VR) system
To cope with the problems mentioned in the previous section, a VR system can be considered as a powerful means of expressing objects. Generally, a VR system is composed of three elements.
(1) Visual information provision device In recent years, head-mounted displays such as "VR goggles" have been popular. They provide visual information to a user and a strong feeling of immersivity.
(2) Haptic device Various types of devices such as tubes with liquid attached to the fingertips, (8) sliders, and rings on fingers, (9) as shown in Fig. 3, have been studied. Most of them have some mechanisms that give local feelings or deviation at the fingertips to emulate the tactile sense of their fingers.  However, such devices can only provide the sensation of touch rather than specify what is being touched.
(3) Tracking device A mechanism called a tracking device that can detect the location of a hand or finger is required. Generally, it detects the relative distance and posture of a user's hand or finger from sensors on a base (a room, a desk, or the body of the user). In many cases, the device tends to require some scaling facilities as shown in Fig. 4. (10)

VR system for tactile educational materials
Although a general VR technology has the potential to be applied for the purpose of this study, there are some issues as follows: • The dependence on visual information is very high.
• The tracking device is too large for individual use by children in a general classroom.
• VR technology was not originally designed to show the dynamic changes in shape, such as motion and growth. Thus, a general commercially available VR system is not suitable for displaying the following: • the sprouting of seeds, the growth of stems, the budding and blooming of flowers, and the formation of fruit, and • the development of a tadpole into a frog. 3D-CAD data are used to express the shapes of objects. These data are easy to model, obtain, manage, and change dynamically.
The goal of our study is to develop an educational tool for visually impaired children to express dynamic changes in shape using VR technology and 3D-CAD data at a centimeter resolution, as shown in Fig. 5. In this paper, to solve the problem on the dynamic capability of the tracker device, optical capture and a haptic device are employed to develop a system as the first step of the study to confirm the principle of the system.  Figure 6 shows an overview of the system. Optical capture is used to detect the position/ posture of the hand, and a haptic device is used to give the tactile feeling to a user in virtual space.

Overview
In the system, the user's hand and the hand in the virtual space are linked. The hand in the virtual space moves according to the information of the real hand via optical capture. When the hand in the virtual space touches the surface of the virtual object, a signal is sent to the haptic device via a controller to provide resistance. Then, the user feels tactile sensation on the object. An image of the system is shown in Fig. 7.

Hardware
(1) Optical capture If a conventional tracker of a VR device, such as Vive, (9) is used, the system becomes too large for the individual use of children in a general classroom. Upon considering the size, interface, and function, optical motion capture is adopted as a tracker. Leap Motion (11) (Leap Motion Ltd.) shown in Fig. 8 is used as the system. It was originally an input device to operate a computer by hand gestures. It easily acquires the hand position and posture information without markers for detecting the hand or finger position. It uses two infrared cameras to detect the position of the user's fingertip in 3D space by image analysis. The detection range is a sector of 500 mm radius subtending an angle of 110º from the sensor and the detection accuracy is 0.2 mm. (12) (2) Haptic device A special haptic device is developed, as shown in Fig. 9, using a servo motor (SG92R, Tower Ltd.) as an actuator, with a finger pad and a wire to connect them. The device was made to be     small (200 mm length) so that it can be worn on both wrists of the user. The motor is driven by a microcomputer with a lightweight and versatile interface (Arduino UNO, Arduino Holding) (13) as shown in Fig. 7.
The principle of the device is shown in Fig. 10. To give tactile sensation, if the virtual object touches the finger in virtual space, a command to pull the wire to raise the finger pad is sent to the motor via the microcomputer and a real force is applied to the real finger. If the relative position between the object and the finger changes, a command is also given to generate tactile sensation on the real finger.

Software
To design virtual space, Unity, (14) a game development platform provided by Unity Technologies, is used as the development environment as shown in Fig. 11. The developer can model/move 3D objects, process physical operations, and detect collisions in virtual space. The position and movement of the hand detected by optical capture are sent to the virtual space to perform collision detection between the virtual object and the hand to send a tactile reaction to the haptic device. As an optional function, a sound alert can be made as required.

Conditions
To confirm the capability of the system, a field experiment was carried out as shown in Fig. 12.
• Setting of the system The haptic devices were set on the left and right index fingers. Outside of the object: haptic device gives NO FORCE/Alert sound OFF Inside the object: haptic device gives FORCE/Alert sound ON • Virtual object Object 1: vertical rectangle plate (100 × 30 mm 2 ) Object 2: horizontal rectangle plate (30 × 100 mm 2 ) Object 3: triangular plate (100 mm in vertical direction) All objects were placed on a reference table.
• Tester Two blind persons and five low-vision persons participated in the experiment. Although children should be adopted as testers in this research, as the first step of the evaluation of the system, adults were adopted as testers because of their greater experience of tactile recognition.
• Method (1) A virtual object is set on the reference table.
(3) The testers try to guess the shape of the object. (4) If their answer is correct, the result is "YES". (5) If they abandon their attempt to guess the shape, take longer than 2 min or incorrectly guess the shape, the result is "NO". (6) The testers proceed to the next object.

Results
The shapes and postures of their hands were correctly detected and displayed on a PC for all testers as shown in Fig. 12. The recognition results are shown in Table 1.
• About half of the testers could distinguish Objects 1 and 2.
• Most of the testers could not identify Object 3, which was a triangle.
• The accuracy rate and the degree of visual impairment have no correlation.
Comments from the testers were as follows: • It is difficult to guess shapes only on the basis of the feeling of the finger tip. • I felt something at the blank part. • It felt uneven for the same part.
• If there are different sounds for each part, they will be helpful. • The triangle is too difficult to identify. • It is hard to feel the force from the haptic device.

Discussion
Although the hands were successfully detected, the results and comments were different from tester to tester. More practice time will improve the recognition. The following are considered as other reasons for misrecognition.
• Generally, humans use all their fingers to touch an object to recognize it, while the system provides sensation to only two fingers. Testers seemed to have difficulty in identifying the shape because of the reduced amount of information. • The distance from the object is not yet expressed by the haptic device. If the depth of the touch is provided, users will be able to identify the shape. • The response of the system may be too slow to reflect reality. • Identifying the reason why the triangle was hardly ever recognized is a future work.

Conclusion
We developed a system that provides VR on the basis of tactile information for the education of visually impaired children, and the following results were obtained.
(1) By performing an experiment on touching an object in virtual space, the possibility of adopting optical motion capture as a tracker for a VR system was found. (2) For more precise recognition, the haptic device giving tactile information to the user must be improved. Future developments include the following: • The number of haptic devices should be increased. • The use of other types of haptic devices, e.g., a vibrational actuator should be investigated. • The proportional control of the actuator must be adopted. • Appropriate sound effects should be considered.