Method of Finger Motion Recognition Based on Polyvinylidene Fluoride Sensor Array

1National Engineering Lab of Special Display Technology, Hefei University of Technology, Hefei 230009, China 2Academy of Photoelectric Technology, Hefei University of Technology, Hefei 230009, China 3School of Instrument Science and Opto-electronics Engineering, Hefei University of Technology, Hefei 230009, China 4Faculty of International Studies, Prince of Songkla University (Phuket Campus), Phuket 83120, Thailand 5School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230009, China 6School of Electronic Science & Applied Physics, Hefei University of Technology, Hefei 230009, China


Introduction
In recent years, research on wearable devices has rapidly developed, (1)(2)(3)(4) and products such as Google Glass and Apple Watch have been designed. Wearable devices create a new experience, while putting a higher demand on human-computer interaction technology. Finger motion recognition is a widely used technique for human-computer interaction.
Today, there are two main types of method of finger motion recognition, namely, finger motion recognition based on computer vision and finger motion recognition based on sensors. Concerning the first type of method of finger motion recognition based on computer vision, (5)(6)(7) Zhang et al. proposed a multiple extraction and multiple prediction (MEMP) network for gesture recognition, which achieved a high recognition rate of 97.01%. (7) However, the methods for finger motion recognition based on computer vision are highly dependent on the external environment such as the ambient light and background. Additionally, the user is limited to the extent that there is no obstruction between the user and the imaging device. The second type includes methods of finger motion recognition based on sensors. Among such methods, devices based on the sensors are often designed as a data glove, such as a data glove with piezoresistive sensors or inertial sensors. (8)(9)(10)(11) Hartmann and Link proposed an approach for online gesture recognition with inertial sensors. (10) The approach utilizes a distance measure based on a dynamic time warping (DTW) online algorithm, which can deal with gestures varying in length and amplitude. Costa et al. showed that a hand glove with piezoresistive sensors can measure the movement of the fingers. (11) Although the use of a data glove makes it easy to collect finger movement data, these devices are inconvenient to carry and will restrain the user's hand. The other type of sensor used at present is the surface electromyography (SEMG) sensor. (12)(13)(14)(15) Mehran and Mahdi proposed using two different sets of EMG features to achieve a high accuracy in recognizing six distinct hand movements. (14) Luo et al. showed that muscle synergies can be well applied to gesture recognition. (15) However, the finger motion recognition based on SEMG sensors has many requirements for wearing, and the experience while using it is poor. In summary, although the existing methods for finger motion recognition can meet the user's requirements, there are some limitations such as the many requirements for wearing, restrictions to the application occasion, and so on.
As finger movement would drive wrist muscles to move accordingly, we proposed a method to indirectly recognize finger motions by detecting motion signals of wrist muscles with pressure sensors. For the above purpose, the pressure sensors should have the characteristics of small size, high sensitivity, and good stability, and need to realize multipoint and dynamic measurement. In principle, the main pressure sensors can be divided into piezoelectric sensor, piezoresistive sensor, capacitive sensor, electromagnetic sensor, and triboelectric sensor. Among them, the electrical viscosity of piezoresistive sensors causes phenomena such as stress relaxation, creep, and hysteresis. This compromises sensor calibration and leads to inaccurate results. (16) The cavity between the two electrodes of the capacitive pressure sensor and the electrical feedthrough out of the cavity complicate the fabrication process, resulting in increased processing difficulties and cost of the devices. (17) Pressure sensing approaches based on electromagnetic transduction principle are often used for remote estimation of the pressure variation. (18,19) As a new sensing technology, triboelectric nanogenerator (TENG) based on triboelectric principle has become a research focus since its first demonstration in 2012, and the self-powered pressure sensor was the straightforward application for TENG-based active sensors. However, there are still some problems. The performance of TENG depends to a large extent on the contact area between friction materials, which will requires more complicated manufacturing process. (20) One characteristic of TENG is its high open-circuit voltage, which will easily lead to breakdown damage of the pre-circuit. (21) As a flexible piezoelectric material, the polyvinylidene fluoride (PVDF) film can fit properly with the human body. Moreover, it has the advantages of having a high piezoelectric constant, wide frequency response, low acoustic impedance, high dielectric strength, and excellent mechanical properties. (22) Moreover, the sensors based on PVDF are widely used in many fields. (23)(24)(25)(26) Hu et al. used the PVDF piezoelectric film to prepare a type of wrist sensor that has small size, good flexibility, and high sensitivity. Furthermore, the sensor was easy to wear and could detect the motion of the wrist when a finger was moving. (27) The focus of this study is to identify the feasibility of finger motion recognition based on wrist motion detection, so we choose the sensor made of PVDF piezoelectric film, which is relatively mature in current technology development, as the detection device for wrist motion detection.
On the basis of the above PVDF sensor, in this paper, we propose a method of finger motion recognition based on wrist motions. First, this method uses PVDF sensors as the motion detection units, which not only solves the problem that the user's hands are restricted by wearing the detection devices on the hand, but also solves the problem that vision-based gesture recognition technology is difficult to use in a mobile environment. At the same time, this method also realizes the noninvasive detection of wrist motion signals. Compared with other existing finger motion recognition technologies, this method is convenient, comfortable, and easy to use. Second, we propose a finger motion recognition algorithm based on short-term energy and prove that short-term energy can efficiently characterize the wrist motion signals. In this algorithm, the endpoint detection of wrist motion signals and motion signal fragments extraction are realized on the basis of short-term energy, and the eigenvectors of finger motion recognition are constructed by using the short-term energy. The experimental results show that the algorithm is effective and its accuracy is up to 96.7%.

Materials and Methods
In this section, we describe a method of finger motion recognition by using a half-cylinder PVDF sensor to detect wrist motion signals. In the first part, we introduce the principle, structure and fabrication technology of the PVDF sensor. In the second part, we study the distribution and optimization of the PVDF sensor array, and build a wearable wrist device (WWD) to acquire wrist motion signals. In the third part, we describe the finger motion recognition algorithm based on wrist motions.

Operating principle
To increase the sensitivity of the PVDF sensor, we fabricated a half-cylinder backing with flexible silica gel. (27)(28)(29) We fixed the PVDF film on the backing to make it form an arch shape.
The PVDF film used in this paper is shown in Fig. 1(a), which is made of a layer of PVDF and aluminum electrodes. The aluminum electrodes are plated on two surfaces perpendicular to the Z-axis. The Z-axis is the polarization direction, so the PVDF film only generates charges in the Z direction. The piezoelectric equation is where S is the strain, s is the elastic compliance under constant electric field, T is the stress, D 3 is the electric displacement, and d 31 , d 32 , and d 33 are the piezoelectric strain constants. The 31-and 33-modes are generally the two common modes in PVDF, as shown in Fig. 1(b). In the 31-mode, a force is applied along the X-axis (1-axis), then charges accumulate on the Z-axis (3-axis) electrodes. In the 33-mode, the applied force direction is parallel to the Z-axis, then charges are generated on the Z-axis electrodes. (30,31) When the backing is flexible, the PVDF piezoelectric film works in the 31-mode. When the piezoelectric film is subjected to a vertical force, the strain will be in the X-direction, as shown in Fig. 1(c). Ignoring the effect of D 3 on the thickness direction, Eq. (1) can be simplified as 3 31 The PVDF sensor proposed in this paper is worn on the wrist. When the wearer's fingers create movements, the movements of the finger joints drive the muscles of the wrist to exert pressure (F) on the sensor. Because the backing is a flexible half-cylinder structure, the stress perpendicular to the surface of the PVDF film will be transformed into tangential (X-direction) strain, and PVDF still works in the 31-mode, (32,33) as shown in Fig. 1 (3) The free charges collected on the surface electrodes of the PVDF film (equal to the charges flowing to the external circuit) are where A 3 is the area of the surface electrode, l and w are the length and width of the PVDF film, respectively.

Sensor structure and fabrication technology
As shown in Fig. 2(a), the PVDF sensor consists of six layers. The first layer is a protective layer composed of a polyethylene terephthalate (PET) film with an adhesive on one side, which can prevent corrosion or wear. The second layer is the PVDF film (area, 20 × 6 mm 2 ; thickness, 50 µm), which is provided by Jinzhou Kexin Electronic Material Co., Ltd. (Jinzhou, China). The PVDF film has been polarized and aluminum electrodes were plated on both sides. Its parameters are shown in Table 1. The third layer is a pressure-sensitive adhesive (PSA) layer. It 21e−12 ε/ε 0 9.5 ± 1.0 D is the film thickness; E is the elastic modulus; G is the shear modulus; d 31 , d 32 , and d 33 are the piezoelectric strain constants; ε/ε 0 is the relative permittivity.
is used to anchor the PVDF film and backing to reduce the relative displacement between them. The fourth layer is the backing, which is a half-cylinder pad (diameter, 7 mm; height, 4.5 mm) made of silicone rubber. It is behind the PVDF film and allows the PVDF film to be fabricated into a desired shape. The fifth layer is a circuit board (area, 17 × 10 mm 2 ; thickness, 0.4 mm), which integrates a differential charge amplifier consisting of an amplifier and a second-order Butterworth low-pass filter. There are two rectangular through-holes (area, 6 × 0.8 mm 2 ) on the circuit board, which are used to pull the PVDF film from the front to the back of the circuit board. The PVDF film was connected to the electrodes on the circuit board through silver glue, so that the charges collected on the surface electrodes of the PVDF film can be conducted into the circuit. The sixth layer is a substrate plate made of plastic (area, 10 × 5 mm 2 ; thickness, 0.4 mm), which is used to anchor the PVDF film on the back of the circuit board. The PVDF sensor is shown in Fig. 2(b), and its size is 17 × 10 × 5.4 mm 3 .

Experimental setup
We analyzed the motions of wrist muscles and configured the PVDF sensor array. Moreover, we built a WWD based on the sensor array.

PVDF sensor array
In general, finger motions are very complicated. There are twenty muscles between the elbow and the wrist, which are roughly divided into the front and back sides. The muscles on the front side control the bending of the wrist and fingers, and the muscles on the back side dominate the stretching of the wrist and fingers. When a person moves different fingers, the active parts of the muscles are different and the intensities of the motions are different. To more comprehensively detect the wrist motion signals, we configured a sensor array based on PVDF sensors.
The sensor array consisted of 3-8 sensors, and the sensors should be placed in the active areas of the wrist muscles. We conducted the experiment to determine the position of each sensor. First, as shown in Fig. 3(a), the circle around the wrist was evenly divided into 16 areas. Then, we fixed identical sensors in these 16 areas, and detected the signals of three finger motions (thumb click, index finger click, and middle finger click). The three finger motions are shown in Fig. 3(b). Figure 4 shows the test results of subject No. 1.
The test results were the basis for determining the wrist PVDF sensor array. In this paper, by taking an array of eight sensors as an example, we determined the areas where the sensors are arranged, namely, areas 2, 4, 5, 6, 8, 10, 13, and 16, and sensors located in these areas were numbered Nos. 1-8, respectively, as shown in Fig. 5.

WWD
On the basis of the above PVDF sensor array, we made a WWD. Its structure is shown in Fig. 6(a), and a photograph of the device is shown in Fig. 6(b). The process of making the wrist wearable device is as follows:   (1) Make and select sensors with a similar performance to form an array, and then make the circuit connection of the sensor array. (2) Use the silica gel to make a wristband that holds the sensor array.
(3) Use the microcontroller unit (MCU) to acquire the wrist motion signals and perform the A/D conversion of the signals. Data from each channel was sampled at 1000 Hz. (4) Transmit wrist motion signals to the computer using an USB-USART adapter board (USB2UIS).

Signal processing algorithm
The above PVDF sensor array can effectively collect wrist motion signals. Before classifying finger motions, it is necessary to acquire the eigenvectors of finger motions using a signal processing algorithm. The process of the signal processing algorithm is shown in Fig. 7.

Signal preprocessing
Although the signals have been filtered by the hardware circuit in the circuit board layer of the sensor, there are still some noise signals in the wrist motion signals, such as power frequency noise and white noise. It is necessary to preprocess the signals so that they can describe the wrist movements more accurately. By comparing the spectra of wrist motion signals acquired from different subjects at different click speeds of fingers, we confirmed that low-frequency signals within 100 Hz are needed. Figure 8(a) shows the waveform and spectrum of the original signals (motion, thumb click; speed, 0.3 s per time; sensor, No. 2), and it can be seen that there are 50 Hz interference noise and high-frequency noise.   According to the characteristics of wrist motion signals, the preprocessing method is as follows. First, center wrist motion signals to remove any dc offset in the signals. Second, filter signals using a low-pass filter to eliminate the high-frequency noise from the signals. Finally, filter signals using a 50 Hz notch filter to remove power-frequency noise. Figure 8(b) is the preprocessed wrist motion signals and its spectrum. It can be seen that the signals waveform has fewer burrs, which can better represent the morphological characteristics of wrist motion signals.

Extraction of motion signal fragments
To recognize finger motions using the above wrist motion signals, it is necessary to segment the signal fragments of each finger motion. We used the endpoint detection algorithm to segment the signal fragments. To improve the efficiency of the algorithm, we used one sensor as a reference to segment the signal fragments of the sensor array. In this work, the difference between the amplitudes of the wrist signals (corresponding to three types of finger motions) acquired by sensor No. 2 is smallest [as shown in Fig. 4], which can ensure that the endpoint detection algorithm can segment the signal fragments accurately. For the above reason, we selected sensor No. 2 as the reference. The method includes the following four steps: Take a higher short-term energy as the threshold MH and use it to roughly segment the active portion of the signals [as shown in Fig. 9(a), the A1-A2 signal segment].
Take a lower short-term energy as the threshold ML and use it to search for signals from A1 and A2 to both sides. The active parts of the lower energy are also added to the finger motion fragment, expanding the range of the motion signal fragment [as shown in Fig. 9(a), the B1-B2 signal segment].
The position of the maximum amplitude of the motion signal fragment is detected, N 1 data are searched forward, and N 2 data are searched backwards. The data in this range are used as the finger motion signal fragment [as shown in Fig. 9(b)]. The values of N 1 and N 2 are determined by actual conditions. The positions of the start and end points of the finger motion signal fragment are taken as the standard and used to segment the signals collected by the remaining sensors in the sensor array.

Construction of eigenvectors
Before using the neural network to recognize the finger motions, we encoded these fragments into 64-digit eigenvectors to characterize the motion signal fragments. The process is as follows.
First, the short-term energy of all the motion signal fragments is calculated to obtain the 64-digit short-term energy fragments. Second, the maximum amplitude of the shortterm energy fragments is searched and used it as the benchmark to quantify the value of the short-term energy fragments into 64-order. That is, the amplitudes of the short-term energy fragments are scaled to the range of [0,63]. The values in the range [0,1) are quantified as 0, and the values in the range [1,2) are quantified as 1, and so on. Finally, the quantized values are taken as the eigenvectors of the finger motion signal fragment.

Finger motion recognition
Because the active parts and intensity of wrist muscles are different when a person moves different fingers, the signals collected by the PVDF sensor array vary accordingly. The eigenvectors of wrist motion signal fragments corresponding to different finger motions also vary accordingly. On the basis of the appearance, we built a four-layer back propagation (BP) neural network for finger motion recognition. First, we determined the dimensions of the input where a is a constant between 0 and 10. Finally, the activation function selected is The activation function is responsible for mapping the input of a neuron to the output.
In the experiment, the input and output values were applied to train the BP network. Then, the trained network was used to classify the finger motions of several sets of input data that are different from those utilized in the training stage.

Results and Discussion
In the experiment, a subject wore the WWD on the right wrist to acquire wrist motion signals. The position of the wearable device is shown in Fig. 5. A total of four subjects participated in the experiments, and each subject performed four sets of experiments at different speeds (1 s per time, 0.75 s per time, 0.5 s per time, and 0.3 s per time). Each set of experiments contains three types of finger motion (thumb, index, and middle finger clicks), and each finger motion was collected 50 times. In these 50 times, 40 were used as the training set, while the other 10 were used as the test set. By processing the wrist motion signals with the algorithm proposed in Sect. 2.3, we extract the motion signal fragments and perform an experiment of finger motion recognition.

Motion signal fragments
We processed the wrist motion signals with the preprocessing algorithm, and obtained signals with good waveform and obvious features. Then, the finger motion fragments were extracted by using the endpoint detection algorithm. Figure 10 shows the signal fragments of three types of finger motion (subject, subject No. 1; speed, 1 s per time).
In Fig. 10, it is clear that the amplitudes and trends of the wrist signal fragments are similar in the same finger motion, while they are quite different in different finger motions. The results of extracting motion signal fragments show that the endpoint detection algorithm is feasible.

Results and analysis of finger motion recognition
A total of 16 sets experiments were performed. The training set of each experiment contains 120 motion signal fragments, and the testing set contains 30 motion signal fragments. We used the training set to train the BP neural network, and used the trained BP neural network to recognize the finger motions of the testing set. Figure 11 shows the results of finger motion recognition for four subjects (at the speed of 1 s per time). Figure 12 shows the results of finger motion recognition for subject No. 1 at the four speeds.
The x-axis indicates the real category of the three finger motions for each subject, where the test set of each category contained 10 trials, and the y-axis indicates the identification results corresponding to each type of test set. As shown in Fig. 11(a), the recognition results of subject No. 1 were as follows: the 10 thumb clicks were identified as ten thumb clicks, the 10 index finger clicks were identified as nine index finger clicks and one middle finger clicks, and the 10 middle finger clicks were identified as ten middle finger clicks.
The experimental results show that the method successfully recognized three finger motions with an accuracy of up to 96.7%, which indicates that the method can effectively detect finger motion and recognize their semantics. The results show that the eigenvectors based on the short-term energy are feasible and efficient.

Conclusions
Aiming at overcoming the problem of finger motion detection and its semantic recognition, in this paper, we proposed a method of recognizing finger motions based on wrist motions. By this method, we studied the distribution and optimization of the PVDF sensor array, and fabricated a WWD. Moreover, an algorithm of finger motion recognition based on short-term energy is proposed. The algorithm realizes the endpoint detection of the wrist motion signals and the extraction of the wrist motion signal fragments, which are then encoded as eigenvectors to recognize the finger motions. The experimental results showed that the three finger motions were successfully recognized with an accuracy of up to 96.7%, which meets the requirement of recognition.
In this method, we chose PVDF sensors as the wrist motion detection units. Compared with other sensors applied to finger motion recognition (for instance, piezoresistive sensors, inertial sensors, etc.), the PVDF sensor has a simple structure and a simple manufacturing process, and it can realize the noninvasive detection of wrist motion signals. This method does not restrain the user's hand like a data glove, and it can be used in multiple environments. In the future, we will improve the current algorithm for better accuracy and enrich the types of finger motion such as double clicks. Furthermore, we need to recruit more test subjects and optimize the sensor array (number of sensors, arrangement, wearing position, and so forth) after fully considering individual differences.