Social Networking Services Support Application Based on Biological Information

These days, Internet services realize computer-mediated social communication as social networking services (SNSs) such as WhatsApp, Facebook, Twitter, and so on, which expand worldwide. Through these SNSs, people express emotions by selecting phrases and characters. These emotional expressions are selected generally on the basis of the user’s subjective idea. However, when users lose self-control, they might send inappropriate expressions to others without proper consideration. To avoid such situations, some observable mechanisms that can detect and communicate unconscious emotions to users before they send the message are desired. Recently, some approaches to detecting unconscious emotions using biological sensors have been reported. These approaches provide procedures to make unconscious emotions observable and communicable to users in real time. We combine a biofeedback method with an unconscious emotion estimation method for detecting mismatch between an unconscious emotion and an outward expression before sending the message. Using this idea, we design and implement a system that detects such mismatch and gives feedback to an SNS user. We carry out an experiment using the proposed system. The results show that the system is helpful for the intended purpose but needs improvement.


Introduction
Psychological research studies show that people exchange emotional expressions in order to adapt to society and have social communication. These days, social communication evolves with Internet services. Computer-meditated social communication is widely used and services, such as WhatsApp, Facebook, Twitter, and so on, are provided. Such services are defined as social networking services (SNSs). Through these SNSs, people express the emotions by selecting phrases and characters. These emotional expressions are selected generally on the basis of the user's subjective thinking, which sometimes can be unreliable. In this case, an expression could be inappropriate or even convey a difference in subjective thinking and expression. (1) Such an expression could cause the receiver to have a wrong impression. Such a wrong impression might cause adverse effects on the relationship between the sender and the receiver. We consider some helpful information for the user to give feedback to avoid such situations.
To achieve this goal, we first consider the possible causes of the user's inappropriate expression. Muraven et al. mention that human emotions subjectively affect expressions, such as anger and sorrow. They also found that there is a tendency for human beings to have less control over themselves while they are experiencing vehement and negative emotions. (2) Lin et al. concluded that mistakes occur when human beings do not recognize their own emotions at the time of typing. (3) In these cases, such emotions would lead to inappropriate expressions. Second, humans sometimes could accurately not recognize their own emotion, and what they expressed on SNSs are not what they truly feel. (4) On the other hand, an expression resulting from an unconscious emotion is based on biological changes, which are affected by external stimulation. Humans themselves cannot manipulate such changes. Because of these two reasons, it is difficult for people to detect their unconscious emotion themselves. To solve this problem, we consider some observable mechanisms that can detect unconscious emotions before users express their emotions on SNSs, which might be helpful for users in communication.
In this research, we provide an observable mechanism to give users feedback on an unconscious emotion before they express their emotion on SNSs. Since the unconscious emotion is independent of the subjective expression of the users, the observable mechanism is dependent on the detection of the unconscious emotion by biological sensors. Several research groups have discussed the detection of unconscious emotions using such sensors as brainwave and heart rate sensors. (5,6) We applied the unconscious emotion estimation method using biological sensors to the system. We also developed a function for comparing the unconscious emotion with the expressed emotion and giving users feedback on the results of comparison. For example, when experiencing anger or distress, the users might use inappropriate words or expressions before they recognize their own emotions. In this case, this system will detect the mismatch between the expressed and unconscious emotions, and feedback the result to the users. We designed and developed a system, and conducted an experiment to show the effectiveness of the system in helping the users. This paper is organized as follows. In Sect. 2, the existing research studies are introduced. In Sect. 3, we propose a system that detects an unconscious emotion on the basis of biological information and provides feedback in an effective manner. Section 4 shows the experiment that we carried out and the results. Finally, we summarize our work in Sect. 5.

Issues
To avoid inappropriate emotion expressions on SNSs owing to unconscious emotions, we considered a method of helping the users by adapting the biofeedback mechanism. In this section, we first discuss biofeedback models and then emotion detection methods.

Biofeedback closed-loop model
The idea of the biofeedback mechanism comes from the area of medication. One of the first studies was carried out by Mandler et al. (5) They defined biofeedback as "the relationship between autonomic response and the subject's reported perception of such response-induced stimulations" and denote it as an autonomic response. Moss and Gunkelman suggested biofeedback as a technique to improve health and performance by measuring an individual's physiological activity and thus supporting humans to change it. (6) On the basis of this definition, biofeedback has been applied in therapy. For example, when unconscious physiological changes occur, patients can get feedback such as lighting or sound to help them realize their unconscious physiological changes.
Using the model in Fig. 1, Janssen et al. developed an affective music player (AMP) to change the user's mood through music. (8) In their research, mood was determined from the user's skin temperature because the skin temperature is related to the valence of the user. (9)(10)(11) Valence is a psychological index that shows whether the person is in a negative or positive mood. Since mood is a long-lasting affect, (12) and music is a long-lasting performance, in their research, they chose "mood" rather than "emotion" (affect is often used as the denominator of mood and emotion). (13) In this research, they applied mood to the biofeedback model. They also evaluated the mood and showed the effectiveness of music to mood. However, the effectiveness was limited for several reasons. First, they did not consider short-time emotion changes. In application scenarios such as SNSs, users present an inappropriate expression in a short time. Second, in such scenarios, the users present an inappropriate expression under intense stimulation. (2,3) As above, it is necessary to consider the affect that is drawn out in a short time or under intense stimulation and apply it with the biofeedback of the model. Also, it is more suitable to apply emotion rather than mood to SNSs.

Unconscious emotion detection method
Emotion estimation using biological information has been studied extensively. Generally, the use of biological information enables the objective estimation of genuine emotions since humans are unable to control their physiological changes such as brain activity electroencephalogram (EEG) Fig. 1. General biofeedback closed-loop model. (6,7) and heart rate changes. Alarcao and Fonseca reported a survey on emotion recognition using an EEG. (14) They described a procedure for detecting unconscious emotions from EEG signals. Ikeda and Sugaya proposed the use of the EEG and heart rate sensors to detect unconscious emotions. (15) They used the two-dimensional coordinates of "emotional expression" on Russell's circumflex model, (16) "pleasant-unpleasant", and "arousal-drowsiness". By displaying unconscious emotions, they proposed a tool with which to deepen emotional self-understanding. Hiramatsu et al. made a stage support system based on such a method. (17) To determine the stage composition, they proposed that the response of the audience can be understood by estimating their emotions using biological sensors. They used the vector decomposition of the emotion classification method based on biological information. However, such methods are rarely used in daily applications.

Abstract
The results of the previous works mentioned above imply that biofeedback is generally used in therapy, but rarely in daily applications. Biofeedback does not usually correspond directly to emotions. The biological information-based emotion estimation method can estimate unconscious emotions. Here, we propose to combine biofeedback with the biological information-based emotion estimation method and apply it to SNSs.

Model
The EEG has been used for several years to estimate unconscious emotions. (14) In most research studies, the unconscious emotion estimation method based on the EEG has been used. Since multichannel EEGs are generally too expensive and complicated to apply in daily life, Ikeda and co-workers (15,18) proposed a method of estimation unconscious emotion based on not only the one-channel EEG but also heart rate, because both type of the sensors they used are inexpensive and convenient. Nowadays, the case of such a sensor is common in our daily life. Ikeda and co-workers applied the results obtained from biological sensors to Russell's circumflex model (15) to estimate unconscious emotions. Russell's circumflex model (16) is known as a structure for classifying emotions. It has two axes of which the vertical axis stands for arousal and the horizontal axis stands for valence. Ikeda et al.ʼs research has advantages on both the sensor and model sides. (18) In our research, we used the same method to estimate unconscious emotions.
Ikeda et al. applied two indexes on arousal and valence. For arousal evaluation, they used a combination of alpha and beta waves from the brain. This index shows the activity states of the central nervous and is used as the arousal value. In Russell's circumflex model, the vertical axis shows arousal in the positive direction and sleepiness in the negative direction. (18) For valence evaluation, they used an index calculated from the heart rate variability (HRV). The HRV is obtained by time-domain analysis. This index shows autonomic nerves for evaluating the sympathetic and parasympathetic states, and is used as the valence value. In Russell's circumflex model, the horizontal axis in the positive direction is pleasant and that in the negative direction is unpleasant. (18) Both sensor values calculated for evaluating the arousal and valence are applied to Russell's circumflex model.
Also, in Ikeda et al.'s research, the accuracy of their emotion estimation method is shown by comparing their emotion estimates. (18) Sugaya and co-workers also used the same method to estimate unconscious emotions. The model they used is shown in Fig. 2. (15,17,18,20)

Sensing method (heart rate and EEG)
On the basis of Ikeda et al.'s research, two types of sensors were used. First, as one of the HRV indices, pNN50 was measured using heart rate sensors. Here, we used the sensor "Pulse Sensor Amped" (21) to detect pNN50 as the valence value. This sensor uses power with 4 mA current at 5 V. It also combines a simple optical heart rate sensor with amplification and noise cancellation circuitry, making it fast and easy to acquire reliable pulse readings.
Second, alpha and beta waves are measured using an EEG. Here, we used Neuro Sky's "Mind Wave Mobile" sensor (22) to calculate the arousal value, which is the difference in intensity between alpha and beta signals. It consists of a headset, a sensor arm, and an ear clip. The headset's reference and ground electrodes are located on the ear clip, and the EEG electrode is on the sensor arm, where it rests on the forehead above the eye (FP1 position).

System design
On the basis of the unconscious emotion estimation method, we used the EEG and heart rate sensors described above to developed our system. The design of our system is shown in Fig. 3. In this system, first, sensors will collect biological information, that is, EEG signals and heart rate. Second, the system will estimate an emotion based on Ikeda et al.ʼs method. (15,18) Third, the user will be asked to input his/her subjective emotion. Last, the system will compare the estimated emotion with the subjective emotion and feedback the result of the comparison to the user. In the system we designed, there are two functions: one is to estimate unconscious emotions and the other is to compare emotions. In Fig. 4, we show the flowchart of the "unconscious emotion estimation" function in (a) and the "emotion comparison" function in (b).
On the basis of the emotion estimation method proposed by Ikeda et al., (18) emotion estimates are divided into four quadrants. After the input measure time, the measurement loop will start. The process shown in Fig. 4(a) runs every second until the measure time runs out. The final result of estimating emotions is determined by weight. Each quadrant stands for a basic emotion, that is, the first quadrant stands for excitement, the second quadrant stands for distress, the third quadrant stands for sadness, and the fourth quadrant stands for relaxation. The word of each quadrant is the word nearest to a bisector on Russell's circumflex model.
After the measurement, the system runs the emotion comparison in Fig. 4(b). Since Russell's circumflex model has multiple emotion words at each quadrant, we put words in the same quadrant as a category, and the name of the category of basic emotions stands for each quadrant. The user will choose one word from all emotion words from Russell's circumflex model. Then, the system will compare the category name of a word user chosen to estimate an emotion. Lastly, the system will show the user the comparison result as feedback.
The system interface is shown in Figs. 5 and 6. Figure 5 shows the start page of the system. The user was asked to input the measured time in seconds. Figure 6 shows the result after measurement. After the measurement, the interface shows several emotion words, and the user can choose one word that describes his/her subjective emotion during the measurement time. After the chosen word is input, the system will give feedback on the comparison result to the user.

Experiment
To verify the effectiveness of the proposed system, a comparative experiment was conducted. In this experiment, we hypothesized a situation for users. In this situation, users were asked to recall their emotions during the experiment and express their emotions in simple words. In the meantime, with the biological sensors, we could detect an unconscious emotion during the emotion recalling. Lastly, we developed a system to compare the expressed emotion and the emotion detected using the biological sensors and gave feedback on the mismatched result to the user. The effectiveness of the system was evaluated through the use of a questionnaire.

Experiment with mapping method
In this experiment, to evaluate the conditions of the presence and absence of feedback on an unconscious emotion, subjects were divided into two groups: with and without biofeedback. The group with feedback was asked to answer the questionnaire after using the system. The group without feedback was asked only to answer the questionnaire. For the experiment, we used our developed system that implemented the designed functions we described in Sect. 3.

Experimental procedure
1. The subject wears the EEG and heart rate sensors at rest for 1 min for baseline measurement. 2. The subject was instructed to read one of the four scenarios from a prepared novel.
To prompt the subject to recall his/her emotions, we asked the subject to recall his/her own similar experiences as he/she reads one of the scenarios (from a novel).
In the experiment, we asked users to express their emotions honestly to remove the influence of false expression. The EEG and heartbeat sensors recorded the EEG signal and pNN50. 3. After the measurement time, the system estimated an emotion using the 2D arousal-valence model. 4. The system compared the estimated emotion with the subject's declared emotion (user's input). 5. Finally, the system fed back the result to the subject. 6. After the experiment, a questionnaire survey was carried out. In this experiment, there were 24 subjects (9 males and 15 females) divided into two groupsthe group with feedback and the group without feedback.
Questionnaires were composed of questions with responses given on nine-point scale questions. The questions are divided into three parts: Part 1 is composed of questions about the effectiveness of the estimated emotional feedback, Part 2 free descriptions, and Part 3 communication ability, (23) which includes ability, introduction, transmission, consultation, discussion, opinion, understanding, and education.

Results and discussion
We analyzed the 24 subjects' data after excluding invalid data of two subjects (females) due to the low stability of the sensor when worn. The average values of experiment results are shown in Fig. 7. Using the results of this experiment, we conducted a paired t-test on the items of the questionnaire in Part 1. The results are shown in Table 1.
A significant difference (*p < 0.05) between the groups with and without feedback was found for a motivation to receive advice about emotional expressions. As for the result of the "motivation to use the system" item, the group with feedback was significantly higher than that without feedback (*p < 0.05). There was no significant difference in "unpleasant experience" and "hesitation before expressing emotions" between the two groups.
Next, in Part 2, free descriptions of several opinions were obtained from the subjects, including whether knowing their emotions enabled them to not only correctly express their feelings but also deepen their awareness of themselves. Lastly, in Part 3, the questions asked were about the communication skill of the subjects. There was a positive correlation between hesitation before expressing emotions and motivation to use the system (r = 0.51). There was a negative correlation between communication ability and unpleasant experience in the group with feedback (r = −0.80). There was a positive correlation between motivation to receive advice about emotional expression and motivation to use the system (r = 0.63).

Discussion
In our experiment, the subjects were randomly divided into two groups with and without feedback. A significant difference (*p < 0.05) between the groups with and without feedback was found for the motivation to receive advice about emotional expression and motivation to use the system items. In the group with feedback, when a mismatch of comparison results occurred, subjects would reconsider their emotional expression. After subjects realized that there might be a mismatch between subjective and estimated emotions, subjects show more motivation to receive advice about emotional expression. Therefore, a significant difference (*p < 0.05) was also found for the motivation to use the system item. There was no significant difference in unpleasant experience and hesitation before expressing emotions between the two groups. The effectiveness of the system at the time of feedback was confirmed.

Conclusions
In this paper, we propose to apply an emotion estimation method to SNSs, to present unconscious emotions to users before they present emotions with inappropriate expressions. This paper simply stimulates the situation where users select emotional verb to express their emotion. Also, an experiment was performed.
Since this paper deals on the service development and evaluation of an innovative SNS, we did not consider the accuracy of the emotion estimation method but used Ikeda et al.'s method that showed the accuracy of the emotion estimation method. However, this point is significantly important in evaluating the service. Therefore, it is necessary to consider accuracy when applying the feedback system to an SNS in practical usage and future works.
Even if we show the effectiveness of our estimate emotion feedback system, there are still unanswered questions. Normally, social networking servers include instant messaging (2) and social networking sites. (25) Instant messaging sites such as WhatsApp and LINE provide instant communication from two users to a group of users. The importance of this type of service is the fluency of dialogs that usually happen in seconds. On the other side, social networking sites such as Facebook, LinkedIn, and Instagram allow users to share their emotion any time. (24) However, when applying the emotion estimation method to both instant messaging and social networking sites, the emotion feedback method at this stage requires a period of time to estimate the user's biological information. This causes a problem of the best time period to estimate an emotion and a problem of feedback timing.
As future plans, we need to consider the questions regarding the following factors mentioned above: 1. the accuracy of the emotion estimation method; 2. the best time period to estimate an emotion; 3. an effective feedback system. We also plan to include adverbs for the input words used in the present stage to the input content of the user in consideration of the discrimination strength. Such a study will identify and feedback the emotional strength. Emotion detection based on biological information can be expected to have a wide range of applications in the future.