Brain Activity Sensing to Verify Effects of Facial Image Reconstruction Puzzle—Verification by Young Students

We have developed an application for cognitive function training in order to cope with the increased number of dementia patients and the shortage of care workers in an aging society. This application generates an image in which a person’s facial image is transformed into a puzzle where the aim is to reconstruct the face. The puzzle is presented as a problem of person estimation to activate brain functions. In this study, we conduct two brain activity sensing experiments using electroencephalograms to evaluate the effect of the facial image reconstruction puzzle on cognitive function. The experiments suggest that the image reconstruction puzzle may affect the frontal lobe.


Introduction
In Japan's aging society, the proportion of people with dementia and dementia reserve in elderly people aged 65 and over is predicted to increase to about 25% in 2025. (1) In addition, the demand for nursing care personnel is increasing, but the supply does not meet the demand, with the gap between supply and demand anticipated to widen in the future. (2) For these reasons, it is necessary to take measures to reduce the burden on caregivers.
In nursing homes, functional training for both exercise and cognition is carried out to maintain the physical function of the elderly. Regarding the prevention of dementia, cognitive function training, such as giving intellectual problems (e.g., calculations and puzzles) to the elderly, is performed. The questions used for cognitive training are usually devised by facility staff. These questions need to be updated every time they are used, placing an additional burden on the staff in addition to their normal care work.
It has been reported that cognitive training such as solving puzzles is the most effective preventive measure for dementia. (3) In addition, mild cognitive impairment (MCI), which is considered as a dementia reserve group, reportedly causes a greater memory capacity decrease for human faces than for buildings. (4) On the basis of these facts, we have developed an application for cognitive function training (hereinafter referred to as the developed application) to reduce the burden on nursing home staff and improve the cognitive function of the elderly. (5) The developed application is based on a puzzle using human facial images.

Overview of the developed application
This section describes the developed application, which automatically detects the face region and generates an image trimmed into a square, as shown in Fig. 1(a), from the input arbitrary human image. After that, the generated facial image is divided into small square areas that are rearranged in a grid pattern to generate a puzzle image as shown in Fig. 1(b). In this study, the image shown in Fig. 1(b) is called the image reconstruction puzzle. In cognitive function training, we presented the image reconstruction puzzle and asked the recipient to guess the original person. The number of squares in the image reconstruction puzzle can be set arbitrarily in the developed application to adjust the difficulty of the puzzle.

Purpose of this study
The purpose of this study is to verify the effect of the image reconstruction puzzle generated by the developed application as cognitive training by sensing brain activity. Cognitive function tests such as the Mini-Mental State Examination (MMSE) and brain activity measurement by electroencephalography (EEG) are used as evaluation methods in this study. In the initial stage, verification is carried out with young students as subjects, since it is difficult to gather elderly people as subjects.
This paper is organized as follows. Section 2 shows the experimental results of effectiveness verification by cognitive function tests. Section 3 describes the results of sensing experiments using brain waves for brain activity measurement. Finally, Sect. 4 concludes this paper and describes future work.

Cognitive function tests
Nouchi et al. (6) utilized cognitive function tests to evaluate different kinds of portable games for cognitive function. The portable games were Nintendo DS games designed for adults to train the brain, which are considered to be good for cognitive function training, and Tetris, a typical puzzle game. The subjects were divided into two groups, with each group playing one of the two types of game. The cognitive function tests were conducted both before and after playing the game, and the cognitive function test scores were compared to evaluate cognitive function. In this case, the game is called an intervention and the test carried out before or after the intervention is called a pre-or post-intervention test, respectively. As the cognitive function tests, MMSE and the Trail Making Test (TMT) are generally used. The former evaluates general cognitive function and the latter evaluates specific cognitive function.
In accordance with the care prevention manual (7) prepared by the Ministry of Health, Labour and Welfare, the following are adopted as cognitive function tests.

MMSE
This tests general cognitive function and is mainly used for screening for dementia. It consists of 11 questions including those on orientation, calculation, writing, and drawing, and the maximum score is 30 points.

Matsui word memory test (immediate playback)
In this test, a subject is asked to memorize ten spoken words and to write them down within 1 min after the reading is complete. The subject repeatedly listens to and writes down the same words four times. The maximum score is 40 points. This test examines the memory function.

Yamaguchi Kanji code conversion test
A subject is asked to convert the presented Kanji (Chinese character) into codes on the basis of the combination of the Kanji representing a color and the corresponding code. As many conversions as possible should be performed within 2 min. This test evaluates attention and execution functions.

Word recall
A subject is asked to recall names belonging to the specified category, such as animals and vehicles, and to give as many names as possible within 1 min. This is a test of language fluency.

Matsui word memory test (delayed playback)
After completing the Matsui word memory test (immediate playback), Yamaguchi Kanji code conversion test, and word recall, the ten words used in the Matsui word memory test (immediate playback) are used for listening and writing again with a time delay. The maximum score is 10 points. This test examines the memory function.

TMT-B
In this test, a paper in which numbers and Kana characters are located irregularly is used. A subject is asked to connect the numbers and Kana characters in order and alternately by lines. The time spent to connect all characters is scored. This test checks the processing function.

Cognitive experiment settings
An experiment using cognitive function tests was performed to evaluate the effect of the developed image reconstruction puzzle. In the experiment, the image reconstruction puzzle played the role of the intervention. The subjects were 11 students in their 20s, who were classified into three groups to determine the effects of the intervention and the type of facial image used in the image reconstruction puzzle. We prepared two types of facial images: celebrity and non-celebrity. The former consisted of people whose faces the subjects may know, and the latter consisted of people whose faces the subjects did not know. There were four people in the experimental group (celebrity), four people in the experimental group (non-celebrity), and three people in the control group.

Experimental group (celebrity)
This group uses the intervention with celebrity facial images.

Experimental group (non-celebrity)
This group uses the intervention with non-celebrity facial images.

Control group
This group does not undergo cognitive training during the intervention period.
The experiment was performed according to the flow shown in Fig. 2. First, all subjects underwent the cognitive function tests described above. Next, the subjects in the two experimental groups performed the image reconstruction puzzle through the training website on the experimental server. Figure 3 shows an example of a puzzle on the training website. The intervention period was one month, and the subjects were asked to solve the puzzle four days a week. They solved one puzzle per day. The time limitation for one question was 1 min. Finally, after the intervention period, cognitive function tests were performed again.  Table 1 shows the pre-intervention scores for each cognitive function test. The postintervention scores are shown in Table 2. The letters A to K in the tables represent the subject labels. Since all subjects had the maximum MMSE score of 30 points, it was not possible to evaluate the increase in post-intervention score due to the ceiling effect. Therefore, the MMSE is excluded in the following comparison between pre-and post-intervention scores. From the results, individual differences can be observed depending on the test. For example, subject A shows a relatively large difference for TMT-B.

Experiment results
From Tables 1 and 2, we focus on the Matsui word memory test (immediate playback), since it shows a smaller individual difference than the other cognitive function tests. Figure 4 presents the scatter plot for the Matsui word memory test. The horizontal axis is the preintervention score and the vertical axis is the post-intervention score. In the figure, the letters A to K indicate the subject labels, and points C and F and points I and J overlap. Categorization by the subject groups can be recognized.    Fig. 5(a), two asterisks are used to show that the null hypothesis is rejected at the 1% level for the experimental group (celebrity). According to these cognitive function test results, the scores in the experimental groups were generally improved after the intervention compared with those in the control group for the Matsui word memory test. This means that the developed image reconstruction puzzle may have some effect on the memory function. Nevertheless, only the Matsui word memory test (immediate playback) of the experimental group (celebrity) showed a significant difference at the significance level of 1%. We analyzed the experiment log data and found that the subjects in the experimental group (non-celebrity) tended to abandon guessing whose face it was before the time limit of 1 min. Because they realized that the facial image in the puzzle was unknown to them, the opportunity to use their memory function was lost. Therefore, it is considered appropriate to use images of celebrities when using the puzzle for cognitive function training.

Effectiveness Verification by Brain Activity Measurement
The cognitive function tests described in Sect. 2 need a somewhat long experimental period from the pre-intervention to the post-intervention. In our case, it was almost one month. On the other hand, it is possible to measure brain activity directly owing to the development of sensor technology.
To evaluate the effect of the image reconstruction puzzle, a direct brain activity measurement experiment is carried out by EEG, which is one of the most popular sensing methods to measure brain activity.

EEG
In neurons of the human cerebral cortex, a faint current is generated by the transmission of information. The potential change caused by this current is observed using an electroencephalogram electrode attached to the scalp. (8) EEG waves are classified into δ, θ, α, and β waves according to their frequency band, as shown in Table 3.
δ waves appear during deep sleep and have the highest amplitude and the slowest waveform among the waves shown in Table 3. θ waves are often observed during slumber, sleep, meditation, or concentration. The appearance of excessive θ waves is also used to diagnose lesions. α waves appear when the eyes are closed or when a person is mentally relaxed. β waves appear in cases of mental excitement, agitation, or stress. Generally, relative to the α waves, the lower-frequency δ and θ waves are called slow waves and the higher-frequency β waves are called fast waves. In the following analysis, we focus on θ waves, because the brain activity derived from concentration is expected to appear while solving the facial image reconstruction puzzle.

EEG measurement experiment settings
An EEG measurement experiment was carried out with two student subjects in their twenties (subjects A and B). The purpose of this experiment is to evaluate the brain activity while the subject tries to solve the image reconstruction puzzle through a visual stimulus. The equipment used in the experiment is shown in Table 4. Figure 6 shows a schematic diagram of the experimental environment.
During the experiment, the subject was asked to sit in front of a monitor on which the visual stimulus was presented, while the experimenter sat in front of a PC in a neighboring room, recording the experimental data from the EEG set. The subject room was shielded for EEG measurement and the subject was asked to wear an EEG cap as shown in Fig. 6. These two rooms are connected by a door, but the door was closed during the experiment. The size of the monitor for the subject was 60.0 × 33.6 cm 2 (27-inch type) and its resolution was 1920 × 1080. Since the visual stimulus was displayed with a size of 600 × 600, the subject was asked to keep sitting at a distance of about 50 cm from the display.
The visual stimuli of the puzzle were presented to the subjects with their eyes open through the monitor. Electrodes were attached to the subjects' head using the EEG cap, and the electroencephalogram was monitored and recorded using an electroencephalograph. Figure 7 shows the locations where the electrodes were applied, which were in accordance with the International 10-20 system, and reference electrodes were applied to both earlobes (A1, A2). The sampling frequency of EEG was set to 1000 Hz. As visual stimuli, we used images of people whom subjects might know, such as politicians, athletes, and entertainers, and images of historical buildings such as temples and shrines. The image categories of the visual stimuli are as follows.
• Facial images: 2 images • Image reconstruction puzzle of facial images divided into 3 × 3: 3 images • Image reconstruction puzzle of facial images divided into 5 × 5: 3 images • Building images: 2 images Among these image categories, two facial images and two building images were used in the normal form without division as visual stimuli. Figure 8 shows the experimental protocol regarding where and how the visual stimuli were presented to the subject sequentially. In this experimental protocol, after a visual stimulus was randomly selected and presented for 30 s, the subject took a rest for 1 min. During the 1 min rest, the subject gave the name of the person or building in the presented image. The combination of the visual stimulus (image) presentation and the rest forms one experimental set. As shown in Fig. 8, ten experimental sets were sequentially repeated in each experiment, and a different image was presented in each experimental set.

Experimental results
From the collected experimental results, the data over the first 3 s were extracted from the 30 s period of visual stimulus presentation, because it was assumed that the subject tried to identify the stimulus more intensively in the initial presentation period. The data over the first 3 s were analyzed to evaluate brain activity by the "band power content rate" method, which corresponds to the content rate of each frequency band in the target EEG data. As fundamental processing, the whole frequency band was set to 0.2-30 Hz, and the power contents of θ, α, and β waves in the whole frequency band were calculated. Namely, after a fourth-order Butterworth filter with a passband of 0.2-30 Hz was applied to the EEG data, the frequency spectrum was calculated using the fast Fourier transform (FFT). As a result, the power spectrum for each frequency (Hz) was obtained.
Then, the power spectra were summed for the α, β, and θ wave frequency bands, and the total power spectrum was obtained by adding all the wave power spectra. Each band power content rate was calculated by dividing each wave's power spectrum by the total power spectrum.
It has been reported that the occurrence of θ waves in the frontal lobe is caused by the activation of attention and concentration functions. (9) Kawasaki et al. pointed out that visual working memory is related to the θ waves in the frontal lobe, (10) and Swaiman et al. reported that θ waves commonly encountered in the frontocentral regions are related to heightened emotional states. (11) Therefore, we focus on the relationship between θ waves and puzzle effects rather than α and β waves.
To evaluate the puzzle effects, we need to determine the baseline. Among the four image categories, the facial image category is selected as the baseline data because the subjects simply watch the image without solving a puzzle. For each image category, the difference in power content rate from the baseline power content rate is calculated for each electrode. Finally, the relative change is obtained by dividing the difference by the baseline power content rate. In other words, despite increases or decreases for other frequency bands, the relative change of the θ wave can be calculated for each electrode of each image category with the facial image category as the baseline. From Figs. 9(a)-9(c) and 10(a)-10(c), we observe an increase in the relative change of the θ wave in the frontal region, namely, Fp1 to F8. Therefore, the electrodes between Fp1 and F8 are categorized as the frontal electrode group, and the other electrodes are categorized as the other electrode group. Table 5 shows the relative changes of the two subjects averaged for the two groups for the image categories.
As shown in Table 5, the relative change of the θ wave increased for both the 3 × 3 and 5 × 5 image reconstruction puzzle categories compared with the baseline facial image category. Therefore, the facial image reconstruction puzzle can be effective because of θ wave activation.    Simultaneously, an increase in the relative change of the θ wave was observed in the building image category. One reason for this is that the building images include a temple and a shrine, which have a difference greater than that between two normal facial images. However, this tendency is weaker for subject B, since the relative changes of the θ wave were 123.09, 108.10, and 39.02% for the 3 × 3 and 5 × 5 image reconstruction puzzle categories and the building image category, respectively. Because of such individual differences, more experiments are needed to clarify individual differences of θ wave activity.

Conclusion
Two experiments were performed to evaluate the activation of brain activity by the image reconstruction puzzle used in the cognitive training application, which was developed to cope with an increased number of dementia patients and a shortage of caregivers.
The first experiment used cognitive function tests. The pre-and post-intervention scores of the cognitive training tests for the subjects were compared, and a significant difference was verified by performing a memory test on a group of subjects, where celebrity images were used in the image reconstruction puzzle. Thus, the developed image reconstruction puzzle has some positive effect on improving the memory function.
The second experiment evaluated brain activity by using EEG sensing technology. The band power content of EEG and the relative change of the θ wave while solving the image reconstruction puzzle were calculated. As a result, an increase in the relative change of the θ wave in the frontal electrode group was observed. Since such an increase is related to the attention function, concentration function, and visual working memory, it can be concluded that solving the image reconstruction puzzle works effectively as cognitive training.
There are two future tasks. The first is to increase the number of subjects in the EEG measurement experiment. It is necessary to confirm the reproducibility of many subjects, such as the fact that the increase in the relative change of the θ wave in the frontal electrode group also occurs for the building images, and the relationship between the difficulty and the increase rate of the θ wave.
The other task is to conduct cognitive function tests on elderly people. In experiments on young people, cognitive function test scores tend to be biased toward high scores. Therefore, it is necessary to conduct an experiment to evaluate the effect of the application on the cognitive function of the elderly.