A Cooperative Game Using the P300 EEG-Based Brain-Computer Interface

Abstract

In this paper, we present a cooperative game, Brainio Bros 300, using a brain-computer interface (BCI). The game is cooperatively controlled by two people using P300-generating color discrimination. The two users advance through the game together, one as the “player” and the other as the “supporter” providing assistance. We assumed that players would be able-bodied, while supporters would include people with severe disabilities. Through experiments using human subjects, we evaluated the subjects’ impressions of the game and its usefulness. The results of the impression evaluation showed that the subjects generally had good impressions, and there were many opinions that such cooperative games are interesting. We also discuss the possibilities of using the P300 BCI.

1. Introduction

A brain-computer interface (BCI) offers a noninvasive means of enabling a human to send messages and commands directly from his or her brain to a computer without moving, by wearing a simple scalp probe (a set of electrodes or sensors) [1, 2, 3, 4]. This noninvasive technology differs from invasive, surgical approaches that can cause irreversible damage to brain tissue.

In this paper, we present a BCI-based cooperative game, Brainio Bros 300. The BCI uses the P300 brain wave [5, 6, 7, 8, 9], a typical electrophysiological response to internal and external event-related potential (ERP) stimuli, measured using an EEG. The P300 wave has proven relatively easy to use for a variety of control-signaling purposes in much recent practical research.

When a human experiences interest in any kind of target, there is a measurable brain activity response. It is known that when a subject recognizes a specified photograph among a series of randomly presented photographs (i.e., the “ah!” response), P300 can be measured around the top of the head. Regardless of the type of stimulus (visual, tactile, auditory, olfactory, gustatory, etc.), P300 appears around 300 ms after the stimulus, which makes it a very useful brain feature that can be used by healthy or handicapped people for controlling an external device or communicating with the environment in real time. Notable applications of a P300 BCI include previous engineering studies in which a locomotive robot [10] and wheelchair [11] were controlled. Some neuroscientific experimental studies have also been conducted using “brain painting” [12, 13] for patient rehabilitation. A P300 BCI has also been used with virtual reality spaces [14] and as an interface for Twitter and Second Life [15].

Simple BCI games can also be useful for helping a user to control his or her brain activity. Games controlled using EEG signals have been designed to improve the power and duration of concentration, increase the speed and accuracy of brain waves, and improve cognitive function [16]. BCI-based games appear in both medical- and entertainment-focused varieties. Medical applications include games that use a steady-state visual evoked potential (SSVEP) to improve the concentration power necessary to control a BCI [17]. Here, SSVEPs refer to the brain signals induced by using a pattern reversal stimulus (i.e., the use of a checkerboard pattern to stimulate vision) or a flash stimulus (i.e., the use of a flickering light source such as an LED to stimulate vision). Medical games to promote the speedy generation of BCI commands, thus improving the user experience, have also been developed [18]. In the entertainment realm, games have been developed using a number of signals, including P300, to play popular games such as Pong and Tetris and to control a dancing robot [19, 20, 21, 22]. Most such games depend on the player’s degree of concentration [22].

In this research, we developed Brainio Bros 300 as a game controlled cooperatively by two people using P300-generating color discrimination. The two users advance together through the game, one as the “player” and the other as the “supporter” providing assistance. The player controls the game character by using a keyboard’s arrow keys to navigate through a series of colored blocks, while the supporter removes blocks obstructing the player’s path by thinking of the appropriate color via the P300 BCI. By cooperating, the participants can reach the end of the game. We assumed that players would be able-bodied, while supporters could include people with severe disabilities.

2. Brainio Bros 300 cooperative game

This section describes the Brainio Bros 300 game, with the discussion divided into two parts: the P300 BCI used and the components and design of the cooperative game.

2.1 P300-based brain-computer interface

We used a new, portable wireless EEG cap called g.Unicorn EEG (g.tec medical engineering, Austria), as shown in Figure 1, for recording brain activity in real time. For our experimental paradigm, we adapted the P300 speller idea to our problem by using MATLAB/Simulink to develop the game-based BCI. Ten dry electrodes were used to record EEG signals at a sampling frequency of 256 Hz. The electrodes were placed according to the international 10–20 system, using the Fz (forehead), Cz (crown), P3, Pz, P4, PO7, Oz, and PO8 (all at the back of the head) locations, with references placed at A2 (earlobe) and Fz (forehead), as shown in Figure 2.

Figure 1.

Calibration of the EEG experiment using the g.Unicorn EEG cap for a P300-based BCI.

Figure 2.

Electrode positions of the eight channels.

Before the cooperative game begins, the first user (designated as the “supporter”) had to put on the EEG cap and perform calibration for sending color commands. We developed a program enabling the user to calibrate six colors: red, blue, green, yellow, purple, and white. The six colors were randomly arranged into two rows of three, with each box flickering 30 times between a photograph of a human face and a color. The subject choses a color and counted in their head how many times that color was displayed. Because research results have indicated that flickering photographs of famous people increases calibration accuracy by increasing the amplitude of the P300 wave, we used images of widely recognizable people, such as Albert Einstein and President Donald Trump. The procedures of the calibration and game phases are detailed below:

• Calibration phase

  1. Select the colors to be used in calibration, as illustrated in Figure 3. For example, red, blue, green, and yellow can be displayed in order, as shown in the figure under “Item selections.”

  2. Begin calibration.

  3. The system directs the user to look at “red.”

  4. The colors randomly flicker. The user mentally adds to the count every time he or she sees the color red. After red has flickered 30 times, the system is configured to stop.

  5. The system records the speed of the response every time the color red was flashed and the EEG information.

  6. This process continues three–five more times for all the other colors. Each color takes 20 s, for a total of around 2 min.

• Game phase

  1. As colors are randomly flashed on the screen, the user looks at the color he or she wants to select and mentally counts its flashes. We used 20 flashes for a balance between selection accuracy and enjoyability of the game.

  2. The system guesses which color the user selected according to how the EEG changes. The EEG readings at these moments are measured, and the instants at which the low-frequency stimuli are displayed are averaged as the trigger.

Figure 3.

Calibration screen for our P300 experimental paradigm.

The information on the selected color was sent by a UDP broadcast on port 1000 and received by a Unity application. As seen in Figure 3, the selected colors were also shown under “Written text,” but this was not actually used in the game.

2.2 Brainio Bros 300 design

The Brainio Bros 300 cooperative game was developed in Unity 2018 and runs on the Windows 10 operating system. It is a game in the style of Nintendo’s Super Mario Bros., in which the player maneuvers a character through various obstacles to reach a goal flag and complete the stage. Along the way, the player encounters blocks that are too high to clear and points where it is impossible to pass, but the player and supporter work together to complete the game.

The game contains blocks of five colors: red, blue, green, yellow, and purple. Figure 4 illustrates a green block obstructing the player, preventing him or her from advancing. In such situation, the player communicates an instruction to the supporter (e.g., “green’s in the way” or “destroy green”), and the supporter then thinks of the color green. By doing so, the green block in the game can be destroyed, as shown in Figure 5. The game was designed so that it cannot be completed without the player and supporter cooperating.

Figure 4.

Obstructed by the green block.

Figure 5.

After destroying the green block.

Because the game would not be enjoyable with only one obstruction, multiple blocks of different colors are placed in one spot, as shown in Figure 4, requiring the player to consider which color to destroy. The game was also designed to offer multiple courses, allowing the player to choose his or her own course. For example, Figure 6 shows a case of two paths. If the red block is destroyed, the player can advance via the blue block, whereas if the blue block is destroyed, the player can jump onto the red block and continue onward.

Figure 6.

Start screen with two ways of advancing.

The game was designed to destroy (i.e., hide) only the one color currently recognized by the BCI, so that only one color could be destroyed at a time. For example, if a red block is first destroyed and then a green block, the game displays the destroyed red block again. Figure 7 shows a case of the player unable to advance with only a blue block displayed, while Figure 8 shows the result with the blue block destroyed, causing the red blocks to reappear. The game was designed as indicated by these images, so that, even though there are no blocks beyond the blue block, destroying the blue block causes the red blocks to reappear, allowing the player to advance.

Figure 7.

Unable to reach the blue block.

Figure 8.

Blue block destroyed, causing red blocks to reappear.

This system, with the combined limitations of using only five colors and allowing only one color to be destroyed at a time, gives the player manual control over which blocks are destroyed. Because an unintended color can be destroyed, it is possible to destroy the player character’s foothold. When the character falls, however, it returns to the start point, from where the player can continue the game.

The game also features obstacles in which blocks move horizontally or vertically. Such blocks were programmed in advance to behave as such in the game in advance. We included such features to prevent the game from becoming a monotonous experience for the users. A flag is beyond all the obstacles. When the character touches the flag, the game is completed.

3. Game experiment

For experiments using an EEG capable of measuring the P300 brain wave, it is extremely difficult at present to prepare an environment supporting easy calibration for many subjects. For example, even preparing only one EEG device capable of measuring P300 requires a considerable cost, and it takes time to perform the calibration. Therefore, this paper considers the possibilities of games for use with such a BCI by focusing on evaluating the Brainio Bros 300 game.

Specifically, we evaluated Brainio Bros 300 in terms of the user’s impression of the game and its usefulness. For our methodology, we used the Wizard of Oz (WOZ) approach. A WOZ system involves a user interacting with a person acting as a computer system (i.e., the “wizard”), allowing for effective simulation of a real system [23].

For Brainio Bros 300, although the supporter wore the EEG device, the game was actually controlled with a keyboard. We thus conducted an evaluation experiment with the supporter as a reference.

The experiment was conducted with 25 students at Future University Hakodate: 19 men and 6 women. Of these students, 12 were designated as supporters and 13 as players. The supporters consisted of 8 men and 4 women, while the players consisted of 11 men and 2 women. The average age of the whole group was 20.76 years, with average ages of 20.75 and 20.77 years for the supporter and player groups, respectively.

To use the WOZ approach, we created an experimental system using keyboard input to destroy blocks as a substitute for the P300 BCI component of Super Brainio Bros 300. In this system, pressing the “R,” “G,” “B,” “Y,” or “C” keys caused the red, green, blue, yellow, or purple blocks, respectively, to be destroyed. For the ease of distinguishing which key corresponded to which color, we applied a sticker of each color to its corresponding key.

We also created a post-experiment questionnaire to evaluate the users’ impressions and opinions of the usefulness of Super Brainio Bros 300. Table 1 lists the details of the questionnaire, which included questions using the semantic differential (SD) method, a five-point scale method, and free responses. For the five-point scale, the responses consisted of “strongly agree,” “agree,” “neither,” “disagree,” and “strongly disagree.” The questions are listed with abbreviated forms in the table.

Table 1.

Questionnaire details.

Table 2 lists the details of each condition for the SD method and the reasons for its selection. Each condition showing a positive impression is filled in gray. On the questionnaire form, the positive and negative responses were distributed between the left and right sides as a counterbalancing measure. The SD method used a seven-stage evaluation, with responses consisting of “extremely” (positive or negative), “very” (positive or negative), “a little” (positive or negative), and “neither.”

Table 2.

Details of the conditions used for the SD method, together with the reasons for their use.

We also used a keyboard, display, notebook PC, desk, chair, and EEG headset as experimental materials, arranged as shown in Figure 9.

Figure 9.

Arrangement of the experimental materials.

We next describe the experimental procedure. The total experiment time was around 30 minutes. The first step was to gather all the subjects in a room and form them into pairs. The participants were instructed to pair with someone they did not know, where possible. As there was an odd number of participants, resulting in one excess player, a researcher acted as the supporter for that player and did not fill out a questionnaire. After pairing the participants, we explained the experiment to the whole group. This included explanations of the following: (1) an overview of the game (a game controlled by brain waves, requiring the player and supporter to cooperate to reach the goal), (2) the role of the player (to control the character by using the keyboard), (3) the role of the supporter (to assist the player by using brain waves), (4) the control method, and (5) an image of the game screen. The players were then taken into separate rooms containing the experimental materials. Each player was seated in the chair indicated by “Player’s chair” in Figure 9 and asked to wait until the supporter arrived.

After the players had been moved, we explained to the supporters that this experiment used the WOZ approach, that brain waves were not actually used, and that they would advance through the game by using the keyboard. The supporters were asked to wear the EEG headset and appear to the player as if they were controlling the game through brain waves. The supporters were then taken to the separate rooms in which the players were waiting, and each was seated in the “Supporter’s chair” shown in Figure 9. As soon as we had confirmed that the player and supporter were both seated, a brief explanation was once again provided to both of them. At this time, the participants were instructed to communicate regarding the color of the block when the player wanted a certain block cleared, when the supporter was trying to clear a block, and so on.

After this instruction, the participants began playing the game. After they had completed it, they returned to the room where the initial explanation had been provided, and they filled out the questionnaires. Once they had completed the questionnaires, the experiment was over. At this point, they were instructed not to reveal the details of the experiment.

Finally, we explain our analysis methods for the experimental data. For the SD method in which pairs of words were displayed side by side (e.g., easy vs. difficult) to assess the impression of the participants, they were asked to assign a score in the range of −3 and 3, with 0 representing “neither”, −3 representing “extremely leaning to the left word”, and 3 representing “extremely leaning to the right word”. We then compiled descriptive statistics and plotted a semantic profile.

Next, the five-point scale was tallied with “strongly agree” as 5 points, “agree” as 4 points, “neither” as 3 points, “disagree” as 2 points, and “strongly disagree” as 1 point. After compiling descriptive statistics for this data, we performed a chi-square test.

Finally, for the free responses, we counted experiences and keywords shared among participants and collected them into overall viewpoints. We also collected strongly held minority opinions as necessary.

4. Results

Table 3 and Figure 10 give the results of the SD method. Table 3 lists the average for each question for each participant role, as well as the difference between the groups. The fields shown in gray indicate a response that implies a positive impression. The left/right distribution of the response fields is the same as on the questionnaire sheet provided to the participants. Averages and average differences with an absolute value greater than 1 are also shown in gray. We also conducted semantic profiling based on the results, as shown in Figure 10.

Table 3.

SD method results.

Figure 10.

Semantic profile of the SD method.

From the group differences indicated by this table and graph, we observed no great differences between the impressions of the supporters and players. The average values in gray show that “substantial,” “cooperative,” “enjoyable,” “happy,” “cute,” “friendly,” and “lively” were evaluated highly by both players and supporters, with “cooperative” evaluated particularly highly. Moreover, the players highly evaluated “satisfying” and “new.”

Because all the data fell within one standard deviation, we could conclude that it was a good set of low-variance data. In the grayed averages, we also see that both supporters and players rated “Was it interesting?” and “Interacting with a child” highly. Players alone rated “Game design” and “Would recommend” highly.

Table 4 and Figure 11 show the five-point scale responses, organized by the participants’ roles, in terms of averages and standard deviations (Table 4) and percentages (Figure 11). Table 4 shows averages greater than 4 in gray. In addition, Table 5 lists the sums of the percentages of “strongly agree” and “agree” responses to each question, as well as the results of the chi-square test. The percentages in gray indicate values above 80%, while the statistical significances in gray indicate a significant statistic at the 5% significance level. The analysis shows that “Would you recommend it?” had a significant result at the 5% level, while the results of the remaining six questions were significant at a significance level of 10%.

Table 4.

Averages and standard deviations for the five-point scale method.

Figure 11.

Percentage results for the five-point scale method.

Table 5.

Percentage sums of the players’ “strongly agree” and “agree” responses and chi-square test results.

Finally, we will discuss the participants’ free responses, including particularly common responses and useful minority opinions. First, out of the people who responded that “it was interesting”, 9 out of the 12 supporters and 8 out of the 13 players gave “cooperative play” as a reason.

For “Other future uses,” 6 out of 12 supporters offered “use as an icebreaker” as a response, as did 4 out of 13 players. In addition, notable minority opinions included “use in brain training,” “communication with foreigners,” and “prevention of dementia in the elderly.”

As for “Points for improvement,” 2 out of 12 supporters mentioned the addition of more complex features and the ability to actually play using brain waves. Of the 13 players, 2 mentioned the ability to use items and the appearance of enemies in the game as points for improvement. Additionally, one minority opinion suggested the capability of the player, too, to use brain waves to control the game.

Finally, for “Any other opinions or feelings,” many supporters and players mentioned that the game was interesting and fun.

5. Discussion

We conducted an impression evaluation of the Brainio Bros 300 cooperative game and its usefulness. First, the results of the SD method showed that the players evaluated the game as “full,” “cooperative,” “enjoyable,” “happy,” “cute,” “friendly,” “lively,” “satisfying,” and “new,” with “cooperative” evaluated particularly highly. On the five-point scale, the game was evaluated highly in terms of both the players’ average values and the combined percentages of “strongly agree” and “agree” for the questions of “Interesting,” “Communicating with a child,” “Game design,” and “Would recommend,” with “Would recommend” showing a particularly strong correlation. Finally, the opinion that the cooperative aspect of play was interesting was particularly widely expressed in the free responses.

One of the biggest advantages of Brainio Bros 300 is the capability for the player, who does not have to wear an EEG cap, to play together with the supporter, who does wear an EEG cap. We believe that playing cooperative games using a P300 BCI could be of significant benefit to people with major disabilities such as amyotrophic lateral sclerosis (ALS), enabling them to play with able-bodied children, family members, and friends, thus deepening connections and communication.

The use of a P300 BCI is an easy-to-measure, noninvasive method. It can also be controlled with a high degree of accuracy without the need for detailed user training in advance. Training the P300 BCI command categories does not take a great deal of time. Most patients, including almost all able-bodied people and even people with severe paralysis, can use a P300 BCI. It also offers a goal-oriented control signal that is particularly suited to situations that do not require a continuous control signal.

On the other hand, one of the game’s limitations is as follows. The P300 BCI is one of the fastest of the currently usable BCIs, but it is still very slow compared to normal input devices such as mice and game controllers. In the current Brainio Bros 300 system, the player must wait for the colors to flash before having the supporter select a color.

With the development of a decoding algorithm that could detect the P300 brain wave to a high degree of accuracy after only one attempt, a command could be sent to the game every second. The results could then be adjusted according to feedback following categorization. For example, in the event of a mistaken command, the response time could be minimized by testing the supporter again. This should also allow the result to be checked once per second. Although Brainio Bros P300 is a game in which players and supporters cooperate, it takes time for a supporter to erase blocks by using P300 control as compared to standard games. Moreover, as it is a game in which two people cooperate, it differs from a competitive game based on speed. For this reason, a player and a supporter must play together and possibly become friends.

We believe that, in the future, if it is possible to reduce the number of flashes while maintaining the current level of accuracy and to increase the number of commands (controllable dimensions), this type of game would be usable in the real world and present an extremely promising interface.

Some problems with using a P300 BCI are that real-time P300 detection can sometimes be inaccurate, as it is easily affected by a number of human sensory phenomena such as attentional blinking, repetition blindness, and change blindness [24, 25, 26, 27, 28]. It is also possible for motivation to impact BCI performance [29], causing the EEG signal pattern to change according to the attention level, fatigue, state of mind, learning, and unsteadiness [1]. A P300 BCI might also not be an effective method for people who cannot control their line of sight [30]. It is thus important to work carefully, given the potential impacts of participants’ state of mind and attention to the problem.

6. Conclusion

In this BCI-based game research, we developed a real-time game, Brainio Bros 300, a cooperative game using a P300 BCI to facilitate two users (a player, who controls the character in the game and does not wear an EEG cap, and a supporter, who uses his or her brain activity to communicate) working together to achieve one goal. We evaluated participants’ impressions of the game and its usefulness and considered the viability of the P300 BCI interface.

In the future, we would like to use noninvasive measurement to investigate more deeply the brain mechanism during a cooperative video game.

Acknowledgments

This research was a first place winner, IEEE Brain Winner, and BR41N.IO SfN Winner at the IEEE SMC2018 BCI Hackathon. On receipt of these awards, we would like to express our gratitude to g.tec and to all the organizers, as well as all the students, who assisted with our experiment.