A Review of P300, SSVEP, and Hybrid P300/SSVEP Brain- Computer Interface Systems

1. Introduction

There are several techniques for measuring brain activities such as magnetoencephalogram (MEG), near infrared spectroscopy (NIRS), electrocorticogram (ECoG), functional magnetic resonance imaging (fMRI), and electroencephalography (EEG). Each technique has some advantages and disadvantages compared to other techniques. For example, in EEG the temporal resolution is high but the special resolution is low compared to fMRI. Because of low cost and portability, EEG has been largely used in both clinical and research applications [1][2][3][4].

One of the EEG research applications is in a brain computer interface (BCI) system. A BCI can provide a new way of communications for special users who cannot communicate via normal pathways. A BCI system can send commands, controlled by brain activity and distinguished by EEG signal processing. There are many features which can be extracted from EEG, for example, six brain rhythms can be distinguished in EEG based on the differences in frequency ranges; delta (1- 4 Hz), theta (4-7 Hz), alpha (8-12 Hz), mu (8-13 Hz), beta (12-30 Hz), and gamma (25-100 Hz). The delta and theta rhythms occur in high emotional conditions or in a sleep stage. The alpha rhythm happens in awake and eyes closed relax condition. The oscillation in alpha rhythm has smooth pattern. The beta rhythm pattern is desynchronized and the condition is the normal awake open eyes. The gamma rhythm can be acquired from somatosensory cortex and mu rhythm from sensorimotor cortex.

BCIs are categorized based on the EEG brain activity patterns into four different types: event–related desynchronization/synchronization (ERD/ERS) [5], steady state visual evoke potentials (SSVEP) [6][7][8], P300 component of event related potentials (ERPs) [9], and slow cortical potentials (SCPs) [6][10]. The focus of this chapter is on P300, SSVEP and hybrid P300-SSVEP BCI systems.

Compared to other modalities for BCI approaches, such as the P300-based and the SCP BCIs, SSVEP-based BCI system has the advantage of having higher accuracy and higher information transfer rate (ITR). In addition, short/no training time and fewer EEG channels are required. However, similar to other BCI modalities, most current SSVEP-based BCI techniques also face some challenges that prevent them from being accepted by the majority of the population. Two important features of each BCI system are information transfer rate and required training time. A general comparison of different BCI approaches is shown in Figure 1.

Figure 1.

A general comparison of SCP, ERD/ERS, P300, and SSVEP with respect to their training time and information transfer rate.

The process of detecting patterns from EEG is divided into three steps [11]: signal pre-processing, feature extraction and classification. The first step is to remove noise such as artifacts or power line noise which is added to EEG. So filtering is the first step in EEG signal pre-processing. Band pass and notch filters are the most common filters utilized in EEG signal filtering.

In the next step, features that are selected in feature extraction step and the type of classifier should be chosen based on the type of BCI. For example, for P300, time domain or time-frequency domain features such as wavelets are appropriate and for SSVEP BCIs frequency domain features are more appropriate. Classifiers such as Fischer’s linear discriminant analysis (FLDA), Bayesian linear discriminant analysis (BLDA), stepwise linear discriminant analysis (SWLDA), and support vector machine (SVM) are utilized [12][13] for P300 classifications. For SSVEP feature extraction and classification, different methods such as the Fast Fourier transform (FFT), the canonical correlation analysis (CCA), stimulus-locked inter-trace correlation (SLIC), and the common special patterns (CSPs) have been used [14][15] [16].

In recent years, the BCI research projects and the number of publications in this area have been increased rapidly [17]. Different areas of research such as new feature extraction methods, new classification techniques, new BCI paradigms, or new approaches for combining different BCI types have been investigated for improving accuracy, reliability, information transfer rate, and user acceptability. Combining different BCI types called a hybrid BCI is a new trend in BCI research which is the main focus of this chapter. In the next sections, the P300 and SSVEP BCI are explained and then different approaches for building a P300-SSVEP hybrid BCI are discussed.

2. P300-based BCI

2.1. The P300 component

Event related potentials (ERPs) are the measurement of brain responses to specific cognitive, sensory or motor events. One of the main approaches towards BCI is based on ERPs. P300 is a major peak and one of the most used components of an ERP. The presentation of stimulus in an oddball paradigm can produce a positive peak in the EEG, 300 msec after onset of the stimulus. The stimulus can either be visual, auditory or somatosensory. This evoked response in EEG is called P300 component of ERP.

2.2. Properties of P300

The spatial amplitude distribution is strongest in the occipital region of brain and is symmetric around central location Cz recorded based on the 10-20 international system [18]. The spatial amplitude distribution of 10-20 international system and the electrodes that P300 is typically recorded from are shown in the following Figure 2. In terms of temporal pattern, P300 wave amplitude is typically in the range of 2 to 5 µV with duration of 150 to 200 msec as shown in Figure 3. Considering the P300 low amplitude relative to background activities of the brain (in the rage of 50 µV), it is clear that P300 detection requires special signal processing. One of the simplest approaches is ensemble averaging EEG over multiple responses to enhance P300 amplitude to identify it while suppressing background EEG activities.

Figure 2.

Recoding of EEG based on 10-20 system and location of the electrodes typically used for P300 detection [18].

Figure 3.

Temporal pattern of P300 component.

P300-based BCI has been used as one of the most widely used BCI systems since 1988 [1]. New advancements in inexpensive and portable hardware made it possible to have real-life application outside of laboratory environment [17][1][20][21][22]. P300-based BCI has been used from controlling a wheelchair for helping disable people to a virtual keyboard for spelling word and interacting with computers. This type of BCI systems possesses the potential to improve the quality of life.

P300-based visual speller paradigms are attracting much attention as they could provide means to communicate letters, words, and simple commands to computer directly from the brain. In the following sections, we will review the classical speller paradigm and discuss current and future trends in this area.

Processing and successful use of P300 wave in a BCI application requires several processing steps. First of all, the recorded EEG data have to be processed to reduce the effect of noise. A feedback mechanism is required where a visible signal is presented in the monitor correlated with the recorded signal. A pattern recognition or classification algorithm has to be developed to identify the P300 wave in the recorded ERP epochs. The algorithm parameters should be adjustable to adapt according to the change of user characteristics [11][17].

Figure 4 shows a typical BCI setting for speller application. Stimulus is presented by random flashing of the characters on the screen. This eventually evokes P300 wave in the recorded EEG. A signal processing technique performs the processing of P300 related information and the classifier contains the pattern recognition algorithm as described earlier [17].

Figure 4.

A typical P300 BCI setup with visual feedback.

The classical paradigm for P300-based BCI speller was originally introduced by Farwell and Donchin in 1988 [1]. This Row-Column (RC) paradigm is the most popular speller format. It consists of 6 × 6 matrix of characters as shown in Figure 5. This matrix is presented on computer screen and the row and columns are flashed in a random order. The user is instructed to select a character by focusing on it. The flashing row or column evokes P300 response in EEG. The non-flashing rows and columns do not contribute in generating P300 [1]. Therefore, the computer can determine the desired row and column after averaging several responses. Finally, the desired character is selected.

Figure 5.

A typical row/column paradigm [1].

It is interesting to note that P300-based BCI did not receive much attention when it was first proposed. However, recent trend is quite different where P300 BCI has emerged as one of the main BCI approaches. The researchers have focused on identifying the scopes of improvement of the traditional paradigm by introducing new ways of flashing, introducing colors, or investigating other ways to enhance the ERPs. Much focus has put on applying advanced digital signal processing techniques and classification methods in order to improve the classification results. Also, there have been several attempts to introduce new paradigms to evoke P300 potentials. Figure 6 shows such a different approach which is called single character (SC) paradigm that only single character is flashed instead of a row or column. The SC paradigm randomly flashes one character at a time with a delay between flashes [17]. The delay in SC speller is longer than the delay in RC speller. Though SC speller is slower than RC speller, SC speller can produce larger P300 amplitude [17].

Figure 6.

Single character paradigm where each character is flashed [14][1].

Checkerboard (CB) speller is another paradigm proposed to overcome a problem associated with RC speller [17]. This drawback is arising from the distraction or inherent noise due to row/column association [17]. CB speller effectively reduces these two limitations as the characters are arranged in a checkerboard style as shown in Figure 7. CB speller also increases ITR [20].

Figure 7.

Checkerboard paradigm [20].

The region-based (RB) paradigm was proposed by Fazel-Rezai et. al. in 2009 [21]. It is a two-level speller where the regions have to flash instead of rows and columns. In the first level, characters are placed in several regions (seven groups) as shown in Figure 8 [17][1][20][21]. The users are instructed to focus attention on a specific character in one of the seven regions. After several flashes the desired region is selected. In the second level, characters are distributed following the same rule used in the first level and each character flashes in similar order. After several flashes, the desired character is identified [21].

Figure 8.

Region based paradigm where a set of characters in level 1 (E) are expanded in level 2 for spelling character “B” (F).

It is reported that RB speller has decreased the adjacency problem significantly [17][1][20][21]. The RB and CB paradigms show new directions in BCI speller paradigms apart from RC speller.

There has been much progress in bringing BCI technology out of lab environment to real-life applications. BCI has widely been studied in helping disable people, for example, enabling controlling a wheel chair using brain signals [22]. The other promising applications are in managing smart home environment, controlling a virtual reality environment, and next generation gaming [12].

3. SSVEP BCI

Electrophysiological and neurophysiological studies have demonstrated increases in neural activity elicited by gazing at a stimulus [23]. Visual evoked potentials are elicited by sudden visual stimuli and the repetitive visual stimuli would lead to stable voltage oscillations pattern in EEG that is called SSVEP.

SSVEP is considered as a concept with two different definitions. Ragan [24] proposed that SSVEP is a direct response in the primary visual cortex. On the other hand, Silberstein et al. [25] assumed that the SSVEP includes indirect cortical responses via cortical-loops, from the peripheral retina, while a cognitive task is performed. SSVEP in this model has a complex amplitude and phase topography across the posterior scalp with considerable inter-subject variability. Although the main mechanism of SSVEP still is unknown, generally SSVEP is considered as a continuous visual cortical response evoked by repetitive stimuli with a constant frequency on the central retina. As a nearly sinusoidal oscillatory waveform, the SSVEP usually contains the same fundamental frequency as the stimulus and some harmonics of the fundamental frequency. For example, when the retina is excited by a visual stimulus at presentation rates ranging from 3.5 Hz to 75 Hz, the brain generates an electrical activity at the same and different frequency of the visual stimulus. The flickering stimulus of different frequency with a constant intensity can evoke the SSVEP in verity of amplitudes, ranging from (5-12Hz) as low frequencies, (12-25 Hz) as medium ones and (25-50 Hz) as high frequency bands [26]. This type of stimulus is a powerful indicator in the diagnosis of visual pathway function, visual imperceptions in patients with cerebral lesions, loss of multifocal sensitivity in patients with multiple sclerosis, and neurological abnormalities in patients with schizophrenia and other clinical diagnoses [26].

In addition to the usual clinical purpose of diagnosing visual pathway and brain mapping impairments, the SSVEP can serve as a basis for BCI. Recently, SSVEP BCI systems have gained a special place in the BCI paradigms continuum because of having a variety of different possibilities. SSVEP BCIs are useful in different applications, especially the ones that need some major requirements as follows [27]:

• Large number of BCI commands is necessary (in SSVEP BCI limitations are mostly defined only by the design).

• High reliability of recognition is necessary (in SSVEP BCI, patterns are clearly distinguishable by frequency).

• No training (or just a short time training for classifier training) is allowed.

• Self-paced performance is required.

A typical SSVEP-based BCI system uses a light-emitting diode (LED) for flickering. SSVEP responses can be measured within narrow frequency bands (e.g. around the visual stimulation frequency. Several numbers of stimuli can be implemented by using not necessarily a wide range of flickering frequencies, as the minimum detectable difference between frequencies is 0.2 Hz [27]. The occipital region is the area where this feature is generated more prominently [6]. The most wide-spread signal processing technique to extract the SSVEP responses of the brain from the raw EEG data is based on power spectral density (PSD) using FFT of a sliding data window with a fixed length. Template matching and recursive outlier rejection have also been used to show the feasibility of SSVEP BCI systems. Other methods which attempt to improve on robustness upon the FFT-based methods are autoregressive spectral analysis, and the frequency stability coefficient (SC) which has been shown to be better than power spectrum for short data windows; although training is necessary for building the SC model. Furthermore, CCA is also an efficient method for online SSVEP-BCI, as the required data window lengths are shorter than those necessary for power spectrum estimation.

Pastor et al.[28] studied the relationship between visual stimulation and SSVEP-evoked amplitudes, showing that the amplitude of SSVEPs peaks at 15 Hz, forms a lower plateau at 27 Hz, and declines further at higher frequencies (>30 Hz) as shown in Figure 9.

Figure 9.

SSVEP amplitude with different flickering frequency [28].

In low-frequency stimulation, SSVEP detection is more accurate. In spite of its favorable detection properties, this band presents two major inconveniences [28],

• According to visual perception studies, stimulation frequencies in this band are rather annoying and tiring for the subject

• The risk for inducing photo epileptic seizures is higher for stimulation frequencies in the 15 – 25 Hz.

A simple solution could be in using higher stimulation frequencies. From empirical and subjective evidence, the threshold could be set to 40 Hz for low stimulation [28].

Ding et al. [23] demonstrated that a person’s attention level modulates his/her SSVEP. Since the SSVEP depends directly on the stimulation frequency of visual flickering, user’s attended target can be identified by analyzing the frequency contents in the induced SSVEP. By tagging different flickers with distinct flickering frequencies, subjects can shift their gaze to their desired flickers. These gaze targets can then be identified using the Fourier spectrum of the measured SSVEP signals. Middendorf et al. [29] designed a flight simulator controlled by two flickering lights that controlled leftwards or rightwards movement with a classification accuracy of 92%. Cheng et al. [30] implemented a SSVEP-based virtual keypad that achieved a mean ITR of 27.15 bits/min using twelve frequency-tagged flickering lights. Using two EEG electrodes positioned at the primary visual cortex, Kelly et al. [31] developed a method allowing participants to interact with a computer game.

Moreover, some visual BCIs have been developed as independent from users’ eye gaze. Allison et al. [6] investigated selective attention using overlapping stimulus to induce SSVEPs difference in an online control study. Zhang et al. [32] also modulated the SSVEP amplitude and phase response by means of shifting covert attention on two sets of random dots with distinct colors, motion direction and flickering frequencies in the same visual field. Trader et al. [33] compared the performance of the Hex-o-Spell and matrix design using covert attention. Their results demonstrated that the Hex-o-Spell is more than 50% better than those with matrix design with covert attention. This SSVEP-based BCI identifies user’s intended targets on calculated Fourier spectra. Nevertheless, the Fourier spectrum requires a time window (e.g., 1 or 2 sec) for computation to achieve sufficient frequency resolution in identifying two distinct gaze targets. Data segment with insufficient length in Fourier spectrum computation usually results in reduction of frequency resolution, which can limit the number of available targets in SSVEP-based BCI. Since BCI performance depends on accuracy and speed, a reliable method for extracting SSVEPs and recognizing gaze targets in an appropriate data segments is crucial.

It has been shown that the refreshing frequency, of a cathode ray tube (CRT) monitor can evoke a clear SSVEP. For SSVEP-based BCI development, the decoding accuracy is the most important factor, and a suitable stimulator is very crucial in this regard [34]. In previous studies, CRT flicker has been the most widely adopted stimulator, the LED flicker has only been reported in a small number of studies, and liquid crystal display (LCD) flicker has not appeared in the literature [23]. Since each of the three kinds of flicker can successfully evoke SSVEP, it is important to investigate the SSVEP differences that result from these different stimulators, and ascertain the type of flicker is most helpful in improving the accuracy of SSVEP-based BCI application. In the selection of the stimulating frequencies in a BCI application, one must ensure that the responses are as unique as possible. Thus, the stimulating frequencies are neither harmonics nor sub–harmonics from each other. From a practical point of view, the advantages of SSVEP BCI systems can be summarized as follows [34]:

• User is allowed to have small eye movements.

• User is capable of mild but sustained attention effort.

• User’s visual system is not engaged in other activities.

• Visual stimulation can be performed by usual equipment like computer display or LED panel.

• Command delays of 1-3 s are allowed.

4. Hybrid BCIs SSVEP-P300 hybrid BCIs

There are some obstacles for BCIs to be more applicable, such as reliability, BCI illiteracy [35], low ITR, and no satisfactory accuracy for all different subjects. In recent years, an extensive amount of work in BCI has been invested based on utilizing the combination of different types of BCI systems, or BCI and non-BCI, called hybrid BCI systems. Overcoming the limitations and disadvantages of the conventional BCI systems is the main goal of hybrid BCI. The focus and attraction toward hybrid BCI field has been extended in recent years. This is shown in Figure 10, based on the Scopus search engine [36], and the keyword ((“hybrid” AND (“BCI” OR “brain computer interface”))) and (“SSVEP” AND “P300”) and limited to “Engineering”, “Neuroscience” and “Computer Science” subject areas.

Figure 10.

The increasing research trend in hybrid BCI area.

In general, the BCI systems can be combined in the way that, each system has separate input signal or the output of one system would be the input of the second system. The systems are called sequentially and simultaneously hybrid BCI, respectively [37]. Figure 11 shows a general block diagram of a sequential and a simultaneous hybrid BCI system. In sequential hybrid BCI, the first system mostly acts as a switch [37]. For this task, one of the appropriate options is SSVEP. SSVEP has high classification accuracy; high information transfer rate does not need training.

Figure 11.

a) Sequential and (b) simultaneous hybrid BCI systems.

One of the main issues in this area is the optimum combination and selection of conventional BCIs. Several combinations of hybrid BCI systems have been introduced [37]. Conventional BCI systems are combined together based on the features of each system and the application of the hybrid BCI. If there are different tasks to be performed by the hybrid BCI, for each task, the more appropriate BCI can be chosen and, depending on the how the tasks are related to each other, the overall system can be combined. Some of the combinations for hybrid BCI that have been studied in recent years are shown in Table 1. Most of the studies in this area are focused on the combinations of BCI systems and few studies are on BCI and other physiological systems or devices.

Papers Numbers	Brain								Non-Brain(Non-Invasive)
Non-Invasive							Invasive
EEG				Non-EEG
P300	SSVEP	ERD	SCP	fMRI	NIRS	MEG	ECoG	EOG	EMG	ECG
[40]; [39]; [38]; [41]
[42]
[43]
[44]
[13]; [12]
[45]; [46]; [47]

Table 1.

The covered cells in each row have been introduced as hybrid BCI systems

In control applications, the BCI systems should be capable to cover multi-tasks. Hybrid BCIs has opened new opportunities for BCI systems to have more intense application in different areas. One of the areas in which hybrid BCI could play an important role is smart home control. There have been several studies done in this field [12][46]. The smart home or virtual environment systems are consisting of several stages and several control command types in each stage. In motor control systems, hybrid BCI shows improvement in accuracy and facilitates control tasks. Control commands, have different characteristics, and are divided to different types. Based on the characteristics, for each type, special types of BCI would be appropriate and for combination of control commands, two or more BCI types would fit. In discrete control commands, the task is the selection of one option from several options. P300 and Mu-Beta are more appropriate for this type of control commands. For series of the commands, continuous commands, ERD and SSVEP are more suitable. Also other features of the conventional BCIs should be considered, e.g., P300 is a slow responded system, but reliable. Mu-beta is fast responded, but not as efficient as P300.

The main enhancement that has been made by hybrid BCI is improvement in the applicability of BCI systems. As presenting two or more BCI type to the user, in the simultaneously combination, the user has the chance to get more efficient respond through utilizing the BCI type that is more appropriate for him or her. It also can decrease the fatigue, as the user can shift to another BCI option. It is shown that the accuracy is improved in the hybrid condition [42][43].

In [39], the hybrid BCI is introduced for Functional Electrical Stimulation (FES) control. Two tasks were considered to be implemented by the BCI systems; selecting one object among three objects and movement imagination to trigger FES. The high true positive rate (TPR) for SSVEP and ERD shows the capability of hybrid BCI for implementing several tasks that can be used in various control fields. SSVEP switch was introduced for smart home control [12]. The selection of control options, displayed on the screen is based on P300 BCI and SSVEP is operated as toggle switch. SSVEP was introduced as a switch for P300-based system [13]

Another BCI that can be introduced as a brain switch is ERS. In [41], post-imagery beta ERS-based brain switch was introduced for activating and deactivating the process of opening and closing the orthosis hand which was operated using SSVEP BCI.

More studies have focused on the simultaneous combinations of the conventional BCIs. As for one task, two or more BCI type are presented at the same time, the difficulty and complexity of performing the task is increased but, on the other hand, the accuracy in most of the cases increases for majority of the users. In addition, the fatigue may decrease, as users can switch between the BCI types that may be more comfortable for them. Another parameter is BCI illiteracy that can be decrease as users have the opportunity of accessing multi approaches [35]. The task of tracking the hint arrow was presented by SSVEP and ERD, and the accuracy was improved in the hybrid condition [38].

In another type of hybrid, more than one source of measurement is presented for one BCI type, for example, EEG and NIRS were acquired simultaneously for the ERD-based BCI [43] EEG and ECG were fused for motor imagery (MI) based BCI system[48]. EEG and EMG were utilized as hybrid in [44]. In this hybrid BCI, improvement in accuracy was shown. In some application areas, the tasks may be divided to two or more parts and each part is implemented by one BCI or non-BCI system. In this way, based on the features of the task, the system is selected. For example, in [42] control commands were divided to two parts and were implemented by EEG and electrooculography (EOG).

One of the issues in hybrid BCIs is that the system may be feasible but not optimum in all features. The hybrid BCI may improve the performance or accuracy but not compared to each of conventional BCI systems. For example, in the simultaneous combination of SSVEP and ERD, as in the conventional SSVEP, the accuracy is enhanced compared to ERD-based BCI system but not a lot changes compared to SSVEP.

P300 and SSVEP BCI were introduced as hybrid in an asynchronous BCI system in [13]. It seems that P300 and SSVEP combination works well as the stimuli for evoking both patterns can be shown on one screen simultaneously. The P300 paradigm considered in this study is a 6x6 speller matrix based on the original P300 row/column paradigm introduced by Farwell and Donchin [19]. Only one frequency is allocated for SSVEP paradigm. Background color was flashed with the frequency slightly less than 18 Hz. This facilitates the SSVEP detection. During the classification, P300 and SSVEP signals are separated by a band pass filter. The SSVEP is utilized as a control state (CS) detection, in the way that, when the user is gazing at the screen, the SSVEP is detected and it is assumed that the user intends to send a command. The system detects P300 target selection and CS simultaneously.

For SSVEP detection, the mean power spectral density (PSD) in the narrow band near the desired frequency and the PSD in the wider range near the desired frequency were utilized in an objective function (these values were subtracted from each other and divided over the PSD value from the wide band) and the function value was compared to a specified threshold. During the data acquisition, the channels for acquiring EEG signal were not fixed for all subjects. For P300 classification, FLDA or BLDA was utilized [14][15]. The experiment was presented as offline and online test. Ten subjects participated in the experiment. Subjects had training runs. In offline test, forty characters were presented for detection, divided to four groups. For better evaluation of SSVEP effect, two groups with and two groups without SSVEP were presented. In control state, subjects were instructed to count the number of time they distinguish the highlighted character. In non-control state (NCS), subjects were instructed to do a mental task like multiplication of two numbers and relax with closed eyes. For four out of five subjects, the accuracy was improved inconsiderably during the presence of SSVEP and P300 detection was not determinate. Between ten characters detection, there was a break and the time of the break depends on the time subjects pressed a keyboard button, and an auditory cue alerted about the finish of NCS time. The average classification accuracy of 96.5% and control state detection accuracy of 88% with the ITR of 20 bits/min were achieved during the offline test. The online test was presented under the semi synchronous condition. The experiment was consisted of blocks with 5 rounds, for detecting each character. SSVEP detection for at least three out of five runs showed the control state detection by the subject and P300 was detected during the control state. If the control state was not detected, the ‘=’ character was shown on the screen. The break time and the auditory alert was the same as offline test. The average control state detection accuracy of 88.15%, the classification accuracy of 94.44% and the ITR of 19.05 bits/min were achieved during the online test. P300 and SSVEP combination was also introduced to control smart home environment in [12]. P300-based BCI was used for controlling the virtual smart home environment and SSVEP was implemented as a switch for the P300 BCI operation. Results from this experiment show that P300 is suitable for discrete control commands and SSVEP is suitable for continuous control signals. The hybrid BCI achieved high accuracy and reliability in all subjects. In this chapter, P300, SSVEP and the hybrid P300 and SSVEP BCI systems were reviewed. The new trend and direction in BCI systems is to use new approaches in stimulating brain patterns such as hybrid BCIs while keeping the system complexity low and user acceptability high.