Hybrid Brain-Computer Interface Systems: Approaches, Features, and Trends

Abstract

Brain-computer interface (BCI) is an emerging field, and an increasing number of BCI research projects are being carried globally to interface computer with human using EEG for useful operations in both healthy and locked persons. Although several methods have been used to enhance the BCI performance in terms of signal processing, noise reduction, accuracy, information transfer rate, and user acceptability, the effective BCI system is still in the verge of development. So far, various modifications on single BCI systems as well as hybrid are done and the hybrid BCIs have shown increased but insufficient performance. Therefore, more efficient hybrid BCI models are still under the investigation by different research groups. In this review chapter, single BCI systems are briefly discussed and more detail discussions on hybrid BCIs, their modifications, operations, and performances with comparisons in terms of signal processing approaches, applications, limitations, and future scopes are presented.

1. Introduction

The spontaneous electrical currents in mammalian brain (rabbit and monkey) were first demonstrated by English Physiologist Richard Caton in 1870s, but the human electroencephalogram (EEG) was discovered in 1924 by German Psychiatrist Hans Berger [1]. The brain waves (neural oscillations) can be considered as biomarkers for wide range of applications from therapeutic to cognitive disorders [2]. The neural activities in brain generate voltages in response to external events or stimuli called event potential (EP). However, event-related desynchronization/synchronization (ERD/ERS) does not require such external stimulation. Interestingly, EP components can be subdivided into steady-state evoked potential (SSEP) and event-related potential (ERP), and ERD/ERS from motor imagination. Eventually, there are three main approaches employed by researchers to study electric signals generated from the brain activities. Following sections will elaborate discussion about these approaches.

1.1. P300 event-related potential

This event-related potential is a function of uncertainty of the external stimuli, and major changes in the positive amplitude of the EEG waveform appears at about 300 ms after the stimulus which is called P300 component of ERP, first used by Sutton et al. [3]. The P300 component of ERP was tested in human by Farwell and Donchin, and their experiment revealed that the rare event elicits P300 which can be used to develop mental prosthesis [4]. Farwell and Donchin proposed alphanumeric BCI speller consisting of 26 alphabets and 10 numbers (0–9) arranged in 6×6 matrix of rows and columns as shown in Figure 1a [4]. In this row-column (RC) paradigm, rows and columns are flashed randomly and the subject is asked to count the number of flashings of rows and columns corresponding to the target character. Flashed row/column containing target stimulus elicits P300 from parietal, occipital, and temporal regions (majorly in parietal) of the brain based on Oddball Paradigm, i.e., occurrence of rare (odd ball) event. The higher amplitude P300 is evoked from stimulus with higher strength and low probability (rare event). However, this paradigm suffers from low information transfer rate (ITR) due to multiple trials.

Figure 1.

(a) RC paradigm with second row flashing [4]. (b) and (c) RB paradigm where seven sets of characters in level 1 (b) is expanded in level 2 (c) to spell a character “B” [6].

Various changes in visual aspects of RC paradigm in terms of background color, character distance, and character size is done [5] to test the system performance. In this experiment, various visual protocols such as black background, white background, large symbol size, small symbol size, larger inter-symbol distance, and smaller inter-symbol distance are tested to observe the performance in RC BCI speller. Visual protocol with white background outperformed all the other protocols, while small symbol size resulted in poor performance.

A region-based (RB) BCI paradigm was designed by [6] to reduce human perpetual errors such as attentional blink, repetition blindness, habituation, and other spatial errors such as crowding effect and adjacency problems. Human perpetual errors in P300 speller was demonstrated by [7] to show the effect of adjacency problems. RB paradigm shown in Figure 1b and c uses seven regions flash at two levels instead of rows and columns. The region containing target character is selected at first level and the target is selected at second level which elicits P300. The number of characters in the RB speller is 49 and the probability of hitting a target is 1/7 which evokes higher amplitude of P300. Thus, accuracy, user acceptability, and ITR are enhanced in RB paradigm than traditional RC paradigm [6].

Only one character flashes in single character (SC) paradigm rather than all the six characters in row or column in RC paradigm [8]. In [9], SC paradigm was compared with RC paradigm for 19 subjects and observed that the classification accuracy was better for RC (85.3%) than SC (77.90%). Further, in [10], four P300 BCI spellers: RC, SC and two RB paradigms were compared, in which characters were based on alphabetical order in one and frequency of use in another. It was observed that accuracy of RB with characters in alphabetical order was highest and SC, the least for six subjects to spell two words WATER and LUCAS in three trials. In addition, whereas, user acceptability was highest for both RB paradigms than RC and SC, and region accuracy was least for central character on seven regions [6].

A checker board (CB) paradigm was proposed in [11], having 8×9 matrix of alphanumeric characters and keyboard commands, and compared the performance with traditional RC paradigm. Eighteen healthy subjects were used for the experiment and it was found that mean online accuracy, mean bit rate, and user acceptability were significantly higher for CB than RC but it suffers from adjacency errors. Other various modifications on standard RC paradigm have been done like a constant character flashing and shape changing which enhances the performances of P300 to some extent [12].

1.2. Steady-state visual evoked potential

The concept of visual evoked potential (VEP) was given by [13] using flash light and calculated evoked EEG signal by averaging to measure visual evoked responses from four parietal and occipital regions of scalp with bipolar electrodes. A clear high amplitude plot after 80 and 145 ms of the stimulus was found. VEPs, due to low stimulus rates, are classified as transient VEPs (TVEPs) and the repetitive high stimulations are under steady-state VEPs (SSVEPs). TVEP responses are during brain resting stage and if visual stimuli duration is shorter, evoked responses by each stimulus overlap each other and SSVEP is generated at steady state of brain excitation [14, 15].

SSVEP based on gaze detection falls into dependent BCI and is not suitable for ALS patients who cannot move their eyes. Gaze-independent SSVEP using LED interlaced square pattern for stimulation has been designed by [16]. People can shift attention among visual stimuli without shifting gaze, called as covert attention and overlapping stimuli can evoke changes in SSVEP which is sufficient to control BCI without gaze shifting like two overlapped images. Training for selective attention like playing certain types of computer games can improve SSVEP performance, and SSVEP systems are suitable to operate in challenging environments with distractions and noises like in homes and hospitals [17].

SSVEP visual stimuli are three main types as categorized below among which LED stimulation results in highest bit rate. All visual stimuli have properties like frequency, color, and contrast which affect SSVEP. Stimuli frequency can be divided into low (1–12 Hz), medium (12–30 Hz), and high (30–60 Hz) bands. Visual fatigues and false positives can occur at low frequency bands, whereas flash and pattern reversal stimuli can provoke epileptic seizures above low frequency bands. Red light has strong SSVEP response at 11 Hz but decreases at other frequency levels. However, the response decreases further for blue and yellow light. The three major types of visual stimuli for SSVEP are categorized as follows [18].

• Light stimuli: light sources are LEDs, fluorescent lamps, Xe lights, etc., and their intensity is measured in candela per sq. meter.

• Single graphics stimuli: rectangle, square, or arrow on computer screen that appear and disappear at specific rate and stimulation rate are the number of full cycles per second called frequency of stimulus.

• Pattern reversal stimuli: periodic alternation of graphical patterns are usually black and white such as line boxes, checkerboards, etc., on computer screen.

The effect of visual distractions in SSVEP is dependent on the level of attention requirement during the task and the nature of distractions. SSVEP amplitude and identification accuracy decreases in dynamic screen condition compared to static condition [19]. Visual stimuli with a frequency resolution of 0.1 Hz were classified with high accuracy sufficient for practical BCI and the factors affecting the SSVEP speller are distance between adjacent stimuli, light source arrangements, stimulating frequencies, electrode arrangements, and visual angles [20].

The frequency response of SSVEP is experimented in [21] using visual stimulation at 14 different frequencies within the range of 5–60 Hz and found that the primary visual cortex follows an activation pattern similar to SSVEP and the SSVEP amplitude peaks at 15 Hz stimulation shown in Figure 2.

Figure 2.

Variation of SSVEP amplitude with respect to stimulus frequency [21].

SSVEP response not only has the same fundamental frequency as stimulus but also includes higher harmonics and use of three SSVEP harmonics has resulted higher classification accuracy than for one or two harmonics [22]. SSVEP-based BCI has many advantages over other EEG-based BCI systems due to the following reasons [16].

• high signal-to-noise ratio

• high information transfer rate

• less susceptibility to eye movements and blink artifacts

• require very little or no training

Asynchronous SSVEP-based BCI using flickering lights was used to control neuro-prosthetic devices for restoration of grasp function in spinal cord injured people [23] and as a functional electrical stimulation for abdominal stimulation to augment respiration in tetraplegia [24]. An emergency call system using SSVEP-based brain switch was developed for ALS patients and they successfully called their guardians by simply starring at stimulus in about 6.56 s, starting the experiment by themselves. This system had fairly good performance when experimented up to 4 weeks. A chromatic visual stimulus with isoluminant red color is used to reduce intensity of the stimulus [25]. SSVEP-based BCI using single flicker stimulus is coded spatially to control four channels for navigation of 2-D computer games. Control channels are coded by their spatial position rather than flickering frequency or phase which may provide alternative route toward a practical SSVEP BCI with reduced visual strain [26]. To reduce visual fatigue from traditional SSVEP using flickering lights, an equal-luminance, ring-shaped, red-green colored checkerboard paradigm is used which produces high SSVEP power around 15 Hz [27]. Most people, despite no prior BCI experience, can use SSVEP BCI system even in a very noisy environment and better performances is observed in young and female subjects [28].

1.3. Motor imagery

Sensorimotor rhythms (SMRs) are synchronized brain waves over sensorimotor cortex in three different frequency bands: μ (8–12 Hz), β (18–30 Hz), and γ (30–200 Hz). EEG recording is mostly limited to μ and β bands. SMR amplitude is higher during idle stage called as event-related synchronization (ERS) and the amplitude decreases when the sensorimotor areas are active due to a certain motor task or even during motor imagery (MI). This decrease in SMR amplitude is called event-related desynchronization (ERD). The ERD signal is used for MI-related BCI. ERS immediately occurs after ERD [29]. For MI tasks, the subjects are instructed to imagine themselves performing a specific motor action without actual motor output and there exists contralateral lateralization of left-hand/right-hand/foot [30].

A novel typewriter “Hex-O-Spell” was presented in [31] using imagined right-hand and right foot movements shown in Figure 3. Five letters or symbols are inside six hexagons surrounding a circle having center arrow. Imagination of right-hand movement turns arrow clockwise and imagination of right foot movement stops the rotation and arrow extends to select a character if the imagination persists longer. A synchronous MI-based “Oct-O-Spell” paradigm is designed by [32] using 2-D cursor control with simultaneous MI tasks and claimed to be feasible with higher efficiency.

Figure 3.

Two states of “Hex-O-Spell” paradigm selecting a character using MI [31].

MI detection is challenging due to low signal-to-noise ratio, but development of advance signal processing enables MI-based BCI to implement various tasks [33]. MI-based BCI was used first time by [34] for stroke rehabilitation in a tetraplegic patient using imagination of foot movement where the patient was able to grasp cylinder with the paralyzed hand.

MI-based BCI is a system that is subject specific and requires data recording and a system training for each new user. Subject-independent MI was developed by training the data acquired from several subjects [35] and a conscious target strengthens ERD in β frequency band [36]. ERD amplitude was higher due to body ownership illusion like moving rubber hand than other visual targets [37].

MI activity acts as a neurofeedback and a feasible part of stroke rehabilitation but may increase moderate fatigue due to external factors like long hours of training session [38]. Neural plasticity can be achieved through neurofeedback [38, 39]. MI-based BCI uses a neurofeedback strategy in poststroke rehabilitation using functional electrical stimulation (FES), robot, and orthosis [40]. Majority of stroke patients can use EEG-based MI [41, 42] for limb rehabilitation [43] and was extended to imagination of tongue movement [44]. MI can be used for a reliable and high performance BCI for both healthy subjects and ALS patients where the user requires less trainings [45]. MI-based BCI can be used for stroke rehabilitation to perform various functions such as controlling computer cursor, processing word, accessing Internet, and controlling environment and entertainment [33]. Without any muscular activities, MI tasks were employed in an experiment to drive a car in 3-D virtual environment [46] and to play video game on virtual ground [47].

There are other methods apart from EEG to measure brain activities such as magneto-encephalography (MEG), electro-corticography (ECoG), functional magnetic resonance imaging (fMRI), and functional near-infrared imaging (fNIR). However, due to noninvasive method, easy experimental setup, low cost, and high efficiency, EEG is most widely used. Although P300, SSVEP, and ERD/ERS are most widely used EEG signals, there are also other brain signals such as slow cortical potentials (SCP) and electrooculogram (EOG) in BCI [29]. Each of these brain signals do not work same for all users. So, a novel approach has been used to combine two or more conventional BCIs to form a hybrid BCI to enhance the overall performance [48].

2. Hybrid BCI and modes of operation

The initial concept of hybrid BCI was used in [49] to incorporate electrocardiogram (ECG) with EEG for autonomous BCI switch ON and OFF operation to analyze whether heart bit rate can be used as an additional communication channel in BCI. P300 was combined to μ and β rhythms from sensorimotor cortex to operate a brain-controlled wheelchair [50]. In [51], hybrid P300/SSVEP system was compared with conventional P300 and SSVEP BCI from 10 healthy subjects and observed improved performance relative to single SSVEP system and the user acceptability was higher for the hybrid which suggested the need for efficient future protocols. A continuous simultaneous hybrid BCI for two dimensional cursor control was introduced in [52] using ERD and SSVEP activity, in which vertical position of the cursor was controlled via ERD with imagined movement and the horizontal position with SSVEP from visual attention, and the overall result suggested that further research is needed to optimize hybrid BCI.

In [53], hybrid BCI systems were reviewed and different possibilities to combine their advantages and disadvantages were discussed. Hybrid P300/SSVEP was used by [54] for GO/STOP command in wheelchair control at simultaneous asynchronous mode and obtained improved performance in terms of detection accuracy and response time. A novel hybrid P300/SSVEP was designed by [55] integrating random flashing and periodic flickering to reduce adjacency problem and habitual repetition, and obtained an online classification accuracy of 93.85% and information transfer rate of 56.44 bit/min from 12 healthy subjects in a single trial. A new hybrid P300/SSVEP was proposed in [56] based on visual approach of shape changing instead of existing color changing and compared the performances with traditional P300, SSVEP, and normal P300/SSVEP hybrid. The new hybrid BCI was compared with normal hybrid and each traditional BCIs, and found better performance with 100% accuracy and 30 bit/min ITR for eight trials with 10 healthy subjects.

A systematic review of hybrid BCI was done by [57] in terms of taxonomy and usability. This review discussed two modes of operation: simultaneous and sequential modes. In simultaneous mode, any two BCI systems (e.g., P300 and SSVEP) work simultaneously controlling two functions at a time and this combined system might achieve higher accuracy and ITR. As explained previously in [52], the hybrid BCI used simultaneous mode which includes ERD (imagined movement) to control the cursor in vertical position and SSVEP to control the cursor in horizontal position. In sequential mode, output of one BCI system is used as the input for another to control various functions of the second BCI system or as a switch in asynchronous mode [57]. These two modes are depicted in Figure 4a and b.

Figure 4.

Hybrid BCI modes of operation [57].

Among all other EEG signals, SSVEP possess a better suitability to combine with P300 [58] for constructing efficient hybrid BCI due to the following reasons [55]:

• SSVEP and P300 both are elicited by visual stimuli, so subjects only need visual attention.

• Both are noninvasive so reduction in experimental setup time, complexity, effort, and cost.

• No mental count is required for SSVEP thus reducing the mind workload.

• Both are measured in different domains (time domain for P300 and frequency domain for SSVEP).

• Both are detected from almost different cranial regions with independency enabling higher accuracy.

The research on hybrid BCI is growing and Figures 5 and 6 illustrate number of publications in this field. The two figures are based on searches at IEEE Xplore [59] and PubMed [60] with keywords: “BCI,” “Hybrid BCI,” “SSVEP and MI,“ “P300 and SSVEP,” and “P300 and MI “published in Conference, Journals, Magazines, Books, and e-books in the fields of “Engineering,” “Psychology,” “Neuroscience,” “Medicine,” and “Computer Science.”

Figure 5.

BCI and hybrid BCI papers in IEEE and PubMed.

Figure 6.

Hybrid BCI papers in IEEE and PubMed together.

Figures 5 and 6 illustrate the number of published articles in IEEE Xplore and PubMed which are added together. They depict the growing numbers of research in hybrid BCI and among hybrid BCIs, number of P300 and SSVEP hybrid is comparatively higher as illustrated in Figure 6.

3. Hybrid BCI classification

The most common signals for BCI are P300, SSVEP, and ERD, and there are various approaches used to combine two or more these signals to make a hybrid to enhance performance. The most common methods for hybrid BCI are discussed below and their classifications based on various parameters are illustrated in Table 1.

Hybrid type	Subj.	Modes of operation	Classifiers	Acc. (%)	ITR (bits/min)	Improvements	Reference
P300 + SSVEP	8	Simultaneous	SVM and DFT	90	22	“Go/stop” control signal with higher accuracy	[54]
P300 + SSVEP	12	Simultaneous	SWLDA and CCA	93	56	Classification accuracy and ITR	[55]
SSVEP + MI	14	Simultaneous	LDA	81	—	Reduction in BCI illiteracy	[61]
SSVEP + MI	12	Simultaneous	CCA	80	15	False activations reduction with binary decision	[62]
SSVEP + MI	3	Simultaneous	SVM & CCA	90	295	Time reduction with higher reliability	[63]
SSVEP + MI	6	Sequential	Filter and Threshold	78	—	Improved performance over conventional FET	[64]
SSVEP + MI	24	Sequential	CSP & CCA	87	—	Enhanced MI performance	[65]
SSVEP + MI	17	Simultaneous	LDA	85	—	Improved performances with single channel	[66]
P300 + SSVEP	10	Simultaneous	BLDA and CCA	90	22	Improved performance and easiness for users	[56]
P300 + SSVEP	10	Simultaneous	SWLDA and CCA	93	31	Reduced stimulation time for P300 and improved ITR	[67]
P300 + SSVEP	10	Sequential	FLDA and BLDA	88	19	Improved classification accuracy and ITR	[68]
P300 + SSVEP	12	Simultaneous	SWLDA	93	34	Transformed frequency locked SSVEP to time locked in a single speller	[69]
P300 + SSVEP	10	Simultaneous	BLDA and CCA	83	12	Improved SSVEP but not P300	[51]
P300 + MI	4	Sequential	SVM and FLDA	82	—	Finished more complex tasks in virtual environment	[70]
P300 + MI	4	Sequential	FLDA	82	—	Less exhaustive and reliable control of robotic devices	[71]
P300 + MI	11	Sequential	SVM	93	—	Enhanced accuracy and lowers trial duration	[72]
P300 + MI	11	Sequential	SWLDA and CSP	92	41	Control applications	[73]
P300 + MI	12	Simultaneous	LDA and CSP	92	—	Enhanced discriminating performances	[74]

Table 1.

Different hybrid BCI systems’ descriptions based on a number of subjects, modes of operation, classifiers used, accuracy, ITR, and improvements in the model.

3.1. SSVEP-MI hybrid

SSVEP and ERD signals are used to form a BCI hybrid that combines visual attention and motor imagination. In [61], 14 healthy subjects (six women and eight men of ages 17–31 years) were chosen to perform three different tasks: MI only (ERD signals generated from left-hand or right-hand movement), SSVEP only (visual signals generated from two flickering LEDs at 8 Hz and 13 Hz), and simultaneous hybrid SSVEP-MI. Linear discriminant classifier was used and the classification accuracy was higher for SSVEP than MI and was highest for the hybrid.

An artificial upper limb was controlled by [62] combining SSVEP and MI in two degrees of freedom (DoF) in which MI controlled grasp function and SSVEP controlled elbow function (flexion and extension) of the limb. The experiment was conducted with 12 healthy subjects (7 male and 5 female) in offline and 7 healthy subjects (4 male and 3 female) in online. In offline experiment, 4 runs each with 40 trials were taken and the subjects were instructed to imagine feet dorsiflexion from two to four runs focusing the two flickering lights 7 and 13 Hz. The online experiment consisted a 2 DoF artificial upper limb and subjects controlled grasp and elbow functions as per instructions.

In [63], SSVEP-MI hybrid was proposed to control the speed (accelerate or deaccelerate the wheelchair based on flashing stimuli of 7, 8, 9, and 11 Hz) and direction (left- and right-hand imageries to control the direction) of a wheelchair in real time. Both virtual and real-time tests were conducted to observe the performance. Three options: straight driving, left and right turns were provided for direction, and accelerate, deaccelerate, or drive options for speed control using eight separate commands: turn left, turn right, drive forward, accelerate, deaccelerate, drive at uniform speed, and turn on or off the switch. Three healthy male subjects (21–30 years old) were participated in the experiment and the classification accuracy was more than 90% using Support Vector Machine (SVM) for MI and canonical correlation analysis (CCA) for SSVEP.

The sequential operation of SSVEP and ERD signals was used in [64] for advanced functional electrical therapy (FET) in which six right-handed healthy subjects (5 males and 1 female, mean age around 25 years) were selected. SSVEP signals from flickering LEDs of frequencies 15, 17, 19, and 21 Hz were used to select the types of grasp: palmar, lateral, and pinch followed by MI which was used as a brain switch that activated the intention of grasp.

In [65], hybrid BCI paradigm was proposed to enhance MI tasks using SSVEP. Twenty-four right-handed healthy subjects aged 23–30 years (19 males and 5 females) were used for the experiment to perform MI focusing on flickering SSVEP, where SSVEP was used initially to train the subjects for MI tasks providing accurate feedback, and afterward, the weight was shifted to MI gradually keeping SSVEP at less weights. Common spatial pattern (CSP) was used for MI and CCA for SSVEP classifications. This paradigm hypothesized that accurate feedback enhances MI.

The multiple channels hybrid was replaced in [66] combining SSVEP and MI in a single channel C3 or C4 improving performance. Seventeen healthy subjects (12 male and 5 female subjects with an average age around 23 years) were preinformed about the experiment to focus simultaneously on flickering visual stimuli of frequencies 15 and 20 Hz, and perform right- and left-hand MI, respectively. The average classification accuracy was around 85% for both channels.

3.2. P300-SSVEP hybrid

An asynchronous control of wheelchair was proposed by [54] combining SSVEP and P300 in which four groups of buttons were displayed, each group having one large central button surrounding eight small buttons with 45° spacing in a circumference of 60 mm radius. The four groups flickered at frequencies 6, 6.67, 7.5, and 8.57 Hz to evoke SSVEP, while 100 ms interval was used to intensify and change the color of large central button to elicit P300. SVM and discrete Fourier transform (DFT) were used for classification of P300 and SSVEP, respectively, with an overall classification accuracy of about 90% from eight healthy subjects (20–31 years) performing “go/stop” control task using a real wheelchair.

A hybrid SSVEP-P300 BCI was proposed by [55] as mentioned earlier to improve spelling accuracy combining random flashing and periodic flickering to evoke P300 and SSVEP, respectively. All the cells of 6x6 matrix were flickered on black background with six frequencies 8.18, 8.97, 9.98, 11.23, 12.85, and 14.99 Hz, and selection of these frequencies were based on the higher SSVEP amplitude and easier target detection while orange crosses were flashed for 120 ms in random manner. Twelve healthy subjects (5 male and 7 female subjects, age 21–29 years) with good visions were used and the performance of the hybrid system was evaluated online using single trial. This experiment claimed to have the best performance.

In [56], a BCI with shape changing and flickering speller was designed, rather than traditional color changing as in [67], combining SSVEP and P300 in which the classification accuracy was improved in each of the individual systems. Shape changing of red boxes to arrows was used for P300 and flickering of those four boxes with frequencies 6, 8, 9, and 10 Hz for SSVEP. Ten healthy right-handed subjects with normal vision (9 male and 1 female subjects, age 22–27 years) were used for five offline experimental sessions having 20 runs of each sessions lasting for 4 s so the one session was 40 min including 10 min rest. The subjects found the new hybrid less annoying.

An asynchronous hybrid BCI combining P300 and SSVEP was proposed by [68] where the information transfer and control state (CS) detection was accomplished using P300 and SSVEP, respectively. This system operated in sequential manner in both offline and online experiments. Ten healthy subjects (7 males and 3 females aged 19–28 years) were participated in both experiments where P300 was elicited from flashing of a 6x6 matrix of 36 characters (A-Z and 0–9), and SSVEP was obtained from flickering of the characters black and white alternatively with frequency around 17.7 Hz. Two classifiers: Fisher’s linear discriminant analysis (FLDA) for first three subjects and Bayesian linear discriminant analysis (BLDA) for remaining subjects were used for P300 classification. Inclusion of SSVEP into P300 improved the classification accuracy.

In [69], same target stimulus was used to elicit P300 and SSVEP blocking (SSVEP-B). A new speller of 3x3 matrix with characters 1–9 was proposed while all the characters flickered at a constant repetitive rate, and each character stopped flickering randomly within a short time. The repetitive flickering elicited SSVEP and the interrupted flickering of a character would elicit SSVEP-B and P300, a rare event at the same time which was detected simultaneously. Twelve right-handed healthy subjects (7 male and 5 female subjects, age 23–36 years) were participated in the experiment. The size and font of characters were changed with a variance of 0.49 ms for event-related potential, and brightness altered between light and dark with about 14.96 Hz for evoked potential. Stepwise linear discriminant analysis (SWLDA) was used for classification and the hybrid system produced better result than the individual systems.

In [51], SSVEP and P300 were combinedly evoked to improve accuracy using four red boxes with black background participating 10 healthy male subjects of age 21–25 years for three runs of 80 trials each in pseudorandom order. P300 was elicited by counting 32 flashings of the four red boxes for 8 times in each trial in the order: top, down, left, and right, and box changed from red to white for 100 ms during each flash while SSVEP was evoked by focusing on the target box which flickered at particular frequencies 9, 6, 10, and 8 Hz for 4 sec in the same order as of flashing, and the subjects needed to count the flashes and focus on flickering simultaneously for the hybrid paradigm. The P300 and SSVEP signals were analyzed separately and the average performances are mentioned in Table 1. This experiment highlighted the need for efficient hybrid as the hybrid performances improved relative to SSVEP but not to P300, and the subjects were comfortable with the hybrid.

A hybrid SSVEP-P300 BCI generating dual-frequency SSVEP was proposed by [67] using 9 panels speller, each panel with 4 characters flickering at different frequencies, and flickering panel and periodically updating characters evoked dual-frequency SSVEP. Ten graduate students (8 male, 2 female subjects, and average age 26 years) participated in both the offline and online experiments and there were improvements in stimulation time and ITR.

3.3. P300-MI hybrid

A hybrid P300-MI was used in [70] to control appliances in a virtual environment, in which P300 was used as a device control to operate control panel of virtual devices and MI as a navigation tool to turn left/right in the virtual environment, and the detection was sequential based on the activation of either MI or P300. Four healthy adults (1 male, 3 female subjects, age 23–25 years) participated and P300 was elicited using oddball paradigm, and MI using left-hand/right-hand movement imagination. The average online classification accuracy was achieved around 82% and authors claimed that more complex tasks in virtual environment could be performed compared to single pattern BCI.

The possibility of combining various brain signals for hybrid BCI was discussed by [71] merging P300 and ERD to control a robotic device such that additional features of one system could be used to improve classification accuracy of another. In this case, object selection from various options, called discrete decision, was done using P300 and movement control of the robot was done using ERD from motor imagination. EEG was filtered to a band 0–10 Hz for P300 classification. Optimum filter band varied with subject for ERD and the filter band was obtained from training data. Principal component analysis (PCA) and CSP was used in feature extraction for P300 and MI, respectively, and the discriminant classification was done by FLDA. Four healthy subjects were experimented with at least three MI trainings before the experiment by each subject with tasks: one out of five P300 symbols (1–5) and one out of two MI hand movements (left/right). Each subject performed one experimental session consisting of 60 trials, 30 trials for P300 in which targets were 1–5 numbers random flashings, and remaining 30 trials for MI where subjects needed to focus only one either on P300 or MI during each trial. Hybrid classification accuracy achieved was about 82% with average MI classification accuracy of 71%.

The selection of a target based on hybrid P300-MI BCI was developed by [72] on 1166×721 pixels monitor using two sequential tasks: cursor movement with the help of ERD from left-/right-hand movement imaginations and selection/rejection of the target from focused attention on flashing to evoke P300 keeping one system passive when other is active. Eleven healthy subjects (10 male, 1 female subjects, and age 22–32 years) were used in both online and offline analysis and each experiment was performed within 2 s with each trial duration for about 18 s. Two targets such as Green Square for selection and Blue Square for rejection were randomly appeared on the screen and the classification accuracy was about 93% in real time.

MI-based brain switch was merged with P300 sequentially by [73] in which right-/left-hand movement imaginations was used for control signal acting as a brain switch to turn ON/OFF P300 speller. Eleven healthy subjects (8 male, 3 female subjects, and age 23–30 years), with a few subjects having previous experience to P300 or MI, were experimented. Offline training was done for P300 and MI before real tests in a 22″ LED monitor containing 6×7 matrix of 26 English letters, 10 numbers, a few special characters, and commands. SWLDA was used for P300 classification and filtration in the alpha-beta rhythm range with two bands discrimination using CSP for MI and the classification accuracy achieved was about 92% for P300.

An efficient approach was proposed by [74] combining MI with P300 in a block diagonal matrix form to improve classification accuracy concatenating features of each paradigm where first-order information was used for P300 and second order for MI. Twelve volunteers (10 male, 2 female subjects, age 22–35 years) were experimented provided with eight flashing buttons and an arrow cue such that screen remained blank for initial 2.25 s and a cross appeared on the screen from 2.25 to 4 s to attract subject’s visual attention. Arrow cue was displayed from 4 to 8 s such that up arrow was for P300 task to focus on the random flashing buttons without MI tasks and right arrow for MI task to make left-/right-hand imaginations. Test data was obtained from second experimental session and the classification accuracy was about 92%.

4. Signal processing approaches

The signal processing steps in BCI are signal acquisition, preprocessing, feature extraction, feature selection, feature classification, and postprocessing. The common classification approaches in hybrid BCI are briefly discussed here.

4.1. Fast Fourier transform (FFT)

FFT is an efficient algorithm for calculation of DFT reducing order of $N^2$ operations to $N{\text{log}}_{2}N$ that decomposes signal from time domain to its individual frequency components whose pairs are given as [75]:

$$X\left(k\right)={\displaystyle \sum _{n=0}^{N-1}}x\left(n\right)W_N^{\mathit{kn}}$$E1

$$x\left(n\right)=\frac{1}{N}{\displaystyle \sum _{k=0}^{N-1}}X\left(k\right)W_N^{-\mathit{kn}}$$E2

where $W_N=e^{-j\left(\frac{2\pi }{N}\right)}$ and N = length [x(n)].

Amplitude vs. frequency is plotted and the dominant amplitude at a particular frequency is obtained for SSVEP signal analysis in BCI. FFT classifier has been replaced by other better classifiers in BCI research.

4.2. Linear discriminant analysis (LDA)

LDA, also known as Fisher’s LDA (FLDA), is a useful classification technique that transforms features into a low-dimensional space with high degree of separation. Suppose, there are a certain set of samples belonging to classes “A” and “B” whose mean and scatterings within each classes are represented as ${\mu }_A$, ${\mu }_B$ and $S_A^2$, $S_B^2$, respectively, and LDA is calculated as [76].

$$\mathit{LDA}=\frac{{\left({\mu }_A-{\mu }_B\right)}^2}{S_A^2+S_B^2}$$E3

This method is simple to use, has low computational cost with high accuracy, and is widely used in number of BCI applications for P300 and MI-related tasks [77]; but, it also suffers from small sample size and linearity problems [78]. BLDA, a Bayesian version of LDA, is useful for P300 in which parameters are estimated with Bayesian regression whose probabilistic output value lies between zero and one. It gives good results for small number of train sets or noise contaminated data and may give poor result for a complex nonlinear EEG data [78, 79]. SWLDA is a stepwise LDA that performs space reduction by selecting suitable features and stepwise analysis to remove least significant features which is an effective classification technique for online classification of P300 [80].

4.3. Support vector machine (SVM)

SVM is used for classification of linearly separable binary data sets that uses a discriminant hyperplane to identify classes and the best choice is the hyperplane that leaves maximum margin from both classes. The kernel generally used in P300 BCI is the Gaussian kernel and the corresponding SVM is Gaussian SVM which is given as:

$$\mathrm{K}\left(\mathrm{x},\mathrm{y}\right)=\text{exp}\left(\frac{-{\left|\left|x-y\right|\right|}^2}{2{\sigma }^2}\right)$$E4

where K(x, y) is kernel function and σ is kernel width.

SVM is simple and stable, and has a low variance which may be a key for low classification error for unstable and noisy P300 signals. SVM produces good results with high-dimension feature vectors and a small training set, thus outperforming LDA, but are generally slower than other classifiers [79]. BLDA is most robust for P300 application compared with LDA and SVM [81].

4.4. Canonical correlation analysis (CCA)

CCA is a multivariate statistical method to analyze frequency components of SSVEP in EEG [82]. It extracts narrowband frequency components of SSVEP in EEG using maximum correlation between reference stimulus signals and EEG signals. Suppose X be the EEG all channels data and $Y_f$ be the reference signals at f Hz stimulus frequency with N number of harmonics, the reference signals $Y_f$ are given as:

$$Y_f=\left(\begin{array}{c}\text{sin}\left(2\mathit{\pi ft}\right)\\ \text{cos}\left(2\mathit{\pi ft}\right)\\ .\\ .\\ .\\ \text{sin}\left(2\mathit{\text{πNft}}\right)\\ \text{cos}\left(2\mathit{\text{πNft}}\right)\end{array}\right)$$E5

The maximum correlation among X and $Y_f$ is obtained as:

$${⍴}_{\mathit{max}}=\text{max}\left[\mathit{\text{correlation coefficient}}\left(X,Y_f\right)\right]$$E6

CCA is more common method for analysis of SSVEP signals in frequency domain that improves SNR, classification accuracy, and ITR [81, 83].

4.5. Common spatial patterns (CSPs)

CSP is used to analyze spatial patterns of MI calculating spatial filters to find optimum variances for two different classes of EEG data. It uses simultaneous diagonalization of two covariance matrices, and the spatially filtered signal Z of a single-trial EEG data is obtained as:

$$Z=\mathit{\text{WE}}$$E7

where E is $N\times T$ matrix of single-trial raw EEG data, N is the number of channels, T is the number of measurement samples per channel, and W is CSP projection matrix. The rows of W are stationary spatial filters and the columns of $W^{-1}$ are common spatial patterns. Spatial patterns of motor action are dependent on the specific region of brain like left-hand movement on right cerebral hemisphere [84]. A higher classification accuracy for multitask learning with very few training samples among 19 healthy subjects was achieved by [85] in which spatial filter was replaced by Laplacian filter in CSP algorithm.

5. Applications

Hybrid BCI is in the state of development and various BCI signals are combined to form a hybrid enhancing performance for numerous experimental-related applications which are summarized in Table 2. Most of the applications are based on wheelchair control. Other applications include use of computer and communication, prosthetics using artificial limbs, advanced functional electrical therapy, monitoring ALS patients, entertainment and care in virtual smart home where MI is used mostly in prosthetics. Although BCI application is potentially safe, it needs regulatory approval before the experiment.

Application type	Specific control	Hybrid type	References
Wheelchair	Direction	P300 + MI	[50]
	“Go” and “Stop” movement	P300 + SSVEP	[54]
	Speed	SSVEP + MI	[65]
	Multi-degree	SSVEP + MI	[86]
	Speed and direction	SSVEP + MI	[63]
		P300 + MI	[87, 88]
	Autonomous navigation	P300 + MI	[89]
Computer cursor	2-D	SSVEP + MI	[52, 90]
		P300 + MI	[72]
Speller	Spelling accuracy	P300 + SSVEP	[55, 68]
		P300 + MI	[71, 73]
Artificial limb	Upper limb	SSVEP + MI	[62, 64]
	Multidimensional robotic arm		[91]
Functional electrical therapy	Advanced	SSVEP + MI	[66]
ALS patients	Communication	P300 + MI	[92]
	Awareness	P300 + SSVEP	[93]
Virtual environment	Smart home	P300 + SSVEP	[94, 95]
	Intelligent nursing bed		[96]

Table 2.

Hybrid BCI applications.

6. Limitations

Phenomenon of acquiring and processing information by human brain is still unknown. A very few hybrid BCI are experimented with target users and most of the subjects are healthy with small sample size. Rehabilitation using BCI is still not used in clinical practice [97]. Various methods have been incorporated to improve accuracy and ITR, and some hybrid with different classifiers combination have shown some improved results, but mostly subject’s specific. Type and design of electrodes have impact on subject’s head which influence EEG signals and demands for high compatible systems. These systems are not free from physical and mental fatigue that challenges their adaptability. Moreover, there are obstacles in EEG acquisition due to electrodes placement, skull muscle movement, environment noises, limitations in hardware, and their calibrations. Two or more tasks need to be performed simultaneously in hybrid that might increase mental workload and cause discomfort to some users. Due to complexity, prior knowledge is required to use hybrid systems for target users. This demands for further research and hybrid BCI is still under development inside laboratory [57, 70].

7. Future scopes

Hybrid BCI has wider future scope and combining three or more signals may result better performances. Optimum combination of signals with high degree of compatibility may be obtained which is accessible to all [98]. Virtual environment-controlled applications [72] may turn to a real one which may provide easy access to target as well as healthy users. These applications may be broadened to people without disabilities too. Various researches are going on to calculate mental workload of armed soldiers, and brain automation control of wheelchair may be extended to control automobiles and airplanes [99]. Efficient algorithms need to be developed in future to make BCI practical with high accuracy and speed which act as a neurorehabilitation for stroke patients suffering from movement and language deficits. Human’s intentions, emotions, and behaviors might be predicted in future using EEG which will ease for identifying fatigue in soldiers during war. It might be used in children to study various psychological measures such as behavior and learning tendency relative to age, and can be extended to animals besides human [97]. The laboratory experiment may be extended to the real world to ease our daily lives. Eventually, these might attract stakeholders to invest in BCI industry to produce commercialized BCI products in future.

8. Discussion and conclusion

Brain is a self-sustained oscillator where individual neurons oscillate at certain harmonics. Major rhythms of motor outputs generate through bifurcation. Several linear (spectral coherence and cross-correlation) and nonlinear (phase synchronization, mutual information and entropy) measures have been adopted to measure the oscillations [100]. Structural and functional connectivity of the brain works in coherence to perform a common action. Structural connectivity relates to the physical connection between different regions of the brain, while functional connectivity is the correlation between various regions over time that shows dynamic behavior [101].

During cognitive tasks requiring attention, certain brain regions become more active while the other regions activity decreases. A flashing or flickering visual stimulus eliciting event or evoked potential (P300 or SSVEP) increases activity in frontal and visual cortex. Due to more repetitive mental tasks, brain activity increases in the specific region, whereas activity in the other regions is reduced. The reason for reduction may be due to unrelated or difficult tasks [102]. This increase in brain activity corroborates growth in working memory of brain illustrating brain dynamic states. Brain changes its state according to the environment similar to an artificially intelligent machine which adapts to learn from the input attributes without being explicitly programmed. So, a BCI illiterate at one point of time may adapt to learn with continuous trials due to dynamic brain states [103].

Human brain is a nonlinear dynamic system behaving as a chaotic and fractal system [104]. Therefore, EEG is a complex, nonlinear, and nonstationary signal. However, EEG signals have been analyzed based on linear/nonlinear and stationary/nonstationary techniques for feature extraction and classification. Fourier transforms, wavelet decomposition, power spectral density, autoregression, CCA, LDA, SVM, and CSP are some of the linear methods for EEG classification. However, only commonly used linear classifiers in hybrid BCI are discussed here. Due to the dynamical nature of the brain and the associated EEG signal, widely used linear approaches are not enough to obtain promising results. Therefore, the nonlinear dynamical behavior of EEG should be carefully considered during brain signal analysis. EEG signals need to be analyzed along with the dynamic states to reveal additional features that cannot be assessed with the linear methods.

EEG signals were analyzed dynamically in [105] to identify and code the attractors related to mental states using artificial neural network. It was shown that binary patterns of attractors resulting from neural firing of identical cognitive or sensory stimuli are similar, but they might appear as distinct features with different stimuli. The chaotic behavior of attractors highlights the fact that the neural signals are coherent. Indeed, brain dynamics has unveiled an emerging area of research to quantify information using attractors and fractals from EEG signals for useful operations applying hybrid BCI. It should be noted that attractors and fractals are the dynamic variables to measure complexity (correlation dimension and Hurst exponent) and stability (Lyapunov exponent and entropy) of EEG data [106, 107, 108]. The phenomenon of brain receiving the sensory inputs, storing the information, and processing output is still unknown. Hybrid BCI can be an efficient tool to transform the interesting brain dynamics into actions.

In this chapter, hybrid BCI is reviewed, and advancements from single BCI system to hybrid BCI systems, associated signal processing methods, usages, shortcomings and future scopes are discussed. The common hybrid systems based on signal combinations as well as operation methods, their performances, and improvements are mentioned. Statistical analysis of BCI and hybrid BCI related to P300 and SSVEP are illustrated based on publications. Transitioning from laboratory to the possible commercial applications is well discussed along with the limitations onward. This review illustrates P300, SSVEP, and MI which are mostly used EEG signals for BCI. Simultaneous operation is very common in P300-SSVEP hybrid and sequential operations are incorporated mostly in MI-related hybrid experiments. Average accuracy and ITR among reviewed hybrid BCI papers are 90% and 28 bits/min, respectively, which demand the need for a more efficient hybrid BCI system.