Brain-actuated Control of Robot Navigation

1. Introduction

Brain-driven robot navigation control is a new field stemming from recent successes in brain interfaces. Broadly speaking, brain interfaces comprise any system that aims to enable user control of a device based on brain activity-related signals, be them conscious or unconscious, voluntary or evoked, invasive or non-invasive. Strictly speaking, the term should also include technology that directly affects brains states (e.g., transcranial magnetic stimulation), but these are not usually included in the terminology. Two main families of brain interfaces exist according to the usual terminology, although the terms are often used interchangeably as well:

• Brain-computer interfaces (or BCIs) usually refers to brain-to-computer interfaces that use non-invasive technology;

• Brain-machine interfaces (or BMIs) often refers to implanted brain-interfaces. This chapter shall use these terms (BCI and BMI) as defined in this paragraph. Other sub-categories of BCIs are discussed below.

The idea of BCIs is credited to Jaques Vidal (1973) who first proposed the idea in concrete scientific and technological terms. Until the late 1990’s the area progressed slowly as a result of work in but a handful of laboratories in Europe and North America, most notably the groups at the Wadsworth Centre (Albany, NY) and the Graz (Austria) group led by G. Pfurtscheller. Aside from the few researchers working on BCIs in the ‘70s and ’80s, slow progress then was largely due to limitations in:

• our understanding of brain electrophysiology,

• quality and cost of recording equipment,

• computer memory and processing speed, and

• the performance of pattern recognition algorithms.

The state-of-the-art in these areas and the number of BCI researchers have dramatically increased in the last ten years or so. Yet, there is still an enormous amount of work do be done before BCIs can be used reliably outside controlled laboratory conditions.

In this chapter, an overview of BCIs will be given, followed by a discussion of specific approaches for BCI-based robot navigation control. The chapter then concludes with a summary of challenges for the future of this new and exciting technology.

1.1. Overview of BCIs

A typical – and simplified – BCI system is illustrated in Fig. 1. In principle, the easiest way for most users to control a device with their thoughts would be to have a computer read the words a user is thinking. E.g., if a user wants to steer a robot to the left, he/she would only need to think of the word ‘left’. However, while attempts at doing this have been made, such approach currently leads to true positive recognition rates that are only slightly above chance – at best. At present BCIs work in two ways (see item A in Fig. 1): either

• brain signals are monitored while the user performs a specified cognitive task (e.g., imagination of hand movements), or

• the computer makes decisions based on the user’s brain’s involuntary response to a particular stimulus (e.g., flashing of an object on a computer screen), although both approaches have been combined recently as well (Salvaris & Sepulveda, 2010).

Figure 1.

Schematic of a brain-computer interface system

Once signals are recorded (which usually includes amplification, common mode rejection and antialising filtering) and digitized, they need to be further processed to increase the signal to noise ratio by applying frequency band filters, spatial filters, and various referencing methods (see step B in Fig. 1 and the Signal Processing section below). At this stage, we often have several hundred to several thousand data points per second, depending on the domain on which the data are analyzed. To reduce both the dimensionality of the pattern recognition task and the amount of irrelevant data (e.g., data that do not carry information related to the mental states of interest) feature selection algorithms (step C) are applied at least during offline testing, although they can also be applied on-line. Once features are extracted/selected, they are fed into a classifier (step D) that will attempt to infer the user’s mental state. Finally, some BCI systems also provide specific feedback to the user (step E), such as a sample of the classifier’s output or a parameter related to the user’s level of concentration, amongst others. Some of the steps described here make use of methods that are common to other areas. Machine learning and feature selection algorithms (e.g., support vector machines, linear discriminant analysis, nerofuzzy inference, genetic algorithms, cluster overlap indices, etc.) used in BCIs are often the same as those applied in other fields such as computer vision, etc., or variations of them. These, as such, will not be discussed in this chapter (but see Lotte et al., 2007, for further information). On the other hand, a number of techniques are specific to the kind of data used in BCIs, i.e., usually suitable for encephalographic (EEG) data. These are discussed in some detail below.

1.2. Types of BCIs

Throughout the years, BCIs have been categorized in several different ways. Most accepted terminology falls under one of the following (Wolpaw et al., 2002):

• Invasive vs. non-invasive.

• Dependent vs. independent.

• Spontaneous vs. evoked vs. event-related.

• Synchronous vs. asynchronous.

So far, these have been studied mostly in a mutually exclusive manner (e.g., either synchronous or asynchronous). They are described in more detail below.

1.2.1. Invasive vs. non-invasive

In a narrow sense, there is an obvious difference between invasive interfaces (i.e., implanted) and those (non invasive) that go on the skin surface or farther from the body. In a strict sense, however, any technology that deposits external elements on the body, be it matter or even photons and magnetic fields, is invasive in that it directly affects the internal physical state of the body. As discussed under Recording Equipment, below, technologies such as near-infrared spectroscopy (NIRS, which deposits near-infrared light on tissue), magnetic resonance imaging (which applies magnetic fields) and positron emission tomography (which requires the administration of a radioactive substance) are all invasive as the very mechanisms by which they work requires that the observed tissue (and surrounding ones) be internally disturbed. For most of these technologies, the effects on the body are well known. For NIRS, however, the effects of the absorbed energy on the brain tissue have not been studied concerning long term and even short term but prolonged use (e.g., rare occasions, but with hours of use every time), and thus caution is recommended at this point in time.

Of the technologies described later in the chapter, only electroencephalography and magnetoencephalography can be considered non-invasive.

1.2.2. Dependent vs. Independent BCIs

BCIs have been classified as either dependent or independent (Wolpaw et al., 2002). A dependent BCI does not require the usual ways of brain output to express the internal mental state (e.g., speech, facial expression, limb movement, etc.), but it does require that some functionality (e.g., gaze control) remain beyond the brain. In practice what this means is that a dependent BCI is not entirely reliant on the brain signals alone. For example (see SSVEP BCIs below), in some cases the user must be able to fixate his/her gaze on a desired flashing object on a computer screen in order for the BCI to determine which object (or choice) the user wants amongst a finite set. This is a problem in principle as a ‘pure’ BCI should not require any body-based functionality beyond being conscious and being able to make decisions at the thought level. However, in practice very few users have no overt (i.e., observable via visual or auditory means) action abilities left. These, so called totally locked-individuals, would not be able to benefit from a dependent BCI, so independent BCIs are needed in this case. On the other hand, most disabled and otherwise functionally restricted people (including some locked in individuals), as well as able-bodied people, have at least some voluntary eye movement control, which motivates further research in dependent BCIs.

Independent BCIs, in contrast, do not require any physical ability on the part of the user other than the ability to mentally focus and make decisions. In other words, even if the user has no voluntary control of any organ beyond the brain, an independent BCI would be able to infer the user’s mental choices by looking at the brain signals. In this case, the signals are independent of whether or not the user has any control of any body parts. Examples of this are mental navigation, mental arithmetic, imagination of limb movements, etc.

1.2.3. Spontaneous vs. evoked vs. event-related

Evoked potentials (EPs) are observable brain responses to a given stimulus, e.g., a flashing letter, a sound, etc., whether the user is aware of or interested in the stimulus or not. EPs are time locked to the stimulus in that the observed brain signal will contain features that are consistently timed with respect to the stimulus. For example, if a user focuses his/her gaze on a flashing character on a computer screen, this flashing frequency (if it is between about 6Hz and 35Hz) will be observable in the visual cortex signals. Other brain signals can be spontaneous, i.e., they do not require a given stimulus to appear. Thoughts in general can be assumed to be spontaneous (although strictly speaking this is still debatable). An example of spontaneous potentials are those related to movement intentions in the sensory motor cortex and are thus not a result of specific input. Finally, a third class of signals is termed ‘event-related potentials’ (ERP). They are related to evoked potentials but also include brain responses that are not directly elicited by the stimulus. I.e., they can include spontaneous or deliberate thoughts such as mental counting, etc. (Rugg and Coles, 1995), but they have a well controlled time window within which brain signals are monitored, whether spontaneous or as a result of a specific stimulus. The term event-related potential is currently seen as more accurate for all but the most restricted stimulation protocols and it is preferred over the term evoked potentials.

1.2.4. Synchronous (cue based) vs. asynchronous (self-paced)

The three subcategories described in the previous paragraph are also referred to using two other terms, synchronous and asynchronous interfaces. BCIs based on EPs and ERPs are synchronous in that they restrict the interaction as the user is only allowed to convey an intention or command to the machine when the machine allows it. I.e., either the monitored signal is a response to a computer-timed stimulus, or it is a mental task executed only when the monitoring computer gives the ‘go ahead’ to the user, typically by means of a tone or an object on the screen. The user, thus, has control over what to convey to the machine, but not when. This is by far the most common approach in BCIs. For example, a common approach is to have the computer give a visual or auditory cue to let the user know that he/she is to perform a mental task (e.g., movement imagination) immediately afterwards. In this case, as in most BCIs, the user is told to stop the task after a few seconds. The computer then uses these few seconds of data to infer the mental state of the user. In another common synchronous approach, users choose from a set of flashing letters (see the P300 and SSVEP approaches later in the chapter). Obviously, in this case computer interpretation of the signals can only be done on data obtained while the object of interest in flashing, the timing for which is very precisely controlled in order to map specific data features to the correct flashing object. Synchronous BCIs are often called ‘cue-based’ as well.

Asynchronous BCIs (e.g., Townsend et al., 2004), on the other hand, use brain signals that are produced any time by the user, with or without a specific computer-controlled stimulus. This makes classification of the user’s intention difficult as the machine first needs to identify whether a deliberate intention-related signal has been produced (the so called ‘onset detection’ problem, e.g., see Tsui et al., 2006) to then be able to identify what intention took place. An alternative is to use continuous classification of the signals with ‘idle’ or ‘no specific state’ as one of the classes, but this may reduce classification performance overall as it adds a class to the number of possible outputs from the translation algorithm. An example of an asynchronous BCI is the use of movement imagination (e.g., left hand movement vs. right hand movement) to steer a robot left or right when the user decides this must be done and not when the robot/computer demands a command. Asynchronous BCIs are often called ‘self-paced’ as well.

The main problem with asynchronous BCIs is the difficulty in determining training and validation labels for the classifier. In essence, if the computer is not able to determine when an onset took place with a precision in the order of a few hundred milliseconds, it is unlikely that the data will be correctly labelled for (re)training purposes. To circumvent this problem, often the user is asked to perform the spontaneous mental task for relatively long periods, e.g., near 10s, so that onset timing errors become irrelevant (see e.g., Sepulveda et al., 2007). However, this approach puts serious limitations on the rate of information transfer between the user and the machine.

Due to the timing and labelling problems with asynchronous (self-paced) BCIs, most work to date has used synchronous (cue-based) interfaces. However, self-paced BCIs are much more natural to the user as they do not require that the subject be paying full attention to a given stimulus or cue. Thus, not only does the user have timing freedom with self-paced BCIs, but he/she is also free to multi-task and interact with the environment beyond the BCI. This is the ultimate aim of BCIs.

1.3. Recording equipment

A number of devices have potential for use in BCIs. Some candidates are briefly discussed below. As we will see, only one class of equipment is currently suitable for widespread use, but, depending on the circumstances, the other devices may be useful as well.

• Functional magnetic resonance imaging (fMRI) (Belliveau et al., 1991): This is a powerful technology used for functional brain mapping based on hemodynamics (i.e., blood flow and oxygenation changes).

  • Pros: It provides excellent spatial resolution.

  • Cons: The equipment is very large, heavy and expensive. Thus, portability is not a possibility for at least a few decades, if ever. The temporal response is slow compared to, e.g., electroencephalography (EEG, discussed on the next page). As fMRI monitors hemodynamics, a few seconds of changed and sustained neural activity go by before significant changes are seen (Raichle et al., 2006). This poses a major problem for real-time BCIs, in which case a particular mental state of interest may have come and gone without being detected. This is because much of BCI-relevant brain activity is of a transient nature (but see SSVEP later in the chapter for an exception to this). On the other hand, if a mental task is maintained for several seconds, in semi or actual steady-state, fMRI will allow detection of this process. An additional problem is the use of strong magnetic fields, which poses safety issues, especially if metals (e.g., electrodes) are in the proximity.

• Positron emission tomography (PET) (Ter-Pogossian et al., 1975): In principle this technology can be used for brain mapping, usually through radioactive oxygen or glucose given to the user.

  • Pros: spatial resolution is better than with EEG.

  • Cons: First and foremost, then use of radioactive substances precludes use of this technology in BCIs, although in extreme cases (e.g., in totally locked in individuals) it can be useful for validating other methods as it is a well established technology, having been available for several decades. Like fMRI, the approach also suffers from poor time resolution, aside from the fact that long delays (up to hours) are required between radioisotope ingestion and the brain imaging procedure. The recording equipment is very large and expensive as well.

• Near infrared sprectroscopy (NIRS) (Wolf et al., 2007): This method too is based on hemodynamics. In this case blood oxygenation changes are linked to the amount of reflected near-infrared light applied on the brain through transmitters on the scalp, the receiver being placed nearby on the scalp as well. The approach is similar to that used in existing sensors using mid-range infrared, but near infra-red has much deeper penetration in tissue (up to a few centimetres), lending itself to brain monitoring.

  • Pros: This is currently the cheapest hemodynamics-based technology available, although the equipment is still more expensive than for EEG. Compared to other hemodynamics devices, NIRS equipment is also fairly portable, and wireless systems have been recently developed as well (Muehlemann et al., 2008).

  • Cons: As in other hemodynamics-based systems, time resolution is poor. On the other hand, different from PET and fMRI, spatial resolution is poor due to significant scattering of the near-infrared light in tissue. NIRS systems are also very sensitive to transmitter and sensor motion and environmental NIR sources.

• Magnetoencephalography (MEG) (Cohen, 1972): MEG records the magnetic fields orthogonal to the electric fields generated by ensemble neural activity, although there is evidence suggesting that the source of detectable magnetic fields in the brain is physiologically different from those generating EEG (Hamalainen et al., 1993).

  • Pros: It has much better time resolution than hemodynamics-based systems. Electrical and magnetic field changes reflect the underlying neural activity within a few milliseconds. Also, in contrast with fMRI, PET and NIRS, MEG only monitors brain signals and does not deposit any matter or energy on the brain. It is thus a truly non-invasive technology in that it does not disturb the object of study.

  • Cons: MEG equipment is still very large, comparable in size with fMRI equipment. It is also very costly.

• Electroencephalography (EEG) (Niedermeyer et al., 2004): This is by far the oldest of all the devices discussed here, having been available at least since the 1920’s (Swartz and Goldensohn, 1998). In EEG, the electrodes are usually placed on the scalp and record the electrical activity taking place in the brain tissue underneath, which reaches the electrode region by volume conductor processes. In order for neural activity to be detectable using EEG, it must be both fairly near the cortical surface and include many millions of cells in synchrony so that the total sum of the activity is large enough to be detected from the scalp. Still, the largest EEG potentials seen under most conditions have amplitudes in the order of a few tens of microvolts.

  • Pros: EEG is the least expensive technology for brain monitoring. EEG systems are also very portable and provide excellent time resolution. Due to their passive nature (from the brain’s point of view), they are very safe as well.

  • Cons: There are two main limitations in EEG systems. One (poor spatial resolution) is inherent while the other (poor usability) can still be tackled. Poor spatial resolution is inherent due to the volume conductor effects through which signals from nearby (even up to a few cm apart) areas are irreversibly mixed together. While various approaches exist to minimize this problem, sub-centimeter EEG features have little meaning. Poor usability stems from the need to use electrode gel to reduce the impedance between the electrodes and the scalp, but there are commercial systems currently available that employ user-friendly wet electrodes or dry (usually capacitive) electrodes, at the expense of signal quality.

• Implanted Brain Interfaces, or Brain-Machine Interfaces (BMIs) (Lebedev & Nicolelis, 2006): Various approaches have been developed that require the implantation of electrodes and arrays thereof to record brain electrical signals much closer to their source than with MEG and EEG. These record local potentials and in some cases signals from as few as tens of cells can be recorded. These brain interfaces have been implanted at various depth levels under the skull, from the surface of the protective tissue surrounding the brain to near a centimetre into the cortex.

  • Pros: The main advantage of implanted devices is that they have the potential to simultaneously provide the best spatial and time resolution.

  • Cons: Very high cost, risky surgery and post-surgery damage risks (including possible irreversible loss of neural tissue) are the main issues. These are currently of such high level, however, that (implanted) BMIs are still experimental, although there is currently at least one user with a working BMI in place (Hochberg et al., 2006).

Of all the devices described above, EEG is so far the best candidate for routine BCI use as it is portable, safe, relatively inexpensive, and has good temporal resolution. For these reasons it is the device of choice in the vast majority of BCI research and development. Hence, hereafter in this chapter all methods under discussion refer to EEG unless otherwise specified.

A typical EEG set up is illustrated in Fig. 2. In the picture electrodes have been placed on the scalp on locations anterior and posterior to the areas of the brain that control limb movements (see Motor Imagery below). In this case differential (bipolar) recordings are obtained for each anterior/posterior electrode pair, providing relevant information on movement-related intentions.

Figure 2.

Typical EEG cap for use in BCIs. In the picture, only electrode locations used for monitoring hand and foot movement imagination are included in the set-up

1.4. Signal processing

The signal-to-noise ratio in EEG signals is significantly <1. This is largely due to the small amplitude of the recorded signals (in the order of microvolts), but also due to a number of other factors, such as: irreversible summation of sources due to volume conduction in tissue, brain multitasking, less than suitable skin-electrode impedances, evoked potentials resulting from unwanted or unaccounted for stimuli, motion artifacts (electrode and cable movement, etc.), environmental noise, strong interference from muscle signals, eye motion artifacts, etc. On the whole this produces a very noisy signal, which, at first glance, has little if any information about the underlying brain function. For the simplest cases, basic features can be extracted (e.g., closing of one’s eyes or relaxing produces a visible amplitude increase in the alpha band – 8Hz to 10Hz), but in most cases sophisticated feature extraction and machine learning algorithms need to be employed to obtain even partially reliable information. A typical EEG signal set is shown on the left panel in Fig. 3. The figure illustrates the recorded signals at various locations on the scalp after they have been submitted to digital filtering and ear-referencing (discussed in the next subsections).

Three characteristics can be easily identified in the EEG plot in Fig. 3:

• the strong correlation between signals from nearby electrodes, thus illustrating the poor spatial resolution mentioned above;

• the lack of obvious events during movement imagination, starting at the green arrows and lasting 3s; and

• the strong artifact caused by rolling of the eyes, shown in the red box.

Notice that while the eye-movement artifact is more prominent in the more frontal areas, it does spread as far as the back of the head.

Figure 3.

EEG time-domain signal sample and standard electrode locations. Left panel: multichannel recordings during two trials (i.e., attempts) of hand movement imagination. Right panel: standard scalp location of electrodes for a 32-channel setup using the 10-20 system (Cooper et al., 1969). Fp: frontal pole, AF: antero-frontal, F: frontal, FC: fronto-central, C: central, T: temporal, CP: centro-parietal, P: parietal, PO: parieto-occipital, O: occipital, z: mid-sagittal line. The (red) box on the left panel shows the effect of eye movement on the signals. The green arrows show when a visual cue was given to the user to begin mental imagination of right hand movement, which lasted 3s

The figure above illustrates how challenging information extraction is even after digital filtering and ear-referencing EEG signals. Without these pre-processing steps, however, the signals are even less usable. We thus proceed to discuss the minimum pre-processing stages.

1.4.1. Frequency band filtering

EEG signal energy is optimal in the 0Hz-80Hz range, although historically most studies have ignored frequencies above about 45Hz (as most processes of interest to the medical community take place below 45Hz). In this range, there are three main sources of noise which must be removed or minimized:

• motion artifacts caused by electrode and cable movements (including slow electrode drift), which are mostly below 0.5Hz;

• mains interference (50Hz in the UK, 60Hz in the USA); and

• muscle signals, i.e., electromyography (EMG, e.g., from jaws, facial muscle twitches, swallowing, etc.), some of which actually overlaps with EEG as EMG produces relevant information between near 0Hz and about 500Hz (up to 1kHz if implanted EMG is recorded).

EMG cannot be fully removed due to the EEG/EMG overlap, but it can be minimized by avoiding muscle contractions in areas near the brain and by applying a lowpass digital filter to the EEG signal (if EMG is simultaneously recorded, it can be used with methods such as independent component analysis to reduce the EMG interference on the EEG signal). Most motion artifacts can be removed with a highpass filter at ~0.5Hz (some researchers will apply a cutoff as high as 5Hz if they wish to ignore the EEG delta band). Mains interference can be eliminated by referencing (see next subsection). But, if no referencing will be applied, a notch or stopband filter must be used to remove mains interference. Overall, both analogue and digital filters are needed as a first step in EEG processing. Typical filters suitable for BCIs are illustrated in Fig. 4, which are shown only as basic guidelines as researchers might use different filter types, orders and cutoff frequencies.

Figure 4.

Typical frequency band filtering of EEG signals

1.4.2. Referencing

EEG signal referencing is the subtraction of the potential recorded at a scalp location (usually already subtracted from a scalp-based common mode rejection point at the hardware stages) from a nearby overall reference location. This is done to remove common environmental noise from the recorded EEG. To this end, the reference point must be near enough to a scalp electrode so that it has a similar noise content, but it should not have any signal sources itself. There are two typical referencing methods:

• Outside the scalp (ear or mastoid referencing): one of the ear lobes or mastoid locations (or the average between the left and right ones) is used as the reference. This is the standard approach for overall removal of environmental noise for most experimental scenarios.

• Scalp average: this is used when the goal is to investigate the difference between one channel and the rest of the scalp. It is useful also for rough localization of function (e.g., movement imagination vs. other tasks) or to study waves that are over several, but not all, channels (e.g., the P300 wave discussed later in the chapter).

Although referencing is very effective in removing common environmental noise, it does nothing to improve spatial resolution, i.e., the difference between signals from adjacent electrodes. To this end, spatial filtering is often used, the most common approaches being bipolar and Laplacean processing, as follows:

• Bipolar: This is a simple subtraction between signals from two adjacent electrodes. It will give a good estimate of activity in the area between the two electrodes. For example, subtraction of channel CP1 from channel FC1 (see Fig. 3 above) gives good information about activity related to right arm movement, whose control is the area between these two electrodes.

• Laplacean: the subtraction of one channel from the ones surrounding it. This is very useful for maximizing spatial resolution, e.g., to distinguish between imagination of movement for different limbs as their control areas are near each other in the cortex. For example, to monitor signals related to foot movement, the Cz signal (Fig. 3) can be subtracted from the average of the signal from the {FC1, FC2, CP1, CP2} electrode set. In this way, the Cz signal would yield less information about irrelevant areas nearby and more about what is directly underneath the Cz electrode.

The bipolar and Laplacean methods are also called referenceless as any previous signal referencing done will drop out during the subtraction process.

Referencing and referenceless methods can also reduce eye-movement artifacts, but often these persist and must be reduced by more sophisticated methods such as independent component analysis (Vigario, 1997), at the expense of processing speed and risking losing relevant information. However, many BCIs ignore eye-movement artifact removal altogether as the pattern recognition algorithms can learn to ignore the artifact and the increase in computer memory use and processing time is often not worth the effort.

2. Approaches for BCI control of robot navigation

As mentioned above, there are many approaches currently under development in BCIs. Some will more easily lend themselves to applications in robot navigation, but almost every approach can be used for this purpose with minor modifications. Due to space and topic relevance restrictions, it is not possible to cover all approaches within this chapter. Instead, in the next subsections the three main candidates for brain-actuated robot navigation using (non-invasive) BCIs are discussed. The subsection will conclude with a discussion of how to employ a particular approach towards BCI control of robot navigation.

2.1. Motor Imagery (MI)

Imagination or mental rehearsal of limb movements is a relatively easy cognitive task for BCI users, especially able-bodied ones. Some individuals will not find this task as straight forward, but most become better at motor imagery (MI) with practice. Another advantage of this approach is that it allows the user to multitask, e.g., he/she does not need to focus on the BCI computer and can thus interact with the environment more freely than with methods such as P300 and SSVEP discussed below.

In addition, MI benefits from the fact that movement-related brain activity is well localized. Several areas of the brain handle movement-related intentions before these are executed, but, at the execution stage, the primary motor cortex (PMC) is the main control center (right panel in Fig. 5). The area immediately posterior to the PMC, the somatosensory cortex, receives sensory information from the equivalent body parts controlled by the PMC. Within the PMC, subregions of the body are distributed in a well localized functional map as well. For example, a cross section of the left primary motor cortex area is illustrated on the right panel in Fig. 5, where the labels indicate which part of the (right side of) the body is controlled. We can clearly see how one might use signals from different channels to be able to distinguish between movements of different body parts, e.g., hand vs. foot. However, the functional map shown below can only be fully explored by means of implanted devices, intra-cortical ones in particular. Due to the volume conductor effects mentioned above, EEG electrodes will also pick up signals from areas near the region underneath the electrode. For example, an EEG electrode on the scalp right above the hand area will likely contain signals related to other areas as well, from arm to neck, at least. This problem, however, can be lessened by applying multichannel recordings and bipolar or Laplacean processing, as discussed above.

Figure 5.

Brain localization of primary movement control

2.1.1. Motor Imagery (MI) towards robot navigation

As mentioned above, motor imagery is an intuitive cognitive task for most, although it may require practice. Control of robot navigation with this method can thus be easily mapped to limb movements. For example (Fig. 6), to steer a robot to the right, the user can imagine or mentally feel (also known as kinaesthetic motor imagery) a movement of the right hand (e.g., hand closing, wrist extension, etc.). The example below shows only three movements. Other motor imagery tasks can be added to increase the number of robot movement classes (i.e., choices). For example, imagination of tongue movement is often used as an additional class.

Figure 6.

Possible use of motor imagery for robot navigation control

Notice from Fig. 5 above that using separate movements of the right and left feet should not be expected to yield two separate classes with EEG as the brain areas that control both feet are next to each other and deep in the brain, which means the same electrode (i.e., Cz in Fig. 3) will pick up signals from both feet.

Motor imagery will produce features that can be used for classification. A common feature used with MI is band power, in which case the power of the (previously pre-processed) signal in specific frequency bands (notably alpha: 8-12Hz and beta: 13-30Hz) yields good classification of right vs. left movements. A similar feature commonly used with MI is event-related desynchronization and synchronization (ERD/ERS) which compares the energy of the signal in specific bands with respect to an idle state (i.e., a mentally relaxed period). In either case the most appropriate electrodes are usually the ones over or near the relevant primary motor cortex areas.

Motor imagery can be used with any timing protocol. It can be used by itself in a cue based approach, in a self-paced system, or used in combination with the P300 approach discussed below (as in Salvaris & Sepulveda, 2010), although the latter has not been applied to robot navigation.

One of the limitations of MI-based BCIs for robot control is that usually a few seconds of EEG data are needed for each control decision and for the motor cortex to fully return to a neutral state. Typically this will give an information transfer rate (from user to robot) of <10 bits/min.

MI approaches similar to the one discussed above have been applied to robot navigation (e.g., Millan et al., 2004).

2.2. P300

This approach falls under the event-related potential category. In this method, the user is presented with a visual array of choices (left panel, Fig. 7, based on Farwell & Donchin, 1988), although sound and touch can be used as the stimulus as well. Typically each row and column will flash for a short period (about 100ms) in a random sequence on a computer screen. When the row or column containing the desired choice flashes, the user adds 1 to a mental counter to signal that a target has flashed. For example, if the user wants to type the letter P using a BCI, she/he will count every time a row/column containing it flashes. On average, when a target row/column flashes, a strong signal is seen (especially in the centro-parietal electrodes, Fig. 3) which will peak at about 300ms after the desired object flashed, hence the P300 name. Until recently it was assumed that eye gaze did not significantly affect P300 responses, but there is now evidence suggesting that this is not the case (Brunner et al., 2010).

The right panel in Fig. 7 illustrates signal differences between target, non-target and near-target events). In most cases, the target P300 peak is only easily distinguishable from non-target events if an average of several trials is performed, and often up to ten target trial responses are needed to have a true positive rate of about 90%.

Figure 7.

Typical P300-based stimulus array and EEG responses (modified from Citi et al., 2004)

2.2.1. P300 towards robot navigation

The array in Fig. 7 is used for communication BCIs (e.g., as a speller) and does not directly lend itself to use in robot navigation control. However, if each object on the array represents a command to a robot, the user can employ the BCI to give the robot a sequence of commands which may include variables such as direction, timing schedule, proportional control parameters (e.g., for angular displacement, speed), etc. But, one of the problems with this interface is that the user must wait for all rows and columns to flash before a new flashing cycle begins. With current standard timing parameters, this would take several seconds per trial, per chosen letter. If, as mentioned above, several trials are used to increase true positive recognition rates, choosing one letter can take more than 10s, which is not suitable in many robot navigation cases. To minimize this problem, an array with less elements can be used, although this will reduce the difference between target and non-target events as this difference increases when target events are much less probable than non-target ones.

An alternative to the P300 standard array and one that is suitable for robot navigation (at least in a 4-direction 2D scenario) is shown in Fig. 8. The figure is based on a system designed for mouse control (Citi et al., 2004), but it can be easily employed for robot navigation. For example, the four flashing objects can represent left/right/back/forward. One limitation that would still exist, however, is that the user must have full attention on the screen showing the 4-object array. In this case, the user would count every time the desired direction flashes, as a result of which the robot would turn in the desired direction.

Figure 8.

P300-based interface for basic robot steering

P300 approaches similar to the one shown in Fig. 8 have been applied to robot navigation recently. Also, Rebsamen et al. (2007) produced a P300 system in which the objects on the monitor are pictures of the landmarks to which the robot (a wheelchair in this case) must go. In that system, the user chooses the end point and the robot uses autonomous navigation to get to it.

The information transfer rate of a P300-based BCI with four classes will yield a higher information transfer rate than with motor imagery, possibly >20bits/minute, but, as mentioned above, it has the disadvantage that it demands the user’s full attention on the visual interface.

2.3. Steady-State Visual Evoked Potentials (SSVEP)

The P300 method above is similar to the SSVEP approach in that the user is presented with an array containing flashing objects from which the user chooses one. However, in the SSVEP method each object flashes at a different frequency, usually between 6Hz and about 35Hz (Gao et al., 2003). When the user fixates his/her gaze on a flashing object, that object’s flashing frequency will be seen as a strong frequency-domain element in the EEG recorded from areas above the visual cortex (occipital areas, Fig. 3). For example (Fig. 9), if the user is interested in number 7 on a number array, fixating his/her gaze on that object (which is this example is flashing at 8Hz) will produce the power spectrum shown on the right panel in Fig. 9, which is an average of five trials (i.e., target flashing cycles).

Notice that the user must have eye gaze control for this approach to work, but, as mentioned above, this ability is retained by the vast majority of potential BCI users, both disabled and able-bodied.

2.3.1. SSVEP towards robot navigation

Using an SSVEP-based BCI for robot navigation control is similar to the case with the P300 method, i.e., a suitable array of flashing objects can be designed specifically for robot navigation.

SSVEP-based BCIs have been used for robot navigation control (e.g., Ortner et al., 2010). The information transfer rate will yield a higher information transfer rate than with motor imagery, >40bits/minute, but, as is the case with the P300 approach described above, it has the disadvantage that it demands the user’s full attention on the visual interface.

Figure 9.

SSVEP BCI. Left: example of a multi-object array in which the object of interest (number 7) flashes at 8Hz. Right: power spectrum of the recorded EEG when the user fixates his/her gaze on number 7 in the interface (notice the strong harmonic components)

2.4. Choosing the most suitable BCI type

The best BCI type will depend on the scenario to be tackled. For example, for robot navigation in an environment for which landmarks are stored in its memory, either the P300 or the SSVEP approaches can be used only when necessary by allowing the user to choose the desired landmark and letting the robot use its autonomous system to get to the landmark. If, on the other hand, the environment is novel or the robot encounters unexpected obstacles, motor imagery can be used for direct control of robot steering. All approaches can be used in combination as well, e.g., using motor imagery for initial navigation while the robot saves information about the environment and then later using P300 or SSVEP to perform landmark-based navigation (as in Bell et al. 2008). Recently, wheelchair navigation control was done using a BCI that relied on the so called error potentials (an involuntary brain response to undesired events; not discussed here) to allow the robot to determine which of the routes it found was suitable to the user (Perrin et al., 2010).

3. Future challenges

In order for BCIs to be routinely used in robot navigation control, a number of factors will need to be improved. Amongst other, the following will need to receive high priority:

• Recording equipment: while systems based on dry electrodes exist, they do not yet give as reliable a signal as standard wet EEG electrodes. The latter require the use of electrode gel or water. Systems that require the use of electrode gel are the most reliable ones, but they require about 1min per electrode to set up, and the gel needs to be changed after a few hours. Water-based systems are quicker to set up, but at present they provide less reliable signals. As dry sensor technology improves, it is likely that such devices will be preferred, especially by users outside the research community.

• Degrees of freedom: The number of classes available in a BCI depends on which type of BCI is used. P300 BCIs can provide a large number of classes (>40 in principle, although clutter will decrease true positive recognition performance), but this will come at the expense of longer processing times for each individual choice. The same applies to SSVEP-based interfaces. MI-based BCIs can provide only a small number of classes at the moment, usually 4 or less if more than 90% true positive rate is desired (although up to 20 classes have been successfully classified in a small but well controlled study, Lakany & Conway, 2005).

• Proportional control: BCI control of proportional variables such as robot angular displacement and speed has received little attention so far. This is an important area of research for future use of BCI-based robot navigation.

• Intuitiveness and user freedom: The most intuitive approach is motor imagery, but more classes would be needed to make this approach more useful in robot navigation control. P300 and SSVEP approaches require full attention on the visual interface and thus give no freedom to the user. Other cognitive tasks have been used in off-line BCI studies (e.g., Sepulveda et al., 2007), but these should be investigated further before they are used for robot navigation.

• Speed issues: If information transfer rate alone is the main concern, SSVEP would be the best choice, followed by P300, but the required focus on the interface will remain a major problem for the future. It will thus be crucial to find fast approaches that rely on motor imagery or other cognitive tasks that are intuitive and give the user freedom from the interface.

4. Conclusions

BCIs have come of age in many ways and are now being used outside controlled laboratory settings. However, a number of limitations in the current state-of-the-art will need to be addressed in order to make this technology more reliable, low cost, user friendly and robust enough to be used for routine robot navigation control. Until then, BCIs will remain largely an experimental tool in robot navigation, or as an additional tool within a multi-modality approach. Nonetheless, brain-actuated or brain-assisted robot navigation control will bring benefits to the field, especially in difficult scenarios in which the robot cannot successfully perform all functions without human assistance, such as in dangerous areas where sensors or algorithms may fail. In such cases BCI intervention will be crucial until a time when robots are intelligent and robust enough to navigate in any environment. But, even then, human control of the robot will probably still be desirable for overriding robot decisions, or merely for the benefit of the human operator. In another case, when the robot is a wheelchair, frequent user interaction will take place, in which case BCIs are essential for at least some form of brain-actuated navigation control.