Open Access is an initiative that aims to make scientific research freely available to all. To date our community has made over 100 million downloads. It’s based on principles of collaboration, unobstructed discovery, and, most importantly, scientific progression. As PhD students, we found it difficult to access the research we needed, so we decided to create a new Open Access publisher that levels the playing field for scientists across the world. How? By making research easy to access, and puts the academic needs of the researchers before the business interests of publishers.
We are a community of more than 103,000 authors and editors from 3,291 institutions spanning 160 countries, including Nobel Prize winners and some of the world’s most-cited researchers. Publishing on IntechOpen allows authors to earn citations and find new collaborators, meaning more people see your work not only from your own field of study, but from other related fields too.
To purchase hard copies of this book, please contact the representative in India:
CBS Publishers & Distributors Pvt. Ltd.
www.cbspd.com
|
customercare@cbspd.com
This paper reviews efforts in a new direction of the EEG research, the direction of EEG emulated control circuits. Those devices are used in brain computer interface (BCI) research. BCI was introduced 1973 as a challenge of using EEG signals to control objects external to the human body. In 1988 an EEG-emulated switch was used in a brain machine interface (BMI) for control of a mobile robot. The same year a closed loop CNV paradigm was used in a BMI to control a buzzer. In 2005 a CNV flip-flop was introduced which opened the direction of EEG-emulated control circuits. The CNV flip-flop was used for BMI control of a robotic arm in 2009, and for control of two robotic arms in 2011. In 2015 an EEG demultiplexer was introduced. The EEG emulated demultiplexer demonstrated control of a robotic arm to avoid an obstacle. The concept of an EEG emulated modem was also introduced. This review is a contribution toward investigation in this new direction of EEG research.
In 1929 Berger carried out research on human electroencephalogram (EEG) and introduced EEG rhythms [1]. In 1973 Vidal [2] introduced the term Brain–Computer Interface (BCI), and he set a challenge of controlling objects external to the human body by using the signals from a human electroencephalogram (EEG). He actually stated two challenges for EEG researches:
develop methods for EEG control of objects not being part of a human body
develop new methods for extracting event-related potentials (ERP) from an EEG, other than the standard averaging method.
Vidal [2] advised the use of various EEG signals, including EEG rhythms and event related potentials. Specifically, he challenged the use of the Contingent Negative Variation (CNV) event related potential.
Response to the BCI challenge was relatively slow in the years after 1973. The first report on control of an object using EEG was given by Vidal himself in 1977 [3]. He designed a responsive BCI, with active movement of eyes to elicit various visual evoked potentials (VEPs), in order to control a 2D movement of a cursor-like object on a screen. In 1988, three reports appeared related to control of objects using EEG signals. Farwell and Donchin [4] used P300 event related potential to choose a letter from a computer screen and write text on that screen. Bozinovska et al. [5, 6, 7, 8] used contingent negative variation (CNV) event related potential to control a buzzer. That work also responded to the second Vidal’s challenge, as it introduced an adaptive filter to extract a time varying CNV potential. That work introduced a new taxonomy of brain potentials [8] which classified CNV as an anticipatory brain potential. Bozinovski et al. in 1988 [9, 10, 11, 12] used changes in EEG alpha frequency band (contingent alpha variation, CαV) to control a movement of a physical object, namely a robot; that BMI solved the long lasting challenge of telekinesis (movement of a physical object with energy emanating from a human brain). And, after the 1973 BCI challenge, it was the first intention driven (rather than response driven) BCI. Those five pioneering BCI efforts appeared before 1990. Examples of works of other authors after 1990 related to this work are [13, 14, 15, 16].
An EEG based BCI setup consists of the following steps: (1) Produce a state in a (human or animal) brain which will be manifested by a particular EEG signal in which a control command is encoded. (2) Record the EEG signal and transmit it to a computer. (3) Analyze the EEG signal and decode the encoded command. (4) Send the decoded signal to a controlled object, such as a visual object, or a sound object, or a physical object with a mass.
EEG is a classical modality of obtaining a brain signal, but other ways of recording brain signals (e.g. magnetoencephalogram) are also being developed. This paper deals only with the EEG modality used in a BCI.
There are two ways of generating EEG-encoded commands to control an object.
Command encoded in an external stimulus-driven EEG response. This method uses an external stimulus (e.g., light flickering) to generate an EEG response which then is used for external object control. This way of conducting a BCI in the early 1990s was named brain-response interface (BRI) [17]. As example, a BRI directs the eye gaze toward a particular part of a computer screen which flickers in particular frequency. When a user eye looks at that part of the screen, the brain reflexively produces a corresponding visual evoked potential in the EEG, which is a function of the stimulus frequency. This method usually requires contraction of the eye muscles in order they to direct the eye gaze to a particular area on the screen, where a particular flicker is generated. An example of such method is the SSVEP (steady state visual evoked potential) method. It is a gaze-based brain-response interface method.
Command encoded in an intention to modulate a particular brain rhythm. This method encodes a command in EEG by an internal intention (i.e., conatively, willingly) rather than by a brain response to an external event. An example is willingly increasing the amplitude of the EEG alpha frequency band (7–13 Hz), by an idle activity (no activity), and decreasing that amplitude due to an engaging activity. For example, if the alpha band is measured from the visual area of the brain (named alpha rhythm), then closing the eyes produces a resting state and increased alpha activity. Other methods also produce relaxation and increased alpha activity. If the alpha band is measured from the motor area (named mu rhythm) then no movement and no imaginative movement produces resting state. If the alpha band is measured from the auditory area (named tau rhythm) then no-sound produces resting state.
Another term used, besides BCI (brain computer interface) and BRI (brain-response interface), is BMI (brain machine interface). It is usually used for control of an object outside of the computer screen, for example a physical object with a mass, such as a robot. With that in mind, the term BMI is used in this paper.
A BCI can be carried out invasively and non-invasively. Invasive BCI records a signal inside a brain, which requires a surgical intervention. Non-invasive BCI records EEG from the scalp, which is outside the brain. For example, the first non-invasive BMI was carried out in 1988 [9, 10] for control of a (mobile) robot, and the first invasive BMI for control a robot (arm) was carried out in 1999 [18]. It is worth mentioning that those two works were the only ones in the 20th century dealing with BMI for moving object with a mass.
An essential objective of the BCI software is to find an EEG feature which can be used as a switch for controlling an object. In addition to an EEG emulated switch, recently other EEG emulated control structures are being explored, such as a flip-flop, demultiplexer, and modem. This paper will be devoted to that research.
Reviews of the BMI efforts (e.g., [19, 20, 21]) are present in the literature. Various robotic devices are being built (e.g, [22]). Many companies are involved in BMI (e.g. Emotiv [23], Kinova [24]).
In the sequel, we will first review the EEG emulated switch for control of a mobile robot. Then we will describe the EEG emulated flip-flop with applications of controlling robotic arms. Then we will describe an EEG demultiplexer and the EEG modem. Some results of current experimental research work using EEG demultiplexer are also shown.
An EEG switch is a control circuit which produces a digital switch output driven by an EEG pattern. After the BCI challenge stated by Vidal [2], the first EEG switch was explicitly described in 1988 [9, 10, 11, 12]. It is shown in Figure 1 [11, 12].
Figure 1.
EEG emulated digital switch.
Figure 1 shows the screen of the 1988 brain-machine interface program [9, 10, 11, 12]. The lower part of the screen shows the EEG signal recorded in a particular BMI session. Inside that session a user may generate alpha wave bursts by some relaxation technique, for example closing/opening the eyes. The robot movement takes place in real time while intentionally generating/blocking the alpha rhythm. In offline analysis mode, the program has a feature of zooming a part of the signal, defined by a line below the signal, as shown in Figure 1. The zoomed segment of the signal is shown in the center of the screen. The upper part of the screen shows the result of pattern recognition method in real time, which recognizes when the EEG signal contains increased amplitude of the alpha rhythm. That produces a switch pattern, actually an EEG emulated Schmitt trigger (e.g., [25]). The recognition software implements a machine learning algorithm in which the learning phase is collecting two distributions, one for amplitude and one for frequency. If both amplitudes and periods between two adjacent extreme points of an EEG signal increase, it is recognized as increased alpha rhythm, and it turns on the switch. If both amplitudes and periods decrease, it turns off the switch. A statistical machine learning method was used. Details are given in a recent review [26].
A human user, in a BCI based on an EEG emulated switch presented in Figure 1, requires a period of training in order to perform the task. The 1988 experiments show that the training requires about 20 minutes.
Let us note that the concept of a “mind switch” was introduced in [14]. Before that, the term “switching devices” [27] was used in relation to the independence of disabled persons. The term “brain-controlled switch” was used in [16]. We use the term EEG emulated switch as part of our work on EEG emulated control circuits.
The BMI task when the EEG emulated switch was used in 1988, was control of a mobile robot moving along a closed line drawn on the floor. Robot movement happens when the user increases the alpha rhythm by closing the eyes. The robot stops when the user opens the eyes and observes how far the robot is from the goal point, a “station” where the robot should stop. At what point to stop was decided by the user based on a visual feedback. The BMI setup is shown in Figure 2.
Figure 2.
The 1988 experimental setup for an EEG control of a robot.
Figure 2 is a translation of the original 1988 block diagram of a BCI [9, 10, 11, 12]. It was the first block diagram of a BCI in the literature. It shows a human user, an EEG amplifier, a computer (PC/XT), an A/D converter, a software for recognition of an alpha rhythm, a D/A converter, an energy amplifier, a robot, and an optical sensor for following the trajectory drawn on the floor. The robot used was a Ellehobby Movit Line Tracer II.
A differential biosignal amplifier was used to record the signal from the Pz site (international 10/20 system), with referential electrode on right mastoid, and ground electrode placed at the forehead. The signal was received in an IBM PC/XT (640 KB, 8 MHz) computer by an A/D converter at 300 Hz sampling rate. The software which recognized the alpha wave was written in Pascal. During the alpha wave presence, the system outputted a logic pulse at 5 volts through a D/A converter. The output signal was amplified on a transistor amplifier which drove the robot motor.
A CNV flip-flop is an EEG emulation of the flip-flop digital circuit based on the contingent negative variation (CNV) event related potential (ERP). The concept of CNV flip-flop was introduced in 2005 [28].
The CNV potential [29] manifests an EEG a mental state of expectation. The CNV potential appears in an experimental procedure known as the CNV paradigm. It is a well-known procedure (e.g., [30]) in which, in an open-loop way, a slow negative potential shift (the CNV) appears in the inter-stimulus interval of the S1-S2 stimulus pair. The negative slow potential shift is interpreted as an expectancy wave and is related to learning and memory. The classical CNV paradigm is an open-loop control system. After a stimulus S1 (visual or auditory), the brain is expecting stimulus S2 and is preparing to produce reaction R on S2. The ERP between S1 and S2 gradually develops to be a recognizable CNV. The CNV paradigm produces a ramp-like potential (the CNV) related to the pair S1-S2, but also produces a number of other evoked, cognitive, and preparatory potentials related to S1 and/or to S2.
The open-loop design for obtaining a CNV potential is given in Figure 3.
Figure 3.
The classical CNV paradigm.
As Figure 3 shows, while EEG is recorded, the user (subject) receives two sequential stimuli, by some time distance apart. The time distance between S1 and S2 (inter-stimulus interval) is fixed, 2 seconds. The time distance between S2 and next S1 (inter-trial interval) is random, 11 ± 2 seconds. Here a three - state buffer represents a trial control, where EEG enters the CNV paradigm. After several repetitions, the subject learns that after S1 follows S2 and starts to expect it. As a result, a special event related potential (ERP) appears between S1 and S2. It is a negative shift of the EEG baseline and was named Contingent Negative Variation (CNV). If a standard ERP averaging is applied, a distinctive ramp-shape potential is visible. In the classical CNV paradigm, the goal was to show the existence of a CNV potential, so the experiment ends after CNV appears, as shown in Figure 3.
In 1988 a feedback loop was introduced in the classical CNV paradigm [5]. In 2005 [28] it was recognized that in such a way a CNV flip-flop is emulated by an EEG. The EEG emulated CNV flip-flop is shown in Figure 4.
Figure 4.
An EEG emulated CNV flip-flop.
As Figure 4 shows, a CNV flip-flop has two binary states, like an ordinary flip-flop, Q and inverse of Q (Q’). In state Q, an EEG signal is recorded as in a classical CNV paradigm, ERP is extracted after each trial, and it is tested whether the ERP is a CNV. In other words, it is tested whether the expectation expect (S2|S1), is developed in the brain. However, as distinct to the classical CNV paradigm, once the CNV potential is recognized inside the recorded EEG, the flip-flop enters the sate no-Q (i.e., Q’). In this state the stimulus S2 is disabled. As a result, the CNV potential will degrade beyond recognition, which will trigger the reactivation of the S2 signal, and the flip-flop will enter the state Q again. The digital outputs Q and Q’ here are used to control S2 but, can also be used to control other external devices.
Figure 5 shows the block diagram of the 1988 CNV-based BCI experiment [5, 6, 7, 8].
Figure 5.
The first CNV-based brain-computer interface, developed in 1988. It shows a control of a buzzer using CNV potential.
As can be seen from Figure 5, the subject generates an EEG which contains an ERP due to the stimuli S1 and S2. The EEG undergoes initial signal processing, after which the procedure of ERP extraction follows. The final CNV pattern recognition procedure tests whether the observed ERP is a CNV. Once the presence or absence of CNV is recognized, the control signal (Enable/Disable) is sent to the controlled buzzer.
The BCI procedure starts with building CNV potential in the subject by generating S1-S2 pairs of sounds. By classical conditioning, an expectation of S2, E(S2) is being built. After repetitions, which are part of the learning process, the expectation to S2 is formed in the subject’s brain, and a CNV is manifested. That event, recognition of a CNV, can be used to control an external device, such as a sound generator, a robot, or something else. In the case that expectation is not built, the CNV will gradually degrade and disappear. That point, recognition of no-CNV (no expectation) event, can also be used to control an external device, in our 1988 experiment to enable the buzzer.
A standard way of building expectation is using a reaction R(S2) to stop the duration of the S2 signal, usually by pressing a button. Pressing a button is not essential, because a subject develops expectation regardless of a motor reaction [31].
Note that the subject could willingly control the process by ceasing to build expectation, i.e., by not paying attention to the S2 stimulus. But in that case, there is no adaptive interaction between the subject and BCI, and adaptive interaction is what makes this BCI interesting.
An important feature of the CNV flip flop paradigm is that it was the first bidirectional adaptive BCI, in which both the human brain and machine are engaged to adapt to each other. This paradigm was used to study adaptive behavior in adaptive learning systems [32].
Moreover, the 1988 CNV based BCI was the first to respond to the second BCI challenge, building a method for extracting an ERP beyond the classical averaging. The need for that appeared because in the CNV flip-flop paradigm the ERP is constantly changing so the classical ensemble averaging is not applicable. An adaptive filter was needed, and the following adaptive filter was implemented.
The feature extraction module extracts the Event Related Potential (ERP). Since the paradigm requests that the obtained signal be time-variant, i.e., it forms and decays, a classical averaging technique is not suitable, so we used our own adaptive filter, namely
ERPst=pERPst−1+qEEGstE1
where.
s is the sample number in a trial (s = 1,2,…, N),
t is the trial number in an experiment (t = 1,2,…,T), and.
p and q are weighted parameters, satisfying p + q = 1.
Several parameters are collected for the current shape of the ERP, particularly important being the regression angle and the amplitudes near S1 and S2. The pattern recognition module decides whether the current ERP can be classified as a CNV. The key parameters are the slope of the regression angle and the ERP amplitude difference near S1 and S2. In the forming phase of the CNV, three consecutive confirmation trials are needed before a CNV appearance can be acknowledged.
From a practical use of a BCI, it is important that the use of a CNV flip-flop does not require separate subject training. The mental development of an expectation state is taking place in the course of the CNV experimental paradigm. In the CNV paradigm, the subject learns to expect. S/he learns that after event S1 comes event S2, and s/he adjusts her/his mental state accordingly. The mental action produces a cognitive state “after S1 expect S2.”
3.1 Non-invasive BMI control of a robotic arm using a CNV flip-flop
In 2009 a BMI was built to control a robotic arm using a CNV flip-flop [33]. The task we considered was the Tower of Hanoi puzzle with two disks, the TOH(2) task.
The Towers of Hanoi, TOH(n) task, is a well known puzzle in Computer Science [34]. It has been pointed out that the solution space has a fractal structure [35]. Given a stack of n disks with different diameters, a tower is defined as a stack of disks in which the smaller disk is always above the larger one. The task is stated as follows: given three spots, A, B, and C, if the initial tower is in the spot A, move it to the spot C, using a “buffer” tower in the spot B. At each step of the task, the concept of a tower is preserved, a smaller disk always being above a larger one. It is known that to solve the TOH(n) problem, the number of required moves of disks is 2n-1.
BMI setup for control of a robot arm to solve TOH(2).
The equipment used consists of a 4-channel biosignal amplifier from Biopac. The subject is connected to the biosignal amplifier with EEG electrodes placed on Cz and mastoid, while the forehead is the ground. A Windows based personal computer was used, as well as a Lynxmotion with 6-degrees-of-freedom robotic arm. We wrote the complete software in C#.
The preprocessing part of the software shows the obtained raw EEG and considers the EOG artifacts. The ERP extraction part extracts the ERP between S1 and S2. The CNV recognition part observes when the ERP builds a recognizable CNV, as well as when the CNV decays beyond recognition, and becomes a non-specific ERP.
The CNV flip-flop recognizes series of appearances and disappearances of the CNV potential, and triggers a behavior execution part, which moves the robotic arm toward the completion of the Towers of Hanoi task.
The robot control software receives a flip-flop signal from the CNV recognition software that a CNV is not recognizable (state Q) or is recognizable (state Q’). The flip-flop activates one of the robot behaviors stored in the memory. If there are 2 disks, i.e., the task is TOH(2), there are 22–1 = 3 behaviors stored. The behaviors 1 and 3 are activated by the state Q’ and behavior 2 by the state Q. Each behavior is a trajectory to move a disk from current spot to the next, at a particular height. The sequence of behaviors is a solution of the TOH task. Behavior-based robotics [36] is a widely used approach in robot control.
Figure 7 is a photo of the experimental setup [37].
Figure 7.
Experimental setup for BMI control of a robotic arm based on a CNV flip-flop.
As Figure 7 shows, the subject having EEG and EOG electrodes, observes the progress of TOH(2) solution, as he oscillated the state of expectation it his brain.
Figure 8 shows our graphical user interface which the experimenter observes during each trial [33].
Figure 8.
An experiment trial of CNV flip-flop for robot control.
The screen shows six rows (channels) out of which the first four are acquisition channels and the last two are mathematically computed channels. The first channel is the EEG acquisition channel, the second is the EMG acquisition from the arm pressing the button, the third is the EOG signal channel, and the fourth is the press-button recognition channel. The sixth channel is the event related potential extracted so far. If an appearance or disappearance of CNV is recognized on that channel, the signal is given to the robot to move and that is recorded on the fifth channel in Figure 8. In this case channel 6 shows a recognizable CNV potential, and that is signaled on channel 5. Note that CNV potential (expectancy state) is recognized before the EMG reaction signal is recognized.
TOH(2) requires 22–1 = 3 moves to complete the task. To see the number of BMI trials needed for solving the TOH(2) task we carried out 4 experiments and obtained results as shown in Table 1 [33].
Experiment
1
2
3
4
Brain state
Robot behavior
Trial number
CNV
behavior1: A to B
16
12
11
6
noCNV
behavior2: A to C
22
23
12
19
CNVagain
behavior3: B to C
26
29
22
22
Table 1.
Experiments of using a CNV flip-flop to control a robotic arm to solve TOH(2).
The data in the Table 1 show the trial number in which the event occurred. For example, in the first experiment, the first appearance of CNV was in trial 16, the first CNV disappearance was in trial 22, and the second CNV appearance was in trial 26. As can be seen from Table 1, in each experiment, the two-disk Towers of Hanoi task was executed successfully within 30 trials, using this brain-machine interface.
3.2 Non-invasive BMI control of two robotic arms using a CNV flip-flop
The next BMI task considered using a CNV flip flop in a BMI setup was collaboration of two robot arms in solving the Tower of Hanoi problem with three disks, TOH(3). The task is depicted in Figure 9 [38].
Figure 9.
The considered TOH(3) problem to be solved by two robotic arms controlled by a CNV flip-flop.
The approach is the following: Robot1 is activated by a CNV appearance event and Robot2 is activated by a CNV disappearance event. Both robots have predefined behaviors. Robots and their behaviors are triggered by a brain state recognition system, which recognizes the existence and non-existence of the brain expectancy state represented by the CNV potential.
If the height of a particular disk is denoted with a number between 1 and 3 (height 1 being the bottom), the needed sequence of robot behaviors can be defined as: A3toC1, A2toB1, C1toB2, A1to C1, B2 to A1, B1 to C2, A1 to C3. Let us note that an artificial Intelligence program was previously written for solving the general TOH(n) problem [39], where from the knowledge was used to solve this TOH(3) problem.
Once the problem is decomposed into a sequence of robot behaviors, the CNV flip-flop generates an oscillatory process that will drive the two robots with corresponding behaviors. Robot1 behaviors are activated whenever the ERP shapes into a CNV, while Robot2 behaviors are activated whenever the ERP loses its CNV shape.
In order to solve the TOH(3) task the number of moves is 23–1 = 7. The research hypothesis for the experimental investigation is that healthy subjects will be able to carry out the oscillatory expectancy process in the brain long enough to solve the TOH(3) problem. The subject should produce the appearance of the CNV four times and the disappearance of the CNV three times. It is assumed that the TOH(3) task gives enough achievement motivation for completing the task.
The experimental setup consists of an EEG-event recognition part and a robot behavior execution part. The event recognition part recognizes the appearance/disappearance of the brain state of expectation, while the behavior execution part activates the controlled devices.
The two controlled robotic arms and the Towers of Hanoi disk set are shown in a photo in Figure 10 [38].
Figure 10.
Experimental setup, two robotic arms and the TOH(3) disk set.
Each robot is controlled by a servo controller connected to the computer by a USB-to-COM cable.
The subject is sitting and observing his/her progress toward the solution of the TOH(3) task, which gives a motivation for achievement. The EEG electrodes are placed on Cz and mastoid, while the forehead is the ground. A personal computer receives the signals and processes them. A Biopac four-channel biosignal amplifier receives the biosignal information from the subject. A USB cable connects the biosignal amplifier to the computer.
The BMI experiment screen. An experimental trial. A developed CNV potential, triggers a behavior in one of the two robots.
As Figure 11 shows, a raw EEG is recorded in channel 1, and EMG and EOG channels are recorded in channel 2 and 3. Channel 4 shows the recognized EMG and channel 5 the recognized CNV signal which is sent to execute Behavior 1 in Robot1. Channel 6 shows the current ERP which is recognized as CNV.
Results of experiments of controlling two cooperating robots by a CNV flip-flop.
As Table 2 shows all the experiments were successful. A human user developed her/his CNV potential in average in the 14th trial, lost it in average in the 22nd trial and so on. The task is completed in average 60 trials of a CNV flip-flop paradigm.
4. Non-invasive BMI control of a robotic arm using an EEG-emulated demultiplexer
EEG emulated demultiplexer is an emulation of a demultiplexer, a 1-to-n serial to parallel converter, a device that receives a serial input and distributes it to n outputs. An example of an EEG emulated 1-to-2 demultiplexer is shown in Figure 12 [40]. It is driven by alpha rhythm.
Figure 12.
EEG emulated demultiplexer.
As Figure 12 shows, a serial EEG signal is divided into two segments (frames). Those two EEG frames, A0 an A1, with encoded intent for commands to external devices, are sent to the EEG demultiplexer. Both frames can be defined as binary channels, but A0 can also be defined as multi-valued channel. The binary channels are used as binary addresses for the address decoder of the demultiplexer, while the multivalued channel is viewed as a data input of the demultiplexer. The EEG demultiplexer contains an address converter and a data converter. The signals enter the redundant 1-to-2 demultiplexer which sends the data channel to one of the two output channels c1 and c2, defined by the address decoder. Those output channels control two devices, for example two motors of a robotic arm. The demultiplexer used is redundant, because for addressing 2 output channels it uses 2 address lines, instead of just one. That has been done to increase the accuracy of EEG addressing (e.g., [41]).
The BMI task considered is shown in Figure 13 [40].
Figure 13.
The BMI task of controlling two motors by a single EEG channel using EEG demultiplexer.
As shown in Figure 13, a robotic arm should move from start region A to goal region B. The horizontal projection of the arm is such that if moved toward goal area B, it would hit an obstacle C along the way. In order to avoid the obstacle, the wrist of the arm should be moved up, so that horizontal projection of the arm is shortened before it reaches the obstacle C. The BMI task for the subject is: from a single EEG channel, generate an EEG pattern that will move the robot arm from A to B, avoiding C. So, the task is to control two motors from a single channel EEG.
Experimental setup for a BMI based on a EEG demultiplexer.
As Figure 14 shows, a subject is sitting in front of a robotic arm and sends EEG commands such that the task of moving the arm while avoiding and obstacle is achieved.
The experimental trial of an EEG demultiplexer controlling a robot is shown in Figure 15 [40].
Figure 15.
The screen of an experimental trial of an EEG demultiplexer.
As Figure 15 shows, a raw EEG is received by the BMI system and is shown in Channel 1. Channels 2 and 3 are not recorded. Channel 5 is the filtered EEG to obtain the alpha rhythm. Channel 4 is the filtered alpha rhythm to obtain a signal which represents the alpha rhythm envelope. That signal is tested against a threshold value, shown in the same channel. Channel 6 contains two frames, each showing a pulse for how long the duration in which the envelope is above the threshold. Also, in the Figure 15 can be seen that the binary value of frames is A1A0 = 01.
The channel 6 is the EEG demultiplexer channel. First the binary values of the frames are computed, in this case A1A0 = 01. That is a command to send the data to the chosen motor. The data are computed from the duration of the signal in frame A0, and a signal to move is sent to the motor. The demultiplexer commands are defined as A1A0 = 00 do nothing, A1A0 = 1X, change motor, and A1A0 = 01 move motor. Thus, control of two motors using a single channel EEG is achieved.
Table 3 [40]. shows an experiment of a BMI using EEG demultiplexer in solving the problem of moving a robotic arm from A to B avoiding an obstacle at point C along the way.
Trial
EEG Demultiplexer
Robot
Threshold θc = 25
Output line
Motors
ak = sgn (Ck - θc)
Motor
Command
Position
C1
a1
C0
a0
1
20
0
0
0
c1
M0
NoOP
127
2
23
0
36
1
c1
M0
Move
112
3
1
0
88
1
c1
M0
Move
89
4
5
0
45
1
c1
M0
Move
76
5
0
0
62
1
c1
M0
Move
60
6
0
0
28
1
c1
M0
Move
53
7
23
0
31
1
c1
M0
Move
39
8
77
1
4
0
c2
M3
Switch
127
9
11
0
43
1
c2
M3
Move
138
10
0
0
47
1
c2
M3
Move
150
11
0
0
35
1
c2
M3
Move
159
12
0
0
44
1
c2
M3
Move
170
13
2
0
55
1
c2
M3
Move
184
14
0
0
75
1
c2
M3
Move
203
15
0
0
65
1
c2
M3
Move
220
16
0
0
65
1
c2
M3
Move
237
17
25
1
17
0
c1
M0
Switch
39
18
0
0
19
0
c1
M0
NoOP
39
19
0
0
54
1
c1
M0
Move
25
Table 3.
Experiment of a BMI using EEG demultiplexer to control a robotic arm to move from point a to point B avoiding an obstacle at point C along the way.
As can be seen from Table 3, the threshold value of the alpha band envelope is set to 25. At trial 1, the frame A1 has a value of 20 < 25, and frame A0 has a value 0 < 25. So the binary values of the input lines to the demultiplexer are a1a0 = 00. The output line of the demultiplexer is c1, which activates motor M0 which is for horizontal movement of the robot arm. The command a1a0 = 00 means “do nothing” and the robot arm stays and its initial position 127, which is in the start region A. In the second trial the subject generates EEG such that C1 = 23 < 25, and C0 = 36 > 25, so the input demultiplexer lines are a1a0 = 01. The currently addressed motor M0 moves from position 127 to position 112. The subject drives the robot arm horizontally, up to position 39. The obstacle is at position 35, so the subject changes the movement to the motor M3 which will move the arm wrist vertically. It should be noted that the subject does not know the internal coordinates of the motors, and s/he only sees the movement in space, and s/he estimates how far the robotic arm is from the visible obstacle. In trial 8 the subject changes the alpha rhythm pattern co that A1 = 77 > 25 and A0 = 4 < 25, i.e., a1a0 = 10 which changes the demultiplexer output and chooses the motor M3 which is still in its initial position 127. In trial 9 s/he moves that motor to position 136. With careful BMI control, the subject succeeds to achieve the goal area in robot coordinates M0M3 = (25, 217), avoiding the obstacle at M0M3 = (< 34, <217). Any position of M3 < 217 would hit the obstacle at M0 < 34.
Figure 16 [42] shows some results of the experiments of a BMI using EEG demultiplexer in controlling a robotic arm, as described above.
Figure 16.
Achievement motivation space for experiments in BMI for controlling a robot arm using an EEG demultiplexer.
The experimental investigation carried out 53 BMI experiments. Successful were 42 of them. Figure 16 shows example of 5 experiments. Here the goal region is marked with symbol
and the avoidance region (obstacle) with symbol
. The participants build behavioral trajectory through the achievement motivation space in order to reach the goal region while avoiding the obstacle. The coordinates are the internal robot coordinates, unknown to the participants. The participants use the view of the robotic arm to navigate the arm using their EEG.
As can be seen from Figure 16, the starting region of robot movement in each experiment is around the coordinate M0M3 = (127,127). Using BMI and controlling generated alpha rhythm in an EEG sentence, various trajectories are achieved toward the goal region M0M3 (>34, >217), avoiding the obstacle region M0M3 = (< 34, < 217).
An EEG emulated modem [40] is a process in which a sentence (message, command) is encoded in an EEG and is decoded at some receiving site, for example in a computer. Figure 17 shows the concept.
Figure 17.
EEG emulated modem.
As Figure 17 shows, the EEG signal is viewed as an EEG encoded sentence, which contains words represented by EEG frames. The sentence is encoded as an EEG modulation. A modulation process usually contains a carrier signal which is a good harmonic signal, modulated by a message. The EEG carrier signal is a stochastic (or chaotic) signal [43], and it has some statistical properties, such as mean value and standard variation, among others. And it can be decoded given some information about the encoding process. For example, if it is known that the message is encoded in the alpha band, then first the alpha band can be filtered out, and the envelope can be obtained containing the message, as it was done in [40].
Here the concept of modulation is wider than the classical harmonic signal modulation. It can be any way of encoding a sentence in an EEG.
The EEG modem is an approach toward application of BCI with a low number of channels, when several devices should be controlled with a minimum number of EEG channels.
EEG emulation of control circuits is a new direction in EEG research. It was introduced in 2005 with the concept of a CNV flip-flop. However, after the Vidal’s BCI challenge in 1973, the first explicit description of an EEG emulated electronic circuit was given in 1988. It was an EEG emulated switch, actually a Schmitt trigger, based on the EEG alpha rhythm. Recently in 2015 the EEG emulated demultiplexer and EEG emulated modem were described. This paper is a first review of this new direction in EEG research.
This research is supported in part by the current grant USDA/NIFA SCX-311-23-17 to the first author. Part of this research was supported by Macedonian-Croatian bilateral research grant to both the authors in the period 2007-2010. The authors would like to express gratitude to Drs. Stanko Tonković, Velimir Išgum, and Ratko Magjarević for their help.
References
1.Berger, H. Ueber das Elektrenkephalogramm des Menschen. Archiv fuer Psychiatrie Nervenkrank, 1929; 87: 527–570.
2.Vidal, J. Toward direct brain-computer communication. Annual Review of Biophysics and Bioengineering. 1973; 157–180.
3.Vidal, J. Real-time detection of brain events in EEG. Proceedings of the IEEE. 1977; 65: 633–641.
4.Farwell, L, Donchin, E. Talking off the top of your head: a mental prosthesis utilizing event-related brain potentials. Electroencephalography and Clinical Neurophysiology. 1988; 70: 510–523.
5.Bozinovska, L, Sestakov, M, Stojanovski, G, Bozinovski, S. Intensity variation of the CNV potential during the biofeedback training guided by a personal computer [abstract] (In Serbian). Neurologijasupppl 2. 1988; 37: p. 76.
6.Bozinovska, L, Bozinovski, S, Stojanov, G, Sestakov, M. Introduction of biofeedback in the CNV paradigm. (In Serbian), In: Proceedings Conference ETAN, Novi Sad, 1989; XII: 93–98.
7.Bozinovska, L, Stojanov, G, Sestakov, M, Bozinovski, S. CNV pattern recognition - A step toward cognitive wave observation, In: L. Torres, E. Masgray, M. Lagunas (editors). Signal Processing: Theories and Applications. Proceedings of Fifth European Signal Processing Conference (EUSIPCO 90); Barcelona. Elsevier Science Publishers, 1990; 1659–1662.
9.Bozinovski, S, Sestakov, M, Bozinovska, L. Control of a mobile robot using alpha rhythm from a human brain, (In Croatian), Proceedings Symposium JUREMA, Zagreb, 1988; 247–249.
10.Bozinovski, S, Sestakov, M, Bozinovska, L. Using EEG alpha rhythm to control a mobile robot. In: G. Harris, C. Walker (editors), Proceedings 10th Annual Conference of the IEEE Engineering in Medicine and Biology Society, track 17, Biorobotics, New Orleans, 1988; 10: 1515–1516.
11.Bozinovski, S, Sestakov, M, Stojanov, G, Bozinovska, L. Bioelectric control of mobile robots, (In Macedonian) Proceedings 6th Yugoslav symposium for applied robotics and flexible automation, Novi Sad, 1989; 237–242.
12.Bozinovski, S. Mobile robot trajectory control: From fixed rails to direct bioelectric control, In: O. Kaynak (Ed.) Proc. IEEE International Workshop on Intelligent Motion Control, Istanbul, 1990; 2:463–467.
13.Keirn, Z, Aunon, J. Man-machine communication through brain-wave processing. IEEE Engineering in Medicine and Biology Magazine, 1990; 55–57.
14.Craig, A, Kirkup, L, McIsaak, P, Searle, A, The mind as a reliable switch: Challenges of rapidly controlling devices without prior learning. In S. Howard, J. Hammond, G. Lindgaard, (Eds.) Human Computer Interaction. London, Chapman and Hall. 1997; 4–10.
15.Searle, A. Electrode performance and signal processing strategies for discrimination of EEG alpha waves: Implementations for environmental control by unconstrained subjects without training. PhD. Thesis, Department of Applied Physics and Technology, Sydney, 2000.
16.Mason, S, Birch, G. A brain-controlled switch for asynchronous control applications, IEEE Trans. Biomedical Engineering 2000; 47 (10): 1297–1307.
17.Sutter, E. The brain response interface: communication through visually induced electrical brain responses. Journal of Microcomputer Applications. 1992; 15: 31–45.
18.Chapin, J, Moxon, K, Markowitz, R, Nicolelis, M. Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex, Nature Neuroscience. 1999; 2: 664–670.
19.Lebedev M. Augmentation of sensorimotor functions with neural prostheses. Opera Medica and Physiologica. 2016; 2 (3):211–227.
20.Lebedev, M, Nicolelis, M. Brain-machine interfaces: from basic science to neuroprostheses and neurorehabilitation. Physiological Review.2017; 97: 737–867.
21.Liu, X, van der Spiegel, J. Brain Machine Interface. Springer Verlag. 2017.
22.Li, W, Jaramilo, C, Li, Y. A brain computer interface based humanoid robot control system. In Proc. IASTED Int. Conf. Robot., Pittsburgh, 2011; 390–396.
23.Emotiv. The introductory guide to BCI (Brain-Computer Interface). [Internet]. 2020; https://www.emotiv.com/bci-guide/
24.Ledoux, M. How existing assistive HMIs could change our near future: Part II [Internet] 2020; https://www.kinovarobotics.com/en/knowledge-hub/how-existing-assistive-hmis-could-change-our-near-future-part-ii
25.Sherz, P, Monk, S. Practical Electronics for Inventors. McGraw-Hill, 2013.
26.Bozinovski, S, Bozinovska, L. Brain-computer interface in Europe: The thirtieth anniversary. Automatika. 2019; 60(1): 36–49.
27.Barnes, M. Switching devices and independence of disabled people. British Medical Journal 1994; 309: 1181–1182
28.Bozinovski A. CNV flip-flop as a brain-computer interface paradigm. Proceedings 7th Symposium on Medical Informatics. Rijeka, Croatian Society of Medical Informatics, 2005;7:149–154.
29.Walter, G, Cooper, R, Aldridge, V, McCallum, W. Contingent negative variation: An electric sign of sensory-motor association and expectancy in the human brain. Nature. 1964; 203: 380–384.
30.Tecce, J, Cattanach, L. Contingent Negative Variation (CNV). In E. Niedermeyer, F. Lopes da Silva (eds.) Electroencephalography: Basic Principles, Clinical Applications, and Related Fields. Urban and Schwarzenberg, 1982; 543–562.
31.Bozinovski A, Tonkovic S, Isgum V, Bozinovska L Electrophysiology of the expectancy process: Processing the CNV potential. Proceedings of the 5th International Conference on Informatics and Information Technology. Bitola. 2007; 129–137.
32.Bozinovski S., Bozinovska L. Anticipatory brain potentials: An electrophysiological insight into the anticipatory behavior of adaptive learning systems. In M. Butz, O. Sigaud, S, Swarup (Eds.) Proc. Workshop on Anticipatory Behavior of Adaptive Learning Systems. Los Angeles, 2004; 1–10.
33.Bozinovski, A, Bozinovska, L. Anticipatory brain potentials in a brain-robot interface paradigm. Proc. 4th International IEEE EMBS Conference on Neural Engineering. Antalya. 2009; 451–454.
34.Meissner, L. The Science of Computing. Waldsworth Publishing Company, 1974.
35.Bozinovski, S. The Artificial Intelligence (In Macedonian) Gocmar Press, 1994.
36.Arkin, R. Behavior-Based Robotics, The MIT Press, 1998.
37.Bozinovski, A. Brain-machine interface - A 21st century dynamic technology: Anticipatory brain potentials and robot control. Proc of the Global Forum: Shaping the Future, Washington, DC, November 8–9, 2010
38.Bozinovski, A, Tonkovic, S, Isgum, V, Bozinovska, L. Robot control using anticipatory brain potentials, Automatika. 2011; 52(1):20–30.
39.Bozinovski, A. Control software for the Towers of Hanoi problem solving, using a manipulative robot – A case study, Proc. 2nd Balkan Confer4ence on Informatics. Ohrid. 2005; 366–378.
40.Bozinovski, S, Bozinovski, A. Mental states, EEG manifestations, and mentally emulated digital circuits for brain-robot Interaction. IEEE Transactions on Autonomous Mental Development. 2015; 7(1): 39–51.
41.Kuekes, P, Robinett, W, Seroussi, G, Williams, S. Defect-tolerant interconnect to nanoelectronic circuits: Internally redundant demultiplexers based on error-correcting codes. Nanotecnology. 2005;(16):1419–1432.
42.Bozinovska, L, Bozinovski S. An achievement motivation space for brain-robot interface. Proc. IEEE SoutheastCon, Huntsville, AL. 2018
43.Dmitriev, A, Kuzmin, L, Laktushkin, A. Amplitude modulation and demodulation of chaotic carriers. In Proc. 12th IEEE Workshop on Nonlinear Dynamics in Electronic Systems. (NDES'2004), Evora. 2004;138–141.
Written By
Liljana Bozinovska and Bozinovski Adrijan
Submitted: 26 July 2020Reviewed: 07 October 2020Published: 11 January 2021
Open access peer-reviewed chapter
