Spelled words by users
1. Introduction
There exist different methods to analyze and study biomedical signals. Some of these methods are based on medical imaging, involving Magnetic Resonance Imaging (MRI), Computed Tomography (CT), Nuclear Scintigraphy, etc. This techniques show the image of a specific part of the human body. But there exits other methods which study the different signals based in 1-dimensional analysis, such as Electroencephalography (EEG), Electrocardiography (ECG), etc. These do not show an image, but they also show relevant information.
These techniques are not only used to detect any anomaly, but in the case of the EEG, it is also possible to develop communication systems by Brain Computer Interface (BCI) [1]. The BCI technology has numerous applications which can improve the quality of life of those people who need external help at the time of communication or controlling their movements.
EEG records the electrical activity along the scalp produced by the neurons. This activity happens since the brain cells communicate each other giving place to tiny electrical impulses.
The impulse begins with a chemical discharge which origins a current in the membrane of the emitting cell. Once the impulse gets the extreme of the connection between cells, the neuron secretes a protein that inhibits or excites another neuron.
After receiving the signal, the neuron releases ions to the outside of the cell. When lots of ions are expelled at the same time they can stimulate other neurons. At the time this wave of ions gets to the electrodes, the ions can attract or push the metal of the electrodes. This difference of pressure can be measured by a voltmeter and the record of this activity along time is the EEG signal.
Different studies show that, with the correct training, it is possible to control different external devices with the BCI technology. In [2], Galán et al. present a study where two users have to control a wheelchair using a system based in BCI. They have concluded that the success has been higher after learning the interaction with the system.
With respect to the communication, the most used technique is the detection of the P300 wave as result of a photostimulus [3]. In these cases, the user has to watch one keyboard where the columns and rows are illuminated randomly. If the letter the person is watching is illuminated, the brain sends a signal as a response to that photostimulus. This answer is called P300.
In [4] the authors use a system based on evoked potential to control one robotic arm. The system illuminates different actions which can be made by the arm. To know which action the user is focused, they detect the P300 and the N2pc.
There are also different applications of BCI technology in different games, such as Mindball [5], which consist on moving a ball depending on the brain activity: the more relaxed and focused the user is, the more the ball will move.
But electrodes are not only used to measure brain signals, but they also register the electrical activity from other parts, such as the heart (ECG) or the skeletal muscles (EMG). It is also possible to pick up the muscular activity, as it could be the measurement of the eye movements (EOG).
We have decided to use an EEG as electrocardiograph (ECG) and as electrooculography (EOG). The used device is the same for both applications: a portable EEG with 4 different channels.
This device was initially acquired to work in one application in which it was necessary to carry a portable EEG. Due to its portability, it can collect data during long periods of time and outdoors.
The electrooculography (EOG) is based on the electrical activity of the eyes movement and it has been used for the user’s communication. There are other applications based on the eye movements to control devices. They detect the user’s iris, so it can know the position the user is looking through the coordinates of the iris. In [6] the study presents a method to control a wheelchair with one camera establishing three points as reference: the eyes and the nose. This three points form a triangle.
The ECG is a register of the electrical activity of the heart in order to know if there is any anomaly depending on the heart behavior.
We have used two different placements of the electrodes:
Bipolar placement: each channel represents the difference between two adjacent electrodes. This montage has been used for the EOG.
Referential or monopolar placement: each channel represents the difference between one electrode and another one established as reference. The ECG has been acquired with this montage.
The chosen electrodes are adhesive because of their facility to be placed both in the face as well as in the breast. The diameter of each electrode is 24 millimeters.
The electrodes are responsible for registering the voltage inside a single cell. An electric impulse, known as the action potential, is an electric discharge which travels along the cellular membrane. The action potential is used to transport the information between the tissues. Although they can be generated for different kind of corporal cells, the most active are the cells of the nervous system [7].
For the monitoring and the posterior analysis we have developed two user’s interface in Matlab: one for the ECG and another one for the EOG.
2. Relation between EEG and EOG
During the register of an EEG, there can appear different signals called artifacts. These signals can be originated by multiple causes: muscular movements, skin impedance, technical problems, etc. One of the causes of these artifacts can be the ocular movement, such as eye blinking. This electrical activity is registered in the frontal region of the brain, being FP1, FP2 the channels where can best be appreciable.
But to register the eye activity it is not necessary to place the electrodes on the head, it can be registered placing the electrodes in the face, which is called EOG.
An EOG is a method to register the eyes movement by placing little electrodes near the eyes muscles. Under normal conditions there exists a potential difference (10µV to 5 mV approximately) between the cornea and the Bruch´s membrane, located on the back of the eye. This is known as cornea-retinal potential. The cornea corresponds to the positive extreme and the retina corresponds to the negative of the dipole.
Figure 7 represents the differential of potential between the cornea and the retina:
Figure 8 show the behavior of the eye as a dipole:
In [8] it is well explained the different kinds of eye tracking systems and the evolution of them. For studying the eyes movement the criteria for divide the different techniques is different. There are three different kinds of eye tracking:
Invasive technique: this technique uses a special contact lens with an incorporated mirror or a magnetic field senor. The data acquired are more detailed because it is in contact with the eyeball, but is more uncomfortable because the user has to wear the contact lens.
Electrical potential: it uses electrodes placed around the eyes to detect the movement [9]. It is a robust method to measure the ocular movements when there are eyes blinking or change of the gaze. The main disadvantage is that it may become uncomfortable and the register can be affected by the electrodes movement. One of the main advantages is that it registers the eye movement with closed eyes so they can be used in the analysis of sleeping problems [10] and [11]. The EOG is within this kind of eye tracking.
Non - invasive technique: there is no contact with the eyes. The eyes movement is located through a camera or an optical sensor. The infrared light generates different reflections of the cornea of the user’s eyes known as Purkinje images [12] and [13].
The montage of the electrodes is bipolar: we register the difference between two electrodes. As the device is a 4 channel EEG, we have used them to perform 2 different positions: one for register if the user is looking on the left or on the right; and a second one to check if the user is looking up or down. The reference electrodes are placed in the ears.
The difference between channels 2 and 3 shows whether the user is looking up (positive) or down (negative). The difference between channels 4 and 5 determines if the user has looked to the left (positive) or to the right (negative).
The final application will have one user interface divided into two tasks:
Communicate between the device and the main application via Bluetooth.
Interpret what the user wants to express.
For the first task, we have used the API provided by the fabricant. To allow the user’s communication, the application will show the same keyboard as Figure 12:
The user moves one circle through the keyboard by moving his eyes to the left, to the right, up or down. Once the user selects “OK”, the application will use the T9 dictionary to interpret the correct word. The dictionary has been programmed taking in count the words which meaning is related with expressing feelings and emotional states.
Figure 13 show the difference between looking up and looking down. The subtraction of CH2 and CH3 accentuates the difference between the eye movements.
In Figure 14, which shows the difference between looking to the left or to the right, it can be seen that the situation is similar than looking up and down: looking to the left produces an increment of the signal value, while looking to the right decrements it.
3. Relation between EEG and ECG
An ECG is the graphical representation of the electrical activity precedent from the heart. It is a non - invasive method to study and to analyse the condition of the heart with the aim of detecting possible anomalies or diseases.
In order to synchronize the cyclical contraction of the heart, the fibers of the cardiac muscle transmit electrical impulses.
The electrical activity is the potential difference generated by the cardiac cells (each cell acts as a voltage generator) which are appreciable on the skin surface, where the electrodes pick up this activity in order to get a graphical representation.
This representation shows 4 significant parts:
“P Wave”: the cardiac cycle begins with the depolarization, which leads on the contraction of the atriums in order to deposit the blood in the ventricles. The sinus node (SA node) indicates the atrium muscles that they have to contract to begin the sequence. The potential action is propagated through the specialized cells of the cardiac muscle to the AV nodule.
“QRS complex”: the wave of depolarization arrives to the ventricles through the Bundle of His and the Purkinje fibers, producing the contraction of the ventricles.
“ST segment”: indicates the time between the end of the contraction of the ventricles and the beginning of the resting period.
“T wave”: indicates the repolarization of the ventricles.
Figure 19 is the representation of a normal ECG. There can be appreciated the different waves explained before.
When the electrical impulse arrives to a cardiac cell it provokes the interchange of ions inside the cell, giving place to a change of polarity. The action potential is the base for the depolarization and polarization of the myocardium: it is what generates the electrical impulse.
The electrodes are places along the trunk of the users, taking the shoulders as reference. Figure 21 shows the position of the electrodes for the correct register of the ECG.
The user’s interface has the aspect of Figure 22. The user can select the channels in both graphs.
4. Practice examples
4.1. EOG
In this section, it will be described how the EOG has been used to move a mark along a keyboard on the screen in order to help the people who are not able to communicate with the rest due a disease.
First of all, the application calibrates the position of the electrodes and establishes the correspondent thresholds to know the position of the eyes. After that, it will show the keyboard the user will use for the communication.
Once the application shows the keyboard, it will do the operations described in Figure 24 and Figure 25 to know if the user has made any action with the eyes.
The signal is obtained during 200ms and then the application calculates the average during this time interval. After that, it compares if there has been an enough increment or decrement to detect if the user has moved the eyes intentionally.
Figure 26 and Figure 27 show the calibration state of the EOG before showing the keyboard. The first two images correspond to looing up (yellow rectangle) and down (green rectangle).
Below, the pictures show the user looking to the left (dark blue) or to the right (cyan).
The EOG has been tested with 5 different users (3 female, 2 male) with ages between 23 and 30. They have been asked to spell 5 different words: WATER, FISH, SUN, DOG and MIND.
4.2. ECG
This section describes the ECG used to monitor the heart activity and to detect possible anomalies. In this case, the montage is monopolar, so we can obtain information from 4 different points (4 channels), establishing as a reference the user’s shoulders. The signals the ECG records are smoothed before being monitored.
After acquiring the signal (Figure 29), it is filtered with a Butterworth bandpass filter between 15Hz and 50Hz to detect the QRS complex (Figure 30).
Once the signal has been filtered, we have established a threshold to facilitate the detection of the R wave (Figure 31):
5. Results
5.1. EOG
We have asked the users to move a circle along a keyboard to spell five different words. We have divided the results into areas: the spelling of the whole word and the spelling of the single letters.
It has been established that if the user expends more than three attempts to move the mark, the objective letter is considered as an error.
Table 1 shows the obtained results for the different users:
|
|
|
|
|
|
WATER | o | o | o | o | o |
FISH | x | o | x | x | o |
SUN | o | o | o | o | o |
DOG | o | o | o | o | o |
MIND | o | o | x | o | o |
Average letter | 78.95% | 100% | 84.21% | 89.47 | 100% |
Average word | 80% | 100% | 60% | 80% | 100% |
Table 2 contains the average of each word by letter and by users. The number inside the parenthesis indicates the total number of letters of each word involving the five sessions (five different users).
|
|
|
WATER (25) | 100% | 100% |
FISH (20) | 65% | 40% |
SUN (15) | 100% | 100% |
DOG (15) | 100% | 100% |
MIND (15) | 90% | 80% |
After establishing different thresholds, it is possible to select the wished letter controlling the movement with the eyes with an average of the 90.53%.
Figure 32 and Figure 33 present the different movements for the spelling of the words DOG and WATER respectively.
The following two graphs are presented where the word FISH is correctly spelled (Figure 34) and wrongly spelled (Figure 35).
It can be seen that an exception has happened with the EOG at the time the user was spelling fish.
5.2. ECG
The ECG allows the detection of the principal waves for the posterior analysis of anomalies.
The ECG shown in Figure 37 presents a premature ventricular contraction after the T wave.
Figure 38 contains an ECG with arrhythmias: the cardiac frequency presents an alteration in the second 83 and the second 86.
The three graphs explained above correspond to three different users, showing the channels acquired for the 4 electrodes. The first one presents a normal ECG while the other two present two kinds of anomalies: premature ventricular contraction and arrhythmia.
Comparing the normal ECG with the rest, it can be seen that there are differences between the form of the wave and the RR intervals. In the second graph, there is a beat which is advanced in respect of a normal frequency. Because of that, the second graph presents few numbers of RR intervals (6 beats in 10 seconds).
Regarding to the third graph, the form of the waves is similar than the normal ECG, but the heart frequency is not constant: it can be appreciated that the heart frequency is not constant due to a little pause between the seconds 83 and 86.
6. Conclusions
As it has been demonstrated, the sensors used in BCI have an extra potential for applications which need high sensitivity.
The signals recorded by an EEG, an EOG and an ECG are bioelectrical signals; that is why the techniques to obtain the electrical activity of the brain, the eye movements and the heart activity are very similar. In the three cases, it is necessary to place electrodes in order to measure the biologic potentials.
The electrical activity of the eyes and the heart movement can be better appreciated than the activity from the brain. Because of that, it has been possible to use one EEG device as ECG and EOG as the resolution to distinguish the different signals is less than in an EEG. On the contrary, it could be not possible to use as EEG a device which has not been designed for that.
The election of the type of electrodes that are needed to be used is very important depending on the signal to measure and the needed resolution. The chosen electrodes are good for the recording of an EOG and an ECG, but their resolution may not be sufficient in some EEG applications.
Regarding to the EOG, the users have communicated the different letters with an average of 90.53%. With the appropriate training, the times of spelling could be reduced and it would be possible to increase the hit rate.
As future work there could be three different lines:
Use the EEG for analyzing more bioelectrical signals, for example EMG.
Analyze the signals in real time in order to detect any possible anomaly.
Improve the results of the EOG adding new functions to the interface and check the most adequate trainings to increase the percentage of success.
Acknowledgments
Finally, we would like to thank to those people who has helped to develop the application, especially to Eneko Lopetegui, because his advices and knowledge.
References
- 1.
Graiman B Allison B. Z Pfurtscheller G 2011 - 2.
Galán F Nuttin M Lew E Ferrez P. W Vanacker G Philips J et al 2008 Sep;119 9 2159 69 - 3.
Guger C Edlinger G 2010 462 94 8 - 4.
Sirvent J. L Azorín J. M Iáñez E Úbeda A Fernández E 2011 Apr;8 2 103 11 - 5.
Lin T. A John L. R 2006 - 6.
Purwanto D Mardiyanto R Arai K 2009 Dec 15;14 3 397 400 - 7.
Bear M. F Connors B. W Paradiso M. A 2007 - 8.
Duchowski A. T 2007 - 9.
Muthmainnah N Noor M Ahmad S 2012 3 5 - 10.
Liang S-F Chen Y-H Kuo C-E Chen J-Y Hsu S-C 2011 2104 2107 - 11.
Tagluk M. E Sezgin N Akin M 2010 Aug;34 4 717 25 - 12.
Talukder A Morookian J Monacos S Lam R Lebaw C Bond A 2004 - 13.
Weigle C Banks D. C 2008