Brain Computer Interface Drone

Abstract

Brain-Computer Interface has emerged from dazzling experiments of cognitive scientists and researchers who dig deep into the conscious of the human brain where neuroscience, signal processing, machine learning, physical sciences are blended together and neuroprosthesis, neuro spellers, bionic eyes, prosthetic arms, prosthetic legs are created which made the disabled to walk, a mute to express and talk, a blind to see the beautiful world, a deaf to hear, etc. My main aim is to analyze the frequency domain signal of the brain signals of 5 subjects at their respective mental states using an EEG and show how to control a DJI Tello drone using Insight EEG then present the results and interpretation of band power graph, FFT graph and time-domain signals graph of mental commands during the live control of the drone.

1. Introduction

The brain computer interface (BCI) technology makes the possible manipulation of embedded systems using signals generated by brainwaves. A characteristic of the BCI system can easily capture brain signals generated by neural activities, it can also recognize differently firing neural activity patterns, and these signals can transform them into useful commands [1]. These commands can be utilized to control the machines or the devices. BCIs are most commonly applied in prosthetic limbs for paralyzed patients, exoskeletons, robotics, autonomous vehicles, virtual keyboard and computer games [2]. The BCI system can be classified as invasive, non-invasive (these are classified based on the location of placement of EEG biosensors). Non-invasive BCIs are based on electroencephalography (EEG) to record the brain activities using a series of biosensors disposed on the scalp will be able to measure the potential generated by the electrical activity of thousands to billions of cortical neurons inside our brain [3]. Our study is focused on noninvasive BCI using an Electroencephalogram. The neocortex is a convoluted surface which resides at the top of the brain. It is about ⅛ of 1 inch thick. It has 30 billion neurons arranged in 6 layers. Each neuron makes around 10,000 synapses with other neurons, which results in around 300 trillion connections in the total [4]. The most common type of neuron in the cortex is the pyramidal neuron, populations of which are arranged in columns oriented perpendicular to the cortical surface. The surface of the cortex is convoluted, with Fissures sulci, Ridges gyri. The neocortex exhibits functional specialization. Each area of the cortex is specialized for a particular function. The occipital areas near the back of the head specialize in basic visual processing [5]. The parietal areas towards the top of the head specialize in spatial reasoning and motion processing [6]. Visual and auditory recognition occurs in the temporal areas (towards the sides of the head) while frontal areas are involved in planning and higher cognitive functions. Inputs to a cortical area mainly come into the middle layers, Outputs of the cortical area leave from the upper and lower layers [7]. Based on these input–output patterns, the cortex roughly acts as an organized network of sensory, motor areas. Coming to the EEG, it is a device that extracts, organizes, and filters the electric signals which exist due to the neural firings (action potential) of the brain it is used for various diagnosing purposes, it is a popular non-invasive technique for recording the neuronal firing using electrodes placed on the scalp. The currents originating deep in the brain due to the firing of the neurons are not detected by EEG because the voltage fields will fall off with the square of the distance from the source [8]. The time domain signal displays the signals from different electrodes in a graph known as electroencephalography. EEG will reflect the summation of postsynaptic potentials occurring due to firing of thousands of neurons which are oriented radially to the scalp but not due to tangential electrodes. The spatial resolution of EEG is poor in a square centimeter range because of the impedance caused due to the presence of skull, scalp, CSF, meanings [9]. These layers’ act as volume conductors and low pass filters to smear the original signals, whereas coming to the temporal resolution is good at the range of milliseconds [9]. This time domain signal from EEG is then converted into frequency domain signal using different transforms in the signal processing such as discrete Fourier transform, fast Fourier transform, etc. The amplified frequencies (according to the fast Fourier transform) which are extracted from brain by electroencephalogram into 4 ranges they are theta (θ) which ranges from 4 Hz to 8 Hz, alpha (α) which ranges from 8 to 12, beta (β) ranges from 12 to 25 Hz and finally gamma (γ) ranges minimum from 25 Hz to maximum of 45 to 75 Hz [10]. After performing many experiments on many patients specifically to observe the type of waves and the amplified frequencies (when will they occur, in what state of patient these waves can be observed) they have presented a generalized form of relation between their frequency ranges and normal human functions. When a person is ready or about to perform tasks or if he/she is in an alert state then more percentage of α frequency waves are generally observed and if a person is task oriented or if he is in a busy state or anxiously thinking or actively concentrating then high percentages of β frequency waves are generally observed, if a person is performing high motor functions, or if the person is switching the activities during multitasking then high percentage of γ frequency waves are observed mostly in the frontal lobe of the human brain. After performing several tests, I was able to predict that in my meditation state a high percentage of θ frequency waves were observed even in a sleeping state where the mind is in a relaxed condition there are high percentages of θ frequency waves. The Emotiv Insight is an EEG Brain wear device which is composed of five sensors that are projected to acquire and measure the key activity from the entire functional areas of the cortex. The device can provide raw EEG Signals, Mental Commands (conscious thoughts), Facial Expressions - Facial mimicry and Measurements of performance of the brain. The principal key characteristics of this scientific design is the dynamic brain-computer interface interactions with more degrees of freedom for controlling physical and virtual objects. The device accurately identifies mental states and emotion such as Engagement, Focus, Excitement, Meditation, Relaxation, Stress [11]. There is a possibility to build brain activity models in real-time based on spatial resolution. A deeper perspective on specific patterns of an individual’s brain activity. The very important problem in EEG processing is low signal to the noise ratio since there are many layers between the neural cortex and the scalp and also due to the artifacts which result with great amplitude, the solution to minimize the noise is we can use filtering techniques and noise reduction techniques to remove the noise from the raw EEG data and extract the brain activity signals [12]. Since the EEG signal is non-stationary signal we use classifiers which are trained on user data (which is limited, this is also another main problem) we can generalize those results poorly to the already trained data on the same individual (different for different individuals because of physiological differences, this also limits the use of EEG applications) at different times. The accuracy might increase as we increase the number of training sessions but generalizing for subjects, i.e., to handle inter-subject variability processing pipelines with domain-specific approaches are used in order to clean, extract relevant features and classify (Riemannian geometry based classifiers, adaptive classifiers) EEG data. The subset of Machine Learning which is Deep Learning is used mainly to extract the features, Recently CNNs (convolutional neural networks) are used to simultaneously extract the feature and the classifier in order to achieve end to end supervised feature learning. Hence the devices use CNNs and recurrent neural networks of 3 to 10 layers in total [13].

2. Common EEG patterns

2.1 EEG

In most of the Neurosurgery hospitals or the hospitals where the diagnosis of the human brain takes place, they mostly use EEG which have very high Temporal and spatial resolutions, devices also occupy huge amounts of space. But in the study of Brain Computer Interface wherein we are required to develop the applications to control the external environment in such a way that the patient or subject or the user must not be facing the adaptability issues to significant extent [14]. If we observe the EEG devices which are used in the Hospitals, they generally require minimum of 2 to 3 hours for just equipping the device or placing the electrode in proper locations over the scalp, where in the contact optimizing fluid needs to be applied all over the head and the electrodes needs to be placed which is a tedious and complex process. But recently the company Emotiv has come up with a very sophisticated and easily adaptable device named as INSIGHT. This Insight device is equipped with very efficient specifications, where in which It has 3 axis gyroscope and 3 axis Magnetometer which are very helpful to remove the artifacts due to the head movements which is absent in the case of hospital EEGs (therefore the patients is instructed not to move their head/to avoid the motor movements). There are 5 important electrodes installed in the device which are made of semi-dry polymer. The electrode locations are 2 in the frontal region, 2 in the temporal region and another is at the central peritoneal region. The nomenclature of these electrodes are given according to the international 10–20 system. Where the frontal left electrode is named as AF3, frontal right electrode is named as AF4, temporal left electrode is named as AT7, temporal right electrode is named as AT8 and peritoneal central electrode is named as Pz plus DRL reference mastoid electrode on the left, the channels with built in digital 5th order sinc filter, bandwidth of 0.5-43 Hz with digital notch filters at 50 Hz and 60 Hz, 2.4 Hz wireless connectivity, 8400uV is the dynamic range (input referred), sequential sampling, 128 samples per second is the sampling rate, 14bit motion resolution, the electrodes are semi dry polymers, with 14bits 1 LSB = 0.51muV (16 bit ADC, 2 bits instrumental noise floor discarded) EEG Resolution. Main principle of EEG devices is the differential amplifier which has 2 inputs Input 1 and Input 2. From both the inputs the information is fed, the information can be mathematical or any signal it gives the output resultant signal which is the relative deference chunk of signal. Mounting the device over the subject’s scalp is a crucial procedure, where saline glycerol solution is applied to the semi dry electrodes in order to maintain the optimum contact quality with the scalp of the head. The electrodes were being placed in active locations according to the International 10–20 system protocol and the optimal 100% contactivity is ensured. The subject is instructed to not to move to avoid motion artifacts and stay focused on mental commands. Connection between the insight and the laptop is achieved by following the EmotivApp protocol, connection can be established either using Insight dongle or through the Bluetooth connection. Authentic Interpretation of EEG requires a very high amount of training and experience in analyzing and predicting the Graphical data. The most important set of rules which needs to be followed while analyzing the EEG data is the type of montages used, the time domain of the subject state compared to the EEG data at that current time in order to check that there are no external noises or movements made by the subject. There are different types of montages such as longitudinal bipolar Montage, longitudinal-transverse bipolar montage, circumferential bipolar montage, temporal bipolar montage, Cz referential Montage, Ipsilateral ear referential Montage in order to analyze the patient’s cognitive state in different ways in order to predict the correct result. In the EmotivePro app in MacBook Air laptop to record raw EEG data for the experiments I have used an Ipsilateral ear referential montage where one input of the differential amplifier will be a DRL reference mastoid electrode and the other input will be any one of the active electrodes. I set up the channel spacing to 400 μV minimum amplitude to −100 μV, maximum amplitude to +100 μV, with a high pass filter.

2.2 EEG pattern in eye blinks

As we know that the cornea is positively charged and the retina is negatively charged, when there is a movement of the eyes towards upwards due to the bell’s phenomenon Figure 1, when eyes are closed this will result in the abnormal signals in the frontal electrodes [15]. There is an upward peak then downward peak when eye is blinked once, I blinked the eye continuously therefore crist and troughs continuously occurred only at the AF3, AF4 frontal electrodes in the Figure 2, this is because of the bells phenomenon when the eyes are closed the cornea moves up due to which there is a change is potential observed near the frontal electrodes, there is an upward peak at the first when the eyes are closed which implies that there is a relative positive potential of AF3/AF4 with the reference electrode and when eyelids opened there is a downward peak which implies that there is a relative negative potential of AF3/AF4 with the reference electrode.

Figure 1.

Bell’s phenomenon during eye blink.

Figure 2.

Crests and troughs observed at AF3,AF4 electrodes in common reference montage EEG data.

2.3 Meditation

To cross check a theoretical aspect of the brain frequencies also in order to validate and prove that the device is authentic, we performed meditation mental tasks for diverse subjects and analyze their signal frequencies of their respective time domain signals using the Fast Fourier Transform (FFT) in the MATLAB.

2.3.1 Test dataset acquisition and observation

The test is being conducted in an anechoic chamber as we can observe at the background of the subjects in Table, in order to minimize the external noise and to isolate the experiment. The subject’s voluntary consent has been taken prior for performing this particular test experiment. Also, the subjects are strictly advised to be in a relaxed mental condition for 10 minutes before the test perusal and before 30 seconds of the performance of the test, the subject’s eye movements (closed state and opened state) also analyzed by the EmotivPro application in order to remove the eye blink artifacts. All subjects have performed the test successfully as instructed.

As the subject’s experiments begin, their mental state is being captured in the camera and the state at which the subjects are present is being noted and presented in the mental state column of Table 1. After ensuring the optimum contact quality, the EEG data following common reference montage is analyzed in the live, with EmotivPro license, it is possible to record the current EEG data of its respective electrodes.

Table 1.

The table displays the specific subject with their respective mental task.

After performing the tests on the subjects, the raw EEG data which is recorded during experimentation is stored in the EmotivPro application cloud. This recorded data is exported to the client server system in the .csv format, Figure 3 these files can be assessed through the links provided in the above Table 1 for the respective subject test.

Figure 3.

The imported data in the workplace.

This .csv file contains the recorded potentials of all 5 channels (AF3, AF4, T7, T8, Pz) at their respective timestamp. This data is copied and imported into the workspace of MATLAB. Database toolbox is used to read, write, import and export the data of .csv files Figure 4. Digital signal processing toolbox is used for converting time domain signal to frequency domain signal.

Figure 4.

The subject’s EEG data in the work space.

The main principle of fast Fourier transform is it converts the time domain signal into frequency domain signal. As the raw data is injected into the FFT in matlab, it analyzes the frequencies of the time domain signals. The xfft vs. absolute part of the FFT gives us the frequency domain signal. In this case, there is a large amount of noise observed in the frequency domain graph for the test dataset. Hence, in order to remove the noise and the high peak which is near to zero, the smoothing filters were applied independently and a band pass filter is also applied between 4 to 45 Hz, as the result of the experiment lies within that particular frequency range.

1. The length of the signal is assigned to variable = nfft,

2. In order to have a good resolution of the signal the nearest 2 power value of the length nfft is assigned to the variable nfft2.

3. The fast Fourier transform filter is applied and assigned to the variable fft.

4. In the absolute part of the fft, we can observe a symmetric curve with frequency x axis and amplitude magnitude of the y-axis.

5. In order to remove the symmetry about the line parallel to the y-axis and only consider the initial half part, we only consider half of the nfft2 length and that half portion is assigned to the variable = fff.

6. For the x component of the fft signal is assigned to the variable xfft, where input values of the sample frequency with half of the length of the data is assigned to xfft.

7. The graphical plot is made with xfft as x-axis and fff as y-axis for that particular time period. Savitzky Golay filter, moving average filter, smoothing filter were applied and analyzed independently to all of the test files. After applying filters to the signals, their respective frequencies are extracted, recorded and saved and the smoothing filter is considered to be the best fit filter at a window of 100 neighboring samples for our test datasets.

2.3.2 Test results

The smoothing filter, band pass filter along with FFT in combination resulted in best output for the frequency domain analysis and the results are observed, noted and represented graphically in Table 2 as there is time domain graph at the top and frequency domain graph at the bottom in the experimental analysis column in Table 2.

Table 2.

This table represents the experimental analysis graphically.

As most of the subjects are performing the meditative test they should experience a very high amount of theta waves they ranged between 5.1 to 7.6 Hz i.e., there will be a maxima peak within the range of 5.1 to 7.6 Hz in the frequency domain signal and also when the subject performs a demanding motor task he experienced a high amount of gamma waves with maxima peak at 35.9 Hz (Table 3).

S. No	File Name	Mental State	Frequency at Maxima (x-axis)	Maxima (y-axis)
1	InsightTestNithin01	Mathematical Calculation	35.9 Hz	45375.0
2	InsightTestMurty05	Meditating	05.5 Hz	17920.0
3	InsightTestNithin06	Meditating	05.6 Hz	18475.0
4	InsightTestNithin07	Meditating	07.6 Hz	24568.0
5	InsightTestNithin08	Lifting	35.9 Hz	13585.0
6	InsightTestSai09	Meditating	05.1 Hz	5016.8
7	InsightTestOnesimus10	Meditating	05.6 Hz	8094.6

Table 3.

This table represents the frequency analysis of subjects numerically.

Hence, the test results prove that the subject’s mental state falls within specific range when a specific mental task is performed. This ends the validation part of the Emotiv Insight EEG device.

3. Common EEG patterns

After connecting the device with the EmotivApp, the EmotiveBCI application is used to train mental commands. After training, the live mode is switched ON to control a cube with our Imagery thoughts. Next those live mental commands should be extracted from the application to integrate with the drone. This is achieved through cortex API documentation, for the ease I have edited the code in python language. Then simultaneously those commands should be integrated to a dji tello drone. This is achieved through dji tello API documentation. Finally a link between the drone and Insight brainware is achieved.

3.1 Mental command training

Giving mental command to the EmotivBCI application will result in the movement of the object in the desired direction. Initially numerous training needs to be done to the device in the initial stage to get good desired output. For example during an object movement test when the subject is thinking to move the cube towards the right direction we can observe this in the Figures 5 and 6 represented below that there is a movement of the cube towards the right direction. Each of the Neutral, moving left, lift, drop mental commands were trained for 10 times.

Figure 5.

The object movement test of the subject, live mode in BCI App when the person is neutral.

Figure 6.

The object movement test of the subject, live mode in BCI App when the person thinks of moving the object left from neutral, the cube has moved left.

3.2 Extraction of mental commands and assigning them to the drone

The python code is built in Atom software editor in Dell Inspiron laptop. Jason, websocket, ssl, time, win32, requests, pyautogui, socket, keyboard, threading are the libraries used in the code. The control of the drone with the computer is achieved using DJI Tello drone protocol Figure 7, where the drone is connected to the local wifi and the laptop should also be connected to the same wifi using a TP-Link USB Wi-Fi Adapter which is used for PC(TL-WN725N) it’s an N150 Wireless Network Adapter for laptop, UDP protocol is used to make an interface between the computer system and dji tello drone, it should be explicit and bind to a local port on our computer where tello can send messages. The functions that listen for messages from tello will be printed on the screen. Connection between the insight and the laptop is done according to the Emotiv App protocol, connection can be established either using Insight dongle or through the Bluetooth connection. After connecting the device with the EmotivApp, the EmotiveBCI application is used to train mental commands. After training, the live mode is switched ON to control a cube with our Imagery thoughts. Next those live mental commands should be extracted from the application to integrate with the drone. This is achieved through cortex API documentation. Then simultaneously those commands should be integrated to a dji tello drone. This is achieved through dji tello API documentation. Finally a link between the drone and Insight brainware is achieved. Coming to the Insight, InsightHandler class is used then the web connection is acquired, In order to get the approval form the Emotiv app we have to generate client id and secret then we have to give the approval in the emotive app. After approving then authorisation happens to generate the token this token will create the session and loads the profile from the cortex app, then we are going to start and stream the mental commands data in our terminal, these commands will be integrated with the our keyboard clicks, these clicks will intern control the drone as a controller, so basically as we are thinking of a mental command which will be controlling our computers keyboard this will intern control the drone.

Figure 7.

Procedure of interfacing EEG with a drone.

4. Result

After several attempts the drone control was modified and whenever the power meter of a mental command crosses 50% then that specific keyboard key will be clicked. For example if the subject is in a neutral state and suppose he is thinking of a lift command and simultaneously if the power meter of that lift mental command is at 50% or crosses 50% then that command will be executed and the computer keyboard key which is assigned to the push will be clicked this will intern transmits the thrust command from the laptop and drone will receive this command and lifts off the floor. In the terminal window and live mode in the Mental command window controlling a cube are represented in the Figures 8–10. The cube is in the neutral position therefore we can observe neutral in the terminal window in Figure 8. When the person is thinking of lifting the cube, the cube changes from its neutral position and moves upwards therefore we can observe change from neutral to lift in the terminal window, Figure 9. The cube is in the neutral position therefore we can observe neutral in the terminal window, Figure 10.

Figure 8.

Cube in neutral state.

Figure 9.

Cube moving upwards.

Figure 10.

Cube moving downwards.

The drone is in the resting state when the mental command of the subject is neutral, Figure 11. The drone takes off when the mental command of the subject has changed from the neutral to lift, Figure 12.

Figure 11.

Drone in the neutral state.

Figure 12.

The drone takes off.

5. Conclusions

EEG patterns such as motor movements, eyes movements, meditation, sleeping tests are recorded and analyzed. A mathematical model is developed in the MATLAB using concepts such as signal processing in order to analyze the theoretical data of the brain frequencies at different mental states. The validation of the device by analysis of raw signal has been performed using signal processing methods such as fast Fourier transform simultaneously applying filters to extract the signal of interest by removing noise. These EEG patterns are analyzed on 5 different subjects and cross validated the data with the theoretical brain frequencies data. After these above experiments, Interface between DJI Tello Drone and Emotiv Insight BCI headset was achieved and the Drone was controlled with 2 mental commands moving up and moving down from the neutral state.

Acknowledgments

Inspiration, motivation and presentation have always played a key role in the success of any venture. I express my sincere thanks to “Prof. S. Murugesan” Associate Professor, Department of Pharmacy, Birla Institute of Technology and Science, Pilani being my supervisor and encouraging me to the highest peak and providing me the opportunity to prepare for this research. I feel to acknowledge my indebtedness and deep sense of gratitude to “Dr. Anantha Krishna Chintanpalli” Assistant Professor, Department of Electrical and Electronic Engineering at BITS Pilani for being co-supervisor whose valuable assistance to me in this course has shaped my present work. I am immensely obliged to my friends for their elevating inspiration, encouraging guidance and kind supervision in the completion of my project. Last, but not the least, my parents are also an important inspiration for me. So with due regards, I express my gratitude to them.

CPZ	Central Peritoneal Zero
Cz	Central Zero
DBS	Deep Brain Stimulation
DCES	Direct Electrical Cortical Stimulation
ECoG	Electrocorticography
EEG	Electroencephalogram/Electroencephalography
fMRI	Functional Magnetic Resonance Imaging
Hz	Hertz
InsightTestNithin06	Insight: Device Name.
	TestNithin: Testing the subject name.
	06: Test number
IZ	Inion Zero
kHz	KiloHertz
LTD	Long Term Depression
LTP	Long Term Potentiation
MEG	Magnetoencephalography
MU	Memory Unit
Nz	Nasion Zero
OS	Optical Stimulation
Oz	Occipital Zero
POZ	Peritoneal Occipital Zero
Pz	Parietal Zero
STD	Short Term Depression
STDP	Spike timing dependent plasticity
STF	Short Term Facilitation
STP	Short Term Potentiation
TMS	Transcranial Magnetic Stimulation
TUS	Transcranial Ultrasound Stimulation