Real–Time Low–Latency Estimation of the Blinking and EOG Signals

Electrooculography biosignals (EOG) are very important for the eye orientation and eyelid movements (blinking) estimation. There are many applications of the EOG signals. Most important applications are related to the medical applications [Duchowski (2007)]. The EOG signal is used for the analysis of eye movement in the selected medical test of the eye related health problems. It is also important for the sleep analysis. The EOG signal has much higher level than the important EEG (electroencephalography) signals and should be removed from the EEG measurements [Duchowski (2007); Shayegh & Erfanian (2006)]. The reduction of the EOG artifacts from EEG is considered by many researchers and it is also important for the practical applications of the EEG–based Human–Computer Interfaces.


Introduction
Electrooculography biosignals (EOG) are very important for the eye orientation and eyelid movements (blinking) estimation.There are many applications of the EOG signals.Most important applications are related to the medical applications [Duchowski (2007)].The EOG signal is used for the analysis of eye movement in the selected medical test of the eye related health problems.It is also important for the sleep analysis.The EOG signal has much higher level than the important EEG (electroencephalography) signals and should be removed from the EEG measurements [Duchowski (2007); Shayegh & Erfanian (2006)].The reduction of the EOG artifacts from EEG is considered by many researchers and it is also important for the practical applications of the EEG-based Human-Computer Interfaces.
Many alternative oculography techniques are available.The applications of the EOG signals for the HCI applications should be considered as one of the available techniques.The most important disadvantage is the long-time stability of the measurements and the influences of the other factors like light sources.The video-oculography (VOG) is interesting alternative, but the long-time influence of the infrared illuminators usually used on the eye has not been well tested.The infrared oculography (IROG) applies a small set of the illuminators (IR LEDs) and IR sensors for the estimation of eye movements.
The recent application of the EOG signal is the computer animation.The estimated orientation and blinking signals are used for the control of eye and eyelid of the human-generated avatar [Deng et al. (2008)].This is specific for the motion capture technique [Duchowski (2007); Krupi ński & Mazurek (2009)] that, for instance, was used successfully in Beowulf movie [Sony et al. (2006); Warner Brothers (2008)].Such a motion capture technique is alternative to the video-based motion capture systems fixed to the human head.
A measured biosignal has two important subsignals: electrooculography and blinking.Both of them should be separated and the interesting parameters should be estimated.The possibilities of application of the HCI system based on EOG depend on the biosignal measurement system and digital signal processing techniques applied to the obtained signals.

Electrooculography signal 2.1 Retina-cornea voltage source
The EOG measurement is based on the voltage measurement that depends on the eye orientation (Fig. 1).The voltage between retina-cornea is about ±1 mV [Northrop (2002); Schlgöl et al. (2007)], which is a very high value in comparison to other biosignals, but it also depends on numerous factors, for instance, the light conditions [Denney & Denney (1984)], the contact between electrodes and skin, which is the source of amplitude instability [Augustyniak (2001); Krogh (1975)].
One of the advantages of the EOG measurements over other techniques is that the field-of-view is not reduced by the glasses that are used as mounting platform for video or other sensors and illuminators.The electrodes are placed around eyes and are distant for the eye safety reasons.

Measurement system configurations
There is no single unique electrodes placement, so many configurations are possible and used in practice [Brown et al. (2006)].Such situation complicates the comparison of different algorithms used by researchers.The properties of the EOG signal change depending on the placement of electrodes, so the estimation technique should be robust.Additional calibration is necessary if the precise estimation is expected.
There are a few electrode configurations (Fig. 2) and in this chapter the 3/4 configuration is assumed: three main electrodes, for two differential measurements: LEFT-UP and RIGHT-UP, and an additional fourth reference electrode (REF).
The 3/4 configuration is minimal one for the full estimation of eye orientation in both directions and blinking too.Further reduction is possible if the single orientation is sufficient for a specific application.Another configuration 7/8 is the maximal variant [Thakor (1999)] that allows the measurement of very precise movement including every eye separately.It is important especially for the medical purposes.The configuration 4/5 is the compromise between both mentioned configurations.The large number of electrodes reduces the long time reliability due to the degradation of skin contact.The reduced number of electrodes is preferred for the applications where the touching of the human face is possible.The wires located especially below eyes in the 4/5 and 7/8 configuration is one of disadvantage.The 3/4 configuration allows the placement of electrodes in less visible parts of a face.
The number of electrodes depends on the acquisition systems.The first number represents the number of active electrodes used for measurements and the second number is the total number of electrodes.The additional electrode (the number 4 in the 3/4 configuration) is the reference electrode (REF).The acquisition systems typically use differential inputs.Such input type is preferred due to better SNR.Two channels are used and the first one is the LEFT-UP and the second one is the RIGHT-UP.The example signals are shown in Fig. 3. High impedance inputs are necessary due to the high resistance of the voltage source.The additional suppression of the 50/60 Hz interference is necessary [Prutchi & Norris (2005)], because the power lines are the source of the biosignal disturbances for the EOG signal, which has bandwidth about 200 Hz.Filtering techniques and appropriate wiring are used for the reduction of power lines interference.High frequency power line interference is omitted if a measurement system has the low-pass properties.The main source of high frequency interference is an incandescent light source.The power line interference is additive to a biosignal, especially, if the measurement systems wires are not shielded.Some portion of the power line interference occurs between electrodes on the human body.
The long time stability of the skin contact is obtained using the adhesive electrodes or an electrogel.The Ag/Ag-Cl electrodes are used typically.Such electrode types are conductive, but the other types (e.g.capacitance-based) are also used.The conductive electrodes support a DC signal.

Measurements settings and signal processing
The EOG biosignal needs a much higher sampling rate in comparison to other biosignal measurement systems.The sampling rate should be about a few hundreds samples per second.The low-pass filtering property of the measurement system is necessary.The AC coupling application used for the suppression of the DC signal is not correct.The DC level and low-frequency components corresponds to the eye orientation.The band-pass filtering in some measurement systems makes the differentiation of signals and the correction of measurements very hard or not possible.The reliable measurements should be obtained using the flat band-pass filters (e.g. the Buttherworth filter), but the phase distortions are introduced by the analog filters.The combined filtering of a signal using the analog and digital filters and higher sampling rates is necessary.
The number of quantization levels (the number of bits per sample) for EOG should be carefully set depending on the DC level processing.Even a 8-bit per sample is enough for not demanding applications with correct DC level maintenance and 50/60 Hz interference suppression before sampling.The higher resolution of quantization is used (e.g.12-16 bits per sample) if both mentioned components are hard to control by electronics.The higher resolution of quantization allows the signal processing using the digital signal processing algorithms.
The higher sampling rate and better quantization create the possibilities of precise observation of the EOG signal, which is important especially for the medical purposes.
The calibration of the system is necessary.A few extreme orientations of eyes are used and the intermediate orientations are interpolated.The HCI system requires the calibration before the beginning of measurements.The non real-time applications support the additional calibration at the end and intermediate calibration if they are necessary.More than single calibration allow the correction of the measurements and improve the acquisition results.
High quality measurements are recommended, but the signal processing of the obtained biosignal is necessary for the separation of the EOG and blinking signals.The estimation of parameters for both signals is necessary.

Blinking
The EOG signal has one important artifact and it is the blinking signal.The blinking occurs if the eyelid makes movements (vertical one).The disturbance depends on the electrode configuration and it is additive for the 3/4 configuration.The blinking pulse is similar to the Gaussian pulse for typical blinking (Fig. 4).
There are also atypical blinking cases when the eyelid moves very slowly or if the eyelid closing time is different than the eyelid opening time.There are much more cases when blinking is non-Gaussian, but they are not considered in typical systems.In this chapter typical blinking is assumed.

Saccades
The saccade is the rapid change (Fig. 5) of eye orientation [Becker (1989); Gu et al. (2008); Mosimann et al. (2005)].The rapid changes require the acquisition of high frequency components of a signal.This is one of the reasons why the necessary sampling rate is higher in comparison to the other systems.The low-pass filtering for the removal of the 50/60 Hz component using the cut of frequency about 30-40 Hz disturbs a saccade signal.
There are also interesting cases, for instance, when the blinking is near to the saccades.This situation is not rare in real measurements, but it is very often not considered by researchers.

Smooth pursuits
During the tracking of a slowly moving object, the eyes move smoothly (Fig. 6).Such movements are named as smooth pursuits.
There are also other features of the EOG measurements, like the microsaccades that are rapid movements of eyes but in the smaller scale.

Filtering techniques
There are many techniques for the separation of the EOG and blinking signals.The main techniques are based on the filtering (linear or non-linear) of blinking pulses.The signal with removed blinking pulses is the EOG one.The subtraction of EOG from the original signal gives a blinking signal.Such operation is applied independently to both channels in the 3/4 system.The pattern recognition techniques are used for the estimation of the position of the saccades (e.g. using differentiation and thresholding).The blinking pulses are detected using thresholding.The independent processing of both channels for blinking signal is necessary if the asymmetric blinking is possible.The typical blinking is related to both eyelids together, but the single eyelid blinking is possible too.
The typical filter used for the separation (Fig. 7) is a median filter and derivative filters that are used for the removal of the pulses [Bankman & Thakor (1990); Juhola (1991); Krupi ński (2010); Krupi ński & Mazurek (2010a); Martinez et al. (2008); Niemenlehto (2009)].The filtering based on a median filter is the simplest technique but not reliable.There are many cases when the blink removal is not possible: a typical case if a blink is after a falling saccade step or before a saccade step.

Blink filtering algorithm input signal
EOG signal blinking signal

Fig. 7. Typical separation technique
Such techniques are useful for real-time systems.The latency is higher than the maximal blinking pulse width.The rising edge of blinking pulse is simple to detect, but without the falling edge of blinking it is not possible to recognize it correctly.The rising edge for saccadic movement is similar.The availability of both edges of blinking is necessary for correct detection.The median filter calculates a median value from available values and is used for the estimation of the EOG signal level.The expected value is obtained if more than 50% values are assigned to the signal part without blinking.The median filter size (a median moving window) determines the latency.
The implementation of the median filter is possible using more efficient way due to the moving window requirements and full sorting is not necessary [Arce (2005)].

Analysis by synthesis technique
The estimation of signal parameters using the synthesis technique is possible (Fig. 8).Such a technique is based on fitting a signal generator model to a signal using the optimization techniques [Krupi ński & Mazurek (2010b;d;e;2011)].The additional constraints are added for improving results and for the reduction of computation time.A correct model related to the specific domain of synthesis is necessary.
The EOG signal and blinking one are well defined in time domain, so synthesis is performed by the comparison of the part (S)o fs e l e c t e ds i g n a l( s) and synthesized one (m).The aim of optimization process is to reduce an error value using, for example, the following fitting criteria: The estimated parameters are related to the selected part of a signal and the number of them depend on the number of detected features.Mazurek (2010c)].Real-time processing is very difficult and additional latency occurs.Some applications does not need the detection of saccades and the EOG signal is a signal that is used directly (e.g. in the motion capture applications or the analysis of point-of-interest).
In [Krupi ński & Mazurek (2011)] such a technique for the EOG and blinking signals is introduced.This algorithm uses evolutionary search with the mutation of a single child [Michalewicz (1996)].The additional gradient optimization is used for the computation time reduction by the local improving of convergence for a blink position and height, a saccade position and the value of the EOG signal level between two saccades.

Singularity processing
Another method of the event detection (a saccade, a blink) is the wavelets transform [Ghandeharion & Ahmadi-Noubari (2009); Reddy et al. (2010)].The saccade and blink are well defined in time domain and they have a limited length in time.
In [Bukhari et al. (2010a;b;2011)] the signal analysis and filtering of eye movements using scalogram and 'db4' wavelets up to details level 10 are considered.In [Reddy et al. (2010)] a few wavelets are compared ('sym8','haar','db4','db10','coif3') for the blink detection, and the best is 'sym8'.The threshold based technique is applied for the blink detection.In [Bhandari et al. (2006)] the signal enhancement techniques using wavelets are proposed.The 'coiflet3' wavelet for denoising is applied.Blinks and saccades are enhanced using the 'haar' wavelet.Discussed wavelets techniques by the authors are not sufficient for HCI systems.Moreover, there is a lack of the analysis of the more complex scenarios, like a saccade near to a blink, what appears in real measurements.
The wavelets are useful for the processing of signals and depend on the applied wavelet so the selected properties of a signal are emphasized [Augustyniak (2003); Mallat (1999); Mallat & Zhang (1993)].The selection of a particular wavelet defines the specific response for singularities too.
The non-isolated singularities need the multifractal analysis.The signals with singularities are analyzed using the singularity spectrum.The EOG signal with blinking is such a kind of signal that has the isolated singularities for most cases.The distance between events of any type is quite large, but both kinds of events may appear in short time and in such a case the limited multifractal properties exist.The analysis of the singularities is the basis of the detection and gives the possibility for real-time processing without median filtering (Fig. 9).
The singularities create the large amplitude values in their cone of influence what is observed in a singularity spectrum.The analysis of singularity spectrum is possible using the detection of local maximum for every scale.The maximal values of the wavelet transform coefficients |Wf(u, s)| are obtained by differentiation and testing the values.The zero value is obtained if a maximum point is found.
The additional conditions are necessary for the removal of non-strict maximum points that appear for the constant value of |Wf(u, s)| for some cases.
The detected maximum points are connected on the every scale.The parameters of a line: a length, an accumulated value over a line, and a slope are used for the detection of the even type and the estimation of parameters.The singularity analysis needs the tracking of the lines starting from the small scale to the largest one.The line length depends on the fitting of the wavelets to the singularity (event).A small length corresponds to the less important feature.The longest lines are taken into account.The length of a line is not only one method for the detection of features.The accumulation of the values along trajectories of accumulated singularity spectra is a technique used in this work.The accumulated value should be higher than a predefined threshold and this value is set by the previous observation of signal behavior.The wavelet shape 'gaus2' (Fig. 10) is applied for the signal processing by the continuous wavelet transform (CWT).
The following CWT formula is used for the computations: where * denotes the complex conjugate, a is a scale parameter, b is a position, and ψ is the selected wavelet.

Example 1 -two blinks and a single saccade
In this example the results of CWT for two blinks and a single saccade are shown.
First blink is slow and second one is fast (Fig. 11a).For the blinks and saccade positions the cones are visible and start from the smallest scale and they extend towards the larger scale (Fig. 11b).The local peaks are detected and multiplied by the wavelets coefficients and the results are shown in Fig. 11c.The accumulated values from the previous step are depicted in (Fig. 11d).The obtained positions are used as the starting positions for the tracking algorithm.The track-before-detect approach (TBD) is used and the simplified spatio-temporal track-before-detect algorithms are used with recurrent processing [Mazurek (2009;2010a;b;c;2011a;b)].There are three motion vectors used due to the high resolution of the scale.There is not need for searching more motion vectors.The result of tracking is shown in Fig. 11e.The tracking of values along lines is important and the accumulated value corresponds to the strength of the event (a blinking height and saccade differential height).The motion trajectory vectors are accumulated too (Fig. 11f).The motion vector near zero corresponds to the blink signal, because a wavelet function is similar.The slope is oriented to the left or right direction depending on the saccade falling or rising edge.The slope counter measures the motion vector (Fig. 11h) and is used together with accumulated values (Fig. 11g) for the threshold-based detection of the event.The detection of the event is shown in (Fig. 11i) where a peak and the peak marks correspond to the event type.

Example 2 -two blinks and two saccades and smooth pursuit
The next example shows the influence of the smooth pursuit on the processing of a signal.The smooth pursuit is very low frequency component of a signal and a very large scale should be considered by the CWT algorithm.The limited scale range allows reducing the influence of the trend from smooth pursuit.The filtering behavior (Fig. 12) is similar to Example 1.
The verification of the algorithm needs the computation of many examples using the Monte Carlo approach.

Performance analysis of wavelets separation technique 6.1 Monte Carlo approach and a signal generator
The Monte Carlo approach [Metropolis & Ulam (1949)] is applied for the tests of wavelets performance depending on a few conditions.The testing of the algorithms using synthetic EOG and blinking signals generator is necessary.Such approach is very good for the testing of algorithm.The tests based on the analysis of the recorded signal are limited by the number of available samples.The representative set of the real samples is necessary with the man-made description of every example.The synthetic technique needs a good generator, but the tests are much more reliable.The samples obtained by the real measurement process are related to the small set of humans.The EOG and blinking signals generator is described in [Krupi ński & Mazurek (2010a)] and used in the papers [Krupi ński & Mazurek (2010b;d;e;2011)] with some additional extensions (the smooth pursuit support).There are two possible techniques for the application of the generator.The generator could be used for any possible values of parameters and also for the limited values of parameters.Additionally, it allows testing the specific cases more deeply.

Test 1 -three saccades and three blinks
This test shows the performance of the algorithm depending on the noise (Fig. 13).
The signal consists of three blinks, three saccades, and the smooth pursuit is not applied.The Gaussian additive noise disturbs the signal.There are many sources of the noises in biosignal measurement systems related to the electrical properties of human body, a contact type and the measurement system.The external radio frequency interference is also the important factor of noise.The 1000 tests were processed.The maximal number of missing blinks is less than 4% for the higher noised samples (Fig. 13a).This figure depicts the reduction of the missed blinks due to the higher probability of assignment the noise peaks to a blink.The amount of overdetected blinks is depicted in Fig. 13b.The estimated position of the blink is disturbed too (Fig. 13c).The noise level does not influence significantly the position error for the standard deviation for about 0.08.The saccade position detector is more sensitive (Fig. 13d).This is expected behavior, because a single noise value may disturb the position of a saccade.The large values of overdetected saccades due to higher standard deviation noise values, creates false saccades (Fig. 13e) or shifts existing ones (Fig. 13f).The curves for the corresponding quality plots are similar for the blinks and saccades.

Test 2 -three saccades, three blinks and smooth pursuit
This is similar test to Test 1.The smooth pursuit signal is added so a trend occurred (Fig. 14).
It is expected that the results are similar to the case without smooth pursuit.The wavelets transform does not support very long time scales so the influence of wavelets processing should not be observed.The smooth pursuit is very low frequency signal and should be processed in similar manner like the constant levels of the EOG signals between neighborhood saccades.This test confirms that the trend does not affect significantly the results.This is very important for the applications, because no additional processing is necessary related to the smooth pursuit removal during estimation.

Test 3 -two blinks only
This test shows the influence of blinks (that are available) and saccades (that are absent) (Fig. 15).
The number of blinks missed is reduced proportionally in comparison to the previous Test 1.There is no influence due to false detected saccades.

Test 4 -two blinks, smooth pursuit
This similar test to Test 3 related to the influence of smooth pursuit (Fig. 16).
The results are similar to the previous test.The smooth pursuit does not influence significantly the algorithm.

Test 5 -two saccades
This is similar test to Test 3, but here exist only saccades without blinks (Fig. 17).
As expected there is no significant influence of blink detection on saccades.

Test 6 -two saccades, smooth pursuit
In this test the influence of smooth pursuit on the detection of saccades is tested (Fig. 18).The plots are similar to the previous Test 5.It means that there is no significant influence of smooth pursuit.
Additionally, in all tests there is no one wrong detection of the saccade slope direction (falling or rising).

Latency of wavelets transform technique
The real-time processing is possible using CWT and signal events estimator algorithm.The most important is the latency of the algorithm for the real-time applications like HCI.It is possible to use the alone EOG signal directly to control a computer.The blinking pulses are used as the additional control signals.The applications of the 3/4 electrode system need the low-latency processing, but the median filters are not suitable.The wavelets transform-based approach discussed previously has important advantage over other discussed techniques.latency of the system is limited by two factors.The first factor is the blinking pulses width.The saccades are immediate signals that are detected using tens of samples, depending on the sampling rate.They are well defined in time domain and occupies a very short time periods.
The long width of blink means that a system is limited only to the typical blinking.The typical blinking takes about 300-400 ms.It means that the length of the wavelet for the largest scale of CWT should be similar in a value or larger.CWT may process data in real-time and update results for every new samples.The forbidden region of detection occurs for the latest samples and is well shown in the right part of the CWT result e.g. (Fig. 11b).The calculation of the correct values in the cone that starts in the boundary samples is not possible.The calculation results (false) are available due to zero padding.The width of half-cone (Fig. 19) is the half of the wavelets width at the largest scale.It means that for typical blinking the latency is about 150-200 ms appropriately.
The tracking algorithm used for the selection of the peak lines is very fast due to the limited number of motion vectors and the high resolution of CWT.

Conclusions and further work
The application of the EOG measurements system has been rising.The EOG systems are used in different and new application areas.The signal processing techniques for real-time processing with high accuracy and low-latency are necessary.There are many processing techniques for the detection and separation of blinking and EOG signals.Most of them are not suitable for more specific cases and new ones are necessary.The recent research shows the importance of the wavelets transform with carefully selected wavelets function.
In this chapter the new wavelets-based technique for the estimation of blinking and saccade time moments using CWT was proposed.The estimation of the blink and EOG signals is important for the real-time HCI systems.Previous work related to the optimization approach [Krupi ński & Mazurek (2010b;d;e;2011)], using the blinking and eye movement model, was not well fitted for the real-time processing.The computation requirements were high and not defined by the number of processing steps due to the applications of the random number generator in the optimization algorithm.The proposed techniques in those papers were based on the evolutionary approach.The reduction of computation time was obtained by the selection of more efficient evolutionary operators.The computation time was also reduced by the reduction of the number of processed samples.The blink and saccade positions were also considered as a starting point for the optimization process near the global minima [Krupi ński & Mazurek (2010b)] and the sensitivity of this approach was considered in [Krupi ński & Mazurek (2010e)].Additionally, the estimation of the smooth pursuit movements was observed to improve the results [Krupi ński & Mazurek (2010d)].
The proposed in this chapter technique for the EOG measurements was based on the basic idea of CWT analysis introduced in [Mallat (1999)].The application of the spatio-temporal tracking of singularity improved the detection of the time moments and types of singularities like blinks and saccades.Wavelet techniques have well defined computation cost that is also constant, what is important advantage of this technique.The analysis of the slope of singularity in CWT image at all levels allowed the detection of a specific feature.
The application of the peak tracking algorithm allowed the detection of such event.The computation cost was quite low and nowadays available processing devices allow the processing signal at low cost.The EOG signal sampling frequency is usually small (100-400 Hz typically) so modern microcontrollers have enough computation power for signal processing, recommended are DSPs (Digital Signal Processors).
In the chapter the signal performance analysis of the algorithm and latency behavior due to wavelets and tracking algorithm were considered.The latency was obtained from the analysis of the blinking pulses and it was possible to reach 200 ms latency for a signal processing part, and if there were no additional latencies related to the measurement system it was also the overall system latency.Future research will be related to the implementation of the proposed approach using DSP.
The reduction of latency using the pattern recognition techniques is possible and will be considered in further research.
The synthetic, generated signals may fill the extended range of possible cases, which is important for the estimation of algorithm performance and finding the incorrectly estimated cases.The knowledge about incorrect cases is the source of important information for researches and developers about the estimation algorithm and further improvements are possible by the analysis of such cases.

Fig. 1 .
Fig. 1.Model of retina-cornea measurements for two different orientations

Fig. 3 .
Fig. 3. Example of the EOG signal -two channels for the 3/4 configuration

Fig. 19 .
Fig. 19.Latency area and latency time The advantage of this technique is the parallel signal separation, estimation, filtering and detection.The signals are separated using the model of one of them or both of them.The estimated signals are properly obtained if the model of both signals (EOG and blinking) is applied.The artifacts that are not assigned to EOG or blinking are considered as noise, but depend on the model quality and may be still interesting, e.g. for further microsaccadic movement analysis.The models are based on discreet events (blinking pulses, saccades) and additional pattern recognition techniques are not necessary.The obtained results are based on Analysis by synthesis technique the set of discreet events and the corresponding values like the height of a blink and an EOG level value.The disadvantage of this technique is computation power requirements [Krupi ński &