Accessing and Processing MEG Signals in Real-Time: Emerging Applications and Enabling Technologies

1. Introduction

Magnetoencephalography (MEG) offers great temporal resolution; however, traditional MEG studies have only exploited this advantage during offline analysis. With the advancement of computing capabilities, the ability to process and analyze MEG data during an experiment, and even in real-time is now a reality. Online MEG processing opens new opportunities for basic research and clinical applications. For example, by providing real-time feedback a subject could learn to modify their neural activity patterns. This real-time neurofeedback training may be useful in rehabilitation, for example to promote neuroplasticity to strengthen or retrain motor-cortical activity after central nervous system damage due to conditions such as stroke or spinal cord injury. Online MEG analysis could also be used for advancing neuroscience research by allowing for adaptive paradigms that could identify which stimuli are most effective, or to determine if more data needs to be collected to fulfill analysis criteria. However, online MEG presents a number of challenges, including managing the complexities of the data acquisition, addressing noise contamination, and dealing with the processing requirements of data analysis methods. This chapter introduces the concept of online MEG and discusses current research efforts to address its challenges while presenting some novel applications to advance basic and clinical research.

Processing of brain activity in real-time is not a new concept. Observing the complex dynamics of the brain on a millisecond scale has provided great insight into how the brain works in healthy individuals and those with neurological disorders. To date, most real-time analysis of brain activity has been with electroencephalography (EEG) using electrodes placed on the scalp. Because EEG is non-invasive and poses essentially no risk of injury, it can be performed on healthy individuals and patients. However, the spatial resolution and effective bandwidth of EEG are limited, because electrical signals passing through the skull and scalp are greatly attenuated, particularly at frequencies above 50 Hz (Nunez & Ramesh Srinivasan 2005). Intracranial EEG, commonly referred to as electrocorticography (ECoG) alleviates this issue of tissue attenuation by placing electrodes beneath the skull closer to the brain. ECoG is used routinely in the diagnosis and treatment of patients with intractable epilepsy (Leuthardt, Schalk, Wolpaw, Ojemann, & Moran, 2004). Cranial surgery is required to place ECoG electrodes and therefore its utility for many research, and even clinical, investigations is limited. MEG provides a safe, non-invasive alternative capable of capturing the high temporal resolution dynamics of the brain activity. However, the magnetic fields generated by neural activity are extremely small and require complex hardware and software to collect. By averaging across many events, traditional MEG analysis improves the signal-to-noise ratio to better emphasize spatial and temporal characteristics. Yet, by averaging across trials, MEG loses its ability for accessing neural activity in real-time and therefore limits the application of this sophisticated technology.

One area where real-time MEG (rtMEG) is being used is the development of brain machine interface (BMI) applications, which require analysis of brain activity in real-time. Most BMI applications are focused on using an individual’s brain activity to directly control the actions of a device. It has been demonstrated that a delay longer than 200 ms between a person’s movement intention and a device’s reaction is noticeable and can be distracting and lead to degraded task performance (Lauer et al. 2000, Welford 1968). This 200 ms delay is considered the maximum time a system can use for collecting and processing the neural data and for driving the device. For these reasons, this chapter considers a system delay less than 200 ms to be “real-time”.

The majority of BMI systems have utilized EEG because it is noninvasive, the equipment is readily available, and the standard signal processing algorithms can be implemented easily in real-time (Mason, Bashashati, Fatourechi, Navarro, & Birch, 2007). EEG-BMI systems have been used successfully to control devices with a few degrees of freedom (Wolpaw & McFarland 2004, McFarland et al. 2010). However, for complex device control a system must be able to localize multiple separable sources of neural activity with high spatial and temporal resolution.

An alternative non-invasive method of assessing brain activity that does not rely on electromagnetics is “real-time fMRI” (rtfMRI) (DeCharms 2008). First established in 1995, rtfMRI combines the high spatial resolution found with recording the hemodynamic response (i.e. Blood-Oxygen-Level Dependence – BOLD) with advanced computational ability for reconstruction (Cox et al. 1995). Real-time fMRI is a valuable tool that can be used for many of the applications discussed in this chapter including many of the adaptive paradigms. However, the temporal resolution of rtfMRI is limited by the biological signals being recorded; the hemodynamic response. The hemodynamic response inherently has dynamics that change only on the order of multiple seconds. This fundamental limitation to the temporal resolution means rtfMRI neurofeedback systems have delays on the order of several seconds, which would be very noticeable to the participant. In addition, the hemodynamic changes recorded with fMRI are not direct measurements of neural activity as are the magnetic fields recorded with MEG. The high spatial resolution is a great advantage to fMRI, but the hemodynamic timing limits the applications rtfMRI can be used for.

While other neural recording technologies have utility across a range of applications, this chapter focuses on MEG for real-time analysis of neural activity with high spatial and temporal resolutions. This chapter covers some of the current and potential applications of rtMEG. Benefits of online analysis of MEG data range from real-time monitoring of data integrity and experiment validity, to clinical neurofeedback paradigms for the rehabilitation of central nervous system injuries and disorders. Also discussed are some of the remaining challenges that occur with real-time analysis of high throughput data as well as some potential solutions that can be applied.

2. Applications for online MEG

2.1. Basic neuroscience research

The ability to access and process MEG data in real-time opens up many new opportunities for basic neuroscience research. Applications as simple as having real-time visualization of neural data would be useful for assessing data quality during an experiment. Commercial MEG systems display the raw, and even filtered, magnetic signals in real-time, but it is difficult for operators to interpret 200-300 channels of data quickly during a scanning session. Often MEG operators are interested in particular features of the data, such as specific frequency bands or sources of activity in specific anatomic areas. Real-time spectral analysis, source-localization and visualization tools would be very beneficial. Visualization can be straightforward, such as displaying the power spectra for individual channels (see section 4.2), or complex such as projecting the MEG data into source space to display neural activity mapped onto the space of the cerebral cortex (see section 4.3). By watching neural characteristics, such as the frequency content in time, and changes in source localization, experimenters can quickly determine if the paradigm needs to be changed or if more data needs to be collected.

A powerful use for rtMEG is in running adaptive paradigms where the experiment progression is determined by the neural data that was collected (MacKay 1992, Chaloner & Verdinelli 1995). Adaptive paradigms can use outcomes from prior trials (e.g. neural responses to specific stimulus classes) to determine what the next stimulus should be. For example, neural activity can be analyzed during an experiment to determine if a task is too easy or too hard for eliciting the required neural activity. In a memory task, for example, experimenters might be able to adjust the difficulty of the paradigm to elicit activity in a particular brain area.

Real-time data analysis is also useful to quickly determine if sufficient amounts of data have been collected for a given stimulus. One common example is localization of motor or sensory cortices where a large number of trials (>100) are typically recorded to ensure that sufficient data is available for offline analysis. Limiting the number of trials can reduce the scan time for each stimulus and ensure sufficient data was collected thereby making the most of the time in the scanner.

Analyzing MEG data during an experiment can also be used to rapidly determine if a subject’s brain activity fits a certain criteria for a study. For example, subjects could be screened for their ability to modulate neural activity during a given paradigm. By analyzing this data in real-time, the amount of time spent on unnecessary data collection and offline data analysis can be reduced. This can allow experimenters to quickly determine whether the experimental paradigms should proceed or if different paradigms should be performed.

2.2. Applications of real-time MEG in neurology and neurosurgery

MEG’s most established clinical use is for providing non-redundant localizing information for epilepsy patients being considered for surgical treatment (Bagic et al. 2011, Bagic et al. 2009, Burgess, Funke, et al. 2011, Stefan et al. 2011). MEG has also received increasing acceptance as a superb non-invasive tool for localizing eloquent cortices in presurgical functional brain mapping in patients with tumors and other operable lesions (Burgess, Barkley, et al. 2011). Further efforts of the MEG community are focused on establishing new clinical indications where dementia (Zamrini et al. 2011), traumatic brain injury (TBI) (Huang et al. 2009, Maruta et al. 2010) and autism (Roberts et al. 2011, Roberts et al. 2010) are considered current front runners among many other neurologic and psychiatric disorders that are being studied (Stufflebeam et al. 2009, C. Stam 2010). These clinical applications have been developed with the idea that MEG data must be processed offline. However, by assessing MEG data during a patient session, many useful tools and new clinical techniques can be developed.

One of the most appealing uses of online MEG analysis would be to provide instant results to clinicians. For clinicians who regularly spend many hours analyzing complex epilepsy cases (Bagic et al. 2011, Burgess, Funke, et al. 2011), having a software package that can process data online and provide at least preliminary localizing information by the time a patient walks out of the magnetically shielded room would be invaluable. However, at least initially, more stereotypical and less labor-intensive clinical applications, such as presurgical functional brain mapping (PFBM) using MEG evoked fields (MEFs) (Burgess, Barkley, et al. 2011), are more likely to be amenable to this type of application. Even if these analysis methods are not automated and would initially require the interaction of an expert operator, they would still provide invaluable time-saving and could directly increase efficiency and quality of patient care. This is especially the case in urgent situations leading to a surgical intervention, when the timeliness of a clinical decision is critical. With quick analysis methods that could begin while a patient is still in the MEG scanner, ideally, neural imaging reports could be sent directly to the operating neurosurgeons’ planning workstations so patients could have a seamless transition from the MEG room to the operating table. Furthermore, neurosurgeons or other collaborating physicians could request additional data as needed while the patient is still in the MEG scanner.

Another very valuable clinical application for online MEG would be to provide MEG clinicians feedback on the quality and quantity of the data that has been collected. By evaluating the MEG data online the clinician could receive instant feedback on when enough data have been collected for mapping a particular functional modality, such as motor or language mapping. Similarly, software could provide indications to the clinician if the data collected were not sufficient for mapping and additional data are needed. This would help ensure that the amount and quality of collected data was sufficient to render an accurate clinical diagnosis. Of course, making rtMEG accessible and accepted by clinicians will require a collaborative effort between the signal processing community to develop automated systems and the clinical experts to guide and validate the development of the specific technology.

2.3. MEG for BMI technology research and development

While unsuitable for portable BMI applications, MEG can play an important role in BMI research and development as it offers a non-invasive, whole-head, and reasonably high-resolution brain interface with real-time capability. For instance, MEG can be used as an approximate surrogate for invasive technologies that place electrodes directly on the brain surface. Several studies have suggested that MEG might share similar spatial and temporal characteristics as direct cortical surface recording (i.e. ECoG) in terms of source localization accuracy and capability to resolve cortical activity represented by amplitudes of different frequency bands (Dalal et al. 2008, Korvenoja et al. 2006, Gharib et al. 1995). It was also demonstrated that movement-related information could be decoded accurately from MEG signals (Georgopoulos et al. 2005, Waldert et al. 2008, Wang, Sudre, et al. 2010). Figure 1 shows the time-frequency responses of a contralateral MEG sensor (a gradiometer) when a participant performed simple center-out wrist movements (Wang, Sudre, et al. 2010). There is a clear decrease in power for the low frequency sensorimotor rhythm (10-30 Hz) and a distinct increase in power for the high-gamma band (60-200 Hz) during movement. These changes in low and high frequency bands are in agreement with previous MEG (Waldert et al. 2008) and ECoG studies (Leuthardt et al. 2004, K. J. Miller et al. 2007, Wang et al. 2009). In addition, Wang et al. (Wang, Sudre, et al. 2010) demonstrated that high-gamma band activity captured by MEG showed directional modulation similar to what was observed previously using invasive recordings in humans (ECoG) (Leuthardt et al. 2004) and non-human primates (local field potentials) (Heldman et al. 2006) and has been used for BMI control.

Figure 1.

Spectrograms of MEG signals recorded from one gradiometer (marked by the black arrow) during overt wrist movement. A total of 84 trials were aligned at target onset (time = 0) and averaged. The color indicates the percent change in spectral power from baseline. The arrangement of the four spectrograms corresponds to the four directions of wrist movements performed: up, down, left and right. The low frequency band shows a decreased power during movement in all directions while the high frequency band shows increased power during movement with directional preference for movement to the right. Inset: The red dots represent four MEG sensor locations (of the 102) whose high frequency band showed directional preference to contralateral wrist movement (p<0.05).

Recently, Mellinger et al. demonstrated that real-time MEG processing could provide control over a computer cursor in one-dimension with the individuals modulating their sensorimotor rhythms (mu and beta frequency bands) using imagined hand and feet movements (Mellinger et al. 2007). In this study, 3 out of 6 able-bodied participants achieved reliable cursor control (accuracy around 90%) with less than 45 minutes of training. More recent pseudo-real-time studies have shown that by using more advanced decoding algorithms only 5 minutes of initial open-loop data is needed to achieve reliable one dimensional device control using hand-related sensorimotor rhythms (Foldes et al. 2011). Limiting the amount of time spent collecting initial data means more time can be devoted to performing neurofeedback training tasks.

The non-invasive nature of MEG and its recording capability, as demonstrated in the above examples, support the utility of MEG for investigating basic neuroscience questions and piloting engineering solutions for BMI research and development. Using MEG, BMI studies can be conducted with large numbers of participants including both able-bodied individuals and individuals with disabilities (e.g. spinal cord injury, degenerative neurological disorders, stroke and etc.), while posing minimum risks to study participants. Important neurophysiology questions can be investigated across multitudes of subjects to better determine the cortical substrates and plasticity for BMI control. For example, what is the most dominant type of movement information in human primary motor cortex; abstract movement information (e.g. movement direction; (Georgopoulos et al. 1986)) or detailed somatotopy representation as established originally by Penfield and colleagues (Penfield & Boldrey 1937)? Furthermore, while non-human primate studies have suggested that motor cortical neurons encoding individual finger movements are generally mixed in anatomical location (Schieber & Hibbard 1993), several recent human ECoG studies have suggested that in human motor cortex there might exist at least some level of somatotopy or separation in finger representation for movements of different fingers (Kubanek et al. 2009). Such matters are further complicated by potential cortical reorganizations induced by corticospinal lesions in the individuals who would benefit from BMI technology (Cramer et al. 2005, Kokotilo et al. 2009). These questions about motor cortical representation of movement and its potential difference between able-bodied subjects and patients with chronic disabilities, are fundamental scientific questions of critical clinical importance for the research and development of BMI systems. MEG, complementary to other invasive and non-invasive neural recording tools, can provide important insights that can effectively guide the design specifications for implantable neural interface electrodes and neural decoding algorithms. Furthermore, real-time MEG provides a safe test-bed for researchers to investigate various neural decoder training paradigms, such as action observation-based neural decoder training paradigms (Tkach et al. 2008, Velliste et al. 2008) and co-adaptive paradigms where a neural decoder is frequently updated in parallel to potential cortical adaption during BMI training and operation (D. M. Taylor et al. 2002).

MEG can also play a direct role in pre-surgical planning and patient training for invasive or minimally invasive BMI systems. For one, MEG can be used to localize cortical areas that are significantly modulated by intended movement direction, and intracranial electrodes can then be implanted at those cortical sites (Wang, Sudre, et al. 2010). Accurate localization is important particularly in individuals who may have cortical reorganization secondary to injury (e.g. spinal cord injury, stroke, and amputation). Also, implantable electrode arrays typically cover only a small cortical area making accurate pre-surgical localization of the targeted implantation site critical. For example, intracortical microelectrode arrays typically cover only a small area of cortex (4×4mm² (Hochberg et al. 2006)) and high density ECoG grids (Wang et al. 2009) may cover an area only slightly larger (15×15mm²). Another direct role for rtMEG is in pre-surgical training of patients who are scheduled to have electrodes implanted for BMI applications. Using rtMEG analysis, participants could practice BMI control with MEG as a surrogate for invasive technology. This training would orient patients to BMI operation and could potentially improve cortical activity modulation, thus improving performance with the implanted BMI. In summary, MEG and especially rtMEG, offers many opportunities for supporting and advancing BMI research and development.

2.4. Neurofeedback therapies for physical rehabilitation

Neurofeedback can be used to help individuals learn to modulate their own brain activity volitionally (Angelakis et al. 2007, Heinrich et al. 2007) and has been applied to many clinical conditions such as epilepsy, anxiety, and even attention deficit hyperactivity disorder (ADHD) using EEG (Angelakis et al. 2007, Heinrich et al. 2007, Monderer et al. 2002, Sterman & Egner 2006, Patrick & Friel 2007). Generally, individuals are provided visual, auditory, or tactical feedback of their neural activity, which allows them to volitionally control their brain activity. Real-time MEG has the capabilities of providing high quality neural signals for neurofeedback training, leveraging the advantages of MEG over other non-invasive brain recording technologies such as EEG or rtfMRI. In addition, rtMEG can be applied to an emerging area of neurofeedback therapy for motor impairment rehabilitation.

Many traditional physical rehabilitation methods strive to improve motor function using strategies that rely on patients having at least some residual muscle strength. However, when there is little or no residual motor function (e.g. after stroke and incomplete spinal cord injury) a therapy using motor–related signals collected directly from the brain may be a better option. By providing feedback of cortical activity, neuroplasticity can be induced which could potentially impart therapeutic benefits on sensorimotor function. BMI training paradigms have been used to induce motor-related neuroplasticity by providing direct neural feedback of individual’s brain activity (Gage et al. 2005, Nijboer et al. 2008, Hochberg et al. 2006, Helms Tillery et al. 2003, Buch et al. 2008, Mellinger et al. 2007).

Some work has been done directly with individuals with paralysis as a proof-of-concept for using MEG as rehabilitation therapy. Buch et al. demonstrated that MEG could be used to allow individuals with chronic and complete hand paralysis due to stroke to control a one-dimensional computer cursor and a hand orthosis by modulating their sensorimotor rhythms (Buch et al. 2008). The combination of neurofeedback and physical practice may have additive rehabilitative effects, although this remains to be fully investigated. In Buch et al., 2008, individuals with complete paralysis achieved an average success rate of 72% during a BMI-controlled ‘grasping’ task. Six out of the 8 participants significantly improved their quality of brain-control across multiple sessions (13-22 sessions) demonstrating the possibility of neuroplasticity. However, in this group of 8 participants no significant improvement in hand-function was observed, which is not surprising since all had cortical lesions resulting in complete paralysis. Performing similar studies on participants with incomplete hand paralysis and using more advanced data processing methods designed specifically to encourage neuroplasticity may result in more improved motor function. In particular, the feedback in this study was updated with a 300 ms delay, which would be noticeable to participants (Lauer et al. 2000, Welford 1968). This long delay between the participant’s brain activity and the corresponding feedback could result in limited neuroplasticity based on Hebbian plasticity mechanisms where tighter temporal coupling results in better plasticity (Muralidharan et al. 2011, Cooper & Donald 2005; Florian 2007, Caporale & Dan 2008). Being able to access and process neural data at a fast rate is critical for achieving a tight coupling in time with the feedback needed to achieve a stronger therapeutic effect. The fast processing necessary for strong neuroplasticity can be achieved with real-time MEG (Sudre et al. 2011). Furthermore, using advanced neural decoders that adapt over time to encourage stronger and more spatially localized activity could be used to improve the neuroplasticity effectiveness.

3. Challenges of online MEG

3.1. System considerations

Whole-head MEG systems contain up to 306 sensors that can be sampled at high rates (typically ≥1000 Hz). Unlike EEG, which has fewer channels and typically lower sampling rate, the high data throughput of MEG data requires more complex data collection hardware and software. For example, MEG machines utilize multiple Digital Signal Processors (DSPs) working in parallel each of which manages the data from a set of sensors. Using multiple small processing units in parallel complicates direct access to the raw data. In the standard workflow, MEG signals are recorded directly to a hard disk and analyzed offline without concern for accessing and processing data at a fast rate (see figure 2). In order to analyze the data as it is collected, special software is needed to access the data stream at a low level before it is stored in data files.

As a typical example, the Elekta Neuromag systems have a dedicated computer that receives, synchronizes, and stores the sensor data received from the DSP units. The compiled data is typically transferred in large ‘chunks’ of about 1 s duration. Such a long delay is unacceptable for real-time applications that operate on neural data at a millisecond scale. It is possible to adjust the speed and manner at which data chunks are transmitted and to directly access the data stream to allow for real-time MEG processing. Some real-time MEG software tools have been created to access the raw data of various MEG machines. One particular software interface has been developed for the Elekta Neuromag and CTF/VSM MedTech systems and is available in the open-source toolbox “FieldTrip” (Oostenveld et al. 2011, Sudre et al. 2011). This software copies small data segments directly from the data stream and puts them into a separate data buffer that can be accessed by other software running real-time applications. These systems copy the raw data leaving the original data stream intact to ensure the standard data saving occurs as it would normally. The buffer can be hosted on the acquisition computer itself or can even be run on a separate computer via a network connection (i.e. TCP/IP protocol) allowing for customizable network architectures. The buffer is shared across a network connection and can be accessed by other computers running analysis software specifically tailored for the real-time applications. Figure 2 illustrates the typical setup for a real-time neurofeedback system.

Tools for working with the rtMEG data buffer, such as routines to write and read from it, are freely available and designed to be straight-forward for researchers to incorporate customized scripts. Specifically, scripts to read from the buffer in Matlab and C are available, as well as functions to use the buffer as a source module in BCI2000 (Schalk et al. 2004). This real-time software has been used with a 306-channel Elekta Neuromag system to demonstrate some basic neurofeedback and source imaging applications as a proof-of-concept (Sudre et al. 2011). In particular, weighted and cortically constrained minimum-norm estimation (WMNE) was applied in real-time with average delays of 44±17 ms (forward head model and source imaging kernel were calculated ahead of time). Results from this online study compared favorably with the results from using standard offline processing methods. A major contribution to the differences between the online and offline results was found to be head movement, but this may be rectified with real-time motion correction (see section 4.1).

Figure 2.

Example system diagram for real-time MEG using rtMEG software and the Elekta Neuromag system. After sensor data is collected and synchronized, it is sent to the acquisition computer and the real-time buffer. The real-time buffer, which can be hosted on any computer, can be accessed via TCP/IP network connection by computers running real-time analysis software. The real-time application computer can process the neural data and provide real-time feedback to the subject and/or experimenter.

3.2. Artifacts and noise

Magnetic fields measured by SQUIDs (superconducting quantum interference devices) contain both the signals associated with neural activity and the noise and artifacts from various sources. The magnetic fields resulting from neural activity are extremely weak (typically 50-500 fT), about 8-9 orders of magnitude smaller than the geomagnetic field generated by the earth and 1-2 orders of magnitude smaller than the magnetic fields generated with eye movement (Hamalainen et al. 1993). Because the magnetic fields resulting from neural activity are inherently small relative to many noise and artifacts, MEG can have a low signal-to-noise ratio. The appropriate detection and removal of noise and artifacts is especially critical for rtMEG analysis where offline techniques, such as averaging across repeated trials are not available for improving the signal-to-noise ratio.

In general, the noise and artifacts observed in MEG recordings can be classified into four broad categories.

• Environment-induced: There are many magnetic sources in the surrounding environment that generate noise and artifacts in MEG measurements. These environment-related noise/artifact sources include, 60 or 50 Hz power lines, electric appliances, moving equipment such as elevators and hospital beds, and geomagnetic field generated by the earth.

• Device-induced: Both the SQUIDs and the electronic circuits of MEG systems can also introduce noise and artifacts. For example, the electronic circuits (e.g. amplifier, analog-to-digital converter, etc.) in MEG systems generate thermal noise, shot noise, and flicker noise. In addition, a large magnetic signal can temporally saturate the electronic circuit, resulting in one or multiple bad channels that contain large artifacts over a short time period (e.g. spurious interference in time domain).

• Biologic-induced: Subjects can also add noise and artifacts that contaminate MEG signals. For instance, a subject may have dental fillings or braces that interfere with the magnetic fields generated by their brain. Furthermore, periodic heart beats, muscle activity (i.e. electromyographic), and eye blinking generate electromagnetic fields adding unwanted components to MEG signals.

• Experiment-induced: Many clinical and neuroscience paradigms may themselves introduce noise and artifacts to MEG. Consider the example where a subject is instructed to perform a movement task in response to a cue. In this case, the corresponding muscle activity and movement can generate electromagnetic fields causing artifacts. These artifacts would be highly correlated with the movement cue and may be mistaken for task-related modulation in neural activity.

In practice, a small noise or artifact can drastically distort the MEG signals and eventually produce misleading conclusions after data analysis. Careful experimental design is a critical first step for mitigating artifact contamination in rtMEG, similar to the case of traditional offline analysis. For example, subjects with metallic dental fillings and braces should be excluded from research studies and subjects should be instructed to avoid unnecessary eye and body movement during the experiment. However, many sources of artifacts cannot be avoided, such as those caused by involuntary physiological behavior (e.g. heart beats and eye blinks) and in studies that require subjects to perform movements. In addition, it is important that the subjects are in a natural state and do not intentionally try to control their normal physiological behavior which could induce unwanted neural activity (Ochoa & Polich 2000).

To combat artifacts and noise, two complementary problems must be addressed: (1) noise/artifact detection, and (2) noise/artifact removal. Often noise and artifacts are not easy to detect because their time and frequency characteristics are not always predictable (e.g. head and eye movements). Even after a noise/artifact is detected, appropriately removing them from the MEG signals is another challenging task. In many situations the neural signals and noise/artifacts overlap in both time domain and frequency domain. In these cases, advanced signal processing algorithms are required to accurately remove the noise and artifacts to improve the signal-to-noise ratio.

Real-time MEG poses a number of additional challenges for both experiment design and data analysis. In particular, noise/artifact removal methods must process data quickly (i.e. high throughput) and maintain low latencies for rtMEG applications. To maintain high throughput, noise/artifact removal must be implemented with low computational complexity for fast processing. To minimize latency, the signal processing algorithms should generate results using only short time windows. For these reasons, special consideration is needed for addressing noise and artifacts in rtMEG.

4. Real-time analysis methods and solutions

4.1. Noise detection and suppression

Real-time MEG analysis poses a number of challenges for noise/artifact removal due to the unique requirements of high throughput and low latency. These requirements play an important role in designing the appropriate algorithms for noise/artifact reduction in rtMEG. However, while many techniques used to address noise and artifacts in offline MEG studies are directly valid for rtMEG other techniques need to be adapted for real-time use. In general, noise/artifact removal techniques fall into two broad categories: hardware-based methods and software-based methods, as summarized in Figure 3.

Figure 3.

A list of some existing methods for noise reduction and artifact removal of MEG.

Hardware-based methods rely on specific devices and instruments to reduce MEG noise and artifacts. For instance, magnetically shielded rooms have been designed to reduce environmental interference (Kelha et al. 1982, Zimmerman 1977), SQUIDs have been used to detect weak magnetic field and achieve improved signal-to-noise ratio (Clarke & Braginski 2006a, Clarke & Braginski 2006b), and gradiometer coils have been used to measure local gradients of the MEG signals by effectively suppressing the “common-mode” noise caused by external sources (Hamalainen et al. 1993). Adding a reference channel is another important hardware-based method. Instead of recording brain signals, a reference channel captures the interference which can be used to remove artifacts and noise from the MEG signals (Vrba & S. E. Robinson 2001, Hansen et al. 2010). Additionally, electrooculography (EOG) and electromyography (EMG) can be recorded simultaneously during MEG measurement and used to detect possible artifact contamination due to eye or body movements (Fatourechi et al. 2007).

Complementary to hardware-based methods, software-based techniques have also been developed to further increase the signal-to-noise ratio. These software-based techniques offer superior flexibility and are cheap to implement. Hence, they are often integrated into the data analysis flow of many practical MEG applications. Many software based noise removal methods are already appropriate for use in real-time and others can be adjusted to work in real-time applications. These software methods typically fit into two categories: approaches that require a priori information and those that use statistical characteristics of the measured MEG data.

Temporal filtering aims to remove the noise and artifacts based on their frequency-domain characteristics. Temporal filtering assumes that the noise/artifacts and the neural signals occupy different frequency bands. Hence, the noise and artifacts can be removed by filtering out the signals within an appropriately selected frequency band. For example, power line noise occurs at 60 (or 50) Hz and can be attenuated by applying a notch filter. Many temporal filters are computationally simple and can be run easily in real-time. The temporal filter method, however, will fail to work if the noise/artifacts and the neural signals are too close to each other in the frequency domain. For rtMEG applications, causality and latency must be taken into account during filter design. First, the output of the filter can only depend on the current and past inputs, meaning that non-causal filters are not applicable in real-time applications. Second, the filter-induced delays must be minimized. Taking N-tap finite impulse response (FIR) filter as an example, the delay is proportional to the filter order. In order to reduce the delay, the filter order should be minimized while simultaneously maintaining the preferred frequency response.

Different from temporal filters, signal space separation (SSS) is a spatial filter technique that is derived from quasi-static Maxwell’s equations (S. Taulu et al. 2005; S. Taulu & Simola 2006). The key idea of SSS is to derive two different subspaces that correspond to brain signals and external interference respectively. By reconstructing the signals for these two subspaces based on measured MEG data, SSS can successfully separate the noise and artifacts from the MEG signals generated by human brain. However, extending SSS to real-time applications is not trivial. As a spatial filter technique, SSS is sensitive to the “bad channels” that contain large non-magnetic interference (e.g. saturated electronic circuits). Spurious interference is often observed for these bad channels (shown in Figure 4) and can distort SSS results. Traditionally, bad channels are detected and removed using statistical analysis of the recorded MEG data over large time windows on the order of a few seconds. This approach results in artifact removal that reacts slowly which can have a detrimental effect in rtMEG applications. To address this issue, a robust SSS (rSSS) algorithm has been developed to extend the conventional SSS algorithm to real-time applications (Guo et al. 2010). The key idea is to apply robust regression to dynamically detect and attenuate corrupted signals in real-time. In addition, a specially designed numerical solver was been developed to minimize the computational cost of rSSS to be suitable for real-time applications.

The aforementioned temporal and spatial filter methods rely on prior knowledge about the MEG signals (e.g. noise frequency and signal subspaces). A number of other software-based techniques attempt to detect and remove noise and artifacts by statistically modeling their characteristics from the measured data. Three common approaches are: principal component analysis (PCA) (Guimaraes et al. 2007, J. W. Kelly et al. 2011), independent component analysis (ICA)(J. W. Kelly et al. 2011, Hyvärinen & Erkki Oja 2000, Vigário et al. 2000, Choi et al. 2005), and signal space projection (SSP) (Uusitalo & Ilmoniemi 1997, Tesche et al. 1995).

Figure 4.

a) Recorded MEG data contain interference where the MEG channel is sporadically saturated. b) SSS is applied to attenuate noise and artifacts in real-time. c) rSSS is applied to attenuate noise and artifacts in real-time. d) Because SSS cannot automatically handle bad channels in real-time, its results vary from the rSSS results mainly when spurious interference occurs

PCA applies eigenvalue decomposition to the covariance matrix calculated from measured data. It uses the resulting eigenvectors to transform the MEG signals to a new coordinate frame where the transformed signals are uncorrelated. Ideally, in the transformed signal space some of the principal components capture the neural signals of interest, while other components represent the sources of noise and artifacts and should be removed (Guimaraes et al. 2007, Fatourechi et al. 2007, Choi et al. 2005). ICA extends PCA and aims to find the signal space where all transformed signals are mutually independent (J. W. Kelly et al. 2011, Hyvärinen & Erkki Oja 2000, Vigário et al. 2000, Choi et al. 2005). As such, ICA offers an alternative way to separate the sources of noise/artifacts from the recorded MEG signals. Finally, SSP applies the idea of signal space decomposition that is similar to SSS, but instead uses statistical characteristic of the measured signals to determine the two subspaces associated with the MEG signals and the unwanted noise/artifacts respectively. Once these subspaces are determined, SSP projects the MEG data onto the signal subspace, thereby removing the components belonging to the noise subspace. All of these statistical methods (i.e. PCA, ICA, and SSP) often require a large amount of measured data for characterization. For the applications where preliminary characterization data are limited, these methods may fail to work or specific modifications are required to improve their accuracy and stability. When rtMEG is considered, these statistical methods can be directly applied to real-time signals as linear weights. However, since the generation of these weights relies on preliminary data that were collected before an rtMEG paradigm, updating the statistical characterization (i.e. the linear weights) may be needed if the signals are not stationary. If the signals are not stationary, adaptive algorithms (He et al. 2004, Selvan & R. Srinivasan 1999) may be useful to dynamically track the signal/noise characteristics over time. Introducing these adaptive algorithms may increase the computational cost and result in system delays.

In summary, many of the standard noise and artifact suppression methods are applicable to rtMEG. Most hardware-based methods and simple software-based methods (e.g. temporal filtering) can be implemented readily in real-time, while the more complicated software-based methods require adaptation (e.g. rSSS) or separate data collection (e.g. ICA). All of the discussed noise/artifact detection and removal methods have unique design trade-offs that must be carefully explored for each real-time experimental paradigm.

4.2. Real-time spectral analysis

Spectral analysis is performed to study the modulation of specific frequency bands contained in a recorded signal. The dynamical properties of neural networks in the brain give rise to ‘rhythms’ or oscillations in several distinct frequency bands, which are often studied offline or in real-time (Nunez & Ramesh Srinivasan 2005, Wolpaw et al. 2002). For example, mu (8-13 Hz) and beta (12-30 Hz) rhythms are typically associated with motor cortex activation during movement and can be used in BMI systems to control devices in real-time (Pfurtscheller & Lopes da Silva 1999, Waldert et al. 2008).

In a typical offline MEG analysis scenario, data are time-locked to a particular event, such as a stimulus onset, and then the frequency content is extracted and averaged across similar repeated events. Trial averaging improves the signal-to-noise ratio in offline analysis, but is typically not possible in rtMEG applications. For example, controlling a prosthesis in real-time requires that control signals be generated continuously (i.e. within a single trial). Figure 5 illustrates an example of a single trial of MEG data that was analyzed in a real-time manner. Shown are the time-varying spectral representations from one MEG sensor over the contralateral sensorimotor area during hand grasp. The frequency content was determined using autoregressive filters on a sliding 300 ms window of data shifted every 50 ms to produce the time-frequency plots. In addition to the example of real-time spectral analysis (left figure), the spectrograms of 10 trials were averaged together demonstrating a distinct and more stable suppression of the mu and beta bands during movement. In rtMEG applications that are not restricted to using only the latest data, averaging each trial with the previous trials can be used to produce a spectral estimate that improves as the experiment progresses. This is useful, for example, when experimenters want to observe the frequency changes as a study is occurring.

If an experimenter wants to analyze brain signals in real-time, special consideration is needed when picking the analysis method. In traditional MEG analysis many minutes of data are available for characterizing events, but in rtMEG only a few milliseconds may be available. For example, in a BMI application it may be desirable to perform a spectral analysis on short duration time windows (e.g. <500 ms). The duration of the spectral analysis window is an important parameter to consider when doing any real-time signal processing. For example, with a short duration analysis window the spectral estimate can be updated quickly resulting in faster neural feedback but potentially limiting the frequency resolution, because there are simply not enough time points to estimate the power spectrum. Conversely, with a longer duration analysis window, a higher frequency resolution is possible, yet the resulting frequency information will not update as quickly and can produce delays in the neurofeedback. This timing-frequency resolution tradeoff is not unique to MEG spectral analysis and has led to the development of different spectral estimation methods. Some of the most commonly used methods are short-time Fast Fourier Transform (FFT), AutoRegressive (AR) filters, wavelets, and band-pass frequency filters (Kay & Marple 1981). Band-pass filters require the design of filters, which have trade-offs to consider such as ringing and roll-off rate that can limit frequency resolution. Fast Fourier Transform is a common spectral estimation method that is easy to use, but works best with long time segments of data. With short analysis windows, FFT has a tendency to result in erratic spectral estimations across adjacent frequencies. AR filters, such as Maximum Entropy Method (MEM), attempt to solve the erratic estimation issue of FFT by effectively smoothing changes in the power spectrum for a more stable estimate (Kay & Marple 1981, E. A. Robinson 1982). This frequency smoothing effect produces accurate estimates of broadband signals, such as gamma frequencies in motor related activity (Bashashati et al. 2007), but poor estimates of narrowband signals. Wavelets are used to help remove the timing-frequency resolution trade-offs for signals with specific and known characteristics. Typically, wavelets are used to encourage higher frequency resolution in the lower frequency bands while preserving higher temporal resolution in the higher frequency bands. However, this leads to poor timing resolution in the low frequencies and poor frequency resolution in the high frequencies added to the fact that wavelets need to be designed for specific signal waveforms.

Due to the large number of channels and high sampling rate found with MEG, performing spectral analysis quickly and often, such as in a real-time application, can be computationally expensive. Thus for rtMEG, the method for spectral estimation and rate at which the analysis is performed need to be chosen based on the computational resources available. For example, if neurofeedback needs to be updated every 50 ms from the power spectrum of over 200 channels sampled around 1000 Hz, the power spectrum method can only afford at most 0.25 ms per channel (though many software packages allow for parallel processing). With numerous signals and limited time for processing, choosing estimation methods that are computationally intensive, such as AR filters, may not be possible.

One method for decreasing the computational requirements of spectral estimation in real-time is to decrease the number of sensors evaluated. This can be done simply by picking a sensor type (e.g. longitudinal gradiometers) or by selecting a region of interest. The sensor locations could be predetermined based on neuroanatomy or using source localization methods based on specific data from the paradigm.

Figure 5.

Spectral analysis of hand grasp from a contralateral sensorimotor sensor using different amounts of data for estimating the spectrum. Shown is the modulation depth in time calculated as the percent change in power during grasping relative to the power during hand rest. In black is the interquartile range for the muscle activity (EMG is processed and standardized to fit the figure). A decrease in the mu and beta frequency bands (dark blue) is seen preceding and during muscle activity. With more trials used to estimate the frequency content the estimate becomes more stable. Yet, in the real-time case (left panel) modulation due to hand grasp can be accurately detected

4.3. Real-time source imaging

One of the major advantages of MEG is the good spatial resolution and wide coverage of the head. Combining these advantages allows for imaging of the relevant sources of neural activity in the brain. By applying source localization online the experimenter can get immediate feedback of the subject’s brain activity in source space. This information can be used to determine if a subject/patient has the expected brain activity and which additional evaluations should be performed next. In addition, a considerable change in the source location could indicate the subject is no longer performing the task, has become desensitized to the task, or their mind is simply wandering and a break is needed.

Many source imaging techniques can be applied in a straight forward manner for real-time MEG applications. In particular, beamformers, such as synthetic aperture magnetometry (SAM), produce sensor-weight matrices which, when applied to the MEG sensor signals as a spatial filter, produce source images (Brookes et al. 2004). These spatial filter weights can be calculated offline before an experiment using previously recorded data and then applied online, during an experiment. Though the calculation of a leadfield matrix from a generic or subject specific head models can take considerable computational time, the final inverse transform is just a linear weighting of the MEG sensors which takes a negligible amount of computation to apply in real-time. This technique was demonstrated by Sudre et al. to perform source localization of ongoing brain activity (i.e. alpha waves during eye opening and closing) in real-time (Sudre et al. 2011).

Spatial filters from source imaging can also be combined with spectral analysis. For example, dynamic imaging of coherent sources (DICS) method performs beamforming with consideration to the frequency domain which can then be applied to the spectral estimates in real-time (Timmermann et al. 2001, Gross et al. 2001). This frequency-focused source imaging technique combined with real-time spectral estimation can be used to increase the signal-to-noise ratio and help accentuate brain activity associated with attempted movement in sensorimotor frequency bands thereby improving neurofeedback for rehabilitation.

5. Real-time feedback systems

The real-time capability of MEG makes it possible for software programs to acquire instantaneous MEG signals and provide feedback to a subject on-the-fly with minimum lag and high update rate (Sudre et al. 2011). Many neural signal processing algorithms and software packages developed previously for other neurophysiological signals, such as EEG, ECoG, local field potentials, and neuronal firing rates, can be adapted to work with real-time MEG signals. In order to provide real-time feedback, MEG signals (or any neural signals) are typically processed in multiple stages (see figure 6). First, signal conditioning is typically applied to neural signals, such as band-pass filtering, removal of line noise and artifacts (see section 4.1 above), and spatial filtering/beamforming (see section 4.3 above). Second, signal processing algorithms are used for real-time feature extraction, most commonly time-frequency analysis in various frequency bands (see section 4.2 above) (Degenhart et al. 2011, Schalk et al. 2004). Finally, decoding algorithms transform neural signal features into control signals for feedback, such as moving a computer cursor, a robotic/prosthetic arm, or other devices.

Figure 6.

Standard processing flow diagram for real-time neurofeedback. Listed are examples of methods possible during each step

Most studies using neural feedback have used decoding algorithms that take a signal classification approach where certain behavioral parameters are used to generate discrete outputs (e.g. decoding neural activity associated with the imagined movement of the right hand and left hand to change a feedback between two states) (Mason et al. 2007). These discrete decoding methods can be implemented in real-time MEG and are valuable for some neurofeedback applications. In addition, proportional neural decoding algorithms that can generate continuous, time-varying control signals can also be done with rtMEG (Mellinger et al. 2007) and may be more suitable for applications such as BMI. Many neural decoding algorithms have been used for real-time BMI control, including population vector algorithm (Georgopoulos et al. 2005, Georgopoulos et al. 1986, D. M. Taylor et al. 2002), optimal linear estimator/multivariable linear regression (Salinas & Abbott 1994, Foldes & D. M. Taylor 2011, Wang et al. 2007), and Kalman filters (Malik et al. 2011, Wu et al. 2006).

At the final stage, the decoded neural signals are used to control various external devices and generate real-time feedback, which can be visual, auditory, tactile, or proprioceptive. These methods are already being piloted with rtMEG to control simple devices such as computer cursors and hand orthoses (Buch et al. 2008) (see section 2.1 above). Using a combination of visual, tactile and proprioceptive feedback of an orthosis or similar device could improve task performance and perhaps promote plasticity for rehabilitation applications. Providing sensory feedback directly to participants by artificially stimulating the peripheral or central nervous system to promote plastic changes in neural function in the brain is a BMI technology currently under development (Wang, Collinger, et al. 2010). However, neural stimulation can introduce additional artifacts that could distort MEG recordings. In many cases the stimulus artifacts could be removed in real-time by ignoring MEG data during stimulation periods or using artifact removal methods (see section 4.1). Yet in other cases, for example stimulation of the cortex directly, removing stimulus artifacts from MEG signals would be particularly challenging.

Advanced ways to display visual feedback are being increasingly used for BMI applications (Marathe et al. 2008). Paradigms using virtual reality models of a human are useful particularly for neurofeedback applications because they can induce a sense of ownership over the virtual body (Bailenson et al. 2003, Stanney et al. 2003) and it was observed that when participants empathize with the virtual person they perform tasks better (Friedman 2001). Furthermore, compared to abstract cursor movement, movement of a virtual arm is likely to evoke much stronger activity in the premotor and motor cortices for observation-based BMI training (Tkach et al. 2008). Paradigms using interactive games are also valuable for improving BMI control by minimizing both boredom and frustration by adjusting the difficulty of the games. Using games for feedback helps participants work harder and remain engaged while still enjoying the experience (Csikszentmihalyi 1991). This is critical to motivate human subjects for repetitive neurofeedback tasks regardless whether the goal is to achieve robust brain control of assistive devices or to promote functional recovery after nervous system injuries.

There are a number of real-time software packages for neural signal processing, neural decoding, controlling assistive devices, and rendering feedback. “BCI2000” is a successful example of real-time software that was developed for BMI (Schalk et al. 2004) and adapted to work with rtMEG toolboxes for Elekta Neuromag and CTF systems (Mellinger et al. 2007, Sudre et al. 2011). Another example is “Craniux” which is written in LabVIEW engineering programming environment (National Instruments, TX, USA) (Degenhart et al. 2011). The Craniux software package takes advantage of the advanced signal processing, real-time data visualization, and distributed processing capabilities offered by the LabVIEW environment. It can operate either in conjunction with BCI2000 or as a stand-alone package for real-time MEG, ECoG, or EEG. Similar to BCI2000, it has a modular architecture for signal recording, signal processing and decoding, and experiment paradigm control. Furthermore, Craniux offers a variety of display options for real-time spectral analysis, neural decoding algorithm parameters, and visualization of neural command signals including a virtual environment and avatar with fully articulated arms and hands as well as real-time Flash-based games (Adobe Systems Incorporated) that keep participants engaged in the neurofeedback task.

6. Conclusions

MEG is an exciting tool for analyzing neural activity in the brain at high temporal and spatial resolution. Using software that can access, manage, and process the high throughput available with MEG can open up many new opportunities for researchers, engineers, and clinicians. Accessing MEG data during an experiment could enable new paradigms employing neurofeedback and improve existing neuroscience and clinical paradigms by relaying processed data to the experimenters while a participant is still in the scanner. Freely distributed tools have been developed to help experimenters tap into the complex data stream of MEG to access data in real-time. Utilizing the real-time noise and artifact removal algorithms being developed and leveraging the analysis methods used in BMI research, rtMEG can become a valuable instrument for many clinical and neuroscience studies. Of course there are many more opportunities for rtMEG developments by signal processing engineers, neuroscientists, and clinicians.