Many disorders, like spinal cord injury, stroke or amyotrophic lateral sclerosis (ALS), can impair or even completely disable the usual communication channels a person needs to communicate and interact with his or her environment. In such severe cases, a brain-computer interface (BCI) might be the only remaining way to communicate . In a BCI, the brain’s electrical activity during predefined mental tasks is analyzed and translated into corresponding actions intended by the user. But even for less severe disabilities, a BCI can improve quality of life by allowing users to control a computer or specially prepared electronic devices, or to stay in contact with friends through social networks and games. P300 evoked potential [2, 3, 4] based BCIs can provide goal-oriented control as needed to operating spelling devices  or control computer games . For navigating in space e.g. moving a computer mouse ), controlling the motion and movement of a robot or a wheelchair, steady state visual evoked potential (SSVEP) [8, 9, 10, 10] and motor imagination [12, 13] based BCI paradigms can be used.
All these applications require integrating the BCI with an external software application or device. This chapter will discuss different ways this integration can be achieved, such as transmitting the user’s intention to the application for execution, updating the options available to the user, related actions and integrating visual BCI stimulation and feedback paradigms with the Graphical User Interface (GUI) of external applications. The current developments and efforts to standardize the application interfaces will be presented.
2. General structure of a BCI
A BCI system consists of several components (figure 1). The first is the biosignal acquisition system, which records the body’s signals (like the EEG) used to extract the user’s intentions and responses to the presented stimuli and feedback. It consists of a set of electrodes and a dedicated biosignal amplifier, which typically directly digitizes the signals and transmits them to the feature extraction system. The feature extraction processes the signals and analyzes them with respect to specific signals like P300, SSVEP or error potentials [14, 15, 16, 17].
The classification determines the intention of the user based on the extracted features, which may reflect that the user does not intend to communicate at that time. The classification results are converted by a dedicated mapping component or appropriate methods into commands, actions and tasks to be executed by the attached applications such as a spelling device [2, 3, 4] or robot . In order to enhance the user’s response to the presented stimuli and to help assess the system’s efficacy, a feedback related to the classification results is presented and the stimulation is adopted accordingly. Furthermore, the stimulation unit provides information about the presented stimuli using trigger signals and events, which are used to synchronously process the input signals and extract corresponding features.
3. Methods for integrating a BCI with an application
There exist three basic approaches to interconnect the BCI with a user application. The following sections, 3.1-3.3, discuss the different designs, their advantages and disadvantages. Each section will present some of the currently available BCI systems and frameworks that use that design. Section 3.4 compares the possibilities to establish standardized interfaces for interconnecting an application with the BCI system.
3.1. Direct integration
The most straight forward approach is to integrate the application within the BCI system. This approach, which is sketched in figure 1, allows developers to hardcode the symbols and feedback presented. In other words, the conversion of the classification results into application commands, tasks and actions, and the application itself, represent a static addendum to the BCI systems. This approach was used for the first proof of concept systems, and can still be found in simple spelling devices such as the P300 speller distributed by g.tec medical engineering along with the g.HIGHsys, a Highspeed Online Processing block set  for Matlab/Simulink™ (Mathworks, USA) and other BCI systems used for demonstration and educational purposes.
Modern BCI frameworks  such as OpenViBE , BCILAB  or xBCI  that are based on this design use a module based approach to allow application developers and designers to integrate their own application within the existing BCI framework.
The advantage of directly integrating the application within the BCI system is that it can be distributed and used as a compact, all-in-one component. Despite the need for appropriate acquisition and processing hardware, no additional interfaces or protocols are required.
The downside of this approach is that application developers and designers require some knowledge of the interpretation of the feature signals and classification results and how to convert them into appropriate commands, tasks and actions. Whenever an application must be added, updated, exchanged or removed, the presentation of stimuli and feedback has to be adjusted.
3.2. External executable component
The limitations of the tight integration approach can be reduced or eliminated by modeling the application as an individual executable that act as an external component of the BCI. As shown in figure 2, all other components like feature extraction, classification, stimulus and feedback presentation remain inside the core BCI system. This design is utilized by BCI systems which use the BCI2000  or the TOBI platform , among other examples.
A well defined interface, such as the TiC output interface  used by the TOBI platform, allows applications to receive information based on the action that the user chooses to execute. If supported by the stimulus and presentation components, applications even may update and modify the choices available to the user through the BCI. Within TOBI, the TiD trigger interface  can be used by the application to initiate the processing of the change requests.
As a consequence, applications become independent from the BCI and may be connected and disconnected to and from the BCI any time. Both the BCI and the applications can, for example, be adapted to the user’s needs independently without affecting each other. On the BCI side, this includes the selection of the acquired biosignals and features, improvements in the algorithms used to assess the user’s intentions and the stimulation and feedback paradigms used. On the other hand, applications may dynamically adopt the available choices to some internal state or provide new commands and actions to the user without the need to modify any of the core BCI components.
As shown in figure 2, converting the feature classification results has to be done by the application. This basically requires that developers and designers of BCI enabled applications have some knowledge of how to interpret these results with respect to the services, actions and tasks offered by their applications. Propagating changes from the internal state of the application to the user is only possible if the BCI is able to reflect these changes by adopting the options presented to the user accordingly.
3.3. Message based
The limitations of the approaches described in the previous sections, 3.1 and 3.2, can be overcome by integrating a dedicated interface module or component within the BCI system. The mapping component collects the classification results and converts them to corresponding application control messages and the connection interface component transmits each of them to the application, where they are interpreted and the requested services, actions or tasks are executed.
Depending on the capabilities of the core components of the BCI and the interface component, the application may request an update of the presented stimuli and feedback based on the application’s internal state. Figure 3 shows the bidirectional case, where the application can acknowledge the reception of the transmitted messages and send update requests to the BCI whenever its internal state changes. The interface component of the connected BCI system decodes the corresponding messages and initiates the required changes and updates of the stimulus and feedback presentation. Hence, the paradigms and input modalities the BCI uses do not matter.
The message based approach provides a loose coupling between the BCI system and the attached applications. It is used, for example, by the intendiX™ system (g.tec medical engineering GmbH, Austria), which is further described in section 4. Structural and technical changes applied to the BCI system, such as modifications of the signal acquisition, feature extraction, classification, stimulus or feedback presentation, have no impact on the attached application. Improvements and modifications of the services, actions and tasks offered by the applications do not require any technical or algorithmic changes to any part of the BCI system. The dedicated interface component, in combination with a well defined, message based and standardized protocol, enables applications to provide new features to the user any time, without modifying any of the BCI components.
In contrast to the approaches described in the previous sections, 3.1 and 3.2, a dedicated mapping component enclosed within the BCI (figure 3) interprets the classification results and converts them to their corresponding application control message. This encapsulation enables developers with little or no expertise in biosignal processing, analysis, classification and interpreting results to develop applications controllable by a BCI.
The resulting decoupling of the application from the BCI can be taken one step further by transparently embedding it within the user interface of any application. This is achieved by overlaying the BCI stimuli and feedback on top of the application’s standard user interfaces. This may be achieved using an overlay module as described in section 4.3.
3.4. Connection design
The previous sections, 3.1-3.3, described the different ways to integrate the BCI within an application. Independent of these different approaches, the dedicated communication and control interface between the BCI and the application can be implemented using two distinct methods. Platforms like TOBI, BCI2000 or the unidirectional extendiX clients described in section 4.1 and in  provide dedicated application programming interface (API) libraries that handle all the connection and data transfer related issues. An application, as shown in figure 4a, calls the API functions to retrieve information about the user’s intention and initiate the update of the presented stimuli and feedback. All the details about the underlying protocol and connection are hidden by the library. This allows administering modifications and extensions to the protocol any time, without affecting the application, unless the interfaces of the API functions need to be adopted to reflect the changes made.
A large variety of programming languages exist to implement applications and the API library. If the application is implemented using a different language than the API, a so called language wrapper library is necessary. Such a wrapper provides access to the BCI client API from within the language used to implement the application.
User data, selections and information about the internal state of the application are stored in dedicated data structures and fields. Hence, it is likely that these differ from ones used by the API library to receive instructions from and to collect update requests to be sent to the BCI. As a consequence, an application has to convert all data to and from its internal data structures before and after calling a BCI client API function. Depending on the intended usage and the related requirements concerning responsiveness, user experience and usage of computational resources, this may cause additional, probably undesired effects from the application.
Figure 4b shows a different way to establish a connection between the BCI system and an application. It is based on a well defined and standardized protocol only. Both the BCI and the application utilize their own, dedicated connection manager to handle the typically network based connection, process all received requests and generate all outgoing messages based on their current state. The connection manager interprets the incoming messages, extracts relevant data and information and converts them directly into the data structures and representations used by the application or the BCI respectively.
Changes to the application’s internal state are directly converted into appropriate messages requesting changes and updates to the stimuli, the feedback presented to the user and the conversion of classification results to be returned to the application. Thus, it does not matter which programming language is used to implement the application. Instead of an API library wrapper, a dedicated connection manager has to be developed. This requires that the communication protocol uses a well defined and properly described handshake and communication procedures to avoid unnecessary workarounds, which might prevent the application from being used with BCI systems supporting a different version of the protocol.
The previous section discussed 3 different approaches to interconnect the BCI with user applications, services and devices. This section presents the intendiX™ system , which is used by the applications and projects presented in section 5.
The intendiX BCI system was designed to be operated by caregivers or the patient’s family at home. It consists of active EEG electrodes to avoid skin abrasion, a portable biosignal amplifier and a laptop or netbook running the software under Windows (see figure 5a). The electrodes are integrated into the cap to allow a fast and easy montage of the equipment so it can be mounted quickly and easily. The software lets users view the raw EEG to inspect data quality, but automatically informs the inexperienced user if the data quality on a specific channel is good or bad.
This control can be realized by extracting the P300 evoked potential from the EEG data in real-time. The characters of the English alphabet, Arabic numbers and icons were arranged in a matrix on a computer screen (see figure 5b). Then the characters are highlighted in a random order while the user concentrates on the specific character he/she wants to spell. The BCI system is first trained on the P300 response of several characters with multiple flashes per character to adapt to the specific person.
During this training, 5-10 training characters are typically designated for the user to copy. The EEG data is used to calculate the user specific weight vector, which is stored for later usage. Then the software automatically switches into the spelling mode and the user can spell as many characters as desired. The system was tested with 100 subjects who had to spell the word LUCAS after 5 minutes of training. 72 % were able to spell it correctly without any mistake .
The speed and accuracy of the classification can be optimized by choosing the appropriate number of flashes manually. A statistical approach could also automatically determine the optimal number of flashes for a desired accuracy threshold. This latter approach could also determine whether the user is paying attention to the BCI system. The statistical approach could have a major advantage: no characters are selected if the user is not looking at the matrix or does not want to use the speller.
The intendiX™ user can perform different actions after spelling: (i) copy the spelled text into an editor; (ii) copy the text into an email; (iii) send the text via text-to-speech facilities to the loud speakers; (iv) print the text or (v) send the text via UDP to another computer by selecting a dedicated icon. The intendiX™ system offers two distinct ways to manage this interaction with external software and control various applications , described below.
The first is the unidirectional extendiX protocol. It is accessible through the dedicated closed source extendiX library encapsulating the extendiX protocol.
For simple control applications, the extendiX batch file starter client allows users to start dedicated batch scripts whenever it receives information about the symbols selected by the user. This approach is suitable for all applications, services and devices that offer dedicated command-line interfaces for controlling their state and executing specific actions.
The intendix™ Painting  program, a small painting program comparable to Microsoft Windows Paint, allows users to create paintings and other images, as well as store, load and print them.
4.2. intendiX ACTOR protocol
For complex control tasks, the intendiX™ Application ConTrol and Online Reconfiguration (ACTOR) Protocol is provided. Compared to extendiX, it offers a bidirectional User Datagram Protocol (UDP) message based connection between the BCI and the application. A short summary of the standardized intendiX ACTOR protocol can be found in  and its detailed description is available from . No dedicated API library is available. All applications have to implement their own dedicated connection manager, which handles all intendiX ACTOR based communication.
The Interface Unit handles the UDP connections to all simultaneously attached applications, services and devices and converts the classification results to the corresponding UDP message string. This dedicated connection manager processes all connect, disconnect requests and status updates received, prepares the next set of stimuli and feedback modalities to be presented to the user and instructs the corresponding components to adjust their outputs accordingly.
The intendiX ACTOR protocol uses eXtensible Markup Language (XML) formatted message strings to exchange information between the BCI and the attached system. Whenever the BCI system is started, it broadcasts a dedicated hello message to identify the available and active applications, as shown in figure 6a. Each client responds by sending an appropriate acknowledgement message. As soon as the BCI has received this message, it will request from the application the list of applications and services available from this client. The BCI will acknowledge the received list of commands, services and actions and report whether it was able to process it successfully.
A similar handshake procedure is used when a newly started application connects to the BCI system. The only difference is that the BCI acknowledges the hello broadcast sent by the application by requesting the list of available commands, as shown in figure 6b.
Attached clients will receive a control message whenever the user has selected a service, action or task (figure 7a). The content of this message is fully defined by the application, which can ask the BCI to change this message and the related symbols, sounds and sequences presented to the user during stimulation and feedback. In this case, configured along with the definition of a single BCI control element, the BCI will pause the presentation of any stimuli and feedback until the application acknowledges the reception and execution of the control message or requests an update of the BCI user interface.
A single UDP message sent to the BCI may contain several distinct requests that are processed as an atomic batch. The presentation of stimuli and feedback that was paused while sending the last control message is not restarted before the last message within this batch has been processed. Each single message within such a batch may request the change of one single stimulus or feedback element, or contain the content of a whole XML formatted configuration file that describes complex BCI screens and groups of stimuli. The latter is used by the IU to determine which stimuli and feedback modalities should be presented to the user. The detailed description of the configuration file format used is available, along with the definitions of the intendiX ACTOR protocol from .
Whenever the BCI receives an update request, it will return an acknowledge message indicating that it is trying to process and execute the update request. After the update is finished, a message indicating whether the request was executed successfully or failed is returned.
Whenever an application, service or device is terminated, it sends a good bye message to the BCI, which acknowledges this message sending a simple bye message. As shown in figure 7b, the same message is used by the BCI to inform connected clients that it will terminate operation.
4.3. intendiX SOCI
For applications, especially virtual reality (VR) applications and remote control of robots, it is desirable to enhance the standard user interface by directly embedding the BCI stimuli (figure 8). The intendiX platform can be configured to remotely display its stimuli and feedback using the intendiX Screen Overlay Control Interface (SOCI) module . The intendiX SOCI system implements a runtime loadable library based on OpenGL . It is implemented as a dynamic linked library (DLL) for Microsoft Windows and as a shared object for LINUX, and can be used by OpenGL based host applications to embed targets for visual stimulation within the displayed scene. The host applications could be virtual reality environments or real-world videos acquired with a camera. Figure 8 presents an example of how to use the intendiX SOCI module to simultaneously control the camera direction and select amongst different actions and objects.
The intendiX SOCI library is able to generate frequency (f-VEP) or code based (c-VEP) SSVEP stimuli, and supports single symbol and row column based P300 stimulation paradigms. It is initialized and fully controlled by the BCI system using a dedicated UDP based network connection. The application only needs to provide information on the network interface and port to be used when connecting to the BCI system and the screen refresh rate, which reliably can be achieved by the user display system. All other parameters are defined by the BCI system during startup and initialization phase. The intendiX SOCI module provides a standardized API, which allows the application to handle the stimulation and feedback devices attached to the user system appropriately.
Figure 9 shows the call sequence that the application must use to augment its own interface with the BCI controls. This sequence has to be repeated for each display, stimulation and feedback device that the BCI user wishes to use. The init function activates the BCI support for the selected device.
Every time before it calls the draw function, the application has to ensure that the openGL environment is initialized properly to display the BCI controls on top of any other graphical element. This is indicated in figure 9 by the call to the “set transformation” pseudo openGL function. After the swap buffer command from openGL has been called, the application has to call the displayed function to indicate that the stimuli have been updated.
When the application terminates, it disconnects the displays from the BCI by calling the reset function for each screen previously attached to the BCI through the init function. As indicated in figure 9, the application has to ensure that the interval between two consecutive calls to the displayed API function strictly corresponds to the screen refresh rate proposed by the application during initialization.
In section 3, the different concepts used to interconnect BCI systems and user applications were discussed. The application interfaces provided by the intendiX™ system were briefly presented in section 4. This section will list some projects and applications that utilize the intendiX ACTOR protocol and the indendiX SOCI API library to control virtual and robotic avatars and ambient assistance systems (sections 5.1, 5.2 and 5.3). Section 5.4 will present some examples in which the BCI was used to control cooperative multi player games such as World of Warcarft by Blizzard Entertainment  and social network platforms like Twitter (Twitter, Inc., USA). All of these applications either use the P300 matrix or a SSVEP paradigm with frequency coded stimuli.
The VERE project  is concerned with embodiment of people in surrogate bodies so that they have the illusion that the surrogate body is their own body – and that they can move and control it as if it were their own. There are two types of embodiment considered. The first type is robotic embodiment (figure 10a), where the person is embodied in a remote physical robotic device and control it through a brain-computer interface. For example, a patient confined to a wheelchair or bed, who is unable to physically move, may nevertheless re-enter the world actively and physically through such remote embodiment. The second type of embodiment (figure 10b) is virtual, where participants enter into a virtual reality with a virtual body representation. The basic and practical goal of this type of embodiment is to explore its use in the context of rehabilitation settings.
The VERE project uses the intendiX ACTOR protocol (section 4.2) to access the BCI output from within the eXtreme Virtual Reality (XVR) environment (VRMedia S.r.l., Pisa, Italy) to control both the virtual and robotic avatars. The BCI is part of the intention recognition and inference component of the embodiment station, which is developed through the VERE project.
The intention recognition and inference unit takes inputs from fMRI, EEG and other physiological sensors to create a control signal together with access to a knowledge base, taking into account body movements and facial movements. This output is used to control the virtual representation of the avatar in XVR and to control the robotic avatar. The user gets feedback showing the scene and the BCI control via either the HMD or a display and the tactile and auditory stimuli provided by the so called embodiment station. Thereby the intendiX SOCI module is used to embed the BCI stimuli and feedback modalities within video streams recorded by the robot (figure 10c, e) and the virtual environment of the user’s avatar.
The user is situated inside the embodiment station, which provides different stimuli and feedback modalities such as visual, auditory and tactile. Figure 10d shows the setup for inducing the illusion of hand movement by mechanically stimulating the flexor and extensor muscles of the hand.
Images courtesy of VERE, Event lab Universitat de Barcelona, Centere National de la Recherche Scientifique France, Perco laboratory Scuola Superiore Sant' Anna Pisa and Institute of Automatic control Engineering at the Technical University Munich, 2012.
Depending on the selected stimuli, the BCI system offers distinct levels of control. These levels range from high level commands, tasks and actions such as turning on the TV or grasping a can to moving the robot within its environment (figure 10c) or teaching the robot new high level tasks such as opening a door (figure 10e). The message based intendiX ACTOR protocol can switch smoothly between the different control levels and control paradigms.
The Ambient Assisted Living (AAL) research programme supports projects that develop technology to compensate for the drawbacks of the aging society by applying modern information and communication technologies (ICTs). The Adaptable Ambient LIving ASsistant (ALIAS) project  is one the projects funded by AAL. It aims to improve the communication of elderly people, thus ensuring a safe and long independent life in their own homes. A mobile robot platform without manipulation capabilities serves as a communication platform for improving the social inclusion of the user by offering a wide range of services, such as web applications for basic communication, multimedia and event search and games.
The ALIAS robot is equipped with sensing devices including cameras, microphones for speech input and a touch-screen (figure 11a) to perceive the user’s input. The robot utilizes different modalities such as audio output, a graphical user interface (GUI) and proactive and autonomous navigation to interact with the user.
A so called dialog manager ensures that the dialog system can be controlled in a reasonable way. It is the central decision making unit for the behavior of the ALIAS robot and its interactions with the human user. It manages the interplay between input and output modalities of the ALIAS robot, communicates with all involved modules of ALIAS and controls them.
Besides the touch-screen, which is used to display the GUI and two independent automatic speech recognition (ASR) systems, a keyword spotter and a continuous context search also operate in parallel. The ALIAS dialog systems support the intendiX ACTOR protocol for receiving input from BCI systems and updating the presented stimuli and feedback online, based on the active state of the robot.
The intendiX SOCI module is used to embed the BCI stimuli within the GUI (figure 11b), aligned to their corresponding buttons. This tight integration of the BCI enables users to easily utilize the ALIAS platform during recovery, such as recover from stroke. The BCI interface of ALIAS allows them to navigate through the different menus, start programs, chat with friends using Skype, call and dismiss the robot or issue an emergency call.
The BrainAble project  conceives, researches, designs, implements and validates an ICT-based Human Computer Interface (HCI). Such an interface is composed of Brain Neural Computer Interface (BNCI) sensors combined with affective computing and virtual environments to restore and augment the two main shortcomings of people with disabilities. It entails inner and outer components. The inner component aims at providing functional independence for daily life activities and autonomy based on accessible and interoperable home automation. The outer component provides social inclusion through advanced and adapted social network services. The latter component is expected to dramatically improve the user’s quality of life.
Within the BrainAble project, the core structures of the intendiX ACTOR protocol were designed, developed and extended. The ACTOR protocol is used to interconnect the BNCI system with the user’s living environment. This includes elements such as lighting, shades, heating, ventilation, audio, video services such as radio or TV, intercoms and many more.
It further provides access to social network and online communication services, thereby augmenting the user’s social inclusion. The user has access to all of these devices, services and social interaction tools and is able to control them through the BCI system.
5.4. Games and social media
The intendiX ACTOR protocol, in connection with the intendiX SOCI API, can also be used to control games such as World of War craft (WoW)  and social media like Twitter (Twitter Inc. USA) or Second Life (Linden Lab, USA).
World of Warcraft is a common Massively Multiplayer Online Role-Playing Game (MMORPG) in which the player controls an avatar in a virtual environment. The BCI system uses an SSVEP paradigm to control an avatar in WoW . For basic movements, selecting objects or firing weapons, four control icons as shown in figure 12 are required. The bottom three icons are used to move the avatar forward and turn left or right. The fourth icon, the action icon, is located top left. It used to perform actions like grasping objects or attacking other opponents. Stimulation is done on the same 60 Hz LCD-display that also renders the game itself.
Twitter (Twitter Inc.) is a social network that enables the user to send and read messages. The messages are limited to 140 characters and are displayed in the author’s profile page. Messages can be sent via the Twitter website or via smart phones or SMS (Short Message Service). Twitter provides also an application programming interface to send and receive SMS.
Figure 13a shows a UML diagram of the actions required to use e.g. the Twitter service. The intendiX ACTOR protocol is used to interconnect the Twitter interface with the BCI. This system uses a standard P300 spelling matrix (figure 13b), which was extended with commands required for Twitter. The two top rows contain symbols representing corresponding Twitter services, and the remaining characters are used for spelling.
Second Life is a free 3D online virtual world that can be accessed through the “Second Life Viewer”, which is free client software. A dedicated user account is necessary to participate in Second Life. One of the main activities in Second Life is socializing with other so-called residents. Each resident represents a person in the real world. Furthermore, it is possible to perform different actions like holding business meetings, taking pictures or making movies, attending courses, etc. Communication takes place via text chats, voice chats and gestures. Hence, handicapped people could also participate in Second Life just like any other user if an appropriate interface is available.
To control Second Life, three different interface masks were developed. Figure 13c displays a screenshot of a Second Life scene. The main mask (figure 13d) offers 31 different choices. The masks for control like ‘chatting’ provided 55 control elements and the one for ‘searching’ 40 selections. Each of the icons represents an actual command to be executed within Second Life. Whenever a certain icon is selected, Second Life is notified to execute this individual action. Thereby, a dedicated keyboard event generator is used to convert messages based on the intendiX ACTOR protocol into appropriate key strokes. Further details on the BCI integration with Twitter and Second Life, including the results achieved by healthy subjects, are discussed in .
Current research projects aim to establish BCI systems as assistive technologies for disabled people, thereby helping them interact with their living environment and facilitating social interaction. These efforts rely on properly interfacing the BCI with supporting systems, devices, services and tools. For example, this interface could embody the user in an avatar or robot, as done within VERE, or in a robotic assistant, as implemented within ALIAS. The design and implementation of the interfaces between the BCI and the application do have an impact on the flexibility of the resulting assistive system or device and thus on the autonomy and independence of the user. Highly flexible interfaces like intendiX ACTOR and intendiX SOCI make it possible to adapt the BCI to the user’s needs while providing a standardized interface for using the BCI as control and interaction device with a large and constantly growing number of applications, assistive services and devices.