Summarized evaluation results for the three applications. Clearly, all of them leave room for improvement. However, end-users would have liked to use the Painting and Gaming applications in their daily life.
In the past 20 years research on BCI has been increasing almost exponentially. While a great deal of experimentation was dedicated to offline analysis for improving signal detection and translation, online studies with the target population are less common. Although BCIs are also developed for entertainment and thus potentially for healthy users, the main focus for BCI applications that are aiming at communication and control are people with severe motor impairment. There is a great need for translational studies that test BCI at home with the target population. Further, long-term studies with users in the field are required to improve reliability of BCI control. The user centred approach appears suitable to foster such studies.
In this chapter we will first define the needs and the gaps for bringing BCI to end-users and explain the model of BCI control which guides our interventions. Then we will describe the user-centered design and report first results of studies that adopted this approach for evaluating BCI applications. Those results led us to develop novel BCI components which we then tested with healthy and severely ill end-users. More specifically, we will introduce the optimized communication interface, the face speller, and remotely supervised BCI controlled brain painting with a locked-in patient in the field. We will end the chapter with summarizing the requirements for improvement and reasons for cautious optimism that the BCI community will be successful in providing end-users in need with reliable and independent BCI controlled applications.
1.1. The needs and the gaps
In 1973 J.J. Vidal posed the question whether “electrical brain signals” can “be put to work as carriers of information in man-computer communication or for the purpose of controlling such external apparatus as prosthetic devices…?“ (p. 157 ). Already in those days Vidal answered the question with a clear
We are thus, facing a
Further, we are confronted with a
2. A model of BCI control
A BCI acquires input from the human brain, mostly its electrical activity recorded with electroencephalography (EEG), which is filtered, classified and transferred to an output signal. This output signal relates to the brain response or pattern of the BCI users and conveys the respective intention of the user. Importantly, the user receives feedback of his or her action and thus, BCIs imply a closed-loop between the system and the user. The output signal can be used to control an application – ideally, one that meets the desire of the user. Four aspects can be identified that contribute to BCI control: (1) individual characteristics of the BCI user, (2) characteristics of the BCI, (3) type of feedback and instruction, and (4) the BCI-controlled application . The individual characteristics of the user include psychological, physiological and neurobiological factors. For example, visuo-motor coordination and motivation have been identified to predict performance with BCI controlled by sensorimotor rhythms  and event-related potentials . Better inhibitory control, i.e. ability to allocate attention and inhibit distracting stimuli, measured as heart rate variability was related to better ERP-BCI performance . The amplitude of the SMR peak at rest and the P300 amplitude evoked in an auditory oddball paradigm were also related to performance with the respective BCI [9, 10]. Further, the location and quantity of neuronal loss due to accident or disease may deteriorate performance. Besides the hardware used, the software components, namely the classifier of the input signal further determines BCI control (for review ). Common spatial pattern technique and stepwise linear discriminant analyses proved to perform well in SMR- and ERP-BCIs [12, 13].
Little research is available on how the type of feedback and instruction provided in a BCI setting may influence performance. From early neurofeedback studies it is known that immediate feedback is superior to delayed feedback which held also true in a BCI context . It may also be the case that a more ecologically valid feedback in a virtual environment outperforms traditional two-dimensional feedback on a computer screen [15-17]. A quite robust finding across BCI types is that visual feedback is superior to auditory feedback [18-20]. In the SMR based BCI instruction to imagine motor imagery kinaesthetically leads to increased performance as compared to visual motor imagery .
Finally, the complexity of the application influences performance. Usually simple spelling tasks are mastered more accurately and faster than environmental control or control of information technology, such as internet [22, 23].
As can be seen, the model offers multiple toeholds for improvement and user feedback. In the following sections we will introduce novel achievements for BCI that improve and facilitate BCI use and are based on feedback provided by end-users within the user-centred approach. Before we detail the novel approaches, the user-centred design and its application to BCI will be outlined.
3. The user centred design and its application to BCI
BCI development demands for close investigation of the end-users’ needs and requirements and of the restrictions that come along with their diseases. The latter restrictions may range from small artefact contamination of the recorded brain signal up to loss of perception modalities, e.g. loss of ocular control as often the case with progression of neurodegenerative diseases. Furthermore, attention allocation may be limited and long lasting training sessions may be too demanding. BCIs are required to accommodate for such restrictions and to offer appropriate solutions, such as switching to auditory or tactile modalities when vision is impaired. Many of these restrictions are not evident when testing systems with healthy users. Furthermore, a system in daily use has to meet other requirements than a system developed for research purpose only, e.g. with regard to hardware setup, software handling and technical support. Bringing BCI technology to end-users’ homes thus, inevitably requires involving them into this developmental processes.
More recently the potential user of a BCI came more into the focus of BCI development and user-centred approaches were adopted [22, 24, 25]. A user-centred approach implies early focus on users, tasks and environment; the active involvement of users; an appropriate allocation of function between user and system; the incorporation of user-derived feedback into system design; and an iterative process whereby a prototype is designed, tested and modified . The user-centred approach was standardized with the International Organization for Standardization (ISO) 9241-210 (Ergonomics of human-system interaction - Part 210: Human-centred design for interactive systems). According to this approach three kinds of requirements have to be taken into account: (1) Business requirements: Here, typically, a specific number is set in terms of how many systems should be sold in a defined time frame. Although our face speller and brain painting (see below) have already been adopted by a company (http://www.intendix.com/) and are thus, available on the market, these products are not yet suitable for daily use in the field. (2) User requirements and functional specification: BCI requirements need to be specified from a user’s point of view, including the functions required to support a user’s tasks, the user-system and interfaces. Usability goals that must be achieved and the approach for system maintenance at the user’s home need to be defined. (3) Technical requirements: It has to be specified how the system will achieve the required functions and what data structure must be available for internal processing for the approach to be successful. Technical constraints need to be defined, such as the maximum data communication speed over a network or the trade-off between good EEG measurement and comfort with regards to the EEG cap. On the basis of these requirements Zickler and colleagues asked experts in using assistive technology (AT), i.e. people with severe motor impairment, what they would consider the most important requirements for BCI . Those requirements were functionality, independent use, and easiness of use (see section on “User-centred improvements of BCI controlled applications”).
Two different approaches to BCI control were subject of evaluation following these standards: BCIs dependent on modulation of sensorimotor rhythms, referred to as SMR-BCI, and on detection of event-related potentials, referred to as ERP-BCI. To better understand the applications and their evaluation, we provide a condensed description of the SMR- and ERP-BCI as implemented for control of the specific applications described below.
BCIs can be established by detecting an active modulation of sensorimotor rhythms (SMR) over sensorimotor areas of the brain. In a resting state, these rhythms are highly synchronised in the alpha (10-12 Hz) and beta (12-30 Hz) bands. When moving or imagining a movement, these rhythms desynchronise, i.e. the power of these frequency bands can actively be modulated by the user. Thus, SMR modulation constitutes a signal for BCI control [27, 28]. Different classes of motor imagery can be selected depending on a user’s individual brain signals and the degrees of freedom that are required for control of an application. In a typical SMR-BCI, users trigger control signals for two classes by either imagining movement of the right or the left hand. Feedback is provided during the imagery tasks to enhance participants’ performance thereby reinforcing correct behaviour. As hand areas are largely separated in the sensorimotor cortex, the evoked patterns are usually well distinguishable. Importantly, it has been shown that people with amyotrophic lateral sclerosis can utilize such modulations of the SMR to operate a BCI . One of the remaining issues, however, is that a large number of participants is not able to achieve sufficient SMR-BCI performance [7, 9, 30, 31]. BCI systems that do not rely on such active modulations of brain signals are available. The most frequently used system is described in the next section.
3.2. Event-related potential (P300) BCI
A typical BCI based on event-related potentials is the so called P300-Speller, providing muscle independent communication on a character-by-character basis ; for recent reviews:  and . A character matrix is displayed on a computer screen and groups of characters (usually rows and columns in a matrix) are highlighted (flashed) in random order. Users focus their attention on the desired field of the matrix (the target) by counting the number of flashes whilst ignoring all other characters (non-targets). This pattern constitutes an oddball-paradigm as target flashes are rare (odd) as compared to the high amount of non-target flashes. For example, in a 6x6 matrix one row and one column contains the target character whereas 5 rows and 5 columns are to be ignored. Each stimulus triggers distinct event-related potentials among which the P300 usually is the most prominent. It is a positive deflection in the EEG which occurs roughly around 300 ms post stimulus. Its latency may strongly vary with paradigms and across individuals (for review ). Yet other ERPs are also elicited, therefore a time window of up to 1000 ms post stimulus (typically 800 ms) is recommended to investigate users’ individual ERPs (i.e., negative and positive deflections at distinct latencies). The characteristic sequence of event-related potentials is identified for each row and each column. The row and column with the most prominent ERPs are selected and the respective letter appears on the screen. It has been shown that 72.8% of N=81 healthy BCI users were able to communicate with 100% accuracy by means of such an ERP-BCI and that less than 3% could not achieve any control . Importantly, these results transfer to individuals with severe motor impairment, e.g. due to neurodegenerative disease, in that the speller can be utilized as a muscle independent tool for communication (e.g., [22, 36-39]; for review ). Since its first description in 1988, the P300-Speller has been used intensively, further investigated and modified in a plethora of research publications leading to new applications for communication and device control (for review, e.g. ).
4. Evaluation of BCI controlled applications
The ISO 9241-201 (2010) defines usability as the “extent to which a … product … can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (ISO 9241-201, 2010, p. 3). Effectiveness refers to how accurate and complete the users accomplish the task. Efficiency relates the invested costs, i.e. users’ effort and time, to effectiveness. User satisfaction refers to the perceived comfort and acceptability while using the product. Context of use refers to users, tasks, equipment (hardware, software and materials) and the physical and social environments in which a product is used (ISO 9241-201, 2010, p. 2) .
To accommodate for these aspects when evaluating newly developed BCI driven applications, a set of measures has been compiled to assess
As another measure of device satisfaction the ATD PA Device Form was used. The Assistive Technology Device Predisposition Assessment (ATD PA) is a set of questionnaires based on the Matching Person and Technology Model (MPT) of Scherer (2007) . It addresses characteristics of an AT-device and asks respondents to rate their predisposition for using the AT under consideration. The questionnaire rates the AT-person match and the expected support in using the device, in other words the expected technology benefit .
As a coarse measure for overall satisfaction with the device, a visual analogue scale (VAS) ranging from 0 to 10 (not at all – absolutely satisfied) was included in the evaluation procedure. An open interview allowed participants to state their opinion about the BCI and its application and recommendations for further development.
4.1. Extended communication
Zickler and colleagues investigated the first prototype in which BCI was integrated into a commercially available AT software . Control of AT was realized by means of the ERP-BCI described above. Participants tested the text entry, emailing and internet surfing options (Fig 2). The oddball paradigm had to be implemented such that these applications provided by the standard software could be controlled. Instead of rows and columns flashing red dots were assigned to each possible selectable item. The red dots then flashed in random order. Participants were able to write a text, send an email and surf the internet for a specific website.
Selection accuracy (
The above described BCI controlled application already goes beyond simple verbal communication and may constitute a step toward inclusion via the world wide web. Some of our patients have been participating in BCI studies for a long time  and stated that they would also like to control other, more entertaining applications such as playing games or painting.
4.2. Brain painting
Together with an artist (Adi Hösle www.retrogradist.com) the letter matrix controlled by the ERP-BCI was transformed into a painting matrix which allowed the user to select shapes, size, colours, and contours and to move a brush on a virtual canvas (Fig 3). One participant stated “Everyone talks about freedom, but the worst oppression is to be locked into my own body. This art form allows me to break from the prison…”. With his painting (see Fig 4) he wanted to illustrate that there is a light at the end of a tunnel.
Four severely motor impaired potential end-users participated in the evaluation study which comprised seven daily sessions. In five of those sessions, participants could freely paint pictures of their choice.
Taken together, users enjoyed painting and painted up to one picture per session. Three users would have liked to use Brain Painting in daily life once or twice a week. They reported high satisfaction with the learnability, ease of use, and reliability of the device. The EEG-cap and system operability clearly required improvement if the BCI application was to be used in daily life .
Four severely disabled end-users - two in the locked-in state – evaluated the gaming application
Taken together, such evaluation studies are first steps toward bridging the translational gap experienced in BCI research and development. Based on these evaluation results we state that to date ERP-BCIs are more effective and efficient for communication and interaction as compared to SMR-BCIs (Table 1). End-users indicated that the speed of the BCI controlled application was too low. Users would have liked to use the Brain Painting application several times a week, but none could imagine to use the BCI for emailing and internet surfing unless substantially improved. Somewhat surprisingly two end-users could imagine playing Connect Four in daily life despite low control. Table 1 summarizes the evaluation results for all applications.
5. User-centred improvements of BCI controlled applications
As outlined above, functionality, independent use, and easiness of use were rated by expert users of assistive technology (AT) as most important for BCI use in daily life. In the next sections we will describe how we addressed and improved these three aspects.
In an effort to bridge the reliability gap and to address speed of the BCI, we changed the stimulation mode of the widely used P300 spelling matrix. In the commonly used ERP-BCI, characters are light flashed and attention to one of the characters will usually elicit a distinct P300  and sometimes other ERP components such as N100 or N200 (e.g., [48-50]). One option to increase reliability of the system is to enhance signal to noise ratio of the recorded ERPs. It is well known that familiar faces elicit characteristic ERPs, among which the N170 and N400f (f for faces, Figure 5) are very reliable ERPs. Thus, instead of flashing the letters of the matrix we overlaid row- and column-wise a famous face (the face of Albert Einstein or Ernesto Che Guevara, ). Figure 5 provides a screenshot from such modified BCI matrix and illustrates the grand average event-related potentials across N=20 healthy participants. Increasing the signal-to-noise ratio by eliciting more target specific ERPs, significantly boosted offline BCI performance. Importantly, these findings were replicated online in a group of possible end-user of BCI with severe motor impairment, e.g. users with amyotrophic lateral sclerosis or spinal muscular atrophy . They benefited to such an extent that even some users who were unable to operate the traditional ERP-BCI, reached an online accuracy of 100% due to the face stimulation. As such it was possible to decrease the number of stimulation cycles without negatively affecting performance, i.e. bit rate was strongly increased. In six online runs, the number of stimulation cycles was decreased from 10 to 6, 3, 2 and 1 (i.e. single trial) stimulation sequences. Performance in N=9 users with neurodegenerative disease was significantly increased in all runs when exposed to the face speller as compared to the classic ERP-BCI. Furthermore, we compared their single trial performance to the online performance of N=16 healthy participants. As usual, performance was significantly worse in the classic ERP-BCI, however, no difference was found for the face speller. These results clearly underline how modifications to the system can diminish performance drops in end-user samples. Zhang and colleagues (2012) reported that inversion of faces may further increase the N170 component and thus, performance in the BCI task. Face motion, face emotion and face familiarity, however, did not affect BCI performance [38, 52]. We conclude that investigating stimulus material other than the classical character highlighting is a very promising direction for addressing speed and reliability of the system.
5.2. Easiness of use
We developed a so-called optimized communication interface which allows for auto-calibration and word completion and is controlled with a user-friendly graphical interface . After the subject is set up with the electrode cap and connected to the BCI by an expert, the calibration process for parameterizing the classifier can be started by pressing a single button on the screen. No familiarity with technical or scientific details of the BCI is required. Data from calibration is automatically analysed in the background, invisible to the user who only receives a feedback on successful or unsuccessful outcome of the calibration. In the latter case, calibration can be performed again with one click. Yet, if successfully calibrated, communication with the P300-BCI can be initiated with another button press. We tested if such a user-friendly BCI implementation can be handled independently by naïve users. All healthy subjects (N=19) handled the BCI software completely on their own and stated that the procedure was easy to understand and that they could explain it to a third person. A text completion option significantly decreased communication speed. We conclude that from a software perspective, a BCI system can easily be integrated into an automated application that allows caregivers, friends or relatives to control such complex systems without prior knowledge at the end-user’s home or bedside.
5.3. Independent use
Finally, to bridge the reliability gap we implemented BCI controlled brain painting for long-term use at the home of a 72-years old locked-in patient diagnosed with amyotrophic lateral sclerosis (ALS) who used to be a painter . The brain painting application, which was successfully tested and evaluated by healthy subjects , as well as patients ( and see above) was embedded into an easy-to-use interface enabling to use the application after a few steps only. Family was trained to set-up the 8-channel EEG-cap and amplifier and to control the brain painting interface. The brain painting software automatically saved the duration of painting time, number of runs, and the paintings, and transfered them to our lab for remote supervision. After every session, satisfaction was rated. In a separate window familiy and caregivers can comment on the session. In doing so, occurring problems can be noticed and remotely solved by our experts via remote internet access. Figure 6 shows the end-user in a brain painting session at her home.
After each session the end-user is asked to rate her satisfaction with the visual analogue scale (VAS) (Figure 7) and after approximately 10 sessions the workload and device satisfaction are assessed with the NASA TLX  and the Extended QUEST 2.0 [22, 42]. Her responses as well as her data can be observed by our experts remotely to allow for system modifications or other interventions if necessary (e.g. advise for recalibration of the system).
In more than 8 months the end-user has been painting in more than 86 BCI sessions with an average paiting duration of 66.2 minutes. Satisfaction with device strongly depended on functioning of the BCI (Figure 7). When implementing a remote-controlled BCI application problems of malfunctioning arise which are immediately visible in the satisfaction rating (e.g., sessions 9 and 17 in Figure 7). Three types of sources for her dissatisfaction could be identified: In most of the cases dissatisfaction was due to technical problems (software/hardware; especially in the first sessions after set-up of the BCI system at the end-user’s home); second, due to problems from the end-user’s side, e.g. low concentration or exhaustion or not being able to realize the desired painting; and third, due to bad control (e.g. due to false cap placement, insufficient electrode gel) or loss of control over time (e.g. due to electrode gel drying).
Also for this locked-in BCI end-user effectiveness, reliability and easiness of use were the most important aspects for device satisfaction. Additionally, she mentioned professional support, specifically during times in which the system was not running properly. With a mean VAS satisfaction score of 6.2, her overall satisfaction is moderate to high. However, there is high variability with lowest satisfaction when the system was not working (early sessions) and when the painting was not as she expected it to be (later sessions). Highest ratings indicate that the system worked properly and that she was satisfied with her painting. Despite initial problems with the BCI, her motivation to continue brain painting has remained high even after more than 80 sessions. The end-user is currently painting 2-3 times a week, but stated that she would like to paint every day, if she could do so. The limiting factor is the available time of the family setting up the BCI. But currently also caregivers and friends are willing to learn the set-up and control the application to enable her to paint more often. In conclusion, our results demonstrate that expert-independent BCI use by end-users in the field is possible and illustrate the important role of family and caregivers when transferring BCI technology from the research environment to the end-user’s daily life. Figure 8 depicts some of her brain paintings.
Taken these results together, we can state that milestones were achieved in bringing BCIs to end-users. BCIs were combined with standard assistive technology, set up of the system including handling of software was facilitated tremendously and spelling speed was increased whilst maintaining high accuracy levels by altering the stimulation mode. In one exemplary end-user with severe motor impairment an application was installed at home such that family and caregivers can set up the system and maintenance and support is provided remotely. With innovative applications to be set-up at the end-users’ home and long-term studies first steps have been undertaken to bridge the translational and reliability gaps encountered when bringing BCIs to end-users. The user-centred iterative process between developers and end-users proved successful and the results are powerful demonstrators that BCIs are well coming of age and can face the transfer out of the lab to the end-users’ home.