Adaptive Automation and the Third Pilot

Abstract

Currently, automation does not take into consideration the cognitive and emotional state of the crew. Rather, automation provides assistance based on explicit and static task assignments, with no adaptive capabilities, even though it is capable of providing higher or lower levels of support depending on the crew state and/or complexity of the operational situation. This chapter presents a new adaptive automation concept which offers an innovative ‘team’ centred approach to solving crew awareness/workload management problems and enhancing flight safety. Partnership underpins the ‘Third Pilot’ approach. The crew (pilot flying and pilot monitoring), automation and the ‘Third Pilot’ are in charge together. Overall, partnership is proposed. This replaces existing paradigms involving dynamic changes in control function, where changes can be autonomously controlled by the system. Moreover, a new multimodal cockpit concept is advanced providing enhanced assessment of crew state/workload.

1. Introduction

Crew task support and information needs vary according to the crew composition, the crews own experience (i.e. familiarity with type, knowledge of the route), the specific flight situation (i.e. traffic and weather), and the crew state (i.e. fatigue, confusion) [1, 2]. With increasing duty time and traffic growth, pilots can benefit from an ‘experience aid’. Ideally, the crew and the ‘experience aid’ (or assistance system) comprise a cooperative system [1, 2]. This cooperative system approach follows the cognitive systems engineering frameworks, as advanced by Hollnagel and Woods [3].

In its current form, automation does not take into consideration the pilot’s cognitive and emotional state. Rather, automation provides assistance based on explicit and static task assignments, with no adaptive capabilities, even though it is capable of providing higher or lower levels of support depending on the crew state and/or complexity of the operational situation [1, 2, 4]. It is argued that safety is optimised when human and automated systems adapt both to each other and to the specific operational context. This guarantees fluent and cooperative task achievement—maintaining safety at all times.

This research was undertaken as part of the Applying Pilots’ Model for Safer Aircraft (A-PiMod) project, funded by the European Commission (Framework Programmes for Research and Technological Development) [1]. The aim of this project was to demonstrate a new approach/concept (and associated technologies) for an adaptive automation and multimodal cockpit which might mitigate and/or reduce human error. The project commenced in September 2013 and finished in September 2016.

2. Background

2.1. Automation

Currently, Pilots share responsibility for different flight tasks with cockpit systems. As defined by Kaber and Prinzel, adaptable systems are systems which require human delegation of task and ‘function authority’ to automation during real-time operational performance (i.e. the task distribution is controlled by the user) [5]. This is different to adaptive automation (AA), which allows for dynamic changes in control function allocations between a machine and human operator based on states of the collective human–machine system’ [6, 7].

Human factors problems with automation have been cited as contributory factors in many air accidents. This includes: Flight Air France 447 (2009) [8], Flight Spanair 5022 (2008) [9], Flight Helios Airways HCY 522 (2005) [10], Flight China Airlines 140 (1994) [11], and Flight Air Inter 148 (1992) [12]. The air accident reports highlight several human factors issues such as automation surprises, reduced situation awareness, workload problems and over-reliance on automation [1].

2.2. Crew errors

Errors are defined as ‘actions or inactions by the flight crew that lead to deviations from organisational or flight crew intentions or expectations’ [13]. Unmanaged and/or mismanaged errors frequently lead to undesired aircraft states. Flight crew errors reduce the margins of safety and increase the probability of adverse events [13]. As documented by the International Civil Aviation Organisation [14], human error is a causal factor in between 60%-80% of accidents and serious incidents. It is stated in the ‘Flightpath 2050, Europe’s Vision for aviation’, that ‘the occurrence and impact of human error’ will be ‘significantly reduced through new designs and training processes and through technologies that support decision making’ [15].

2.3. Theoretical starting point

In line with a cooperatives systems approach, the question of automation status/communication and its role in the execution of the flight, should to be framed from a team perspective and linked to a risk assessment of the mission and the selection of a course of action. Thus, the theoretical starting point for addressing human factors issues with automation (specifically, teamwork, task distribution, authority, situation awareness, error detection and workload management), is the assessment of the crew/automation/aircraft/environment state in relation to the achievement of the mission level goal (i.e. mission level risk assessment), and the identification of a suitable task distribution at cockpit/agent level, to achieve this [1, 2]. So conceived, automation has a role in relation to (1) real-time risk assessment, (2) the identification of a course of action, (3) the selection and subsequent implementation of a course of action, and (4) the identification of an appropriate task distribution based on the crew state [1, 2].

Further, it is argued that there is relationship between addressing human factors issues with automation (specifically, teamwork, situation awareness, task distribution, authority, error detection and workload management) and improving crew interaction with cockpit systems. The provision of a multimodal concept can support the above. In addition to (1) allowing for flexible interaction with cockpit systems and (2) providing a means to communicate with the crew in relation to crew state and decision support, the multimodal cockpit has a role in relation to (3) supporting the better assessment of crew state/workload (information inputs re crew activity/interactions).

2.4. Methodological background

Stakeholder involvement in programme evaluation has been recognised as one of the most effective approaches to enhancing the use of evaluation findings, and ensuring the validity of the evaluation activities [16]. Stakeholder involvement is defined as the participation of (programme) stakeholders in any phase of an evaluation [17]. Stakeholder involvement can vary with regard to diversity in stakeholder selection for participation, the control of technical evaluation decisions and the depth of stakeholder participation in the programme/project evaluation process [18]. Stakeholders are conceived as invaluable source of knowledge, perspectives, information on context and needs. Drawbacks of stakeholder involvement are also reported. This includes the feasibility of implementing a successful participative study. For example, time, cost, involvement from (disadvantaged) groups and skills required from an evaluator in facilitation and ‘good listening’ [19].

The involvement of stakeholders to accomplish given tasks by participating in common activities has been central to ‘Community of Practice’ concepts [20]. ‘Community of Practice’ members engage in a set of relationships over time around some particular area of technical knowledge or skill associated with the given tasks. This allows the members of a specific ‘Community of Practice’ to generate a sense of joint enterprise and identity by sharing a practice—doing things together, developing a sense of place, common goals. In Wenger’s analysis, three characteristics are crucial to define a ‘Community of Practice’: (1) the ‘domain’—which specifies the identity of COPs with the specific competence and commitment the stakeholders engage; (2) the ‘community’—stakeholders build their relationship interacting in joint activities, sharing information and common objectives and learning from each other; and (3) the ‘practice’—stakeholders share a repertoire of resources (experiences, stories, tools, ways of addressing recurring problems) which help forming the practice with time and sustained interaction [20].

3. Research design

3.1. Objective

The aim of the A-PiMod project was to demonstrate a new adaptive automation concept (and associated technologies) enabled by a hybrid of three elements: (1) multimodal pilot interaction, (2) operator modelling and (3) real-time risk assessment. It is anticipated that the introduction of this new automation concept will reduce human error—making substantial progress in relation to aim of reducing the accident rate by 80%. Table 1 below provides a description of the high level project goals

No	A-PiMod high level project goals
1	To reduce accident rate by 80%
2	To achieve a substantial improvement in the elimination of and recovery from human error
3	To mitigate the consequences of survivable accidents
4	To support smart pilot assistance

Table 1.

A-PiMod high level project goals.

3.2. Overview

The overall design/evaluation methodology combines formal HMI design/evaluation activities (i.e. interviews and simulator evaluation), informal HMI design/evaluation approaches (i.e. participatory design activities), along with an integrated stakeholder approach to evaluation [4]. Overall, 23 COP sessions and two phases of simulator evaluation have been undertaken. The first phase of simulator evaluation involved eight participants, while the second phase involved 12 participants. This has been reported in more detail in [1, 2].

3.3. Community of practice

The concept of ‘Community of Practice’ as proposed by Wenger [20] underpins the A-PiMod ‘Community of Practice’ approach. The A-PiMod ‘Community of Practice’ involved stakeholders who shared technical knowledge and skills associated with relevant functions in the Air Traffic Management (ATM system), and broader aviation related domain. Overall the role of participants in the A-PiMod ‘Community of Practice’ concept was characterised as a ‘participatory’ approach. Members engaged in a range of validation/evaluation activities on a continuous/regular basis, through the run-time of the project.

The panel of stakeholders in A-PiMod included both ‘primary users’ (i.e. internal stakeholders representative of each project partner), and ‘all legitimate groups’ (i.e. external stakeholder representative of the aviation-related industry and Flight operational system). Stakeholder participation involved consultative interaction along with engagement in technical research tasks.

Figure 1 below depicts the composition of the community of practice – with attention to the levels of expertise of both internal and external stakeholders. Internal stakeholders are represented in blue. External stakeholders are represented in amaranth. The overlapping levels of expertise are indicated by the red dotted line.

Figure 1.

A-PiMod Community of Practice: Stakeholder Competency and Knowledge (Source: Cahill et al. [2]).

3.4. Simulator evaluation

Two sets of simulator evaluation were undertaken. In both cases, the evaluation involved a mixed-method approach with the administration of semi structured interviews, simulator observations, collaborative workshop sessions and questionnaires. Each set of evaluations involved two crew members and elapsed over 2 days. Overall, a scenario-based evaluation approach was adopted. Simulator evaluation involved the use of the DLR simulator—the GECO system.

In day 1, participants were first briefed about the overall procedures, consent was obtained, and profile information was elicited. Then participants undertook a semi-structured interview regarding the current state of automation and HF still-open issues. After this both participants obtained training, in relation to interacting with the new adaptive automation multimodal concept/technologies in the simulator. The training was delivered with the support of slides, software training tools, and some hands-on training in the simulator (i.e. using the A-PiMod user interface displays such as the MC-M Display and the MultiModal ND). Then, hands-on training in relation to using the GECO simulator was provided. After this training session, participants undertook the specific simulator evaluation session, addressing the adaptive automation and multimodal concept/technologies. After the simulator sessions, participants were asked to complete a questionnaire evaluating the adaptive automation and multimodal concept and technologies. Further, a specific simulator session was undertaken with a focus on the interaction with the MultiModal ND. This involved a collaborative session to evaluate the operational and safety aspects pertaining to the multimodal cockpit.

In day 2, participants were asked to evaluate the adaptive automation and multimodal concept and technologies. This involved individual post evaluation individual semi-structured interviews. Then a collaborative workshop session was undertaken. This focussed on three distinctive scenarios (i.e. (1) supporting routine operations, (2) avoidance of conflict of taxi/way or apron, (3) Incapacitation (VETO)) and to what extent A-PiMod could support the Pilots in relation to workload management, error identification, situation awareness, teamwork, and how this would have an impact on the error reduction. After this, the participants completed an individual questionnaire related to benefits analysis.

3.5. Assessment of benefits/safety impact

In order to assess the potential safety impact of the new automation concept (and allied multimodal cockpit), a systematic approach was applied. This involved the elicitation of structured feedback from pilots and experts, using the Total Aviation Risk model [21]. Specifically, structured feedback was captured in relation to potential change factors for base events in this risk model [21]. The total aviation system risk model contains 425 base events and 51 end states. The particular structure of an event sequence diagram and its connected fault trees depends on the scenario considered. The probability change factors of all impressionable base events are determined in the safety assessment on the basis of information gathered in multiple workshops with members of the Community of Practice (COP). In the workshops the participants discussed the kinds of mechanisms facilitated by the innovative concept which may increase or decrease the probability of a particular base event in a scenario. The specific details of the quantification of safety impact have been reported in another paper [22].

4. Proposed adaptive automation concept

Overall, the objective is to provide assistance to the flight crew when and if required. Automation acts as a third crew member providing information and task support to crew –safeguarding the mission level goal [1, 2].

The third pilot approach involves providing (1) decision/information support, and (2) workload support. This follows from the hypothesis that information underpins good decisions, which in turn results in safe aircraft states. Further, it follows the philosophy that if there in increase in workload, certain functions should be shifted to automation, to reduce the cognitive/physical burden on the crew.

The overall approach involves continuously monitoring the operational situation and the allied crew/automation state, to determine the best distribution of task activity between the crew and automation [1, 4]. It is anticipated that this will ensure the safe completion of the flight. If there is an increase in workload, certain functions will be shifted to automation, to reduce the cognitive/physical burden on the crew. Also, automation is used to support information management tasks (i.e. information gathering and information assessment). This in turn has an impact on safety. Real-time feedback is provided to the Flight Crew via a cockpit user interface (i.e. HMMI Interaction Manager), so that the crew at all times understand the status of (1) the operational situation, and (2) the joint crew automation system. This ensures full situation awareness, which in turn impacts on mission safety. All of the above ensures that the aircraft remains in a safe state. This in turn has consequences at a process level (i.e. both single and multiple flight levels).

The team comprises the Flight Crew (namely the Pilot Flying and the Pilot Monitoring), the ‘Third Pilot’ and automation. Accordingly, the third pilot is conceived as a virtual team-member [1, 2, 4]. All team members co-operate in relation to making and executing mission level decisions. Mission level decisions are enacted at the mission level and translated into new cockpit level tasks that have to be distributed between the crew and automation, and then performed by them. The system continuously monitors the operational situation and the allied crew/automation/aircraft state, to determine the tasks the team has to perform together, and how to best distribute them between the crew and automation.

The new cockpit technology (i.e. automation and associated systems) allows us to answer the following questions:

• Is the joint crew/automation system in a safe state?

• Is there a potential for a safety critical aircraft state (i.e. now and/or the near future)?

• Do the crew (Pilot Flying and Pilot Monitoring) require support in terms of increased levels of automation?

• Do we need to adjust the level of automation?

• Do the crew require information/decision support?

A-PiMod flags potential risks—providing operational guidance in relation to managing those risks. Pilots have final control, but are responsible and accountable for their decisions and actions [1]. The crew are responsible for assessing the risk status of situation and the appropriate course of action. As such, the crew are not required to follow the decision support provided by the third pilot. This decision support is an aid, not a requirement. The crew can over-ride system proposals/decisions, except in certain critical situations (i.e. incapacitation).

A-PiMod adopts a team centred approach as opposed to a crew centred approach. Specifically, is assists the Flight Crew in relation to information and workload management (i.e. intervention if over and/or under loaded). Automation is conceived as an extension of the pilot’s ability to carry out an action(s). Automation provides decision and information support. In principle, we are focussing on the outcome. That is, we are considering what is best for the safe and efficient completion of the mission, as opposed to adapting to human needs. As indicated in the architecture concept (see below), if the Pilot Flying/Pilot Monitoring is overloaded and this threatens the completion of the mission, the task distribution is adapted at the agent level [1, 2].

As reported previously, we propose partnership as opposed to dynamic changes in control function (such that changes can be managed autonomously by the system) [1]. The A-PiMod architecture describes the means for the adaptive completion of Mission Level tasks and their distribution between the crew and automation. This includes the real-time analyses of both the pilot’s state (situation awareness and workload) and mission risks [1]. Apimod will permanently assess what has to be done by the cockpit (mission and its context = > cockpit level tasks), distributing it between the crew and automation, and assessing if these agents are correctly performing the tasks they have been assigned (i.e. recover from a stall, avoid ground obstacles, etc.). Task distribution is the product of a situation management process [1]. Here, the pilot flying and pilot monitoring, along with other automated processes cooperate to assess the situation, its risks, what has to be done (cockpit level tasks) and the associated risks [1]. This results in a task distribution.

As described in the architecture concept, in situations where the flight crew are overburdened, task distribution is adapted at the agent level [1]. Automation adapts to crew states and capabilities, so that all required cockpit-level tasks are performed [1]. It is this architecture that guarantees the safe completion of the flight.

The proposed automation concept addresses the key decision requirements as defined in the safety case [1, 2, 4]. For more information, please see Table 2 below.

No	Decision requirement	How supported by automation
1	Authority	Boundaries for automation set in relation to role of Pilot and associated decision authority
2	Information	Supports information acquisition and analysis
3	Time	Automation provide real time updates as to status of situation (both current and future) so have time to anticipate potential future problems and how might manage them
4	Judgement	Automation provides decision assistance – providing feedback on potential course of actions and risk associated with each.; Automation can gather decision support information from actors outside the cockpit – if collaborative/consultative decision
5	Teamwork	Human agents/crew and automation are conceived as a team; Better collaboration between team members – enhance situation awareness and increase safety; Cockpit as a cooperative system; Support teamwork in terms of information sharing: (1) information sharing between crew and automation, and (2) information sharing between cockpit and agents outside the cockpit
6	Feedback	Automation provides feedback (updated information picture) in relation to the outcome of decisions taken and next steps

Table 2.

Automation provides support for decisions.

Underpinning the Third Pilot concept is the A-PiMod multimodal cockpit concept [1, 2]. The multimodal cockpit (1) allows for monitoring of crew interactions with cockpit systems, (2) supports crew communication with automation (i.e. via the MCM Display) and (3), enables voice and touch interaction with cockpit displays [1, 2].

5. Proposed technical architecture

The A-PiMod architecture allows for (1) adapting the mission based on current circumstances and (2) the subsequent organisation of the cockpit (task distribution between the crew and automation) and (3) the circulation of information between the crew and cockpit systems (including automation) to the current and future situation(s) [23]. The starting point in the architecture is to adapt the mission permanently, and to translate a given mission configuration (e.g. decision to divert to closest airport) into tasks that have to be executed by the ‘team’. The tasks are then distributed between the team members (crew and automation) based on their current states and capabilities.

Figure 2 below depicts the A-PiMod architecture. This is based on a three layer hierarchy of tasks [23].

Figure 2.

Task Hierarchy.

The architecture is a ‘control’ architecture; it’s about the distributed control of a given object. In this example, this refers to the overall aircraft state including navigation and trajectory. Tasks at a given level are translated into the tasks of the level below based on the context of their execution [23]. At the mission level, the context of execution includes (1) the state of the aircraft and (2) the traffic and environmental context in which the aircraft is flying (i.e. weather, ATC, traffic). At the cockpit level, context refers to (1) the status of the cockpit agents (namely pilots and automation) and (2) cockpit equipment (i.e. displays).

As detailed in Table 3 below, this control architecture is elaborated in relation to three levels. This includes the mission level, the cockpit level and the agent level. These levels are hierarchical—each level is a decomposition of the level above. All actions occur at the agent level. The upper levels are about deciding what tasks the different agents perform, based on the current contexts, at the mission and cockpit levels.

Level	Description
Definition of mission level tasks	Single flight level—mission phase/sub-phase, context, A/C state—the context the aircraft is in
Definition of cockpit level tasks	Tasks that need to be performed by all (i.e. crew and automation) to achieve mission goals given specific context elaborated
Definition of agent level tasks	Actions undertaken by crew or automation to achieve mission goals in specific context

Table 3.

Architecture levels.

The key level is the cockpit level. We are trying to see what the cockpit as a whole (i.e., all of its agents—be they human or machine) has to do at a given point to achieve the Mission Level Tasks (in the current situation). Then, what the cockpit as a whole has to do is distributed between the agents available in the cockpit.

It should be noted that the broader ATM system level is not explicitly referred to in the architecture. This is because the other aircraft are not under the control of the architecture. They are part of the context in which the architecture is flying (i.e. achieving its control tasks).

As indicated in Figure 3 below, several technical components have been specified, linking to the overall architecture concept.

Figure 3.

A-PiMod Architecture (Source: Cahill et al. [2]).

As outlined in Table 4 below, the A-PiMod architecture consists of seven components and two separate software systems [22].

Component	1.	Situation determination @Mission Level
	2.	Risk assessment @Mission Level
	3.	Situation modification@ Mission Level
	4.	Task determination @Cockpit Level
	5.	Situation determination @Cockpit Level
	6.	Task distribution @Cockpit level
	7.	Risk assessment @ Cockpit Level
Separate Software Systems	1.	Crew state inference
	2.	HMMI interaction manager

Table 4.

A-Pimod architecture.

A key feature of the architecture is the distinction between components and modules:

• A component is a functional unit that performs specific tasks (e.g. risk assessment at the mission level). In the A-PiMod architecture a component is always a small cooperative system made of the crew +1 module.

• A module is a software system that acts with the crew as a team in this specific functional unit. The module provides assistance to the crew in the performing the component’s tasks

Thus regarding risk assessment at the mission level, it will always be performed together by the crew and the dedicated module, acting as a small team for that purpose. Both the crew and the module have the common goal to achieve the function assigned to the component. This is what brings great flexibility to the architecture. It allows each component to be executed exclusively by the crew (‘manual’ mode), e.g. the crew solely assess the risks, by the module (‘automated’ mode), e.g. the module solely assess the risks and the crew has no say in that evaluation, or by both the crew and module (‘mixed’ mode), e.g. the module produces a risk assessment and the crew acknowledge or reject it. This is true for all seven components in the architecture. Each is made of the crew and a dedicated module. Given the crew can participate to all seven components, the crew superposes the seven functions associated with the individual components, and this is exactly what human pilots do. They perform all these operations permanently, without being aware of that dissociation between seven elementary functions. They are thus part of the situation assessed by the Situation Determination/Modification @Mission Level component. All these tasks are permanently revised, distributed and executed, based on the context.

In addition, visual analysis of pilots’ behaviour is recorded to infer human operator’s (pilot’s) mental state, stress level, and general workload. The following instruments/technologies are used to obtain information about the pilot’s behaviour:

• Eye tracking

• Gesture recognition

• Head pose

6. Pilot interaction in the cockpit and associated user interfaces(s)

6.1. Overview

Crew interaction with cockpit systems is simple and user friendly. In addition to traditional controls, pilots interact with the system using voice and touch. This interaction is tracked by the system (i.e. what tasks performing, level of fatigue, involvement in activity). This form of tracking is referred to as ‘crew state monitoring’ [1]. Crew feedback is provided via a new cockpit user interface (Mission and Cockpit Level Management Display—MCMD). This provides information about (1) the risk status of the operational situation (this includes an assessment of the status of joint crew/automation system) and (2) what to do (including the provision of best options/alternatives based on different ‘technical’ contributing factors such as fuel remaining, the weather status at alternate airports and so forth) [1, 2]. As noted previously, the crew can over-ride system proposals/decisions, except in certain critical situations such as crew incapacitation.

6.2. The A-PiMod MC-M display

The A-PiMod Mission and Cockpit Management Display (MC-M Display) enables communication between the Flight Crew and the new adaptive automation technologies [1]. The MC-M Display supports both mission and cockpit management tasks. As depicted in Figure 4 below, the proposed MCMD features two related sub-displays—the mission and cockpit level displays [21]. In the cockpit level display, one can see the tasks assigned to the crew and to automation. The software allows the crew to change that distribution—both manually (crew) or automatically (automation). In keeping with concepts of authority/pilot control, all task distribution changes have to be approved by the crew before becoming active.

Figure 4.

The MC-M Display (Source: Cahill et al. [1]).

6.3. Multimodal interaction

During the A-PiMod project a multimodal system prototype was developed to explore possibilities of multimodal interaction in context of flight deck. From the perspective of human-machine interaction, the flight deck interfaces should be able to accommodate large number of diverse tasks while maintaining high level of efficiency, usability and uncompromised safety. In A-PiMod multimodal interaction was demonstrated through the Multi-Modal Navigation Display (ND) [1, 2].

A number of other systems are linked to the crew state estimation and crew task determination components, for the purpose of inferring (1) the pilot’s mental state, stress level and general workload, and (2) predicting potential errors, missed events and/or overlooked information. This includes eye tracking, gesture recognition and head pose tracking technology [1].

7. Crew state monitoring

Crew state monitoring (that is, focussing the Pilots attention on their state along with that of their crew member—and on the current and future state of the aircraft) is an important function of the third pilot. If fatigued, pilots may either forget or omit to review all the safe options. The 3rd crew member (automation) will consider all safe options and prompt the crew in regard to possible options, to ensure that a safe decision is made. In this context, a key challenge is how to get the two human crew members to share their ‘current state’ with the 3rd crew member [1]. Such an exchange should be meaningful and informative but not self-incriminating in any post hoc analysis [1]. Normal human interactions can easily accommodate this. For example, such information might be imparted as part of pre-flight social interactions/conversation. However, this is hard to replicate in relation to human/machine (i.e. third pilot) interaction

The assessment of crew state goes beyond issues of workload. It concerns many factors including crew experience, flight hours, crew familiarity with the proposed route and departure/landing airports, training background and so forth. The crew state might be assessed as less optimal in situations when the two crew members are unfamiliar with the route. From an operational/Pilots perspective, the starting point for crew state monitoring is the crew briefing/flight planning. Depending on the crew situation, this might occur a week before the flight and/or at the time of the pre-flight, flight planning and briefing task. In addition, knowledge of the joint crew status is and any issues linked to this is required [1, 2]. Potentially, the crew might need to report their status/state before the flight commences [1, 2]

The means/basis by which crew state information (i.e. eye tracking data, gesture data, voice, and touch data) is used to assess the crew state needs to be carefully considered. The assessment of this data depends on what we already know about the crew (for example, location in roster and expected level of fatigue). For example, if the crew are not looking at the correct area of the screen, and/or are blinking their eyes a lot, it may not be a problem. In this case, the crew might be very familiar with the route and may be more or less fatigued (i.e. location in roster—first day or last day). However the system might interpret this behaviour differently if the crew are unfamiliar with the route, and on the last day of their roster (i.e. expect higher level of fatigue)

If A-PiMod is to become a trusted (and relied upon) 3rd crew member, it needs to interact with the crew in a manner consistent with ‘normal’ human-human interaction [1]. Potentially, to be fully integrated in the ‘team’, it needs to engage in some form of ‘social’ interaction, and not only technical interactions [1, 2]. This latter issue has not been explored in A-PiMod and requires further consideration

How emphatic a team member is automation? When and how can it articulate its concerns, and potentially, over-ride the crew member’s decisions? In most (but not all situations), the cockpit crew have the final authority and can veto automation. This approach reflects an ‘adaptable systems’ logic. However, there may be situations (identified by the crew monitoring technology) where it is necessary for automation to ‘take charge’ to ensure flight safety (for example, situations of crew incapacity). Accordingly, three different levels of crew state monitoring can be defined. For example, (1) passive support, (2) active support and (3) intervention/over-ride [1]. In this sense, A-PiMod represents an adaptive automation approach. Arguably, a fully adaptive automation approach requires modelling and assessment of both the crew and aircraft state [1, 2]. The A-PiMod system represents a significant advancement in relation to the modelling of crew state (i.e. touch, voice, gesture, use of multimodal displays and so forth). However, aircraft state modelling potentially necessitates integration with aircraft systems and wider ATM and ground systems. This has not been demonstrated in the A-PiMod project

8. Innovation

It is proposed that the Third Pilot/A-PiMod system (1) reflects a mix of the logic associated with adaptable systems and adaptive automation, while (2) also providing something new (i.e. multimodal cockpit concept).

In relation to (1), we are

• Providing a framework for crew-automation cooperation for all activities occurring in the cockpit involved in the completion of a flight

• Extending concepts of assistance (as defined by adaptable systems), where the crew are in conceived to be in control all of the time

• Utilising certain features of adaptive automation—that is, providing task support to the crew following an assessment of crew state (i.e. inferences about crew situation awareness and workload)

In general, this new automation concept is predicated on concepts of partnership. The crew and automation are in charge together. As reported previously, in principle, the crew are in charge (concepts of professionalism and responsibility). However, there will always be particular safety critical situations when automation can take charge (i.e. fully adaptive).

In terms of (2), a new multimodal cockpit concept has been advanced. This enables improved assessment of the crew state/workload and provides a means to provide task and decision support information to the crew. Further, it allows for touch and voice interaction with cockpit systems. Evidently, the application of multimodal in the cockpit is not new. However, the application of crew interaction data (i.e. crew voice and touch interaction in the cockpit), as an input to crew state monitoring is innovative.

The third pilot has different modes of operation. This includes (1) passive monitoring, (2) active monitoring and (3) over-ride. It is anticipated that (1) and (2) will be the most typical operational modes. That is, providing task and decision support to pilots based on an understanding/assessment of the crew state and specific workload requirements at a given point in time. In certain non-routine situations (3) will be required (i.e. fully adaptive). The partnership/third pilot concept is defined in relation to the combination of these three modes of operation. In the future, mode (3) might be become routine, whiles modes (1) and (2) might become less routine. Either way, the advancement of this concept will require considerable effort in relation to both development and certification [1].

9. Assessment of benefits and safety quantification

9.1. Assessment of benefits/impact

The expected benefits/impact were validated through extensive field research including 27 COP sessions, Validation Cycle 1 (VC1), Validation Cycle 2 (VC2) and evaluation with stakeholders at a demonstration day event [1, 2]. Overall, validation activities indicate that the ‘Third Crew Member’ may yield many operational and safety benefits. As defined in Table 5 below, these benefits can be grouped at different levels (i.e. cockpit, task, process and ATM System).

No	Benefit	Benefit level
1	Improved access to information—faster access	Cockpit
2	Improved mechanisms to interact with information (i.e. voice, touch and gesture)
3	Provide variability in way of the automation control selection (i.e. via voice, touch and gesture)
4	Better interaction between automation and Pilots—clear communication lines, clear understanding of who does what and, when/how transfer work between both
6	Flexible and natural interaction with cockpit systems (MM)	Task
7	Decision support—how to proceed in safety critical situation and/or normal situation
8	Provide task support to crew in relation to assessing the situation and deciding on a course of action
9	Reduce workload in complex/high risk situations
10	Pilots always know ‘what is going on’ (i.e. reliable situation awareness)
11	Complementary ways in which to interact with information (i.e. MM interaction)
12	Reduction in workload
13	Improvement in information management—support information acquisition and information analysis
14	Improvement in situation awareness
15	Improvement in situation assessment
16	Provision of new information (i.e. status of operation, status of automation and routing advice)
17	Improvements in the elimination of and recovery from human error
18	Support for error identification and broader TEM
19	Error reduction
20	Error mitigation
21	Error avoidance—avoidance of safety critical situations
22	Better interaction between automation and Pilots—clear communication lines, clear understanding of who does what and, when/how transfer work between both
23	Completion of mission in accordance with the flight plan	Process
24	Safe landing
25	Reduction in accident rate
26	Help mitigate the consequences of survivable accidents
27	Indirect support—flight punctuality
28	Indirect support—management of delays
29	Indirect support—more efficient flights (and allied impact on fuel consumption)
30	Reduction in accident rate	System/ATM
31	Indirect support—flight punctuality
32	Indirect support—management of delays
33	Potential link to SESAR SWIM concept (i.e. sharing information across ATM network)
34	Potential to extend to support single crew operations

Table 5.

Operational and Safety Benefits.

It is anticipated that this third pilot approach will enhance flight safety, especially in abnormal situations. The third pilot will not remove human error. Rather, it will reduce it. This is largely attributable to improvements in error detection and error management. A cockpit that is designed with the A-PiMod approach in mind will extend automation capabilities in an adaptive way, to the extent necessary to support a safer flight. Potentially, such an adaptive automation approach might prevent many accidents. In this regard, we might consider the AF 447 accident [8]. As indicated in the accident analysis research with COP members and VC2 participants, all 19 participants indicated that A-PiMod would have played a significant role in preventing this accident.

9.2. Safety quantification

As reported previously, it is expected that the A-PiMod concept might lead to a reduction in the probability of fatal accidents by 43% [1, 21].

The assessment of safety impact concerns what has been advanced from a conceptual perspective (i.e. A-PiMod concept), as opposed to the technological progress (set of tools developed in the A-PiMod project, which reflects a particular implementation of the concept) [1, 21]. It is these set of tools that were validated in simulator evaluations [1, 2, 4]. These tools can be viewed as a first technical instantiation of the A-PiMod system. The sophistication, scope and integration of the tools can be improved in future research and development [22]. Once implemented, the expected impact might be further assessed.

10. Conclusions

In the perfect world, A-PIMOD would provide an airline with the most experienced team (crew and automation), capable of dealing with any situation (routine, non-routine, safety critical), where skill level is constant, across all weather/routes/airports/time zones. The third Pilot helps avoid dramas. The Third Pilot/A-PiMod will not eliminate human error. Rather, it will reduce error (human error is a reality and errors will always happen), and help manage them if they occur. In this way, the third pilot approach can be conceptualised in relation to ‘Smart Pilot Assistance’.

A-PiMod will enable the automation system to account for pilots’ emotional and cognitive states. For example, normal, tired, stressed, overloaded and incapacitated. Together with a thoroughly designed adaptive multimodal cockpit, this new technology will significantly improve the safety of flight, especially in abnormal situations and during situations of crisis management.

It should be noted that the assessment of safety impact has been undertaken for the A-PiMod concept, rather than for its particular implementation as achieved in the A-PiMod project. The A-PiMod concept is an advanced adaptive automation concept for a multi-modal cockpit, wherein the automation is seen as a third pilot, and crew and automation continuously adapt to each other and the context, such that safety is maintained. In the course of the A-PiMod project a particular implementation of the concept was achieved by development of a set of tools, and these tools were used in validation experiments (i.e. validation cycle 1 and validation cycle 2), in a flight simulator context. This set of tools can be viewed as a first technical instantiation of the A-PiMod system, and the sophistication, scope and integration of the tools can be improved in future research and development.

Acknowledgments

The research received funding from the European Commission’s Seventh Framework Programme (FP7/2007-2013) under grant agreement N. 605141—Applying Pilot Models for Safety Aircraft (A-PiMod) Project. We would like to thank member of the A-PiMod Project Team for their collaboration in this research. Further, we would like to thanks our COP members (particularly, Paul Cullen, William Butler, Martin Duffy and Stephen Duffy) for their invaluable input. Critically, the emerging adaptive automation concept is predicated on extensive feedback in relation to flight crew experience with automation (and associated problems).