Air Traffic Complexity as a Source of Risk in ATM

2.1 Definition and purpose

The Random House dictionary defines complexity as “the state or quality of being complex; intricacy”, and complex as “composed of many interconnected parts; compound; composite”, “characterized by a very complicated or involved arrangement of parts, units”, and “so complicated or intricate as to be hard to understand or deal with”[4]. While this example uses complicatedto define complex, some other sources argue that there is a major difference between the two. Collins English Dictionary states that [5]:

On the other hand, Cilliers, in his seminal book on the topic, claims exactly the opposite [6]:

One example of such thinking is presented by Snowden in [7]. He claims that the aircraft can be considered complicated due to many parts. Once disassembled and analyzed, the function of all parts and their relationships can be determined. Human organizations and systems are, on the other hand, complex. They are made up of many interacting agents, with agent being any component of the system with identity. Agents can have multiple identities based on the context, i.e., a person can assume group identity or switch between formal and informal identities based on the environment. As these identities change, the components of the system change, the rules an agent follows change, and interactions between the components change. This makes it impossible to distinguish between the cause and effect because they are intertwined [8].

In the context of air traffic control, complexity was rarely clearly defined, perhaps due to assumed common knowledge. One notable exception is Meckiff (et al.) who stated that the air traffic complexity can be most easily defined as difficulty of monitoring and managing a specific air traffic situation[9]. It is intuitively clear that it is easier for the air traffic controller to monitor the airspace sector in which aircraft trajectories do not intersect and there are no level changes than the sector in which there are a lot of merging traffic flows and aircraft often change levels. As such, air traffic complexity could also be defined as a number of potential aircraft-aircraft and aircraft-environment interactions during a given time frame. Not all of these interactions require the same level of attention, urgency, or, ultimately, controller workload to resolve.

Complexity is notthe same as traffic density. Obviously, the number of aircraft in a sector (also known as density, traffic load, or traffic count) directly influences the air traffic complexity. This number, however, is not the only indicator of the level of complexity, especially if one wishes to compare different sectors of airspace [10, 11, 12]. Two traffic situations can have equal density but vastly different complexity. Due to two different types of interactions, some researchers have chosen to make a distinction between airspace complexity(also static, structural) and air traffic complexity(also dynamic, flow complexity [13]) which is influenced by the airspace complexity. This distinction will be used in this chapter as well. Unless explicitly stated, complexity will from now on refer exclusively to air traffic complexity.

Complexity is nota synonym for workload, although it has been proven multiple times that the increase in complexity results in increase in workload which in turn limits the airspace sector capacity [14, 15]. Mogford et al. [11] reviewed numerous research articles in search of complexity and workload relationship. They concluded that the complexity is actually a source factor for controller workload (Figure 1). However, complexity and workload are not directly linked. Their relationship is mediated by several other factors, such as equipment quality, individual differences, and controller cognitive strategies [11].

Controller cognitive strategies can be improved through training and experience that is readily seen when comparing experienced and inexperienced controllers. However, if one takes into consideration an average controller with average training, only two avenues to reduced controller workload remain—increasing equipment quality and decreasing complexity.

2.2 Previous research on air traffic complexity

Complexity was a common research topic since the early days of modern ATC operations. First papers that mention complexity were written in the early 1960s [16]. Since then, dozens of papers and reports were written on the topic of complexity—excellent reviews of those papers were written by Mogford [11] and Hilburn [17]. Instead of writing a completely new literature review, this chapter will present important research paths, ideas, methods, and facts, which are relevant to the present research.

It needs to be noted that most of the early research was conducted in order to better define factors that affect workload. Today, most of those factors, with present understanding and definitions, would probably be called complexity factors. Some studies were nonempirical and lack exact definitions and measurement methods for complexity indicators. Those studies were excluded from this short review to give more room to those studies with experimentally validated complexity factors.

Schmidt [18] approached the problem of modelling controller workload from the angle of observable controller actions. He created the control difficulty index, which can be calculated as a weighted sum of the expected frequency of occurrence of events that affect controller workload. Each event is given a different weight according to the time needed to execute a particular task. Though the author conducted extensive surveys to determine appropriate weights and frequencies for various events, this approach can only handle observable controller actions, which makes it very limiting.

Hurst and Rose [19], while not the first to realize the importance of traffic density, were first to measure the correlation of expert workload ratings with traffic density. They concluded that only 53% of the variance in reported workload ratings can be explained by density.

Stein [20] used Air Traffic Workload Input Technique (ATWIT), in which controllers report workload levels during simulation, to determine which of the workload factors influenced workload the most. Regression analysis proved that out of the five starting factors, four factors (localized traffic density, number of handoffs outbound, total amount of traffic, number of handoffs inbound) could explain 67% of variance in ATWIT scores. This study showed the importance of localized traffic density which is a measure of traffic clustering. Technique similar to ATWIT will be used throughout the next three decades, including a modified ATWIT scores that will be used in this research.

Laudeman et al. [21] expanded on the notion of the traffic density by introducing dynamic density which they defined as a combination of “both traffic density (a count of aircraft in a volume of airspace) and traffic complexity (a measure of the complexity of the air traffic in a volume of airspace).” Authors used informal interviews with controllers to obtain a list of eight complexity factors to be used in dynamic density equation. The only criterion was that the factors could be calculated from the radar tracks or their extrapolations. The intention was to obtain an objective measure of controller workload based on the actual traffic. Their results showed that the dynamic density was able to account for 55% of controller activity variation. Three other teams [13, 22, 23] working under the Dynamic Density program developed additional 35 complexity indicators (factors), which were later successfully validated as a group by Kopardekar et al. [24]. Unfortunately, it was later shown that the complexity indicator weights were not universal to all airspace sectors, i.e., they had to be adjusted on a sector-by-sector basis [25]. This shortcoming, while making the dynamic density technique difficult to implement for operational purposes, has no influence if one wishes to compare two concepts of operations under similar conditions (similar sector configuration). Furthermore, same authors [24] suggested that, due to possibly nonlinear interactions between complexity factors, the dynamic density performance could be improved by using nonlinear techniques such as nonlinear regression, genetic algorithms, and neural networks.

Almost the same group of authors will use multiple linear regression method 5 years later to determine which subset of complexity indicators will correlate well with the controller’s subjective complexity ratings [26]. After extensive simulator validation, results of this study showed that there are 17 complexity indicators that are statistically significant. Top five complexity indicators were sector count, sector volume, number of aircraft under 8 NM from each other, convergence angle, and standard deviation of ground speed/mean ground speed. Similar work was done by Masalonis et al. [27] who selected a subset of 12 indicators and Klein et al. [28] who selected a subset of only 7 complexity indicators, though with less extensive experimental validation.

In a similar vein, Bloem et al. [29] tried to determine which of the complexity indicators had the greatest predictive power in terms of future complexity. The authors concluded that there is a significant difference in predictive power of different complexity indicators. To complicate the matter further, they concluded that the subset of the complexity indicators that had the best predictive power changed depending on the prediction horizon.

To calculate potential impact of air traffic complexity on workload and costs, in 2000 the EUROCONTROL has given the same set of traffic data to UK National Air Traffic Services (NATS) and the EUROCONTROL Experimental Centre (EEC) with a task of independently devising a method of measuring the level of service [30]. While NATS has estimated ATS output (the service provided), the EEC has estimated the ATS workload needed to deliver the service. Both “were found to produce reasonably consistent results,”with an additional note that further analysis should be done before the final parameters for determining ATS provider costs are established. By 2006 EUROCONTROL’s Performance Review Commission finalized the complexity indicators to be used for ANSP benchmarking [31]. For this method the European airspace is divided into 20 NM X 20 NM X 3000 ft. cells, and for each cell the duration of potential interactions is calculated. Aircraft are “interacting” if they are in the same cell at the same time. The ratio of the hours of interactions and flight hours is the so-called adjusted density. In addition, the “structural index” is calculated as a sum of potential vertical, horizontal, and speed interactions. The final complexity score is calculated as a product of adjusted density and structural index. All in all, only four complexity indicators are used for this analysis, and no validation of any sort was presented in the report. It was noted, however, that shifting the starting position of the grid by 7 NM caused the ANSP ranking to change dramatically (up to 16 places in an extreme case). Nonetheless, this method is still used for ANSP benchmarking.

First to consider measuring complexity during TBO were Prevot and Lee [32]. They coined the term trajectory-based complexity (TBX) which is a measure of complexity in TBO. The basis of the TBX calculation is a set of nominal conditions—nominal sector size, nominal number of transitioning aircraft, and a nominal equipage mix. Any difference to nominal operations causes a modification to the TBX value. Authors do not explain the method to determine the nominal conditions except that they can “be defined through knowledge elicitation sessions on a sector by sector basis or based upon more generic attributes.” The TBX value is then a number of aircraft that would produce the same workload under the nominal conditions as do aircraft under real conditions (e.g., the TBX of 20 means that the workload is equal to the aircraft count of 20 under nominal conditions even though there are actually only 16 aircraft in the sector). The advantage of this method is that it gives a single complexity value that can be easily related to aircraft count and is thus very user-friendly and self-explanatory (unlike many other complexity metrics). However, this study included only six complexity indicators with weights that were determined in an ad hoc manner and hardly any validation with actual subjective complexity. Only one of those complexity indicators was indirectly related to TBO (number of aircraft with data-link). Many human-in-the-loop simulation runs were performed in which the controllers had to give workload scores which were then compared with TBX value and simple aircraft count. While the authors claim that the subjective workload score correlated better with the TBX value, there was no objective correlation assessment presented. Finally, the authors have not compared the effect of fraction of TBO aircraft on air traffic complexity.

In a subsequent paper by same authors, the relationship between workload and data-link equipage levels was explored [33]. It was concluded that the workload ratings correlated much better with the TBX score than with the aircraft count for varying data-link equipage levels.

Prandini et al. have developed a new method of mapping complexity based exclusively on traffic density [34]. This method is applicable only to the future concept of aircraft self-separation and does not take into account the human factors at all.

Gianazza [35, 36, 37] proposed a method for prediction of air traffic complexity using tree search methods and neural networks. This method is based on the assumption that the air traffic complexity in historic flight data increased prior to the splitting of the collapsed sector into two smaller ones and decreased prior to collapsing the sectors into a larger one. The neural network was trained using this historical data, and then it could predict future increase in air traffic complexity. Tree search method was then used to determine the airspace configuration which yields lowest workload and complexity for the given air traffic pattern.

Lee et al. [38] have proposed that airspace complexity can be described in terms of how the airspace (together with the traffic inside it and the traffic control method) responds to disturbances. The effect of disturbances on control activity needed to accommodate that disturbance is what defines complexity in their opinion. The more control activity needed, the more complex the airspace is. They propose a tool, airspace complexity map, which should help to plan the airspace configuration and the future development of ATM.

In Radišić et al. [39], authors used domain-expert assessment to test the effect of the trajectory-based operations (TBO) on air traffic complexity. ATCOs were recruited to perform human-in-the-loop (HITL) simulations during which they were asked to provide real-time assessment of air traffic complexity. Linear regression model was used to select, among 20 most used complexity indicators, those indicators which correlated best with subjective complexity scores. Six indicators were used to generate a predictive linear model that performed well in conventional operations but less so under TBO. Therefore, the authors defined and experimentally validated two novel TBO-specific complexity indicators. A second correlation model combining these two novel indicators with four already in use generated much better predictions of complexity than the first model. Nonetheless, the best correlation that was achieved was R = 0.83 (R²-adjusted = 0.691). In subsequent work, the authors attempted to achieve better prediction by using artificial neural networks; however, similar results were obtained. This indicates that there is some variation in subjective complexity scores provided by ATCOs that cannot be explained by traffic properties. Indeed, it might be the case that ATCOs introduce a degree of noise into the complexity scores due to difficulty of maintaining the consistent scoring criteria [40].

Wang et al. [41] in their work used network approach to calculate air traffic complexity based on historical radar data. Their assumption is that air traffic situation is essentially a time-evolving complex system. In that system aircraft are key waypoints; route segments are nodes; aircraft-aircraft, aircraft-keypoint, and aircraft-segment complexity relationships are edges; and the intensities of various complexity relationships are weights. The system was built using a dynamic weighted network model.

Xue et al. [42] in their work analyzed three complexity indicators for simulated UAS traffic: number of potential conflicts, scenario complexity metric, and number of flights. Scenario complexity metric is based on cost of pairwise conflict which is defined as deviation from the original path. To perform analysis on around 1000 scenarios at different density levels, authors had to develop a UAS simulator. Analysis was done using Pearson and ACE statistics methods.

Future concept of operations will involve usage of far wider range of air traffic controller tools; therefore, it is expected that new complexity indicators related to interaction of controllers and equipment will have to be developed. Furthermore, novel complexity assessment methods are needed due to limits of current techniques.

2.3 Complexity estimation methods

In this section, several air traffic complexity estimation methods will be examined in greater detail. All complexity estimation methods are based on the traffic data which describes a traffic situation. Since the complexity is a psychological construct, the most relevant estimator of complexity in a given traffic situation is the air traffic controller. The air traffic controller can look at the traffic data and decide whether a traffic situation is complex or not. All other methods are just attempts at approximating the level of complexity as estimated by the controller. The main problem with expert-based estimation is the inconsistency between controllers, where one controller gives a different complexity estimate than the other. Therefore, most other methods seek ways to make the complexity estimate without human input. Ideally, those other methods would be validated by comparing them to the expert, i.e., controller’s estimate; however, this is not always the case.

Three main methods of control-based (i.e., based on ATCOs’ experience of complexity as a driver for workload and, subsequently, limiting factor of airspace capacity) air traffic complexity estimation will be presented here:

• Expert-based air traffic complexity estimation—where an expert, in most cases an air traffic controller, gives their estimate of the complexity

• Indicator-based air traffic complexity estimation—where the values of complexity indicators, derived from traffic data, are used to determine the level of complexity

• Interaction-based air traffic complexity estimation—where the complexity is estimated on the basis of the number of aircraft interactions in a given airspace cell (this method could be broadly defined as a very narrow indicator-based complexity estimation method due to a very low number of indicators)

• Others—methods based on other principles, such as counting the number of clearances [43], evaluating proximity based on probabilistic occupancy of airspace [44], measuring sensitivity to initial conditions of the underlying dynamic system called Lyapunov exponents (i.e., assessing predictability of traffic) [45], and many others

4.1 Human reliability assessment

This section will provide a brief overview of the HRA methods and their development over the years; however, for a more thorough review of HRA methods, one can find more information in [46, 50]. Only those methods that are in some way relevant to HRA in aviation will be considered.

HRA can be defined as “any method by which human reliability is estimated” [51], and it is generally presented as having three main parts: (1) identifying possible human errors and contributors, (2) modelling human error, and (3) quantifying human error probabilities. These methods were first developed in nuclear power safety systems.

In the early models of HRA, human was often considered as just another part of the system. For example, in [52], a technique for human error-rate prediction(THERP) was developed based on the techniques used in nuclear power plant risk management, i.e., a straightforward event tree analysis was performed. Each human action (e.g., reading a display, operating a lever) was given a human error probability (HEP) as a probability with a value from 0 (least probable) to 1 (most probable). Sample of values for different errors can be seen in Table 1. The values assigned to each error type came from authors’ experience and from earlier studies performed in the defense sector.

THERP also specified performance shaping factors (PSF) which were used to modify the nominal HEPs based on the context of the action (e.g., time pressure, human-machine interface, etc.). A list of possible PSFs for one error is given in Table 2. One can notice that there are no error multipliers associated with each PSF. It is the duty of the assessor to define the maximum affect that each PSF could have on HEPs. Criticism of THERP was mostly that it was too difficult to apply because of quite detailed decomposition of tasks that it relied on a database of HEPs which was never really validated and that it took very broad and casual definitions of human performance factors.

Another version of this type of model was done in human error assessment and reduction technique (HEART) [53]. The database of HEPs was much smaller and more generic, so it was more flexible and easier to apply than THERP. Instead of highly detailed errors, the focus is on a handful of generic task types for which probabilities of failure are given (Table 3). This simplification has made the HEART technique much more accepted outside the nuclear power industry for which the THERP was designed.

The author has identified the human factors he found relevant by searching the human factors literature and assigned relative weights to them, identified impacts of errors, and suggested a set of human error data which should enable higher reliability of the system. Instead of calling them PSFs, the author called them error-producing condition (EPC) and provided the multipliers for each. Multipliers are used to increase the nominal human unreliability in cases where there are circumstances that increase the probability of human error. Some of the EPCs are shown in Table 4.

More generic error types have led to confusion when trying to apply it to a specific industrial application. This problem was addressed by developing specialized versions of HEART for specific industries. One such derivative will be discussed shortly.

These models are characterized by defining two broad categories of errors: errors of omission (when human operator fails to make an action) and errors of commission (when operator makes a wrong action). These simplifications were later put, at least partially, into the context of actual human behavior which knows many other ways of committing an error. For example, [54, 55] included contextual effect such as stress, organizational culture, and tiredness into the model, whereas [56, 57] also included the possible variation in operator’s responses and recovery actions undertaken once the errors have been noticed. By taking into account the context of human behavior, these techniques have made a qualitative step forward in comparison to the THERP and HEART, so they are generally called second-generation HRA techniques. This did not, however, improve their adoption in the industry because simpler and more flexible techniques, such as HEART, are more usable and sustainable. For this reason, the first HRA technique developed specifically for ATM was based on HEART technique. It was developed in 2008 and named Controller Action Reliability Assessment (CARA) [58].

4.2 Human reliability assessment in ATC

Compared to HEART, CARA’s generic task types were developed to better suit the needs of HRA in ATM (Table 5). To make sure that the task types are in line with the commonly used models of ATCO tasks, the basis for task development was found in EUROCONTROL’s studies. Literature and ergonomics database reviews were undertaken to find the data which supports new values of HEPs for each generic task type. Where more than one error probability for a given task was found in the literature or the databases, geometric mean was used to establish a single value. Furthermore, uncertainty bounds of each HEP were determined using the single sample t-test [59].

EPCs used in CARA were, like general task types, developed by adjusting EPCs from HEART and other techniques (most notably SPAR-H [60] and CREAM [61]). To ensure that the CARA EPCs closely follow the well-established contextual structure used in ATC, they were modelled to fit the Human Error in ATM (HERA) [62] classification structure. For initial consideration, CARA EPCs’ maximum affect values were taken from HEART, SPAR-H, and CREAM by selecting the most similar EPCs and then picking the one with the highest value (Table 6). It is expected that with further refinement of underlying data, the maximum affect values will be adjusted to better suit the actual values in ATC.

For the first time here, one can see that the traffic complexity was taken into account (EPC 17) with maximum affect of 10. CARA User’s Manual provides additional information about this EPC, adding three anchor points for this EPC [63]:

• Higher than normal traffic levels with some non-routine conflicts to solve (EPC multiplier 0.1)

• Higher than normal traffic levels with some non-routine conflicts requiring constrained solutions; possibility of secondary conflicts (conflict resolution can lead to a second conflict) (EPC multiplier 0.5)

• High traffic levels with unusual patterns of traffic requiring problem solving and a number of future conflicts requiring resolution (EPC multiplier 1.0)

EPC multipliers are used to scale the EPC affect from its maximum value to the actual value for the situation that is being assessed, thus getting the actual effect (AE). As is the case with many HRA techniques, some expert opinion is needed here to determine where the assessed scenario falls on the scale of 0.1–1.0. An example of human error risk calculation is given in the next section.

4.3 Using CARA to assess the effect of complexity on ATCO error risk

To better show how CARA is used to assess the effect of complexity on ATCO error risk, a simple example will be used. In this example, we suppose that the ATCO is working on an en route sector with moderately high air traffic complexity. Weather is calm and there are no failures in any of the air or ground equipment. In these conditions, we might want to assess the probability that the ATCO will not notice a conflict.

To do this, we select a generic task type (GTT) that best suits our situation. Here, it is C1. Identify routine conflictwith HEP of 0.01. Appropriate EPC to select in this case is the EPC 17: traffic complexitywith maximum affect of 10. Also, we use our expertise to determine that the current traffic situation is moderately complex, so we use EPC multiplier to determine the assessed effect (AE) equal to 0.4. Calculating the probability (P) of ATCO’s failure to detect the conflict is then calculated using Eqs. 1–3.

The result shows that the probability of ATCO failing to notice a conflict in a moderately complex situation is 0.046 or 4.6%. The −1 and +1 in Eq. 1 are added to ensure that the resulting EPC is more than 1 without needlessly increasing the EPC (e.g., if only the final +1 was added). Conversely, the probability of ATCO identifying a conflict is equal to 95.4%. These probabilities are valid for a situation with only one ATCO; however, en route ATC operations are usually performed with two ATCOs handling a sector (planning and executive ATCOs). The probability that both ATCOs will fail to notice the conflict is equal to 0.046 x 0.046 = 0.0021 which is to say that approximately 1 in 500 conflicts in moderately complex traffic situations will not be identified (step 1 in Figure 2). Fortunately, ATC tools, such as short-term conflict alert (STCA), will sound the alarm in that case, and the ATCO will have the opportunity for a timely recovery.

This calculation showed how to use CARA to determine probability of a single event. Events can be chained into probability trees to calculate the probability of a sequence of events. Building on the previous example, we can calculate the probabilities of further events after the conflict was identified or after a conflict was missed. First possibility, and a more probable one, is that the conflict was identified. Next step for ATCOs is to solve it. Let us assume that this task can be assigned to the D1. Solve conflict which includes some complexityGTT which is assigned HEP of 0.01. Using a GTT with the same HEP as in previous example, in combination with same EPC for traffic complexity, will yield the same error probability of 0.046 (step 2 in Figure 2). If ATCO notices that the conflict is not solved, they will make another attempt to solve it (step 3 in Figure 2). This can be considered a recovery action for the previous error (not solving the conflict). It is up to the assessor to analyze the traffic situation and operational procedures to determine how many attempts an ATCO could have before the STCA alarm rings. Modelling of additional tools, such as separation tool which helps ATCO to determine whether the conflict resolution action was successful or not, can assist the assessor in determining the most accurate sequence of events.

If the conflict was missed or the ATCO could not solve it in time, STCA will sound the alarm. This usually occurs 2 min before the loss of separation. ATCOs’ response to the STCA can be modelled using the B3. Respond to unique and trusted audible and visual indicationGTT which is assigned HEP of 0.0004. Due to short time until loss of separation, it is reasonable to use EPC number 5: time pressure due to inadequate time to complete the taskwhich is assigned maximum affect value of 11. Since this GTT only relates to noticing and responding to the STCA, the actual effect of this EPC will be on the lower side, so the multiplier is set to 0.2. Calculation of the error probability is then made with Eqs. 4–6.

This calculation shows that the probability of not noticing the STCA alarm will be 0.12% (step 4 in Figure 2). Once the ATCO notices the STCA, they will make another effort to solve the conflict. This time, the appropriate GTT is D2: complex and time pressured conflict solutionwhich is assigned HEP value of 0.19 with confidence interval between 0.09 and 0.39. The assessor should use expert guidance to determine which value should actually be used; in this example, 0.15 will be used. In addition, assessor could add two EPCs, one for time pressure ((5) time pressure due to inadequate time to complete the task) and one for complexity ((17) traffic complexity); however, CARA User Manual states that the EPC 5 should not be combined with GTT D2 and neither should EPC 5 and 17 be used together [63]. This prevents overly pessimistic results. Therefore, only EPC 17 will be included in the assessment. Like in previous steps of this example, we will use 0.4 as EPC multiplier to determine the assessed effect. The calculation is given by Eqs. 7–9.

This calculation shows that, in complex traffic situation, the probability of a conflict not being solved under time pressure (STCA alarm) will be 69% (step 5 in Figure 2). In comparison, if the traffic is not complex, the probability of failure will be only 15%. Obviously, assessor should adjust the values of GTTs and EPCs to better suit the situation being assessed, so these probabilities are in no way final.

Finally, the probability of each outcome can be calculated by multiplying the probabilities of each event that led to that outcome. For example, if one wishes to calculate the probability that the conflict will be solved only after two failed attempts and an STCA alarm, step 5 in Figure 2, they should multiply probabilities of all events leading to that outcome as seen in Eqs. 10–12.

The last step in this process is to sum up all the probabilities of a favorable outcome (conflict solved) versus all the probabilities of an unfavorable outcome (loss of separation). In this example, the probability of the favorable outcome is 99.71% versus the probability of an unfavorable outcome which is 0.29%.

To better appreciate the effect of traffic complexity on the risk of human error, comparison with the traffic situation which is not complex can be made by excluding the traffic complexity EPC from the calculation. This calculation is omitted here for brevity, but the same method without the traffic complexity EPCs yields probability of a loss of separation below 3.5 × 10⁻⁵ per conflict (approximately 1 in 28,600 conflicts). That is two orders of magnitude less probable than in the case with moderate complexity (0.29% or 1 in 345). On the other hand, if the traffic is highly complex, the assessor might use higher EPC multiplier for complexity, all the way up to 1. In that case, the probability of an unfavorable outcome, i.e., loss of separation, is 2% (1 in 50) which is 7 times more probable than in the example above (Table 7).

4.4 Using simulations to assess the effect of traffic complexity on risk

In addition to CARA, another method for assessing risks related to air traffic complexity is by conducting simulations. Simulation is a core method for ATM research and training, with different purposes requiring different levels of fidelity and simulation scope. Fidelity refers to the level of similarity between the simulated environment and the actual operations. Simulation scope can be broadly divided into strategic and tactical simulations. Strategic simulation tools (e.g., EUROCONTROL’s NEST) are used to analyze current and forecast future ATM situation on a global level. On the other hand, tactical simulation tools are used to accurately simulate ATC operations on a sector level (e.g., ATCoach by UFA or Micronav’s BEST Radar Simulator) [64]. For studies involving human factors, tactical real-time human-in-the-loop simulations provide the most reliable results.

Most representative results are produced when the simulator satisfies these requirements:

• Realistic working environment

• Accurate and versatile aircraft models

• Representative ATC tool operation

• Built-in stochasticity

• Human voice communication

• Research-level data logging

• Suitable meteorological model

• Suitable system and sub-system failure modelling

We used HITL simulations to assess the effect of trajectory-based operations (TBO) on air traffic complexity; for more information about that study, see [39]. Here we will provide a brief description of the methodology used and additional analysis of human errors made during that experiment. This will enable comparison of the simulation with the results obtained from CARA.

4.4.2 Simulation results and comparison with CARA

Overall, 88 simulator runs were performed, each lasting for approximately 50 min. Though it is very difficult to ascertain the number of potential and actual conflicts, the frequency of STCA alarms and loss of separation occurrences can be used to assess the risk that air traffic complexity introduces. Before going into further details, it must be noted that the probabilities presented herein are accurate only for this particular set of scenarios in this particular airspace controlled by these particular ATCOs, even if the sample size issues are disregarded. These probabilities should not be used for making real-life operational decisions and are presented here as an example of the human reliability analysis that can be produced from real-time HITL simulations.

In Figure 3, all 88 simulation runs are plotted, showing scenario complexity and number of STCA alarms for each. Blue dots represent simulation runs which had only STCAs, whereas red dots show those runs in which loss of separation also occurred. ATCOs were not allowed to give additional complexity scores once the loss of separation occurred, thus preventing that event from influencing their opinion. Separation minima were 5 NM horizontally and 1000 ft. vertically. Complexity scores were calculated as an average of the ATCO’s subjective complexity scores made during the peak 20 min of the simulation run [39]. Correlation coefficient between these two variables, complexity and number of STCAs, is 0.71, which indicates a somewhat strong correlation.

First thing to notice is that most of the simulator runs, 58 out of 88, finished with zero STCAs. Of the remaining 30, only 5 were in mediumtraffic load scenarios, i.e., scenarios with traffic loads equal to current peak traffic. The remaining 25 were all in hightraffic load scenarios which were designed with 15% higher peak traffic loads.

Next thing to notice is that, even though the complexity scores are highly subjective, it is very rare to have scenarios with complexity higher than 4 and no STCAs (only 4 out of 33 or 12%). This indicates that the ATCOs are bunching most of the scenarios into the lower half of the scale, perhaps underestimating the actual difficulty of managing the traffic situations.

In terms of HRA, it is interesting to calculate the probability that the STCAs will be resolved before the loss of separation occurs. Overall probability of human error in this case is only 0.155 (11 out of 71) compared to the figure calculated by CARA in the example presented in the previous section, which was 0.69. Surprisingly, this probability will not change much even if the scenarios were filtered by complexity. For example, for scenarios with complexity above 5, the probability of an STCA turning into a loss of separation is 0.175 (10 out of 57). For scenarios with complexity above 6, the probability is only slightly higher at 0.189 (7 out of 37). Here, ATCOs obviously show significant compensatory effects which should be included into CARA or modelled more precisely by assessors using the existing GTTs and EPCs.

On the other hand, the probability that the simulation run will contain at least one loss of separation rises sharply with complexity. For the lower half of the complexity scale, this probability is zero. If we consider all scenarios with complexity score equal to or above 4, the probability of loss of separation is 0.33 (11 out of 33). For scenarios with the score equal to or above 5, the probability is 0.5 (10 out of 20), and for scenarios with the complexity score above 6, the probability is 0.538 (7 of 13). This shows that even though the probability of an STCA turning into loss of separation is lower than expected by CARA, the number of conflicts rises to the level at which the loss of separation becomes extremely probable.

As for the Cynefinframework, it could be applied here only in broad brushes. One could argue that the first quarter of the complexity scale in these simulations maps to the simpledomain because there are no STCAs. Second quarter, with only a couple of STCAs which were quickly resolved, perhaps maps to the complicateddomain. The third quarter could be mapped to the complexdomain because there are many STCAs, but only two were not resolved in time. Finally, the last quarter of the scale arguably maps to the chaoticdomain due to high probability of loss of separation which indicates that the ATCOs had lost the immediate control of the situation. Notwithstanding the Cynefinframework, it is clear that the ATM system should be designed to keep the complexity in the lower half of the scale and serious efforts are needed to achieve this in the face of the rising traffic demand.