Coordination Demand in Human Control of Heterogeneous Robot

1. Introduction

The performance of human-robot teams is complex and multifaceted reflecting the capabilities of the robots, the operator(s), and the quality of their interactions. Recent efforts to define common metrics for human-robot interaction (Steinfeld et al., 2006) have favored sets of metric classes to measure the effectiveness of the system’s constituents and their interactions as well as the system’s overall performance. In this chapter we follow this approach to develop measures characterizing the demand imposed by tasks requiring cooperation among heterogeneous robots.

Applications for multirobot systems (MRS) such as interplanetary construction or cooperating uninhabited aerial vehicles will require close coordination and control between human operator(s) and teams of robots in uncertain environments. Human supervision will be needed because humans must supply the perhaps changing goals that direct MRS activity. Robot autonomy will be needed because the aggregate decision making demands of a MRS are likely to exceed the cognitive capabilities of a human operator. Autonomous cooperation among robots, in particular, will likely be needed because it is these activities (Gerkey & Mataric, 2004) that theoretically impose the greatest decision making load.

Controlling multiple robots substantially increases the complexity of the operator’s task because attention must constantly be shifted among robots in order to maintain situation awareness (SA) and exert control. In the simplest case an operator controls multiple independent robots interacting with each as needed. A search task in which each robot searches its own region would be of this category although minimal coordination might be required to avoid overlaps and prevent gaps in coverage. Control performance at such tasks can be characterized by the average demand of each robot on human attention (Crandal et al., 2005). Under these conditions increasing robot autonomy should allow robots to be neglected for longer periods of time making it possible for a single operator to control more robots.

Because of the need to share attention between robots in MRS, teloperation can only be used for one robot out of a team (Nielsen et al., 2003) or as a selectable mode (Parasuraman et al., 2005). Some variant of waypoint control has been used in most of the MRS studies we have reviewed (Crandal et al., 2005, Nielsen et al., 2003, Parasuraman et al., 2005, Trouvain & Wolf, 2002) with differences arising primarily in behavior upon reaching a waypoint. A more fully autonomous mode has typically been included involving things such as search of a designated area (Parasuraman et al., 2005), travel to a distant waypoint (Trouvain & Wolf, 2002), or executing prescribed behaviors (Murphy and Burke, 2005). In studies in which robots did not cooperate and had varying levels of individual autonomy (Crandal et al., 2005, Nielsen et al., 2003, Trouvain & Wolf, 2002) (team size 2-4) performance and workload were both higher at lower autonomy levels and lower at higher ones. So although increasing autonomy in these experiments reduced the cognitive load on the operator, the automation could not perform the replaced tasks as well.

For more strongly cooperative tasks and larger teams individual autonomy alone is unlikely to suffice. The round-robin control strategy used for controlling individual robots would force an operator to plan and predict actions needed for multiple joint activities and be highly susceptible to errors in prediction, synchronization or execution. Estimating the cost of this coordination, however, proves a difficult problem. Established methods of estimating MRS control difficulty, neglect tolerance and fan-out (Crandal et al., 2005) are predicated on the independence of robots and tasks. In neglect tolerance the period following the end of human intervention but preceding a decline in performance below a threshold is considered time during which the operator is free to perform other tasks. If the operator services other robots over this period the measure provides an estimate of the number of robots that might be controlled. Fan-out works from the opposite direction, adding robots and measuring performance until a plateau without further improvement is reached. Both approaches presume that operating an additional robot imposes an additive demand on cognitive resources. These measures are particularly attractive because they are based on readily observable aspects of behavior: the time an operator is engaged controlling the robot, interaction time (IT), and the time an operator is not engaged in controlling the robot, neglect time (NT).

This chapter presents an extension of Crandall’s Neglect Tolerance model intended to accommodate both coordination demands (CD) and heterogeneity among robots. We describe the extension of Neglect Tolerance model in section 2. Then in section 3 we introduce the simulator and multi-robot system used in our validation experiments. Section 4 and 5 describes two experiments that attempt to manipulate and directly measure coordination demand under tight and weak cooperation conditions separately. Finally, we draw conclusion and discuss the future work in section 6.

2. Cooperation demand

If robots must cooperate to perform a task such as searching a building without redundant coverage or act together to push a block, this independence no longer holds. Where coordination demands are weak, as in the search task, the round robin strategy implicit in the additive models may still match observable performance, although the operator must now consciously deconflict search patterns to avoid redundancy. For tasks such as box pushing, coordination demands are simply too strong, forcing the operator to either control the robots simultaneously or alternate rapidly to keep them synchronized in their joint activity. In this case the decline in efficiency of a robot’s actions is determined by the actions of other robots rather than decay in its own performance. Under these conditions the sequential patterns of interaction presumed by the NT and fan-out measures no longer match the task the operator must perform. To separate coordination demand (CD) from the demands of interacting with independent robots we have extended Crandall’s Neglect Tolerance model by introducing the notion of occupied time (OT) as illustrated in Figure 1.

Figure 1.

Extended neglect tolerance model for cooperative.

The neglect tolerance model describes an operator’s interaction with multiple robots as a sequence of control episodes in which an operator interacts with a robot for period IT raising its performance above some upper threshold after which the robot is neglected for the period NT until its performance deteriorates below a lower threshold when the operator must again interact with it. To accommodate dependent tasks we introduce OT to describe the time spent controlling other robots in order to synchronize their actions with those of the target robot. The episode depicted in Figure 1 starts just after the first robot is serviced. The ensuing FT preceding the interaction with a second dependent robot, the OT for robot-1 (that would contribute to IT for robot-2), and the FT following interaction with robot-2 but preceding the next interaction with robot-1 together constitute the neglect time for robot-1. Coordination demand, CD, is then defined as:

$$\text{CD }=1-\frac{{\displaystyle \sum FT}}{NT}=\frac{{\displaystyle \sum OT}}{NT}$$ E1

where, CD for a robot is the ratio between the time required to control cooperating robots and the time still available after controlling the target robot, i.e. the portion of a robot’s free time that must be devoted to controlling cooperating robots. Note that the OT associated with a robot is less than or equal to NT because OT covers only that portion of NT needed for synchronization. A related measure, team attention demand (TAD), adds IT’s to both numerator and denominator to provide a measure of the proportion of time devoted to the cooperative task, either performing the task or coordinating robots.

2.1. Measuring weak cooperation for heterogeneous robots

Most MRS research has investigated homogeneous robot teams where additional robots provide redundant (independent) capabilities. Differences in capabilities such as mobility or payload, however, may lead to more advantageous opportunities for cooperation among heterogeneous robots. These differences among robots in roles and other characteristics affecting IT, NT, and OT introduce additional complexity to assessing CD. Where tight cooperation is required as in the box-pushing experiment, task requirements dictate both the choice of robots and the interdependence of their actions. In the more general case requirements for cooperation can be relaxed allowing the operator to choose the subteams of robots to be operated in a cooperative manner as well as the next robot to be operated. This general case of heterogeneous robots cooperating as needed characterizes the types of field applications our research is intended to support. To accommodate this case the Neglect Tolerance model must be further extended to measure coordination between different robot types. We describe this form of heterogeneous MRS as a MN system with M robots that belong to N robot types, and for robot type i, there are m_i robots, that is $$M={\displaystyle \sum _{i=1}^N{m}_i}$$ . Thus, we can denote a robot in this system as R_ij, where i = [1, N], j = [1, m_i]. If we assume that the operator serially controls the robots for time T and that each robot R_ij is interacted with l_ij times, then we can represent each interaction as IT_ijk , where i = [1, N], j = [1, m_i], k = [1, l_ij], and the following free time as FT_ijk , where i = [1, N], j = [1, m_i], k = [1, l_ij]. The total control time T_i for type i robot should then be $$T_i={\displaystyle \sum _{j,k}\left(I{T}_{ijk}+F{T}_{ijk}\right)}$$ . Because robots that are of the same robot type are identical, and substitution may cause uneven demand, we are only interested in measuring the average coordination demand CD_i, i=[1, N] for a robot type.

Given robots of the same type R_ij, j = [1, m_i], we define OT_i ^* and NT_i ^* as the average occupation time and interaction time in a robot control episode. Therefore, the CD_i for type i robot is

$$C{D}_i=\frac{1}{m_i}{\displaystyle \sum _{j=1}^{m_i}C{D}_{ij}}=\frac{1}{m_i}{\displaystyle \sum _{j=1}^{m_i}\frac{l_{ij}O{T}_i^{*}}{l_{ij}N{T}_i^{*}}}=\frac{O{T}_i^{*}{\displaystyle \sum _{j=1}^{m_j}{l}_{ij}}}{N{T}_i^{*}{\displaystyle \sum _{j=1}^{m_j}{l}_{ij}}}$$

Assume all the other types robots are dependent with the current type robots, then the numerator is the total interaction time of all the other robot types, i.e. $$O{T}_i^{*}{\displaystyle \sum _{j=1}^{m_j}{l}_{ij}}={\displaystyle \sum _{\begin{array}{l}type=1\\ type\ne i\end{array}}^{N}IT}$$ .

Figure 2.

Distribution of (IT, FT).

For the denominator, it is hard to directly measure NT_i ^* because the system performance depends on multiple types of robots and an individual robot may cooperate with different team members over time. Because of this dependency, we cannot use individual robot’s active time to approximate NT. On the other hand, the robots may be unevenly controlled. For example a robot might be controlled only once and then ignored because there is another robot of the same type that is available, so we cannot simply use the time interval between two interactions of an individual robot as NT. Considering all the robots belonging to a robot type, the population of individual robots’ (IT, FT)s reveal the NT for a type of robot. Figure 2 shows an example of how robots’ (IT, FT) might be distributed over task time. Because robots of the same capabilities might be used interchangeably to perform a cooperative task it is desirable to measure NT with respect to a type rather than a particular robot. In Figure 2 robots R₁₁ and R₁₂ have short NTs while R₁₃ has an NT of indefinite length. F(IT, FT), the distribution of (IT, FT) for the robot type, shown by the arrowed lines between interactions allows an estimate of NT for a robot type that is not affected by long individual NTs such as that of R₁₃. When each robot is evenly controlled, the F(IT, NT) should be $$m_i\times {\left(I{T}_i,F{T}_i\right)}^{*}$$ where (IT_i, FT_i)^* is the (IT, FT) for each type i robot, $${\left(I{T}_i,F{T}_i\right)}^{*}=\frac{T_i}{{\displaystyle \sum _{j=1}^{m_i}{l}_{ij}}}$$ . And when only one robot is controlled, F(IT_i, NT_i)^* will be the (IT_i, FT_i) for this robot. Here, we introduce weight $$w_i=\frac{{\displaystyle \sum _{j=1}^{m_i}{l}_{ij}}}{m_i\underset{j=1}{\overset{m_i}{\max }}\left(l_{ij}\right)}$$ to assess how evenly the robots are controlled. $$w_i\times m_i$$ is the “equivalent” number of evenly controlled robots. With the weight, we can approximate F(IT_i, NT_i) as:

$$F\left(I{T}_i,N{T}_i\right)\approx w_i\times \left(m_i{\left(I{T}_i,N{T}_i\right)}^{*}\right)=\frac{{\displaystyle \sum _{j=1}^{m_i}{l}_{ij}}}{\underset{j=1}{\overset{m_i}{\max }}\left(l_{ij}\right)}\times \frac{T_i}{{\displaystyle \sum _{j=1}^{m_i}{l}_{ij}}}=\frac{T_i}{\underset{j=1}{\overset{m_i}{\max }}\left(l_{ij}\right)}$$

Thus, the denominator in CD_i can be calculated as:

$$N{T}_i^{*}{\displaystyle \sum _{j=1}^{m_i}{l}_{ij}}=\left(\frac{T_i}{\underset{j=1}{\overset{m_i}{\max }}\left(l_{ij}\right)}-I{T}_i^{*}\right){\displaystyle \sum _{j=1}^{m_i}{l}_{ij}}=\frac{{\displaystyle \sum _{j=1}^{m_i}{l}_{ij}}}{\underset{j=1}{\overset{m_i}{\max }}\left(l_{ij}\right)}{T}_i-I{T}_i^{*}{\displaystyle \sum _{j=1}^{m_i}{l}_{ij}}=\frac{{\displaystyle \sum _{j=1}^{m_i}{l}_{ij}}}{\underset{j=1}{\overset{m_i}{\max }}\left(l_{ij}\right)}{T}_i-{\displaystyle \sum _{type=i}IT}$$

where $${\displaystyle \sum _{type=i}IT}$$ is the total interaction time for all the type i robots.

In summary, we can compute CD_i as:

$$C{D}_i=\frac{{\displaystyle \sum _{type\ne i}IT}}{\frac{{\displaystyle \sum _{j=1}^{m_i}{l}_{ij}}}{\underset{j=1}{\overset{m_i}{\max }}\left(l_{ij}\right)}{T}_i-{\displaystyle \sum _{type=i}IT}}$$ E2

3. Simulation environment and multirobot system

To test the usefulness of the CD measurement, we conducted two experiments to manipulate and measure coordination demand directly. In the first experiment robots perform a box pushing task in which CD is varied by control mode and robot heterogeneity. The second experiment attempts to manipulate coordination demand by varying the proximity needed to perform a joint task in two conditions and by automating coordination within subteams in the third. Both experiments were conducted in the high fidelity USARSim robotic simulation environment we developed as a simulation of urban search and rescue (USAR) robots and environments intended as a research tool for the study of human-robot interaction (HRI) and multi-robot coordination.

3.1. USARSim

USARSim supports HRI by accurately rendering user interface elements (particularly camera video), accurately representing robot automation and behavior, and accurately representing the remote environment that links the operator’s awareness with the robot’s behaviors. It was built based on a multi-player game engine, UnrealEngine2, and so is well suited for simulating multiple robots. USARSim uses the Karma Physics engine to provide physics modeling, rigid-body dynamics with constraints and collision detection. It uses other game engine capabilities to simulate sensors including camera video, sonar, and laser range finder. More details about USARSim can be found at (Wang et al. 2003; Lewis et al. 2007). Validation studies showing agreement for a variety of feature extraction techniques between USARSim images and camera video are reported in (Carpin et al., 2006a), showing close agreement in detection of walls and associated Hough transforms for a simulated Hokuyo laser range finder (Carpin et al., 2005) and close agreement in behavior between USARSim models and the robots being modeled (Carpin et al., 2006b, Wang et al., 2005, Pepper et al., 2007, Taylor et al., 2007, Zaratti et al., 2006). USARSim is freely available and can be downloaded from www.sourceforge.net/projects/usarsim.

3.2. Multirobot Control System (MrCS)

A multirobot control system (MrCS), a multirobot communications and control infrastructure with accompanying user interface, was developed to conduct these experiments. The system was designed to be scalable to allow of control different numbers of robots, reconfigurable to accommodate different human-robot interfaces, and reusable to facilitate testing different control algorithms. It provides facilities for starting and controlling robots in the simulation, displaying camera and laser output, and supporting inter-robot communication through Machinetta, a distributed mutiagent system with state-of-the-art algorithms for plan instantiation, role allocation, information sharing, task deconfliction and adjustable autonomy (Scerri et al. 2004).

The user interface of MrCS is shown in Figure 8. The interface is reconfigurable to allow the user to resize the components or change the layout. Shown in the figure is a configuration that used in one of our experiments. On the upper and center portions of the left-hand side are the robot list and team map panels, which show the operator an overview of the team. The destination of each of robot is displayed on the map to help the user keep track of current plans. On the upper and center portions of the right-hand side are the camera view and mission control panels, which allow the operator to maintain situation awareness of an individual robot and to edit its exploration plan. On the mission panel, the map and all nearby robots and their destinations are represented to provide partial team awareness so that the operator can switch between contexts while moving control from one robot to another. The lower portion of the left-hand side is a teleoperation panel that allows the operator to teleoperate a robot.

4. Tight cooperation experiment

4.1. Experiment design

Finding a metric for cooperation demand (CD) is difficult because there is no widely accepted standard. In this experiment, we investigated CD by comparing performance across three conditions selected to differ substantially in their coordination demands. We selected box pushing, a typical cooperative task that requires the robots to coordinate, as our task. We define CD as the ratio between occupied time (OT), the period over which the operator is actively controlling a robot to synchronize with others, and FT+OT, the time during which he is not actively controlling the robot to perform the primary task. This measure varies between 0 for no demand to 1 for maximum demand. When an operator teleoperates the robots one by one to push the box forward, he must continuously interact with one of the robots because neglecting both would immediately stop the box. Because the task allows no free time (FT) we expect CD to be 1. However, when the user is able to issue waypoints to both robots, the operator may have FT before she must coordinate these robots again because the robots can be instructed to move simultaneously. In this case CD should be less than 1. Intermediate levels of CD should be found in comparing control of homogeneous robots with heterogeneous robots. Higher CD should be found in the heterogeneous group since the unbalanced pushes from the robots would require more frequent coordination. In the present experiment, we measured CDs under these three conditions.

Figure 3.

Box pushing task.

Figure 3 shows our experiment setting simulated in USARSim. The controlled robots were either two Pioneer P2AT robots or one Pioneer P2AT and one less capable three wheeled Pioneer P2DX robot. Each robot was equipped with a GPS, a laser scanner, and a RFID reader. On the box, we mounted two RFID tags to enable the robots to sense the box’s position and orientation. When a robot pushes the box, both the box and robot’s orientation and speed will change. Furthermore, because of irregularities in initial conditions and accuracy of the physical simulation the robot and box are unlikely to move precisely as the operator expected. In addition, delays in receiving sensor data and executing commands were modeled presenting participants with a problem very similar to coordinating physical robots.

Figure 4.

GUI for box pushing task.

We introduced a simple matching task as a secondary task to allow us to estimate the FT available to the operator. Participants were asked to perform this secondary task as possible when they were not occupied controlling a robot. Every operator action and periodic timestamped samples the box’s moving speed were recorded for computing CD.

A within subject design was used to control for individual differences in operators’ control skills and ability to use the interface. To avoid having abnormal control behavior, such as a robot bypassing the box bias the CD comparison, we added safeguards to the control system to stop the robot when it tilted the box.

The operator controlled the robots using a distributed multi-robot control system (MrCS) shown in Figure 4. On the left and right side are the teleoperation widgets that control the left and right robots separately. The bottom center is a map based control panel that allows the user to monitor the robots and issue waypoint commands on the map. On the bottom right corner is the secondary task window where the participants were asked to perform the matching task when possible.

4.2. Participants and procedure

14 paid participants, 18-57 years old were recruited from the University of Pittsburgh community. None had prior experience with robot control although most were frequent computer users. The participants’ demographic information and experience are summarized in Table 1.

	Age		Gender		Education
18~35	"/35	Male	Female	Currently/Complete Undergraduate		Currently /Complete Graduate
Participants	11	3	11	3	2		12
	Computer Usage (hours/week)				Game Playing (hours/week)
<1	1-5	5-10	"/10	<1	1-5	5-10	"/10
Participants	0	1	2	11	8	4	2	0
	Mouse Usage for Game Playing
Frequently			Occasionally			Never
Participants	9			4			1

Table 1.

Sample demographics and experiences.

The experiment started with collection of the participant’s demographic data and computer experience. The participant then read standard instructions on how to control robots using the MrCS. In the following 8 minutes training session, the participant practiced each control operation and tried to push the box forward under the guidance of the experimenter. Participants then performed three testing sessions in counterbalanced order. In two of the sessions, the participants controlled two P2AT robots using teleoperation alone or a mixture of teleoperation and waypoint control. In the third session, the participants were asked to control heterogeneous robots (one P2AT and one P2DX) using a mixture of teleoperation and waypoint control. The participants were allowed eight minutes to push the box to the destination in each session. At the conclusion of the experiment participants completed a questionnaire about their experience.

4.3. Results

Figure 5 shows a time distribution of robot control commands recorded in the experiment. As we expected no free time was recorded for robots in the teleoperation condition and the longest free times were found in controlling homogeneous robots with waypoints.

Figure 5.

The time distribution curves for teleoperation (upper) and waypoint control (middle) for homogeneous robots, and waypoint control (bottom) for heterogeneous robots.

The box speed shown on Figure 5 is the moving speed along the hallway that reflects the interaction effectiveness (IE) of the control mode. The IE curves in this picture show the delay effect and the frequent bumping that occurred in controlling heterogeneous robots revealing the poorest cooperation performance.

Figure 6.

Team task demand (TAD) and Cooperation demand (CD).

None of the 14 participants were able to perform the secondary task while teleoperating the robots. Hence, we uniformly find TAD = 1 and CD = 1 for both robots under this condition. Within participants comparison found that under waypoint control the team attention demand in heterogeneous robots is significantly higher than the demand in controlling homogeneous robots, t(13) = 2.213, p = 0.045 (Figure 6). No significant differences were found between the homogeneous P2AT robots in terms of the individual cooperation demand (P = 0.2). Since the robots are identical, we compared the average CD of the left and right robots with the CDs measured under heterogeneous condition. Two-tailed t-test shows that when a participant controlled a P2AT robot, lower CD was required in homogeneous condition than in the heterogeneous condition, t(13) = -2.365, p = 0.034. The CD required in controlling the P2DX under heterogeneous condition is marginally higher than the CD required in controlling homogenous P2ATs, t(13) = -1.868, p = 0.084 (Figure 6). Surprisingly, no significant difference was found in CDs between controlling P2AT and P2DX under heterogeneous condition (p=0.79). This can be explained by the three observed robot control strategies: 1) the participant always issued new waypoints to both robots when adjusting the box’s movement, therefore similar CDs were found between the robots; 2) the participant tried to give short paths to the faster robot (P2DX) to balance the different speeds of the two robots, thus we found higher CD in P2AT; 3) the participant gave the same length paths to both robots and the slower robot needed more interactions because it trended to lag behind the faster robot, so lower CD for the P2AT was found for the participant. Among the 14 participants, 5 of them (36%) showed higher CD for the P2DX contrary to our expectations.

5. Weak cooperation experiment

To test the usefulness of the CD measurement for a weakly cooperative MRS, we conducted another experiment assessing coordination demand using an Urban Search And Rescue (USAR) task requiring high human involvement (Murphy and Burke, 2005) and of a complexity suitable to exercise heterogeneous robot control. In the experiment participants were asked to control explorer robots equipped with a laser range finder but no camera and inspector robots with only cameras. Finding and marking a victim required using the inspector’s camera to find a victim to be marked on the map generated by the explorer. The capability of the robots and the cooperation autonomy level were used to adjust the coordination demand of the task. The experiment was conducted in simulation using USARSim and MrCS.

5.1. Experiment design

Three simulated Pioneer P2AT robots and 3 Zergs (Balakirsky et al., 2007), a small experimental robot were used. Each P2AT was equipped with a front laser scanner with 180 degree FOV and resolution of 1 degree. The Zerg was mounted with a pan-tilt camera with 45 degree FOV. The robots were capable of localization and able to communicate with other robots and control station. The P2AT served as an explorer to build the map while the Zerg could be used as an inspector to find victims using its camera. To accomplish the task the participant must coordinate these two types robot to ensure that when an inspector robot finds a victim, it is within a region mapped by an explorer robot so the position can be marked.

Figure 7.

Urban search and rescue task.

Three conditions were designed to vary the coordination demand on the operator. Under condition 1, the explorer had 20 meters detection range allowing inspector robots considerable latitude in their search. Under condition 2, scanner range was reduced to 5 meters requiring closer proximity to keep the inspector within mapped areas. Under condition 3, explorer and inspector robots were paired as subteams in which the explorer robot with a sensor range of 5 meters followed its inspector robot to map areas being searched. We hypothesized that CDs for explorer and inspector robots would be more even distributed under condition-2 (short range sensor) because explorers would need to move more frequently in response to inspectors’ searches than in condition-1 in which CD should be more asymmetric with explorers exerting greater demand on inspectors. We also hypothesized that lower CD would lead to higher team performance. Three equivalent damaged buildings were constructed from the same elements using different layouts. Each environment was a maze like building with obstacles, such as chairs, desks, cabinets, and bricks with 10 evenly distributed victims. A fourth environment was constructed for training. Figure 7 shows the simulated robots and environment.

A within subjects design with counterbalanced presentation was used to compare the cooperative performance across the three conditions. The same control interface shown in Figure 8 allowing participants to control robots through waypoints or teleoperation was used in all conditions.

Figure 8.

GUI for urban search and rescue.

5.2. Participants and procedure

19 paid participants, 19-33, years old were recruited from the University of Pittsburgh community. None had prior experience with robot control although most were frequent computer users. 6 of the participants (31.5%) reported playing computer games for more than one hour per week. The participants’ demographic information and experience are summarized in Table 2.

	Age		Gender		Education
19~29	30~33	Male	Female	Currently/Complete Undergraduate		Currently /Complete Graduate
Participants	18	1	7	12	11		8
	Computer Usage (hours/week)				Game Playing (hours/week)
<1	1-5	5-10	"/10	<1	1-5	5-10	"/10
Participants	0	1	5	13	13	4	1	1
	Mouse Usage for Game Playing
Frequently			Occasionally			Never
Participants	14			2			3

Table 2.

Sample demographics and experiences.

After collecting demographic data the participant read standard instructions on how to control robots via MrCS. In the following 15~20 minute training session, the participant practiced each control operation and tried to find at least one victim in the training arena under the guidance of the experimenter. Participants then began three testing sessions in counterbalanced order with each session lasting 15 minutes. At the conclusion of the experiment participants completed a questionnaire.

5.3. Results

Overall performance was measured by the number of victims found, the explored areas, and the participants’ self-assessments. To examine cooperative behavior in finer detail, CDs were computed from logged data for each type robot under the three conditions. We compared the measured CDs between condition 1 (20 meters sensing range) and condition 2 (5 meters sensing range), as well as condition 2 and condition 3 (subteam). To further analyze the cooperation behaviors, we evaluated the total attention demand in robot control and control action pattern as well. Finally, we introduce control episodes showing how CDs can be used to identify and diagnose abnormal control behaviors.

1. Overall performance

Figure 9.

Found victims (left) and explored areas (right) by mode.

Examination of data showed two participants failed to perform the task satisfactorily. One commented during debriefing that she thought she was supposed to mark inspector robots rather than victims. After removing these participants a paired t-test shows that in condition-1 (20 meters range scanner) participants explored more regions, t(16) = 3.097, p = 0.007, as well as found more victims, t(16) = 3.364, p = 0.004, than under condition-2 (short range scanner). In condition-3 (automated subteam) participants found marginally more victims, t(16) = 1.944, p = 0.07, than in condition-2 (controlled cooperation) but no difference was found for the extent of regions explored (Figure 9).

In the posttest survey, 12 of the 19 (63%) participants reported they were able to control the robots although they had problems in handling some interface components, 6 of the 19 (32%) participants thought they used the interface very well, and only one participant reported it being hard to handle all the components on the user interface but still maintained she was able to control the robots. Most participants (74%) thought it was easier to coordinate inspectors with explorers with long range scanner. 12 of the 19 (63%) participants rated auto-cooperation between inspector and explorer (the subteam condition) as improving their performance, and 5 (26%) participants though auto-cooperation made no difference. Only 2 (11%) participants judged team autonomy to make things worse.

2. Coordination effort

Figure 10.

Typical (IT,FT) distribution (higher line indicates the interactions of.

During the experiment we logged all the control operations with timestamps. From the log file, CDs were computed for each type robot according to Equation 2. Figure 10 shows a typical (IT,FT) distribution under condition 1 (20 meters sensing range) in the experiment with a calculated CD for the explorer of 0.185 and a CD for the inspector of 0.06. The low CDs reflect that in trying to control 6 robots the participant ignored some robots while attending to others. The CD for explorers is roughly twice the CD for inspectors. After the participant controlled an explorer, he needed to control an inspector multiple times or multiple inspectors since the explorer has a long detection range and large FOV. In contrast, after controlling an inspector, the participant needed less effort to coordinate explorers.

Figure 11.

CDs for each robot type.

Figure 11 shows the mean of measured CDs. We predicted that when the explorer has a longer detection range, operators would need to control the inspectors more frequently to cover the mapped area. Therefore a longer detection range should lead to higher CD for explorers. This was confirmed by a two tailed t-test that found higher coordination demand, t(18) = 2.476, p = 0.023, when participants controlled explorers with large (20 meters) sensing range.

We did not find a corresponding difference, t(18)=.149, p=0.884, between long and short detection range conditions for the CD for inspectors. This may have occurred because under these two conditions the inspectors have exactly the same capabilities and the difference in explorer detection range was not large enough to impact inspectors’ CD for explorers. Under the subteam condition, the automatic cooperation within a subteam decreased or eliminated the coordination requirement when a participant controlled an inspector. Within participant comparisons shows that the measured CD of inspectors under this condition is significantly lower than the CD under condition 2 (independent control with 5 meters detection range), t(18) = 6.957, p < 0.001. Because the explorer always tries to automatically follow an inspector, we do not report CD of explorers in this condition.

As auxiliary parameters, we evaluated the total attention demand, i.e. the occupation rate of total interaction time in the whole control period, and the action pattern, the ratio of control times between inspector and explorer, as well. Total attention demand measures the team task demand, i.e.; how hard the task is. As we expected paired t-test shows that under subteam condition, participants spent less time in robot control than under short sensing range condition, t(18)=3.423, p=0.003. However, under long sensing conditions, paired t-test shows that participants spent more time controlling robots than under the short sensing condition, t(18) = 2.059, p = 0.054. This is opposite to our hypothesis that searching for victims with shorter sensing range should be harder because the robot would need to be controlled more often. Noticing that total attention demand was based on the time spent controlling not the number of times a robot was controlled we examined the number of control episodes. Under long and short sensing range conditions two tailed t-tests found participants to control explorers more times with short sensing explorers, t(18)=2.464, p=.024, with no differences found in frequency of inspector control, p=.97. We believe that with longer sensing explorers participants tend to issue longer paths in order to build larger maps. Because the sensing range in condition 1 is five times longer than the range in condition 2, the increased control time under the long sensing condition my overwhelm the increased explorer control times. This is partially confirmed by a paired t-test that found longer average control time for explorers and inspectors under the long detection condition, t(18)=3.139, p=.006, t(18)=2.244, p=.038, respectively. On average participants spent 1.5s and 1.0s more time in explorer and inspector control in the long range condition. The mean action patterns under long and short range scanner conditions are 2.31 and 1.9 respectively. This means that with 20 and 5 meters scanning ranges, participants controlled inspectors 2.31 and 1.9 times respectively after an explorer interaction. Within participant comparisons shows that the ratio is significantly larger under long sensing condition than under short range scanner condition, t(18) = 2.193, p = 0.042.

3. Analyzing Performance

As an example of applying CDs to analyze coordination behavior, Figure 11 shows the performance over explorer CD and total attention demand under the 20 meters sensing range condition. Three abnormal cases A, B, and C can be identified from the graph. Associating these cases with recorded map snapshots (Table 3), we observed that in case A, one robot was entangled by a desk and stuck after five minutes; in case B, two robots were controlled in the first five minutes and afterwards ignored; and in case C, the participant ignored two inspectors throughout the entire trial. Comparing with case B and C, in case A only one robot didn’t function properly after five minutes.

Figure 12.

Found victims distribution over CDexp and TAD.

6. Conclusion

We proposed an extended Neglect Tolerance model to allow us to evaluate coordination demand in applications where an operator must coordinate multiple robots to perform dependent tasks. Results from the first experiment that required tight coordination conformed closely to our hypotheses with the teleoperation condition producing CD=1 as predicted and heterogeneous teams exerting greater demand than homogenous ones. The CD measure proved useful in identifying abnormal control behavior revealing inefficient control by one participant through irregular time distributions and close CDs for P2ATs under homogeneous and heterogeneous conditions (0.23 and 0.22), a mistake with extended recovery time (41 sec) in another, and a shift to a satisficing strategy between homogeneous and heterogeneous conditions revealed by a drop in CD (0.17 to 0.11) in a third.

As most target applications such as construction or search and rescue require weaker cooperation among heterogeneous platforms the second experiment extended NT methodology to such conditions. Results in this more complex domain were mixed. Our findings of increased CD for long sensor range may seem counter intuitive because inspectors would be expected to exert greater CD on explorers with short sensor range.

	5 minutes snapshot	10 minutes snapshot	15 minutes snapshot
A		the robot on the center of the map was stuck
B	the two robots on the upper map were never controlled since then
C	the two robots on the upper left corner were totally ignored

Table 3.

Map snapshots of abnormal control behaviors.

Our data show, however, that this effect is not substantial and provide an argument for focused metrics of this sort which measure constituents of the human-robot system directly. Moreover, this experiment also shows how CD can be used to guide us to identify and analyze aberrant control behaviors.

We anticipated a correlation between found victims and the measured CDs. However, we did not find the expected relationship in this experiment. From observation of participants during the experiment we believe that high level strategies, such as choosing areas to be searched and path planning, have significant impact on the overall performance. The participants had few problems in learning to jointly control explorers and inspectors but they needed time to figure out effective strategies for performing the task. Because CD measures control behaviours not strategies these effects were not captured.

On the other hand, because the NT methodology is domain and task independent our CD measurement could be used to characterize any dependent system. For use in performance analysis, however, it must be associated with additional domain and task dependent information. As shown in our examples, combined with generated maps and traces CD provides an excellent diagnostic tool for examining performance in detail.

In the present experiment, we examined the action pattern under long and short sensing range conditions. The results reveal that it can be used as an evaluation parameter, and more important, it may guide us in the design of multiple robot systems. For instance, the observation that one explorer control action was followed on average by 2 inspector control actions may imply that the MRS should be constructed by n explorer and 2n inspectors.

In the weak cooperation experiment, the time-based assessment showed higher coordination demand under a longer sensing condition. The control times evaluation reported more control times, which implies a higher coordination demand in the shorter sensing condition. This difference illustrates how the measurement unit, control time or control times, may impact the HRI evaluation. Usually, the time-consuming operations such as teleoperation are suited to time-based assessment. In contrast, control times may provide more accurate evaluation to the one-time style operations such as command issuing. Improving the Neglect Tolerance model to suit control times based evaluation should be an area for future work.

In summary, the proposed methodology enables us evaluate weak or tight cooperation behaviors in control of heterogeneous robot teams. The time parameter based measurement makes this methodology domain independent and practical in real applications. The lack of consideration of domain, other system characteristics and information available to the operator, however, makes this metric too impoverished to use in isolation for evaluating system performance. A more complete metric for evaluating coordination demand in multirobot systems would require additional dimensions beyond time. Considering human, robot, task and world as the four elements in HRI, possible metrics might include mental demand, situation awareness, robot capability, autonomy level, overall task performance, task complexity, and world complexity.