## 1. Introduction

Teleoperation, or remote tele-manipulation of robots in inaccessible environments such as deep sea and outer space [1, 2], includes a wide variety of applications such as micro- and nano-teleoperation [3], tele-surgery [4], etc. A network-based teleoperation system involves distant interactions between human operators and remote robotic systems [5, 6]. In addition, swarms of robots are widely employed in complex tasks that cannot be performed by a single robot or in tasks that are better achieved by cooperation of robots such as localization in formations [7], target tracking [8], mapping and localization [9], object pushing [10], area exploration for search and rescue [11], etc. When swarms of mobile robots are teleoperated, specific network requirements should be satisfied in order to guarantee a minimum quality of control, which results in efficient task execution. Research done on teleoperated systems showed that constraints such as bandwidth and CPU processing cause the Quality of Service (QoS) to degrade to an extent that may severely affect performance [12, 13]. To address this problem, various bandwidth management algorithms have been presented for distributed multimedia systems in order to maintain a performance that guarantees an adequate QoS [6]. However, the literature rarely tackled the problem of managing bandwidth based on sensory feedback and the quality of collaboration among robots. Accordingly, a real-time dynamic optimized bandwidth management for teleoperation of collaborative robots is introduced in this paper. The proposed method accounts for interesting events (IEs) and the change in the quality of collaboration (QoC) between robots in order to optimize the allocation of bandwidth between acting agents, where necessary, in a given environment. The developed optimization technique showed outstanding performances when implemented on a system of two collaborating humanoid robots, and thus could be considered a basis for a framework for highly complex algorithms implemented in systems involving real-time bandwidth optimization, where multiple users control multiple collaborating robots.

Different types of resource management algorithms are used to solve the bandwidth allocation problem in robotics systems. Such applications in networked control systems fall into two main categories: static [14] and dynamic [15] bandwidth allocation. Static methods cannot adapt to changes in the system state (surrounding environment, collaboration quality, etc.). Alternatively, dynamic bandwidth allocation algorithms increase performance at the cost of increased computation. Mourikis et al. [7] address the problem of resource allocation in formations of mobile robots localizing as a group. The goal is to determine the frequency at which each individual sensor should be used in order to attain the highest possible localization accuracy. The set of frequencies mentioned is obtained by solving an optimization problem that maximizes the accuracy matrix expressed in terms of the sensors' frequencies. However, the problem is solved offline and the algorithm does not account for any dynamic events that might occur. Sugiyama et al. [8] propose a bandwidth reservation algorithm for multi-robot systems in a target tracking mission. The interesting information, corresponding to a survivor’s detection, is sent to the base station with wideband signals such as dynamic picture images. The final call is left to operators to decide whether the received images indicate a real victim, by allowing/preventing the corresponding robot to reserve the bandwidth affecting the flow of various signals from other robots to the base station. In this approach, the operator's intervention is crucial in allocating bandwidth and thus the allocation process is not fully automated. Xi et al. [5] developed a bandwidth allocation mechanism based on online measured task dexterity index of dynamic tasks so that operators can control remote manipulators efficiently and smoothly even under poor network quality. However, the executed task is simple and does not require the collaboration of multiple robots to be performed. Thus, the quality of collaboration factor is not considered in the bandwidth allocation. Finally, in [10], a bandwidth management algorithm is introduced and the rate of feedback is regulated based on the amount of activities occurring in the environment. The work shows that during complex tasks, the operator's performance is affected by the rate of feedback of information. It is also confirmed that a higher sampling rate is required to maintain the same level of performance obtained when the environment is less dynamic. Yet, the implemented algorithm does not impose any constraint on the total bandwidth of the system. In addition, the notion of monitoring changes in QoC to allocate bandwidth is not mentioned since the task execution only requires the use of a single robot. To the best of the author's knowledge, there was limited research addressing bandwidth management for the specific application of collaborative robots teleoperation. In 2015, Ricardo and Guilherme designed a Dynamic Bandwidth Management Library to control the frequency of individual sensors present in a robotic environment performing a certain task [16]. This work is seeking a universal Dynamic Bandwidth Management Library designed to be used on a system with a variable number of heterogeneous robots performing any collaborative task that requires communication transactions such as the exchange of sensor data between involved agents.

Accordingly, the main contribution of the work presented in this paper is in accounting for (a) IEs occurring in the robotic swarm's environment and (b) changes in QoC among the swarm of robots in real-time optimized bandwidth management of teleoperated collaborative robots. Consequently, assessing the multi-robot swarm dynamics, the stability of the robotic swarm and the effects of packet loss and transmission delay under the proposed algorithm falls out of the scope of this paper. Factoring the latter into the proposed algorithm is possible but will alter the emphasis from the main contribution of dynamic optimized bandwidth management.

A literature review of the most relevant work in bandwidth management was presented in Section 1. The general problem is formulated in Section 2. The formulation is then implemented on an application in which an operator drives two collaborating robots. Section 3 describes the experimental set-up and the corresponding results. Finally, conclusions are presented in Section 4.

## 2. Problem formulation

The focus of the work presented is on real-time dynamic managing of the ‘User’ to ‘Robot’ and ‘Robot’ to ‘User’ (U2R/R2U) and ‘Robot’ to ‘Robot’ (R2R) communication channels, where actuation commands, system state and sensory data including video frames are exchanged. A general formulation of the problem that can be applied to swarms with a variable number of collaborating robots is presented first and is then implemented on a formation of two humanoid robots performing a collaborative task. The formulation is concluded by presenting the solution of the optimization problem.

### 2.1. The general formulation

The goal of the optimized dynamic bandwidth allocation algorithm is to optimize communication at each time event based on information related to the occurrence of *IEs* in the robot's surroundings and to changes in *QoC* among robots. The idea comes from the fact that when sudden changes occur in the robot's environment, the teleoperator needs to be updated more frequently in order to retain the same level of performance. In other words, the rates of information exchanged between operator(s) and robots are updated based on changes in task conditions that may affect the performance severely. Similarly, the increase in *R2R* communication compensates for any decay in the performance of the collaborative task of the robotic swarm. Hence, if the environment is less dynamic and the robots are collaborating well, the corresponding communication rates will be decreased. On the other hand, if the environment is more dynamic and the robots are collaborating poorly, the corresponding communication rates will be increased.

Let the collaborative task be executed by *n* robots and let *x*_{i}, where

Where the elements of X are classified into 3 sets of rates:

Feedback Rates: R2U communication rates

Collaboration Rates: R2R communication rates

Command Rates: U2R communication rates

The rates *x*_{i} are subject to practical constraints that bound each of them with a minimum and maximum value *x*_{imin} and *x*_{imax}, respectively. In addition, the sum of bandwidth consumed by all channels is bounded by the total bandwidth of the system *B*_{max}. Hence, for all the rates, we have the following constraints:

*w*

_{i}'s are weights associated with each rate corresponding to the rate of information (bps) sent at each time event on each channel. We define the vector

*P*as an m-dimensional observation vector that is composed of two main sets of components such that each element in

*P*is an observation related to interesting events occurring in the robot environment or to the quality of the collaborative task executed by the robotic swarm. The elements of

*IE,*{

*p*

_{1},

*p*

_{2},…,

*p*}, and of

_{t}*QoC,*{

*p*

_{t + 1},

*p*

_{t + 2}, …,

*p*}, are variables that dictate the choice of the communication rates. All the elements of

_{m}*P*are normalized in the interval [0: 1]. If the

*i*observation is reflecting a slightly changing environment or a high collaborative performance, then

^{th}*p*

_{i}would be close to 0. On the contrary, if the

*i*observation reflects a highly changing environment or significant degradation in the collaborative performance, then

^{th}*p*

_{i}would be closer to 1.

Distance to obstacles, speed and displacement of robot are potential examples of *IEs* that could be monitored in order to allocate bandwidth. Moreover, any observation that tracks error in a collaborative task could also contribute to the bandwidth management algorithm.

In the formulation presented in this work, a mapping equivalent to the one presented in our previous work [17] is applied, however, with additional constraints that transform the problem from a simple matrix multiplication to a linear optimization problem. The new constraints bound the set of feasible communication rates *x*_{i} depending on the choice of the minimum and maximum rate for each channel *x*_{imin} and *x*_{imax} and the maximum bandwidth of the system *B*_{max}.

In this mathematical formulation, *s*_{i} is defined in the interval [0: 1] for all

Then, an r-dimensional vector *S* is defined as follows:

Hence, at each time instant, the algorithm solves for the *s*_{i}'s and then uses the mapping in (3) to get the rates *x*_{i}'s.

*S*is related to

*P*using the mapping matrix

*M*as shown in (7): where

In (7), the elements of the matrix *M* are selected based on the relation between the observations and the rates. Each row of the matrix *M* can be interpreted as the weights of the observations in *P* affecting the corresponding rate in *X*. Since *s*_{i}'s are selected in the range [0: 1], then choosing the sum of the coefficients in each row of *M* to be equal to 1 ensures that the result of multiplying any row of *M* by the vector *P* represents a weighted average of the observations that results in a value in the [0: 1] range. Thus, for a specific environment, *M* can be initialized once at the beginning of the set of trials. However, in order to improve the performance, *M* could also be updated dynamically based on the quality of the task previously executed. Thus, an improvement in the overall performance is achieved while maintaining an equivalent level of bandwidth consumption.

Since the sum of bandwidth consumed on all channels is bounded by the maximum bandwidth of the system, *B*_{max}, the allocation of the rates on different channels will be formulated as a linear optimization problem. Thus, the problem formulation becomes:

Minimize:

subject to andSince *x*_{i} = *a*_{i}*s*_{i} + *b*_{i}, then Eq. (9) can be written as follows:

which is equivalent to

The constraint in (12) can be expressed in terms of matrix multiplication as follows:

Therefore, for any collaborative task, it is sufficient to set the parameters in Eq. (13) in order to define the linear optimization problem. Thus, optimization techniques can be applied to solve the defined problem.

### 2.2. Mathematical formulation

In order to solve the aforementioned problem, a simple change of variable is first performed. We let:

Therefore, the problem formulation becomes minimize:

subject to andSince the problem is an L1 norm problem, it needs to be slightly modified in order to get rid of the absolute value that complicates the solution of the problem. Thus, the problem can be translated to minimize:

subject to and and where T is an r-dimensional vector containing all the t_{i}'s, which are dummy variables that are introduced to avoid the use of absolute value in the formulation and replace it by a simple minimization of a summation. Hence, at each time instant, the constraint matrix is formed, and then the optimization problem is solved. Since the problem is linear, it is solved efficiently using interior point method. In the experiments performed in this work, the optimization problem was solved in 10–15 iterations for an average time period nearly equal to 100 ms. Thus, by scheduling the rates' update at every 5 s, each communication rate is computed by averaging the values calculated in the last 50 iterations. On the other hand, it is worth noting here that the internal model matrix

*M*grows quickly with the increase in the number of robots used, which would affect the complexity of the problem and the speed of convergence. Therefore, in the following section, the developed optimization technique was tested on a system of two collaborating humanoid robots. The proposed method could be considered a basis for a framework for developing highly complex algorithms for systems involving real-time bandwidth optimization, where multiple users control multiple collaborating robots in various scenarios.

### 2.3. Dynamic optimized bandwidth algorithm experimental verification

In order to illustrate the use of the formulation, we apply it for the case, where an operator drives two collaborating robots. The mission consists of navigating a delimited path to reach a predetermined destination while avoiding obstacles and preserving a formation. The formation is characterized by a distance of 60 cm that separates the two robots, while keeping alignment nearly zero as in **Figure 1**.

The robots' feedback includes visual and haptic data reflecting the environmental conditions. Ultrasonic sensors mounted on each robot allow the detection of obstacles in the navigation path. Each returns an integer value, indicating the distance to the nearest detected obstacle, which is fed back to the operator in the form of haptic feedback. In addition, the camera mounted on top of each robot provides visual feedback of the area in front of the formation. In order to allocate bandwidth based on changes in task conditions, *IE*s such as the distance to obstacles with respect to each robot and the speed of the swarm are monitored. Also, to maintain a high performance at high speeds, the speed of the swarm is one of the dynamic events that are monitored. Moreover, *QoC* factors are measures of how well the robots are collaborating together to efficiently accomplish the predetermined task. Specifically, the errors in position (Δ*x* and Δ*y*) between the robots indicating the deviation of the robots from the required formation are the quantities reflecting the change in the quality of the collaborative task, which need to be monitored. During the task execution, the visual and haptic feedback rates, the rates of commands and the rates of *R2R* communication are allocated based on real-time observations related to occurrence of *IEs* in the environment and to changes in *QoC* between robots. Hence, the elements of *X* are defined as follows:

*x*_{1}: Rate of visual feedback from R1*x*_{2}: Rate of haptic feedback from R1*x*_{3}: Rate of visual feedback from R2*x*_{4}: Rate of haptic feedback from R2*x*_{5}: Rate of R2R collaboration information exchange*x*_{6}: Rate of commands generation

The observations in *p*_{i}'s are also defined below:

*p*_{1}: Distance to obstacle from R1*p*_{2}: Distance to obstacle from R2*p*_{3}: Speed of formation (Combined from R1 and R2)*p*_{4}: Positioning error in the horizontal direction Δ*x**p*_{5}: Positioning error in the vertical direction Δ*y*.

Each element of *P* is normalized with respect to the maximum value that the sensors could measure (in case of *p*_{1} and *p*_{2}), to the maximum value that the system can reach (in case of *p*_{3}) or to the maximum allowed error beyond which the task built upon the formation would start being affected (in case of *p*_{4} and *p*_{5}). It is worth mentioning that *p*_{3} is introduced in order to detect any sudden changes in the system dynamics that could be captured by monitoring the speed of the formation, which would also require significant changes in video feedback.

Knowing that the maximum distance detected by the robots' sensors is 2.5 m, *p*_{1} and *p*_{2} are defined as follows:

Given that *v*_{1} and *v*_{2} are the average forward/backward and sideway speed of the formation, respectively, and knowing the maximum forward and sideway speed that the robots could reach is 6 and 4 cm/s, respectively, *p*_{3} is defined as follows:

Finally, *p*_{4} and *p*_{5} are defined with respect to positioning error in the horizontal (Δ*x*) and vertical (Δ*y*) directions, respectively, as follows:

Additionally, the maximum allowed positioning error in the horizontal and vertical directions

Furthermore, the experiment is performed for a maximum bandwidth *B*_{max} equal to 1.4 Mbps. This value is deemed to be convenient for such small swarm with very few sensory data and video frame to exchange. Moreover, the imaging resolution of the robot's cameras is 160 × 120 × 3, which implies that each frame is formed of 3 matrices and thus, *w*_{1} = *w*_{3} = 160 × 120 × 3 × 8 *bits* = 460,800 *bits*. As for the four remaining rates, the size of the data packets sent to/by each agent is considered to be 1500 *Bytes*. Hence, *w*_{2} = *w*_{4} = 1500 × 8 = 12000 *bits*. But since the collaboration information exchanged requires both robots to send one packet, and since the operator generates a packet for each robot as command, then *w*_{5} = *w*_{6} = 2 × 12000 *bits* = 24000 *bits*. Consequently, *W* is represented as follows:

In addition, the matrices *A* and *B* are set based on the allowed minimum and maximum of these rates. During experimentation, a frequency of 1 Hz is allocated as minimum for all rates. As for the maximum rates of cameras, it is equal to the total bandwidth from which the minimum consumption of all other sensors is removed. For this application, *A* and *B* are calculated to be as follows:

Finally, a fixed matrix *M* is adopted in the implementation as described earlier. Each row of *M* contains the weights of the observations that affect the corresponding rate. The weights for all set of observations corresponding to each rate are allocated in a way to have the sum of each row remain equal to 1. It is worth mentioning that for the speed of the formation, weights of 25% are allocated in the rows of *M* corresponding to all the *R2U* and *U2R* rates (*x*_{1} to *x*_{4}), since it was found to be the least dynamic observation. However, the rate of commands was deemed to be mostly dependent on the speed of the formation *p*_{3}. Thus, *M* would be as follows:

In the experiments, the aforementioned values of *A, B, M, W* and *B*_{max} are adopted while solving the optimization problem and allocating rates.

## 3. Experimental set-up and results

In this section, the suggested algorithm is evaluated against two static allocation algorithms: ‘Equal Bandwidth’ and ‘Equal Rates’. The equal bandwidth method allocates bandwidth equally among the different streams, while the equal rates method divides the available bandwidth such that the different streams would have equal rates of transmission.

The experimental set-up is composed of two NAO humanoid robots driven in a certain formation. An operator drives the two robots using a force feedback joystick (Microsoft SideWinder Force Feedback 2) and communicates with the robot through a router connected to a PC as shown in **Figure 2**. The operator sends real-time commands to the agents that return back haptic feedback as proximity measures and visual feedback from cameras mounted on each robot. The signal flows within the different components of the system as illustrated in **Figure 3**. The collaborative task that is to be accomplished by the robots is to maintain a certain formation while traversing an environment and avoiding collision with potential obstacles. The formation is characterized by a fixed distance (D = 60 cm) separating the two robots, while maintaining the error in the vertical direction nearly zero. Accurate position of the humanoids is calculated using the aid of the Inertial Unit built in the robots, which is made of 2-axis gyrometers with 5% precision (*angular speed* ∼500°/s) and a 3-axis accelerometer with 1% precision (*acceleration* ∼2G).

### 3.1. Testing scenarios

#### 3.1.1. Scenario 1: path with no obstacles

The first scenario consists of driving the swarm in the space delimited by dashed lines, shown in **Figure 4**, and reaching the final destination (dashed rectangular region on the right side) with no obstacles in the path. Obviously, with the absence of obstacles, the shortest path in this case is moving from ‘Start’ to ‘End’ in a straight line. The operators tend to adopt the shortest path, the straight line in this case, to reach the final destination.

#### 3.1.2. Scenario 2: obstacle in front of robot 1

In the second scenario, an obstacle is detected in front of Robot 1 (dashed square) when driving the formation in the delimited area as shown in **Figure 5**. The operator should steer to the left in order to avoid the collision with the obstacle. The swarm is forced to steer toward the left since by steering in the opposite direction Robot 2 would then exit the delimited path.

#### 3.1.3. Scenario 3: obstacle in front of robot 2

The third scenario features an obstacle in front of Robot 2 (dashed square) when driving the formation in the delimited area. In this case, the operator steers to the left in order to avoid the collision of Robot 2 with the obstacle as shown in **Figure 6**.

### 3.2. Testing and results

In order to evaluate the suggested algorithms, for each scenario, teleoperators drove the swarm under the Equal Bandwidth, Equal Rates and Optimized Bandwidth method. In each trial, the following performance parameters are collected: the completion time in *seconds*, the average speed of each robot in cm/s, the average deviation of each robot from the shortest path (Esp1 and Esp2) in , the average error in the formation in the horizontal and vertical directions (*Δx* and *Δy*) in *cm*, the maximum errors in both directions as well as the average bandwidth in Mbps consumed. The first static algorithm divides the available bandwidth equally among the 6 communication channels, whereas the second applied algorithm allocates equal rates to all communication channels. The computed rates for each channel for the static algorithms are reported in **Table 1**. It is worth noting that since the image frame size is much greater than the other data exchanged, allocating that equal bandwidth to all communication channels reduces the cameras' frame rate to around 1 Hz, while allocating equal rates to all communication channels increases frame rates to 1.5 fps; however, it drops the rates of all other data exchanged by a factor of five.

x_{1} | x_{2} | x_{3} | x_{4} | x_{5} | x_{6} | Total bandwidth (Mbps) | |
---|---|---|---|---|---|---|---|

Equal bandwidth | 1 | 8 | 1 | 8 | 8 | 8 | 1.43 |

Equal rates | 1.5 | 1.5 | 1.5 | 1.5 | 1.5 | 1.5 | 1.42 |

The path in front of the formation can be visualized by the cameras located on the forehead of each robot. Robots R_{1} and R_{2} navigate inside a delimited path while avoiding obstacles to reach the final destination. Moreover, a force feedback that corresponds to the distance to obstacles in front of the formation is calculated based on values measured by the ultrasonic sensors mounted on each robot. In order to evaluate the suggested algorithm, four teleoperators drove the formation under the three allocation methods in the three defined scenarios for a total number of runs equal to 36. Under each scenario, the bandwidth methods were *randomly* selected for each driver. Additionally, two training runs were performed by each user in order to get familiar with the task performed and experiment the haptic and visual feedback before executing the official runs. Results for the three mentioned scenarios are recorded in **Tables 2**–**4**. Runs, which included visible slippage by the robots, were repeated in order not to bias the results.

Referring to **Table 2**, at an average bandwidth consumption less than that of both static algorithms, dynamic optimized bandwidth allocation method results in a better performance in the first scenario. With a reduction in bandwidth consumption of around 70 Kbps, the operator performs better when using the proposed dynamic algorithm than when applying the static ones. With the dynamic bandwidth algorithm, the average trial duration is 1.9 s less than the best static allocation method. Moreover, the driving performance improved significantly, since the parameters measuring the average error with respect to the shortest path improved in addition to the average speed of both robots. The instantaneous error to the shortest path has decreased by around 0.15 cm for both robots, while the average speed of both robots is around 0.35 cm/s higher. As for the parameters reflecting the quality of the executed collaborative task, we remark that the dynamic algorithm performs better than both static methods. The average errors in the horizontal and vertical directions are smaller as well as the maximum error is in both directions. For instance, the average horizontal error decreased by 10% for around 0.05 cm, while the average vertical error decreased by 40% (0.19 cm). It is worth noting that for most parameters the proposed method has a lower standard deviation indicating a more consistent performance.

In Scenario 2, the advantage of dynamic bandwidth allocation is also demonstrated by the collected results in **Table 3**. With a reduction in bandwidth consumption of around 70 Kbps, the operators perform better when using the proposed dynamic algorithm than when applying the static ones. With the dynamic bandwidth algorithm, the average trial duration is 3.7 s less than the best static allocation. Moreover, the driving performance improves significantly, since the parameters measuring the average error with respect to the shortest path improve in addition to the average speed of both robots. The instantaneous error to the shortest path decreased by around 0.4 cm, while the average speed of both robots is around 0.35 cm/s higher. Additionally, the maximum errors in the horizontal and vertical directions decreased by around 0.35 cm. It is worth noting here that the runs performed with ‘Equal Rates’ method lead to a better average horizontal and vertical error; however, with this method, high peaks of errors are reached in both directions that almost reach the tolerated bounds of 2.5 and 5 cm.

The rates of visual feedback of robots R_{1} and R_{2} during Scenario 3 for the different adopted bandwidth algorithms are presented in **Figure 7**. Additionally, the rates of haptic feedback of robots R_{1} and R_{2} and the collaboration and commands rate during Scenario 3 for the different bandwidth algorithms are presented in **Figures 8** and **9**, respectively.

Furthermore, we examine the percentage of runs in which the suggested algorithm outperforms the two static algorithms for each performance parameter in all scenarios. In other words, we count the number of times a user performed better according to a parameter when adopting the dynamic algorithm versus when driving with each of the static algorithms. Percentage of best performance for task duration, average speed of each robot (Speed R1, Speed R2), error of each robot with respect to the shortest path (Esp1 and Esp2) and maximum alignment and separation errors in the formation are recorded in **Table 5**. From the collected results, it can be seen that operators perform better with the dynamic allocation algorithm than the static algorithms at a minimum of 67% of the runs (Esp1 and Esp2 with Equal Bandwidth and Max horizontal/vertical error with Equal Rates). The suggested algorithm reaches a success rate of 92% for the speed R1 with respect to ‘Equal Rates’ static allocation.

Finally, the advantages of the proposed dynamic bandwidth optimization and management scheme over legacy bandwidth management schemes are clearly expressed in the results in terms of performance improvement and conserving network resources. Since the proposed algorithm is scalable and not limited to a single task, the improvement in performance is greatly realized in critical situations, where the collaborative task requires high levels of accuracy especially in cases involving human safety.

## 4. Conclusions

In this work, dynamic optimized bandwidth management in teleoperated collaborative robotics is tackled. The focus was on managing all communication channels, where actuation commands, system state and sensory data are exchanged. This was achieved by monitoring the interesting events occurring in the robots' environment and the changes in quality of collaboration among them. Effective completion of the collaborative task with lower bandwidth consumption and better performance was accomplished proving that the proposed method could be the basis of a framework for developing more complex algorithms applied to highly complex systems.