QoS Control in Remote Robot Operation with Force Feedback

Abstract

Recently, many researchers focus on studies of remote robot operation with force feedback. By using force feedback, since users can touch remote objects and feel the shape, weight, and softness of each object, the efficiency and accuracy of operation can be largely improved. However, when the haptic information such as force and/or position information is transmitted over a QoS (Quality of Service) non-guaranteed network like the Internet, QoE (Quality of Experience) and stability may seriously deteriorate. Therefore, it is important to carry out QoS control and stabilization control together to solve the problems. In this chapter, we mainly focus on QoS control. We also introduce our remote robot system with force feedback which we constructed to study QoS control and stabilization control by experiment. In the system, a user operates a remote industrial robot with a force sensor by using a local haptic interface device while monitoring the robot operation by a video camera. We handle two types of operation; operation with a single remote robot system and that between two remote robot systems. We explain several types of QoS control which we have proposed so far for remote robot operation with force feedback. Finally, we discuss the challenges and future directions of QoS control in remote robot operation with force feedback.

1. Introduction

Recently, many researchers focus on studies of remote robot operation with force feedback in which a user operates a remote robot having force sensors by using a haptic interface device while monitoring the remote operations by a video camera [1, 2, 3]. By using force feedback, since users can touch remote objects and feel the shape, weight, and softness of each object, the efficiency and accuracy of operation can be largely improved [4]. Therefore, the remote robot operation with force feedback is expected to be used in many areas such as remote surgery, disaster rescue, and outer space. However, when the information about force and/or position is transmitted over a QoS (Quality of Service) [5] non-guaranteed network like the Internet, QoE (Quality of Experience) [6] may seriously deteriorate [3, 4] owing to the network delay, delay jitter, and packet loss. Furthermore, as the network delay increases, the reaction force becomes larger, and unstable phenomena such vibrations of the robot and device may occur more often [7, 8, 9]. To solve the problems, we need to carry out QoS control and stabilization control together [4]. In this chapter, we mainly focus on QoS control at the application layer. The QoS control alleviates the influences of network delay, delay jitter, and packet loss on QoE.

We also introduce our remote robot system with force feedback which we constructed to study the QoS control and stabilization control by experiment. In the system, a user operates a remote industrial robot with a force sensor by using a local haptic interface device while monitoring the robot operation. We handle two types of operation; operation with a single remote robot system and that between two remote robot systems. We explain several types of QoS control which we have proposed so far for remote robot operation with force feedback.

In this chapter, first, we explain the remote robot system with force feedback in Section 2. Next, we introduce expected applications of the remote robot system with force feedback in Section 3. Then, we outline the problems to be solved for the applications in Section 4 and describe the QoS control which is used to solve the deterioration problems owing to the network delay, delay jitter, and packet loss in Section 5. Finally, we discuss the challenges and future directions of QoS control in Section 6 and conclude the chapter in Section 7.

2. Remote robot system with force feedback

2.1 System configuration

The configuration of the remote robot system with force feedback is shown in Figure 1. The system consists of two terminals called the master terminal and slave terminal. Each terminal consists of two PCs, and the PCs are connected to each other via a switching hub.

Figure 1.

Configuration of remote robot system with force feedback.

At the master terminal, a 3 DoF (Degree of Freedom) haptic interface device (3D Systems Touch [10]) is connected to PC for haptic interface device, and another PC is used for video. At the slave terminal, one of the two PCs is used for a web camera (produced by Microsoft Corp., and video resolution is 1920 × 1080 pixels), and the other PC is used for industrial robot. The industrial robot consists of a 6 DoF robot arm (RV-2F-D by Mitsubishi Electric Corp. [11]), a robot controller (CR750-Q [11]), and a force sensor (1F-FS001-W200 [12]). The force sensor is attached to the surface of the flange of the robot arm. The force sensor is connected to the robot controller via the force interface unit.

2.2 Remote operation

A user at the master terminal can operate the industrial robot at the slave terminal by using the haptic interface device while watching video (coding scheme: Motion JPEG, average bit rate: 4.5 Mbps). The default position of the haptic interface device is set to the origin, and the position corresponds to the default position of the industrial robot [13].

The master terminal updates the position information, calculates the reaction force, and outputs the reaction force every millisecond. The master terminal also transmits the position information to the slave terminal by User Datagram Protocol (UDP). At the slave terminal, the command information which is based on the position information received from the master terminal is sent to the industrial robot every 3.5 milliseconds by the real-time control function [14]. The force information is also acquired by the real-time control function, and the information is transmitted to the master terminal by UDP.

The reaction force ${\mathit{F}}_t^{\left(\mathrm{m}\right)}$ applied to the haptic interface device at time t (t$\ge$ 1) is calculated as follows:

$${\mathit{F}}_t^{\left(\mathrm{m}\right)}=K_{\mathrm{scale}}{\mathit{F}}_{t-1}^{\left(\mathrm{s}\right)}$$E1

where ${\mathit{F}}_{t-1}^{\left(\mathrm{s}\right)}$ denotes the force received from the slave terminal (note that we use only 3 DoF of force here), and $K_{\mathrm{scale}}$ is a force scale which is set to 1 in this paper. Furthermore, since the maximum force applied to the haptic interface device is 3.3 N [10], the reaction force is set to 3.3 N when the calculated force is larger than 3.3 N.

The position vector ${\mathit{S}}_t$ of the industrial robot outputted at the time t (t$\ge$ 2) is calculated as follows:

$${\mathit{S}}_t={\mathit{M}}_{t-1}+{\mathit{V}}_{t-1}$$E2

where ${\mathit{M}}_t$ is the position vector of haptic interface device received from the master terminal at time t,${\mathit{V}}_t\left(\mathit{=}{\mathit{M}}_t-{\mathit{M}}_{t-1}\right)$ is the velocity vector and $\left|{\mathit{V}}_t\right|\le V_{\text{max}}$, and $V_{\text{max}}$ is the maximum movement velocity. That is, in order to operate the robot arm safely, the maximum movement velocity is limited to $V_{\text{max}}$ ($V_{\text{max}}$ = 5 mm/s [13] in this chapter).

In this chapter, we handle two types of operation, operation with single remote robot system and that between two remote robot systems. In the latter operation, we deal with two types of work (carry together and hand delivery). In carry together, two industrial robots carry an object together. In hand delivery, an object was hand-delivered between the two industrial robots.

3. Expected applications

As shown in Figure 2, the remote robot system with force feedback is expected to be used in various areas.

Figure 2.

Expected applications.

3.1 Remote surgery/rehabilitation

In order to solve the problems of imbalance of medical resources, remote surgery/rehabilitation using the remote robot system with force feedback is an effective method. Also, the system can be used for remote surgery training for medical interns.

3.2 Work in dangerous areas

It is difficult for human to work in danger areas such as deep sea and outer space. Therefore, we can employ robots to work in the danger areas instead of humans. By using the remote robot system with force feedback, we can control the robot which works in danger areas from a remote safe area. We can improve the efficiency and accuracy of work by force feedback.

3.3 Disaster rescue and relief

The remote robot system with force feedback can also be used for rescue and relief from disasters such as earthquake and concentrated downpour. In this case, the remote robot can be rescue robot or drone, and the system can be used to help people, to distribute goods for disaster victims. Also, it can be used to confirm disaster situation.

In these applications, it is difficult for only robots to work because the situations are unknown in advance. Thus, human’s support is needed. This means that we need robots to help humans and robots also need human’s supports. Therefore, the cooperation among humans and robots is needed.

4. Problems to be solved

In this section, we explain the problems to be solved for widespread applications of the remote robot system with force feedback.

4.1 Problem of cooperation

There exist several types of cooperation using the remote robot system with force feedback, for example, cooperation between human and human, robot and human, robot and robot. In the cooperation between human and human, the will transmission between humans is important, and efficient transmission methods should be established for the cooperation. In the cooperation between human and robot, it is necessary to consider whether humans support robots or robots support humans, and how to support each other more efficiently. In the cooperation between robot and robot, we have two cases; in one case, the robots can cooperate by communications between the slave terminals; in the other case, they can do with force sensors because they are connected to each other through an object when they carry or hand-deliver the object. The cooperation is important when speedy control is needed.

4.2 Problem of will transition

In the cooperation between human and human, it is important to transmit users’ will (for example, movement directions and speeds) to each other, and will transmission using haptic may reduce the transmission time, and it is possible to transmit wills in delicate manipulation work in which it is difficult to transmit wills only by traditional methods (i.e., wills transmitted by audio and video) [15]. Therefore, it is necessary to establish an efficient method to transmit/determine wills by haptic for the cooperation.

4.3 Network delay, delay jitter, and packet loss

As described in Section 1, when the information about force and/or position transmitted over a QoS non-guaranteed network like the Internet, the reaction force may become large and QoE may be seriously deteriorate owing to the network delay, delay jitter, and packet loss. It is necessary to carry out QoS control to solve the problems.

4.4 Unstable phenomena

In the remote robot system with force feedback, the system may be unstable since there exists a control loop between a haptic interface device and a remote robot (see Figure 3). As the network delay increases, the movement of the remote robot is largely later than that of the haptic interface device, the reaction force becomes larger, and unstable phenomena such vibrations of the robot and device may occur more often. Also, in remote cooperation, there exists more loops in the cooperation systems and the unstable phenomena become more complex and difficult problems [16]. It is important to carry out stabilization control to solve the problems.

Figure 3.

Control loop between haptic interface device and robot.

4.5 Cooperation in case of emergency

In the remote cooperation, there exist emergency cases in which network interruption occurs and users may be not able to control remote robots to do collaborative work. In this case, remote robots need to intercommunicate with each other and finish cooperation based on force sensors. That is, we need to handle the case of emergency and establish effective methods in the case.

From the above, there are many problems to be solved. Here, we mainly focus on QoS control which alleviates the influences of network delay, delay jitter, and packet loss.

5. QoS control

QoS control is effective for solving the problems occurred by network delay, delay jitter, and packet loss. As described in the previous section, in the remote robot operation, there exists a control loop between a haptic interface device and a robot. We need to carry out QoS control in the loop to improve the QoE. This means that we need to carry out QoS control at a haptic interface device terminal and/or at a robot terminal. There are many types of QoS control such as traffic management and control, error control, spatiotemporal synchronization control (we can carry out media synchronization control or causality control to achieve spatiotemporal synchronization), consistency control, adaptive reaction force control [4], and position control using force information [17]. We also introduce serval types of QoS control which we previously proposed for the remote robot operation.

5.1 Media synchronization control

Media synchronization control is used to solve the problems occurred by network delays and delay jitter. The control can be grouped into intra-stream synchronization control, inter-stream synchronization control, and inter-destination (or group) synchronization control [18].

Intra-stream synchronization control is used to preserve the timing relation between media units (MUs, which are information units for media synchronization) [19] in a single media stream. There are several types of intra-stream synchronization control, for example, Skipping [19], Virtual-Time Rendering (VTR) [19], and so on. Skipping outputs MUs on receiving the MUs, and when the sequence number of a received MU is smaller than that of the last-output MU, the control discards the received MU. VTR has a virtual-time axis which can be contracted or expanded dynamically according to the network delay, and MUs are output along the virtual-time axis.

In multimedia applications, if we only carry out intra-stream synchronization control for each media stream separately, the temporal relationship among media streams may be disturbed and QoE may be deteriorated. In order to solve the problem, we need to carry out inter-stream synchronization control. The VTR can be used for intra-stream and inter-stream synchronization control. Under the control, one media stream is handled as the master stream and the others are dealt with as slave streams. VTR carries out only the intra-stream synchronization control for the master stream, and it exerts the inter-stream synchronization control after carrying out the inter-stream synchronization control for the slave streams.

In remote cooperation, in order to improve the efficiency of cooperative work, it is important to output MUs simultaneously at different terminals. Group (or inter-destination) synchronization control outputs each MU simultaneously at different terminals. We proposed three schemes for inter-destination synchronization control (i.e., the master–slave destination scheme, synchronization maestro scheme, and distributed control scheme) [4].

5.2 Causality control

Causality control keeps the causal (i.e., temporal order) relationships among events. Here we introduce two typical examples of causality control; one is the Δ-causality control [20], and the other is the adaptive Δ-causality control [21].

In the Δ-causality control, each MU has a time limit which is equal to the generation time of the MU plus Δ seconds for preservation of the real-time property. The control output the MU at the time limit, and if the MU is received after the time limit, it is discarded because it is considered useless. The adaptive Δ-causality control dynamically changes the value of Δ according to the network load. The control does not discard an MU received after the time limit and uses the MU for prediction.

5.3 Adaptive reaction force control

As the network delay increases, the reaction force applied to a haptic interface device becomes larger and the output quality of haptic media becomes deteriorated. The adaptive reaction force control [4] can be used to solve the problem. We calculate the reaction force based on the spring-damper model [22] or depending on the force sensed by the force sensor. In the spring-damper model, the reaction force consists of the elasticity and viscosity. The elasticity is force exerted by deformation of a spring or rubber, for example. When a spring is pushed or pulled. The elasticity is proportional to the depth of a spring when the spring is pushed, and it is calculated by multiplying the depth by the elastic coefficient. The viscosity is force or resistance exerted by fluids, for example, when we move an object through the fluids (e.g., water and oil). The viscosity is proportional to the relative velocity (i.e., the velocity of the object relative to the fluids), and it can be calculated by multiplying the relative velocity by the viscosity coefficient. The adaptive reaction force control includes the adaptive viscosity control [23], adaptive elastic control [24], and adaptive viscoelasticity control [25]. The adaptive elastic control dynamically changes the elastic coefficient according to network delay, the adaptive viscosity control dynamic changes the viscosity coefficient according to the network delay and the velocity of the haptic interface device, and the adaptive viscoelasticity control combines the two types of control.

5.4 Position control using force information

In order to reduce the force applied to an object operated in cooperative work between the remote robot systems with force feedback, the robot position control with using force information is proposed [17]. The proposed control moves the robot by taking advantage of human perception of force direction by experiment. The control finely adjusts the robot position dynamically in the direction where the force is reduced.

Since the remote robot system with force feedback is delay sensitive [26], we apply Skipping to the system at both master and salve terminals. This means that Skipping is applied to the operation with a single remote robot system and that between the two remote robot systems. For the operation between the two remote robot systems, we apply the adaptive reaction force control, adaptive Δ-causality control, and position control using force information for the remote cooperation between two remote robot systems with force feedback. That is, we apply the control for the cooperation between users (i.e., each user operates a haptic interface device to control a remote robot to do collaborative work), and for the cooperation between the user and robot. We also applied the position control using force information for the cooperation between the two robots. We also investigate the effects of the control by experiment.

6. Challenges and future directions of QoS control

As described in the previous section, although we proposed serval types of QoS control for the remote robot system with force feedback, there still exist many challenges. In this section, we discuss the challenges and future directions of QoS control.

6.1 Integration of QoS control and stabilization control

In order to achieve stable and high quality of service, we also need to carry out stabilization control, and it is important to integrate the QoS control with stabilization control. To integrate the QoS control with stabilization control, we can carry out QoS control and stabilization control independently, or integrate QoS control into stabilization control for the system. We investigate the effects by integrating the position control using force information as QoS control with stabilization control with filters [9], and experimental results show that the effect when we carry out QoS control in the loop of stabilization control is better than that when we carry out QoS control and stabilization control independently. It is important to investigate the effect by using other types of QoS control and stabilization control.

6.2 Multilateral control

In this chapter, we introduced the QoS control only in a communication loop, which is between a haptic interface device and a remote robot (see Figure 3). However, in the remote operation using multiple systems, there are multiple loops caused by communication in the systems (see Figure 4), and there exist interrelationships among the loops. This means that we need to carry out multilateral control for QoS as well as stabilizations and it becomes complex and difficult.

Figure 4.

Control loops in remote robot operations.

6.3 Application of big data, cloud computing and AI technologies

In order to improve the efficiency of QoS control, we need to take account of many factors, for example, contents of work, movement speed, room temperature and wind [27]. Therefore, big data [28], cloud computing [29], and AI (Artificial Intelligence) [30] technologies such as neural network, fuzzy theory, and genetic algorithm can be useful methods for efficient control. The necessary information for QoS control can be transmitted to a cloud server, and the information can be combined as big data for analysis and applied as training data and evaluation data. Efficient QoS control can be expected by using AI after studying the training data. Also, in order to solve the problem of AI computing, we can apply AI chips [31], which realizes edge AI computing, to the remote robot terminal to improve the efficiency of QoS control.

6.4 Others

Since we need to transmit the necessary information for QoS control to a cloud server, it is important to consider the safety and security of data. Also, in many situations, we need to use movable robots as remote robots. This means that we may need to consider the QoS control in wireless and/or mobile networks. This is because a 5G network [32] which is wideband and low latency becomes available and the possibility of the application over the mobile network increases.

In addition, we need to carry out QoE assessment to investigate the effects of QoS control and to clarify how to set parameter values optimally under each type of the control as well as QoS assessment at lower layers. QoE subjective assessment is the most important because the assessment can reflect end users’ opinions directly [4], [33, 34, 35].

7. Conclusions

In this chapter, we focus on QoS control for remote robot operation. We introduce our remote robot system with force feedback which we constructed to study QoS control. We also present the expected applications and the problems to be solved for widespread application of remote robot system with force feedback. We mainly focused on the problems of network delay, delay jitter, and packet loss. We explain several types of QoS control which we previously proposed to solve the problems. Finally, we also discuss the challenges and future directions of QoS control.

For the future plan of our study, we need to solve the problems described in Section 6.

Acknowledgments

The authors thank Prof. Hitoshi Watanabe of Tokyo University of Science, Prof. Hitoshi Ohnishi of the Open University of Japan, Prof. Takanori Miyoshi of Nagaoka University of Technology, and Prof. Takashi Okuda of Aichi Prefectural University for their valuable discussions.

Conflict of interest

The authors declare no conflicts of interest associated with this chapter.