Open Access is an initiative that aims to make scientific research freely available to all. To date our community has made over 100 million downloads. It’s based on principles of collaboration, unobstructed discovery, and, most importantly, scientific progression. As PhD students, we found it difficult to access the research we needed, so we decided to create a new Open Access publisher that levels the playing field for scientists across the world. How? By making research easy to access, and puts the academic needs of the researchers before the business interests of publishers.
We are a community of more than 103,000 authors and editors from 3,291 institutions spanning 160 countries, including Nobel Prize winners and some of the world’s most-cited researchers. Publishing on IntechOpen allows authors to earn citations and find new collaborators, meaning more people see your work not only from your own field of study, but from other related fields too.
To purchase hard copies of this book, please contact the representative in India:
CBS Publishers & Distributors Pvt. Ltd.
www.cbspd.com
|
customercare@cbspd.com
By using remote robot systems with force feedback, we can largely improve the efficiency and accuracy of work among the systems over a network. In such a system, a human can operate a remote robot by manipulating a haptic interface device while monitoring the movement of the robot arm and perceiving force applied to an object touched/moved by an arm of the robot having a force sensor. The remote robot systems with force feedback are expected to be used in many areas such as outer space, deep sea, nuclear power plants, and disaster areas, which humans cannot enter easily. In these situations, three types of cooperation among humans and robots (that is, between humans and robots, between humans, and between robots) are highly demanded. In this chapter, we introduce our remote robot systems with force feedback and describe the three types of cooperation among humans and robots in the systems. We also explain QoS (Quality of Service) control and stabilization control as our challenges and solutions for effective cooperation. Furthermore, we discuss future directions of the cooperation.
*Address all correspondence to: huangpg@gifu.shotoku.ac.jp
1. Introduction
In recent years, many researchers pay attention to remote robot systems with force feedback [1, 2, 3, 4, 5, 6]. In such a system, a human can operate a remote robot by manipulating a haptic interface device while monitoring movement of the robot arm with a video camera and perceiving force from an object touched/moved by an arm of the robot having a force sensor. Since the user can touch and feel the shape, weight, and softness of the object, the efficiency and accuracy of remote operation can be largely improved [7]. Therefore, the remote robot systems with force feedback are able to be used in various areas such as outer space, deep sea, nuclear power plants, and disaster areas, in which humans cannot enter easily. In these applications, it is important to make cooperation among humans and robots with force feedback efficiency. We focus on the cooperation among humans and robots with force feedback in this chapter.
When the control information (i.e., force and/or position) is transmitted over a network that does not guarantee QoS (Quality of Service) [8], owing to network problems such as network delay, delay jitter, and packet loss, the QoE (Quality of Experience) [9] may be seriously deteriorated. Also, unstable phenomena such as vibrations may occur [10, 11, 12], and cooperation may not be well. To solve the problems, we need to carry out QoS control and stabilization control. QoS control alleviates the influences of network delay, delay jitter, and packet loss, and stabilization control suppresses instability phenomena such as vibrations of the robot and haptic interface device. We mainly focus on the problems and their solutions for effective cooperation among humans and robots with force feedback.
In this chapter, first, we explain the cooperation among humans and robots with force feedback in Section 2. Next, we introduce the remote robot systems with force feedback in Section 3. Then, we outline the problems to be solved in the cooperation among humans and robots with force feedback in Section 4 and explain our solutions for effective cooperation in Section 5. Finally, we discuss the future directions of the cooperation among humans and robots in Section 6 and we conclude the chapter in Section 7.
2. Cooperation among humans and robots with force feedback
In a remote robot system with force feedback, a human can operate a haptic interface device to control/support a remote robot while watching the video received from the remote terminal. From the haptic interface device, position information which is used to control/support remote robot is transmitted to the robot, and the position information of robot arm and force information sensing by force sensor attached to the robot is transmitted to the haptic interface device for outputting reaction force (see Figure 1, in which a toggle clamp hand is attached to the force sensor. We can attach various types of hands).
Figure 1.
Remote robot system with force feedback.
The remote robot systems with force feedback can be used for cooperation among humans and robots. Figure 2 shows examples of cooperation between the two remote robots. In Figure 2(a), the two remote robots collaboratively carry and move an object (wooden stick) from one position (initial position) to another position (destination). In Figure 2(b), one robot (robot arm 1) moves an object (wooden stick) toward another robot (robot arm 2), and hands over the object to robot arm 2. In Figure 2(c), the two remote robots carry an object (cardboard box) by putting the object from both sides and moving the object from one position to another position. Cooperation can be widely used in many areas, and several types of cooperation exist. As shown in Figure 3, the cooperation among humans and robots can be grouped into cooperation between human-human, human−robot, and robot−robot. In this section, we introduce expected applications using the remote robot systems with force feedback and explain the types of cooperation.
Figure 2.
Cooperation images between two robots.
Figure 3.
Cooperation among humans and robots.
2.1 Expected applications
The cooperation among humans and robots in the remote robot systems with force feedback is expected to be used in various areas such as dangerous areas in which humans cannot enter easily (for example, outer space, deep sea, and radioactively contaminated areas), disaster areas (disaster rescue and relief), and remote medicine.
2.1.1 Applications in dangerous areas
It is dangerous for humans to work in dangerous areas such as outer space, deep sea, and radioactively contaminated areas. We can use robots to work in such areas instead of humans entering the areas while the humans control the robots in safe areas.
2.1.2 Disaster areas (rescue and relief)
After the outbreak of disasters such as earthquakes and concentrated downpours, rescue and relief are the most important. However, it may be difficult for humans to enter the hard-hit areas. We can control remote robots to confirm disaster situations for making effective rescue plans. Also, humans in safe areas can control remote robots to enter the areas for rescue and relief.
2.1.3 Remote medicine
There exist large health disparities in different regional/national areas. Using remote robot systems with force feedback may be an effective method to solve the problem. The systems can be used for remote surgery and remote rehabilitation. Also, they can be employed for remote surgery training for medical interns.
2.1.4 Other areas
The remote robot systems with force feedback are also expected to be used for various applications such as industrial factories and home delivery in other areas. For industrial factories, humans can control remote robots to enter extreme environments (for example, high temperature and/or high-pressure environments) or limited working areas which are difficult for humans for production work or inspection and maintenance. For homedelivery, humans can remote control drones or movable robots to deliver packages.
In these applications, it is difficult to conduct work with only robots if the situations and environments are unknown in advance and may be always changing. Therefore, human’s support is needed [6]. This means that we need robots to help humans, and robots also need human’s support. That is, in the applications, cooperation between a human (local user) and a robot (remote robot) in each system, that between humans, that between robots (remote robots), and that among humans and robots (remote robots) are needed. We explain the three types of cooperation in the next subsection.
2.2 Types of cooperation
As described in the previous subsection, robots can help humans, and robots also need humans’ support. We explain the cooperation among humans and robots in this subsection.
Cooperation between human and robot
In this type of cooperation, a human operates a haptic interface device to control/support a remote robot. Instead of the human, a robot does the work at a remote side and sends back the information about reaction force sensed by force sensor, and the information about a remote environment by other sensors such as video cameras and microphones. The information can help the human to know the remote situation. Also, it is difficult for the robot to do work in a complex environment independently. Therefore, human’s support is needed for the robot. The human can control or adjust the movement of the remote robot to help the robot conduct the work.
Cooperation between robots
In dangerous areas or disaster areas, wireless communication can be an effective method for communication between humans and robots. However, wireless communication is largely affected by weather, obstacles, and distances; this means that large network delays or network troubles may occur. These problems may make humans difficult to control or support remote robots to deal with abrupt changes in robots’ positions. In this case, since humans require time to support the remote robots, cooperation between robots using force sensors is needed for quick response.
Cooperation between humans
In this cooperation, multiple humans operate haptic interface devices to control remote robots to do work. In this case, to do the work smoothly and effectively, cooperation between humans is important. In cooperation, humans need to start to operate remote robots at the same time at the beginning of the cooperation work, keeping almost the same movement speed in the cooperation. Also, when multiple humans operate the remote robots to move an object, efficient cooperation may be needed for position’s fine adjustment to the goal position. Therefore, in cooperation, humans need to transmit their wills (e.g., the movement direction and movement speed) to each other for efficient cooperation.
Cooperation among humans and robots
For work that cannot be conducted by only one remote robot, multiple robots may be needed. The robots should be operated by multiple humans. In this case, we need the above three types of cooperation.
In this section, we introduce the remote robot systems with force feedback constructed in our study.
3.1 System configuration
The configuration of the remote robot systems with force feedback is shown in Figure 4. Each system consists of a master terminal and a slave terminal. Each terminal has two PCs; that is, PC for haptic interface device and PC for video at the master terminal, and PC for industrial robot and PC for video (web camera) at the slave terminal. At the master terminal, a haptic interface device (3D Systems Touch [13]) having 3 DoF (Degree of Freedom) is connected to PC for haptic interface device. At the slave terminal, an industrial robot is connected to PC for industrial robot, and a web camera (produced by Microsoft Corp., and the video resolution is 1920 × 1080 pixels) is connected to PC for video. The industrial robot has a 6 DoF robot arm (RV-2F-D [14] by Mitsubishi Electric Crop.), a robot controller (CR750-Q [14]), and a force sensor (1F-FS001-W200 [15]) attached to the tip of the robot arm. The force sensor is connected to the robot controller through a force interface unit (2F-TZ561 [15]). Also, a toggle clam hand/a electric hand is further linked to the force sensor.
Figure 4.
Configuration of remote robot systems with force feedback.
In each system, the master terminal inputs the position information of the haptic interface device and calculates and outputs the reaction force for the haptic interface device every millisecond. The master terminal also transmits the position information to the slave terminal by UDP (User Datagram Protocol). At the slave terminal, based the position information received from the master terminal, a position vector is calculated and sent to the industrial robot every 3.5 milliseconds by the real-time control function [16]. The force information sensed by the force sensor is transmitted to the master terminal, and the master terminal calculates the reaction force outputted by the haptic interface device based on the received force information every millisecond as described previously.
At the master terminal, the reaction force Ftm outputted through to the haptic interface device at time t (t≥ 1) is calculated as follows:
Ftm=KscaleFFt−1sE1
where KscaleF is a force scale which is set to 1 in this chapter, and Ft−1s denotes the force received from the slave terminal (note that we use only the three axes (the x-, y-, and z- aexs) of force here). Furthermore, since the maximum force outputted through the haptic interface device is 3.3 N [13], the reaction force is set to 3.3 N when the calculated force is larger than 3.3 N.
At the slave terminal, the position vector St of the industrial robot outputted at the time t (t≥ 2) is calculated as follows:
St=KscalePMt−1+Vt−1E2
where KscaleP is a coefficient for mapping of workspace, Mt is the position vector of the haptic interface device received from the master terminal at time t, Vt=St−St−1 is the velocity vector of the industrial robot, and Vt≤Vmax, where Vmax is the maximum movement velocity (Vmax = 5 mm/s [16] in this chapter) to operate the robot arm safely.
3.2 Cooperation methods
The remote robot systems with force feedback are used for serval types of cooperation, and we here introduce the cooperation methods.
3.2.1 Human-robot
As described in Subsection 2.2, the cooperation between human and robot, a human operates a haptic interface device to control/support a remote robot. In this type of cooperation, a single remote robot system is used, and the human can feel the reaction force sensed by the force sensor attached to the remote robot arm. By feeling the reaction force and watching video received from remote side, the human can support/control the remote robot by sending position information which is used to move the remote robot.
We conduct the cooperation between human-robot in our previous work [12, 17]. In Ref. [12], the authors handle work of pushing a ball and identifying what kind of ball is pushed according to the softness without watching video. In Ref. [17], a human can manipulate a robot arm with a pen to write characters by using a haptic interface device, and the authors make a comparison among different types of stabilization control which we will explain in Section 5.
3.2.2 Robot-robot
As described previously, the remote robot systems can be used for the area such as disaster areas and deep see, in which communication failure, and unforeseen events and abrupt change of position may occur frequently. In this case, robots cannot receive the support/control information from humans. The cooperation between robot and robot can be used to solve the problem by working independently without humans’ support. Also, the cooperation can be used under the ordinary environment. In order to conduct the cooperative work independently, the robots can cooperate with each other according to the force information sensed by force sensors and/or by transmitting position information transmitted between the robots.
In Ref. [18], the authors deal with cooperation work in which an object (a wooden stick) is carried together by using the two remote robot systems with force feedback. In the cooperation, according to the force sensed by force sensor, one robot adjusts the position in the direction of reducing the force applied to the object to cooperate with another robot. We also suppose that mobile robots suddenly move up and down largely and employed the enhanced control of the robot position control using force information to against sudden and large position change in the vertical direction [19]. We further investigate the effects of the enhanced control by employing one axis among the six axes for the robot movement, and we regard the industrial robots as mobile robots (i.e., pseudo-mobile robots). In Ref. [18], the two remote robot systems are used to do collaborative work of carrying an object (a wooden stick) together. In the cooperation, each slave terminal (robot side) also sends the position information of robot arm to that of the other system (robot side). In Ref. [20], two humans operate the two remote robot systems to do cooperative work of carrying an object. In the cooperation, the position information is transmitted between robots to make the cooperation smooth.
3.2.3 Human-human
When multiple humans control remote robot systems with force feedback to do work, the cooperation between humans is important. In this type of cooperation, communication between humans is necessary. Humans can transmit their wills (e.g., the movement direction and movement speed) to each other by using sound, video, and/or force information (i.e., position information which can be used to calculate the reaction force outputted through a haptic interface device) and make the cooperation smooth.
In Ref. [21], one of two humans operate a remote robot to carry an object together with the other human by using a haptic interface device. In the cooperation, the position information of the haptic interface device is transmitted to calculate the reaction force outputted through haptic interface device. In Ref. [22], one human operates a haptic interface device to control a remote robot to carry and move an object (a wooden stick), and hand over the object to another remote robot which is controlled by another human.
3.2.4 Humans-robots
For work which cannot conduct successfully by only one remote robot system, multiple remote systems with force feedback and multiple humans may be needed. When multiple remote robot systems with force feedback are employed to do collaborative work, cooperation among humans and robots is needed. That is, communications between human and robot, between robot and robot, and between human and human are required.
In order to achieve effective cooperation, many problems should be solved. In this section, we explain the problems by grouping them into QoS problems and stabilization problems.
4.1 QoS problems
As described in Section 1, when the control information (i.e., force and/or position) is transmitted over a network which does not guarantee QoS like the Internet, owing to the network problems such as the network delay, delay jitter, and packet loss, the following problems (e.g., QoE deterioration and poor cooperation) may occur.
Fluctuation of reaction force
As the network delay increases, the reaction force outputted through the haptic interface device may become larger. That is, it is more difficult for a human to operate the device.
Out of temporal synchronization (known as media synchronization [23])
Owing to the network delay and delay jitter, media synchronization (i.e., the temporal relationship of media units (MUs), which are information units for media synchronization [24]) problems may occur. The problems of temporal synchronization can be grouped into problems of intra-stream synchronization, inter-stream synchronization, and inter-destination (or group) synchronization [23].
Out of spatial synchronization
In the cooperation, robots need to move/operate an object at the same altitude, same angle, and same speed. However, humans and robots are at different locations, out of spatial synchronization (i.e., robots move/operate the object in different altitude, different angle, and different speed) may occur, and cooperation may not be well. This is because if spatial asynchrony occurs, the force applied to the operating object becomes large, and the object may be damaged by the large force. That is, we need to keep temporal and spatial synchronization high.
To solve the problems, we need to carry out QoS control, and we will introduce the QoS control in Subsection 5.1.
4.2 Stabilization problems
In the remote robot system with force feedback, as the network delay increases, the reaction force becomes larger, and instability phenomena such as vibrations of the robot and haptic interface device may occur more frequently [12]. Furthermore, when multiple remote robot systems are used for cooperation, there are multiple loops caused by communication and force sensors in the systems (see Figure 5), and there exist close relationships among loops. Therefore, instability phenomena may become more often and more complex.
Figure 5.
Control loops in remote robot systems with force feedback.
In this section, we introduce solutions which can be used to solve the problems described in Section 4 for effective cooperation.
5.1 QoS control
In order to solve the QoS problems, we need to carry out QoS control. Various types of QoS control have been proposed [5]. We explain serval types of QoS control which can be used to solve the QoS problems for the cooperation among remote robot systems.
Temporal synchronization control
Temporal synchronization control can be used to solve the problems caused by the network delay and delay jitter [23] and following types of control are included. In the control, we suppose that the global synchronization clocks [25], which have the same value and advancement, are used at all the terminals for simplicity. We can adjust the clocks by Network Time Protocol (NTP) [26] or Global Positioning System (GPS) easily.
Intra-stream synchronization control
The control is used to keep the timing relation between MUs such as video pictures and voice packets in a single media stream. Serval types of intra-stream synchronization control such as Skipping [24], Virtual-Time Rendering (VTR) [24] have been proposed.
In the remote robot systems with force feedback, the control can be used to keep the timing relation of each media stream such as haptic/force media and video for each system.
Inter-stream synchronization control
The inter-stream synchronization control is used to preserve the temporal relationship among multiple media streams. VTR can also be used for inter-stream synchronization control. Under the control, one media stream is selected as the master stream, and the others are handled as slave streams. For the master stream, we carry out only the intra-stream synchronization control, and we exert the inter-stream synchronization control which adjusts the output timing of each slave stream to that of the master stream after carrying out the intra-stream synchronization control.
In the remote robot systems, haptic/force streams and video may be transmitted between the master and slave terminals in the cooperation between human-robot/humans-robots; the haptic stream and voice may be transmitted among the master terminals (i.e., humans’ sides) in the cooperation of humans-humans. In these cases, multiple streams are transmitted among the terminals, and we need to carry out the inter-stream synchronization control. Normally, the most important stream such as the haptic/force stream can be selected as the master stream under the control.
Inter-destination synchronization control
The inter-destination synchronization control can be employed to output each MU simultaneously at different terminals (destinations). When multiple remote robot systems with force feedback are used to do collaborative work, if each MU cannot be outputted simultaneously at different terminals, the cooperation may not be well because humans may watch different displayed video images of cooperation work and feel different reaction force. Therefore, we need to carry out the inter-destination synchronization control to improve the efficiency of cooperation work. Also, the control can be used for spatial synchronization.
Adaptive reaction force control (adaptive elasticity control)
In each remote robot system with force feedback, the reaction force of the haptic interface device is normally calculated from the force sensed by force sensor (see Eq. (1)). When the master terminal receives position information from the slave terminal, the spring-damper model [27] can be used to calculate the reaction force. In the spring-damper model, the reaction force includes the elasticity and viscosity. The elasticity is force exerted by deformation of a spring or rubber. For example, when a spring is pushed or pulled, the elasticity is proportional to the depth of the spring, and it is calculated by multiplying the depth by the elastic spring coefficient. The viscosity is force or resistance exerted by fluids. For example, when we move an object in the fluids (e.g., water and oil). The viscosity is proportional to the relative velocity (i.e., the velocity of the object relative to the fluids), and it can be calculated by multiplying the relative velocity by the viscosity (damper) coefficient. The reaction force is calculated as follows:
Ftm=−KsPt−1m−Pt−1s−Kdvt−1m−vt−1sE3
where Ks is the spring coefficient, Kd is the damper coefficient, Pt−1m is the position vector of the haptic interface device at the master terminal, Pt−1s is that at the slave terminal, vt−1m is the velocity of the haptic interface device at the master terminal, and vt−1s is that at the slave terminal. By using this model, a human can still control remote robot because the human can perceive the reaction force even if other parts of the robot with force sensors hit an object and do not move further. Note that the sensors do not sensed force in this case.
The adaptive reaction force control [7] includes the adaptive viscosity control [28], adaptive elastic control [29], and adaptive viscoelasticity control [30]. The adaptive elastic control dynamically changes the elastic coefficient according to network delay. The adaptive viscosity control dynamic changes the viscosity coefficient according to the network delay and the velocity of the haptic interface device. The adaptive viscoelasticity control exerts the two types of control.
Robot position control using force information
The robot position control using force information [31] is proposed to reduce the force applied to an object carrying by two remote robot systems in cooperative work of carrying the object. The control finely adjusts the position of the robot arm in the direction which leads to small reaction force.
Adaptive Δ-causality control
The adaptive Δ-causality control dynamically changes the value of Δ which is used for outputting each MU simultaneously (i.e., the MU is output at the time limit of generation time plus Δ seconds) according to the network load (i.e., the network delay and delay jitter) [7]. If the MU is received after the time limit, it is discarded as an obsolete MU. In the remote robot system with force feedback, the control can be used to output of the position information simultaneously among multiple terminals.
In [20], the authors propose the global Δ-causality control and compare the proposed control with the local adaptive Δ-causality control by experiment. The local adaptive Δ-causality control is partially applied to the remote robot systems (i.e., the adaptive Δ-causality control is carried out between the two robots, or the control is carried out in each remote robot system). To output each MU simultaneously at each system, we need to carry out the global adaptive Δ-causality control in which the control is globally performed (i.e., the adaptive Δ-causality control is carried out between the two robots and in each remote robot system). Experimental results show that the global control is more effective than the local control for cooperation.
The control can also be used for spatial synchronization.
Force adjustment control
The force adjustment control is proposed for cooperative work (see Figure 2(c)) of carrying an object by putting the object from both sides between two remote robot systems [32], and the control is used to suppress large force applied to the object and to avoid dropping it because of too small force. When the force applied to the object carried by the two robot arms is larger than a threshold value, each robot arm moves in the direction to reduce the force by a certain distance. When the force is smaller than another threshold value, the robot arms adjust in the direction to increase the force.
Based on the characteristics of the above QoS control, we can summarize the relationship between QoS control and the three types of cooperation in Table 1.
Human-robot
Robot-robot
Human-human
Temporal synchronization control
◯
◯
◯
Adaptive reaction force control
◯
—
◯
Robot position control using force information
◯
◯
—
Adaptive Δ-causality control
◯
◯
—
Force adjustment control
◯
◯
—
Table 1.
Close relationship between QoS control and cooperation.
For the temporal synchronization control, the intra-stream synchronization control is needed for each terminal in the cooperation of human-robot, robot-robot, and human-human; the inter-stream synchronization control is required for the cooperation of human-robot in which multiple streams are transmitted between the master and slave terminals; the inter-destination synchronization control is necessary for the cooperation of robot-robot and human-human, in which the same types of operation are needed at the terminals.
The adaptive reaction force control can be used to reduce the large reaction force caused by network delay in the cooperation of human-robot (i.e., position difference which is used to calculate the reaction force becomes large as network delay increases) and human-human.
The robot position control using force information can be used to reduce the force applied to an object carrying by two robots in the cooperation of human-robot and robot-robot.
The adaptive Δ-causality control can be employed to alleviate the influence of difference in network delay between the haptic interface device and robot in the cooperation of human-robot and can be used to reduce the influence of difference in network delay between the robots in the cooperation of robot-robot.
The force adjustment control can be used to adjust the force applied to the operating object in suitable range of values, and the control can be used for the cooperation of human-robot and robot-robot.
For the cooperation of humans-robots, since the cooperation including the three types of cooperation, all the above types of control can be utilized.
5.2 Stabilization control
In order to achieve stable and high quality of the cooperation using the remote robot systems with force feedback, we need to carry out stabilization control as well as QoS control. This subsection introduces serval types of stabilization control which are employed in our remote robot systems.
Stabilization control with filters
The stabilization control with filters uses the wave filter in combination with the phase control filter [12]. The stabilization control with filters makes the remote robot system with force feedback stable against large network delays.
Stabilization control by viscosity
In the stabilization control by viscosity [33], the instability phenomenon is suppressed by using viscosity. The control uses the coefficient to restrict the movement distance of the industrial robot to some extent compared with the movement distance of the haptic interface device.
Reaction force control upon hitting
In order to solve the vibration problem when the remote robot hits/touches a hard object, the reaction force control upon hitting [34] is proposed. The reaction force control upon hitting gradually increases the reaction force to prevent a robot arm from jumping when the arm hits a hard object (i.e., when the reaction force calculated based on the force information received from the force sensor attached to the remote robot arm is larger than a threshold value).
Stabilization control by viscosity
Since the viscosity can be used to suppress vibration, the adaptive viscosity control can also be used as stabilization control. In the control, viscosity is generated by decreasing the movement distance of the industrial robot by a certain amount proportional to the movement velocity [33].
In the previous section, we have introduced serval types of QoS control and stabilization control to solve the problems in the cooperation among humans and robots. However, there still exist many challenges. In this section, we discuss the future directions of QoS control and stabilization control for further stable and efficient cooperation.
Integrated QoS control
In the cooperation using multiple remote robot systems with force feedback, we need to carry out multiple types of QoS control for the systems. If we carry out the QoS control independently, QoE may be deteriorated owing to excessive or insufficient effects of the control [7]. Therefore, we need to carry out different types of QoS control in an integrated manner like the adaptive QoS control [35]; for example, we need to carry out error control [5] together with traffic control [5] to improve the efficiency. Note that the error control increases the network traffic volume. In such a way, if we perform the QoS control independently, excessive or insufficient effects of the control may occur.
Global QoS control
In the cooperation among multiple remote robot systems, it is important to carry out QoS control for each system and/or each terminal. However, it is also necessary to carry out QoS control for the multiple systems globally (i.e., we need to take account the whole systems when we carry out the QoS control). This is because multiple systems are used to conduct one work together. In Ref. [20], as described in Subsection 5.1 (4), we also found that the global adaptive Δ-causality control is more effective than the local adaptive Δ-causality control in the cooperation between the two remote robot systems.
Synthesis of QoS control and stabilization control
Since we need to carry out QoS control and stabilization control for remote cooperation using multiple remote robot systems, it is important to integrate the QoS control with stabilization control to achieve stable and high quality of systems. We can integrate QoS control into stabilization control for the system. For example, we can carry out QoS control in the loop of stabilization control [6].
Application of AI
Many factors such as contents of work, movement speed, network delay, room temperature, and wind may affect the effects of QoS control and stabilization control [36]. In order to achieve efficient cooperation (stable and high QoS), big data [37], cloud computing [38], and AI (Artificial Intelligence) [39] technologies such as neural network, fuzzy theory, and genetic algorithm can be useful methods for taking account of the factors.
In this chapter, we focused on the cooperation among humans and robots in remote robot systems with force feedback. We explained the expected applications and the four types of cooperation by using the remote robot systems. We also introduced our remote robot systems with force feedback and how to cooperate among the systems. We further explained QoS (Quality of Service) control and stabilization control as our challenges and solutions for effective cooperation. Furthermore, we discussed future directions of the cooperation.
The authors thank Prof. Takanori Miyoshi of Nagaoka University of Technology, Prof. Takashi Okuda of Aichi Prefectural University, Prof. Hitoshi Ohnishi of the Open University of Japan, and Prof. Hitoshi Watanabe of Tokyo University of Science for their valuable discussions.
The authors declare no conflicts of interest associated with this chapter.
References
1.Ohnishi K. Real world haptics: Its principle and future prospects. The Journal of the Institute of Electrical Engineers of Japan. 2013;133:268-269 (In Japanese)
2.Music S, Salvietti G, Dohmann P, Chinello F, Prattichizzo D, Hirche S. Human–robot team interaction through wearable haptics for cooperative manipulation. IEEE Transactions on Haptics. 2019;12(3):350-362
3.Singh J, Srinivasan A, Neumann G, Kucukyilmaz A. Haptic-guided teleoperation of a 7-DoF collaborative robot arm with an identical twin master. IEEE Transactions on Haptics. 2020;13(1):246-252
4.Haruna M, Kawaguchi N, Ogino M, Akino T. Comparison of three feedback modalities for haptics sensation in remote machine manipulation. IEEE Robotics and Automation Letters. 2021;6(3):5040-5047
5.Ishibashi Y, Huang P. Improvement of QoS in haptic communication and its future. The IEICE Transactions on Communications. 2016;J99-B:911-925 (In Japanese)
6.Huang P, Ishibashi Y. QoS control in remote robot operation with force feedback. In: Robotics Software Design and Engineering. London, UK: IntechOpen; 2020. pp. 1-13
7.Huang P, Ishibashi Y. QoS control and QoE assessment in multi-sensory communications with haptics. The IEICE Transactions on Communications. 2013;E96-B:392-403. DOI: 10.1587/transcom.E96.B.392
8.ITU-T Rec. I. 350 General Aspects of Quality of Service and Network Performance in Digital Networks. 1993
9.ITU-T Rec. G. 100/P. 10 Amendment 1, New appendix I - Definition of quality of experience (QoE). 2007
10.Miyoshi T, Maeda Y, Morita Y, Ishibashi Y, Terashima K. Development of haptic network game based on multi-lateral tele-control theory and influence of network delay on QoE. Transactions of the Virtual Reality Society of Japan, Special Issues on Haptic Contents. 2014;19:559-569 (In Japanese)
11.Qian Q, Toyoda Y, Ishibashi Y, Huang P, Tateiwa Y. Effect of stabilization control on cooperative work between remote robot systems with force feedback. International Journal of Mechanical Engineering and Robotics Research (IJMERR). 2020;9(3):415-420
12.Huang P, Miyoshi T, Ishibashi Y. Enhancement of stabilization control in remote robot system with force feedback. International Journal of Communications, Network and System Sciences (IJCNS). 2019;12(7):99-111
14.http://www.mitsubishielectric.co.jp/fa/products/rbt/robot/lineup/manual/f/bfp-a8899k.pdf (In Japanese)
15.http://dl.mitsubishielectric.co.jp/dl/fa/members/document/manual/robot/bfp-a8940/bfp-a8940Z.pdf (In Japanese)
16.http://dl.mitsubishielectric.co.jp/dl/fa/members/document/manual/robot/bfp-a8080/bfp-a8080e.pdf (In Japanese)
17.Ye R, Ishibashi Y, Huang P, Tateiwa Y. Comparison of stabilization control for writing characters in remote robot system with force feedback. In: Proc. the 3rd International Conference on Computer Communication and the Internet (ICCCI), Nagoya, Japan. 2021
18.Ito S, Ishibashi Y, Huang P, Tateiwa Y. Effect of robot position control using force information: Human versus robot with force sensor. In: Proc. the 9th International Conference on Information and Education Technology (ICIET), Okayama, Japan. 2021. pp. 257-261
19.Ishibashi Y, Huang P, Psannis KE. Enhanced robot position control using force information for mobile robots: Influences of obstacles on cooperative work. In: Proc. 2021 IEEE 9th International Conference on Information, Communication and Networks (ICICN), the 4th World Symposium on Communication Engineering (WSCE) Held as Workshop of ICICN, Xi’an, China. 2021. pp. 555-559
20.Hameedha N, Ishibashi Y. Effects of local and global adaptive Δ-causality control on cooperative work between remote robot systems with force feedback. ITE Transactions on Media Technology and Applications (MTA). 2022;10(1):1-7
21.Ye R, Ishibashi Y, Huang P, Tateiwa Y. Comparison of collaboration methods between users in remote robot systems with force feedback. In: Proc. the 7th International Conference on Computer and Communications (ICCC), Chengdu, China. 2021. pp. 1052-1056
22.Toyoda Y, Ishibashi Y, Huang P, Tateiwa Y, Watanabe H. Efficiency of cooperation between remote robot systems with force feedback: Comparison with cooperation between user and remote robot system. International Journal of Mechanical Engineering and Robotics Research (IJMERR). 2020;9(6):900-905
23.Ishibashi Y, Tsuji A, Tasaka S. A group synchronization mechanism for stored media in multicast communications. In: Proc. 16th IEEE International Conference on Computer and Communications (INFOCOM), Kobe, Japan. 1997. pp. 693-701
24.Ishibashi Y, Tasaka S, Hasegawa T. The virtual-time rendering algorithm for haptic media synchronization in networked virtual environments. In: Proc. the 16th International Workshop on Communications Quality and Reliability (CQR), Okinawa, Japan. 2002. pp. 213-217
25.Huang P, Ishibashi Y. Simultaneous output-timing control in networked games and virtual environments. In: Handbook on Multimedia Synchronization. Switzerland: Springer International Publishing; 2018. pp. 149-166
26.Mills DL. Internet time synchronization: The network time protocol. IEEE Transactions on Communications. 1991;39(10):1482-1493
27.3D systems, OpenHaptics toolkit programmer’s guide. version 3.2, 2013
28.Komatsu Y, Ohnishi H, Ishibashi Y. Adaptive control of viscosity in remote control system with force feedback. In: Proc. IEEE International Conference on Consumer Electronics - Taiwan (ICCE-TW), Chinese Tapei. 2017. pp. 237-238
29.Suzuki S, Matsunaga K, Ohnishi H, Ishibashi Y. Influences of network delay variation on haptic perception under adaptive reaction force control. In: Proc. the 3rd IEEE Global Conference on Consumer Electronics (GCCE), Tokyo, Japan. 2014. pp. 669-673
30.Abe T, Komatsu Y, Onishi H, Ishibashi Y. QoE assessment of adaptive viscoelasticity control in remote control system with haptic and visual senses. In: Proc. IEEE International Conference on Consumer Electronics - Taiwan (ICCE-TW), Chinese Tapei. 2018. pp. 133-134
31.Ishibashi Y, Taguchi E, Huang P, Tateiwa Y. Robot position control with force information in cooperation between remote robot systems. In: Proc. the 5th International Conference on Control, Automation and Robotics (ICCAR), Beijing, China. 2019. pp. 147-151
32.Kato H, Ishibashi Y, Ohnishi H, Huang P. Force Adjustment Control for Carrying Object between Remote Robot Systems with Force Feedback. Fukui, Japan: IEICE Technical Report, CQ2022–13. 2022 (In Japanese)
33.Rikiishi T, Ishibashi Y, Huang P, Miyoshi T, Ohnishi H, Tateiwa Y, et al. Effect of Stabilization Control by Viscocity in Remote Robot System. Chiba, Japan: IEICE Technical Report, MVE2017–19. 2017 (In Japanese)
34.Arima R, Huang P, Ishibashi Y, Tateiwa Y. Softness Assessment of Objects in Remote Robot System with Haptics: Comparison between Reaction Force Control upon Hitting and Stabilization Control. Tokyo, Japan: IEICE Technical Report, CQ2017–98. 2018 (In Japanese)
35.Ishibashi Y, Nagasaka M. An adaptive QoS control scheme of avatars in distributed virtual environments. The Journal of The Institute of Image Information and Television Engineers. 2006;60(11):1835-1839
36.Wang X, Huang P, Ishibashi Y, Okuda T, Watanabe H. Influence of network delay on QoS control using neural network in remote robot systems with force feedback. In: Proc. 2020 the 9th International Conference on Networks, Communication and Computing (ICNCC), Tokyo, Japan. 2020
37.Hashem IAT, Yaqoob I, Anuar NB, Mokhtar S, Gani A, Khan SU. The rise of ‘big data’ on cloud computing: Review and open research issues. Information Systems. 2015;47:98-115
38.Memos VA, Psannis KE, Ishibashi Y, Kim BG, Gupta BB. An efficient algorithm for media-based surveillance system (EAMSuS) in IoT smart city framework. Future Generation Computer System. 2018;83:619-628
39.Ramesh A, Kambhampati C, Monson J, Drew P. Artificial intelligence in medicine. Annals of the Royal College of Surgeons of England. 2004;86:334-338
Written By
Pingguo Huang and Yutaka Ishibashi
Submitted: 06 July 2022Reviewed: 04 August 2022Published: 04 October 2022
Open access peer-reviewed chapter
