A Real-Time Bilateral Teleoperation Control System over Imperfect Network

Abstract

Functionality and performance of modern machines are directly affected by the implementation of real-time control systems. Especially in networked teleoperation applications, force feedback control and networked control are two of the most important factors, which determine the performance of the whole system. In force feedback control, generally it is necessary but difficult and expensive to attach sensors (force/torque/pressure sensors) to detect the environment information in order to drive properly the feedback force. In networked control, there always exist inevitable random time-varying delays and packet dropouts, which may degrade the system performance and, even worse, cause the system instability. Therefore in this chapter, a study on a real-time bilateral teleoperation control system (BTCS) over an imperfect network is discussed. First, current technologies for teleoperation as well as BTCSs are briefly reviewed. Second, an advanced concept for designing a bilateral teleoperation networked control (BTNCS) system is proposed, and the working principle is clearly explained. Third, an approach to develop a force-sensorless feedback control (FSFC) is proposed to simplify the sensor requirement in designing the BTNCS, while the correct sense of interaction between the slave and the environment can be ensured. Fourth, a robust-adaptive networked control (RANC)-based master controller is introduced to deal with control of the slave over the network containing both time delays and information loss. Case studies are carried out to evaluate the applicability of the suggested methodology.

1. Introduction

Teleoperation, which allows a human operator to interact with the environment remotely, extends humans’ sensing, decision making, and operation beyond direct physical contact. Since its introduction in the late 1940s, teleoperation systems have been deployed worldwide in numerous domains ranging from space exploration, underwater operation, and hazardous assignment to micro-assembly, and minimally invasive surgery.

In common, control schemes for teleoperation systems can be classified as either compliance control or bilateral control. In the compliance control [1, 2], the contact force sensed by the slave device is not reflected back to the operator, but is used for the compliance control of the slave device. On the contrary, in the bilateral control [3–5], the contact force is reflected back to the operator. The operator is able to achieve physical perception of interactions at the remote site similar to as directly working at this site. Consequently, it improves the accuracy and safety in teleoperation. Thus, the bilateral control has drawn a lot of attention.

Figure 1 shows a generic configuration of a bilateral teleoperation system which includes five components: operator, master, communication network, slave, and environment. The master is capable of acquiring information about the desired manipulation actions and assigning the tasks to the slave. It normally consists of an input component, such as a joystick, console, or tactile device, and a force-reflecting mechanism (FRM) to exert the reflected forces on the human operator. The slave is a robotic device that takes the place of the human operator to carry out the required tasks at the remote environment. It is usually equipped with sensors to acquire information about the task process to feed back to the master.

Figure 1.

Generic configuration of a bilateral teleoperation system.

There are two common control architectures of bilateral teleoperation systems: position–position and position–force architectures. In the first approach, the master position is passed to the slave device, and the slave position is passed back to the master side. The reflected force applied to the operator is derived from the position difference between the two devices and, therefore, this approach is not desirable in cases of free motion. In contrast, the position–force approach uses directly the force measured at the remote site rather than the position error. In this architecture, the contact force, sensed by a force/torque sensor mounted on the slave device, is scaled by a force-reflecting gain (FRG), and this scaled force is reflected back to the operator via the master device. This method then provides the operator a better perception of tasks execution at the remote site.

In order to derive sufficiently the FRG, two important tasks are required: first, to detect the environment and, second, to determine the contact force at the slave site. Many studies have been carried out to optimize the FRG [4, 5]. Although the reported algorithms showed some remarkable results, there remain some drawbacks such as how to determine the FRG appropriately with unknown environments; and especially, it is difficult and expensive to attach proper sensors (force/torque/pressure sensors) to detect the loading conditions. Additionally, the use of these kinds of sensors is cost-ineffective and difficult to be installed in practice. Especially, it is easy to be damaged when the system operates under hazard conditions. Incorporation of teleoperation and force feedback requires bidirectional information exchange between the master and slave via the network. In contrast to the advantages such as cost saving, power consumption, easy implementation, and maintenance, a networked control system (NCS) leads to two major problems, inevitable random time-varying delays and packet dropouts because of restrictions imposed by the transmission data rate and channel bandwidth. Due to sensitivity of bilateral teleoperation systems to time delays and packet dropouts, even a small time delay can destabilize the system [6].

Disregarding the packet loss problem, numerous methods have been proposed [7–9] to minimize the bad effect of time delay. Herein, the controllers were designed based on the assumptions that time delay was constant [7], bounded [8], or had a probability distribution function [9]. Moreover, these assumptions just considered that the time delay in the closed control loops was less than one sampling period while in practice, it is random and irregular. To compensate large and uncertain delays, other studies suggested adaptive control schemes using variable sampling periods, which were based on neural networks (NNs) or prediction theories [10–12]. Although the performances were improved, there were requirements on acquiring the real delay data for the training processes, which were not appropriately discussed.

To adapt with NCSs compromising not only time delays but also packet dropouts, many important methodologies, such as state feedback control [13], robust control [14–16], predictive and model-based predictive control [17, 18], were proposed. By using these techniques, the NCS performances were remarkably improved over the traditional methods. However in most studies, time delays and packet dropouts were assumed to be priorly known and bounded to design the controllers. The sampling period was always fixed in these studies and, subsequently, limited the control performances. Furthermore, computation delay normally at the master side is also one important factor affecting directly the system performance. Recently, an advanced robust variable sampling period control approach has been developed for nonlinear systems containing both the kinds of delays and packet dropouts [19–21]. The applicability of this method was proved through real-time experiments.

Therefore to fully overcome the above-mentioned problems, a simple and cost-effective real-time bilateral teleoperation networked control system (BTNCS) is introduced in this chapter. The advanced concept for designing a bilateral master-slave teleoperation networked control system is proposed, and the working principle is clearly addressed. Next, an approach to develop a force-sensorless feedback control (FSFC) is proposed to simplify the sensor requirement in designing the BTNCS while the correct sense of interaction between the slave and the environment can be ensured. To deal with the networked control of the slave, a robust-adaptive networked control (RANC)-based master controller is suggested to compensate for the problems of time delays and information loss. Case studies are carried out to evaluate the applicability of the suggested methodology.

2. Bilateral teleoperation networked control system

Figure 2.

Proposed architecture of the BTNCS.

Without loss of generality, the architecture of the BTNCS is suggested in Figure 2. The design for this BTNCS should address the following issues:

• The local master (LM) controller functions as the main control unit of the system to ensure that the slave manipulator could execute robustly and accurately the tasks given by the operator via a human interface (HI), disregarding the impacts of networked communication problems and environments. The local slave (LS) controller is, therefore, an optional design. It can be just a buffer or a zero-order holder (ZOH) to receive the control input derived by the LM controller and distribute sequentially to the manipulator.

• There is no force/pressure/torque sensor to be attached at the manipulator end to minimize the cost and risks. Only an internal sensor, as displacement sensor, is used to monitor the manipulator trajectory and send back to the LM controller to form a closed control loop.

• The interaction between the manipulator and environment is, therefore, estimated by the FSFC module at the master side. Here, the FSFC should consist of two parts: first, an environmental interaction (EI) estimator to detect the environment characteristics as well as to derive properly a reflecting force required to be applied to a physical device in the HI module (as a joystick); second, an FRM controller to drive the FRM to generate the desired reflecting force.

• The FRM is suggested to be constructed using pneumatic rotary actuators. This use brings some advantages over the traditional design with DC electric motors. Compared with an electric motor, a pneumatic actuator provides a higher ratio of force-mass, and can produce larger reflected force without using any reduction mechanism, such as gearbox. In addition, with the pneumatic solution, the FRM is able to work under safe conditions without damage from the operator.

• Due to having different control modules and other functions at the master side, computation delays ${\tau }^{com}$ need to be taken into account when designing the LM controller.

• The communication network has unavoidable delays, ${\tau }^{ca}\text{and}{\tau }^{sc}$, and packet dropouts, represented by “virtual” switches, S^ca and S^sc, in the forward and backward channels, respectively. S^ca (S^sc) is opened (or 1) when a packet loss event exits.

The following are attempted to address the two important issues in designing the BTNCS which are the FSFC and the LM controller.

3. Force-sensorless feedback control

3.1. Design of FSFC

As stated in Section 2, the FSFC compromises the two main modules: EI estimator and FRM controller. The FRM controller can be selected as a typical feedback controller in which the reference input is the desired reflecting force sent from the environment classifier and the feedback signal can be the force/torque/pressure at the physical device of the HI module.

The EI estimator is suggested as in Figure 3. This estimator with two inputs and one output contains four blocks: Learning Vector Quantitative Neural Network (LVQNN) classifier, slave dynamics, environment dynamics, and fuzzy-based FRG tuner. Without loss of generative, the interactive environment can be represented by two factors: damping c_e and stiffness k_e. Thus, The LVQNN classifier is firstly designed with two inputs and two outputs to classify the environment. The two inputs are the command and response from the slave while the two outputs are predicted values of the environment damping and stiffness, ${{\displaystyle \widehat{c}}}_e\text{and}{{\displaystyle \widehat{k}}}_e$, respectively. Next, these predicted values are fed into both the environment dynamics and fuzzy-based FRG tuner. Synchronously, the slave command is also input to the slave dynamics. The interaction between the slave dynamics and environment dynamics—loading force—is, therefore, estimated and denoted as ${{\displaystyle \widehat{F}}}_e$. Meanwhile, the fuzzy-based FRG tuner is designed with two inputs, ${{\displaystyle \widehat{c}}}_e\text{and}{{\displaystyle \widehat{k}}}_e$, and one output which is value of the FRG. Finally, the desired reflected force, F_dr, is derived from the estimated loading force and the FRG.

Figure 3.

Configuration of the proposed environmental interaction estimator.

3.2. LVQNN classifier

3.2.1. Learning vector quantitative Neural Network

NN is one of the powerful artificial intelligent techniques that emulates the activity of biological NNs in the human brain. LVQNN is a hybrid network which uses advantages of competitive learning and bases on Kohonen self-organizing map or Kohonen feature map to form the classification [22].

Figure 4 shows a generic structure of an LVQNN with four layers: one input layer with m nodes, first hidden layer named competitive layer with S₁ nodes, second hidden layer named linear layer with S₂ nodes, and one output layer with n nodes (in this case, S₂ ≡ n).

Figure 4.

Structure of the LVQNN.

The core of the LVQNN is based on the nearest-neighbor method by calculating the Euclidean distance weight function, D, for each node, n_j, in the competitive layer as in the following:

$n_j=D\left(X,W_1\left(j\right)\right)=\sqrt{{\displaystyle {\displaystyle \sum _{i=1}^m}{\left(X\left(i\right)-W_1\left(j,i\right)\right)}^2}},j=1,\mathrm{..},S_1$E1

where X is the input vector; W₁(j,i) is the weight of node j^th in the competitive layer corresponding to element i^th of the input vector.

Next, the Euclidean distances are fed into function C which is a competitive transfer function. This function returns an output vector o₁, with 1, where each net input vector has its maximum value, and 0 elsewhere. This vector is then input to the linear layer to derive an output vector o₂, where each element corresponded to each node of the output layer and computed as

$Y\left(k\right)=o_2\left(k\right)=k_W\left(k\right)n_2\left(k\right)=k_W\left(k\right){\displaystyle {\displaystyle \sum _{j=1}^{S_1}}{W}_2\left(k,j\right)o_1\left(j\right)},k=1,\mathrm{..},n,\left(n\equiv S_2\right)$E2

where W₂(k,j) is the weight of node k^th in the linear layer corresponding to element j^th of the competitive output vector; k_W(k) is linearized gain of node k^th in the linear layer.

In the learning process, the weights of LVQNN are updated by the well-known Kohonen rule, which is shown in the following equation:

$\{\begin{array}{l}{W}_1^{t+1}\left(j\right)=W_1^t\left(j\right)+\mu \left(X-W_1^t\left(j\right)\right)\text{IF: }X\text{ is classified correctly}\\ W_1^{t+1}\left(j\right)=W_1^t\left(j\right)-\mu \left(X-W_1^t\left(j\right)\right)\text{IF: }X\text{ is classified incorrectly}\end{array},j=1,\mathrm{..},S_1$E3

where μ is the learning ratio with positive and decreasing with respect to the number of training iterations ($n_{iteration}$), $\mu =n_{iteration}^{-1}$.

3.2.2. Design of LVQNN classifier

Here, the LVQNN classifier is designed and implemented into the FSFC to distinguish different working environments at the slave side in an online manner. To enhance the given task with the limited number of input information, the input vector of the classifier is constructed as a vector of current and several historical values of four signals: the slave driving command, $\left\{X_{ds}^{\left(0\right)},X_{ds}^{\left(-1\right)},\mathrm{\dots },X_{ds}^{\left(-g\right)}\right\}$, slave response, $\left\{X_s^{\left(0\right)},X_s^{\left(-1\right)},\mathrm{\dots },X_s^{\left(-p\right)}\right\}$, and their derivatives, $\left\{d{X}_{ds}^{\left(0\right)},d{X}_{ds}^{\left(-1\right)},\mathrm{\dots },d{X}_{ds}^{\left(-h\right)}\right\}$and $\left\{d{X}_s^{\left(0\right)},d{X}_s^{\left(-1\right)},\mathrm{\dots },d{X}_s^{\left(-q\right)}\right\}$, respectively, while the final outputs are the environment damping and stiffness, ${{\displaystyle \widehat{c}}}_e\text{and}{{\displaystyle \widehat{k}}}_e$.

For applications to classify the environment which is varied with both the damping and stiffness values, the output from the classifier should be the mix of classes with different ratios. In order to enhance this task, a so-called smooth switching algorithm is proposed here. The environment class is, therefore, determined by the smooth combination of the current class detected by the LVQNN (Y) and the previous class using a forgetting factor, λ

$class\left(t\right)=\lambda \times class\left(t-1\right)+\left(1-\lambda \right)\times Y\left(t\right)$E4

Similarly, the estimated values of the damping coefficient and stiffness are produced by

$\{\begin{array}{l}{{\displaystyle \widehat{c}}}_e\left(t\right)=\lambda \times {{{\displaystyle \widehat{c}}}_e|}_{class\left(t-1\right)}+\left(1-\lambda \right)\times {{{\displaystyle \widehat{c}}}_e|}_{Y\left(t\right)}\\ {{\displaystyle \widehat{k}}}_e\left(t\right)=\lambda \times {{{\displaystyle \widehat{k}}}_e|}_{class\left(t-1\right)}+\left(1-\lambda \right)\times {{{\displaystyle \widehat{k}}}_e|}_{Y\left(t\right)}\end{array}$E5

Additionally, in order to avoid influences of noises on the classification performance, the forgetting factor is online tuned with respect to the changing speed of the classifier outputs:

Step 1: Set the initial value for the forgetting factor, λ=0.5; define a small positive threshold, $0<{\gamma }_1<{\gamma }_2$, for the classifier output changing speed, $v_Y$, which is defined by the number of sampling periods when Y continuously changes.

Step 2: For each step, check $v_Y$and update λ by comparing $v_Y$with γ using the following rule:

1. + If $v_Y=0$, Then $\lambda \left(t+1\right)=\lambda \left(t\right);$

2. + Else If $\left(v_Y>{\gamma }_2\right)$, Then $\lambda \left(t+1\right)=\lambda \left(t+1\right)/2$ and reset $v_Y=0$;

3. + Else If $\left(v_Y\ge {\gamma }_1\right)\&\left(v_Y\left(t\right)\le {\gamma }_2\right)$, Then $\lambda \left(t+1\right)=\lambda \left(t+1\right)\times 2$ and reset $v_Y=0$;

4. + Otherwise, $\lambda \left(t+1\right)=\lambda \left(t\right).$

3.2.3. An illustrative example

3.2.3.1. Test rig setup

To evaluate the effectiveness of the LVQNN classifier, an experimental system has been designed as in Figure 5. One joystick with one Degree of freedom (DOF) is used to generate driving commands for the slave. An FRM which employs a valve-driven mini pneumatic rotary actuator and two bias springs is attached to the opposite side of the joystick handle. The slave employs an asymmetrically pneumatic cylinder as its manipulator. The displacement of the cylinder rod is sensed by a linear variable displacement transducer (LVDT). The interaction between the master and slave is enhanced by the communication module which can perform either wire-by-wire or wireless protocol. To generate environmental conditions for the slave manipulator, compression springs with different stiffness values are installed in serial with the cylinder rod. An air compressor is used to supply the pressurized air for both the FRM and the slave. A compatible PC and a multifunction data acquisition device are employed to perform the communication between the PC and master. The system specifications are listed in Table 1.

Figure 5.

Design layout for the experimental BMST system.

Parts	Type	Component characteristics
Rotary actuator	CRB1BW15 90-D	Max. torque: 0.9 Nm
Pressure sensors	SDE1-D10-G2-W18	Pressure range: 0–10 bar
Pneumatic cylinder	CDC-20	Stroke: 100 mm, bore: 20 mm, rod: 8 mm
Servo valves	MPYE-5-1/4-010B	Control voltage range: 0–10 VDC
LVDT	Novotechnik TR100	Measurement range: 0–100 mm
Springs	Case 1, Case 2, Case 3	Randomly selected

Table 1.

Specifications of the system components.

Input number	Number of nodes in the hidden layer
	20	25	30	35	40
20	80.35	80.50	81.48	82.17	81.29
24	81.14	81.19	82.26	81.48	82.78
28	80.61	80.03	81.66	81.12	81.69
32	75.34	80.64	85.64	81.00	81.80
36	81.03	81.58	81.93	81.85	80.49
40	79.30	80.98	80.90	80.13	80.22
	Goodness of fit [%]

Table 2.

Learning success rate of the LVQNN classifier [%].

3.2.3.2. LVQNN classifier training and verification

In order to train the network, the prior task is acquiring the target data. Real-time teleoperation experiments without force feedback concept were performed on the test rig. Both the three springs mentioned in Table 1 were used to generate the environmental conditions. For each condition, a trajectory for the slave manipulator was randomly given by the operator. The slave information, including the valve driving command and cylinder rod displacement, with respect to each spring was acquired with a sampling period of 0.02 s.

Next, each of the acquired slave data sets was used to perform the input vector, while the correspondingly selected spring was used as the target output class (1, 2, or 3). To investigate performance of the LVQNN classifier with respect to different structures, several trainings were performed with the selected data set by varying the number of inputs from 20 to 40, and the number of hidden neurons was changed from 20 to 40. After the training process, the results (goodness of fit [%]) of the LVQNN are analyzed in Table 2. It shows that the most suitable LVQNN structure was realized with 32 nodes in the input vector and 30 nodes in the competitive layer. The learning success rate in this case was highest with 85.64 [%].

Real-time teleoperation experiments were performed in order to investigate the ability of the LVQNN classifier in practice. Other three experiments with the three loading conditions were carried out with the same cylinder trajectories given from the master using an open-loop control. The classification results were then achieved as displayed in Figure 6–8. The results imply that the proposed classifier could online detect well the loading conditions and, therefore, is capable of producing precisely decisions on the environment characteristics.

Figure 6.

Real-time classification of the optimized LVQNN with respect to spring case 1.

Figure 7.

Real-time classification of the optimized LVQNN with respect to spring case 2.

Figure 8.

Classification performance of the optimized LVQNN with respect to spring case 3.

4. Robust-adaptive networked control

4.1. Problem and RANC design concept

In this section, the RANC approach for a generic system is introduced to support the design of the LM controller. Consider a discrete-time plant [20, 21]:

$\{\begin{array}{l}{x}_{k+1}=A{x}_k+B{u}_k+E{d}_k\\ y_k=C{x}_k\end{array}$E6

where $x_k\in R^n$is the state vector, $u_k\in R^m$is the control input, $y_k\in R^p$is the controlled output, $d_k\in R^d$is the environment impact, and A, B, C and E are matrices with appropriate dimensions.

The RANC-based LM controller in Figure 2 is then designed based on the following issues [20, 21]:

• Both the actuators and controller are event-driven.

• The sensors are time-driven with variable sampling period T. For step (k + 1)^th, this sampling period T_k+1 is online optimized depending on the total system time delay τ_k+1 to make sure:

  $\{\begin{array}{l}{\tau }_{k+1}={\tau }_{k+1}^{com}+{\tau }_{k+1}^{ca}+{\tau }_{k+1}^{sc}\\ T_{k+1}\ge {\tau }_{k+1}>0\end{array}$E7

• Sets of continuous packet dropouts {p_d} are online detected and bounded by ${{\displaystyle \overline{p}}}_{k+1}$:

  ${{\displaystyle \overline{p}}}_{k+1}=\sup \left(p_d\right),1\le d<k+1$E8

Using the results in references [20, 21], the configuration of an NSC using the proposed RANC controller is clarified in Figure 9. Herein, the RANC mainly consists of five modules: Time Delay and Packet Detector (TDPD), Time Delay Predictor (TDP), Variable Sampling Period Adjuster (VSPA), Quantitative Feedback Theory (QFT), and Robust State Feedback Controller (RSFC). The TDPD module is firstly used to detect the network problems at the current state. This information is then sent to the TDP to perform one-step-ahead prediction of system delays which are the inputs of the VSPA to adjust effectively the sampling period. The two modules, QFT and RSFC, are employed to construct the so-called hybrid controller to compensate for the influences of delays and packet dropouts. A smart switch (SSW) is employed to switch the hybrid controller to QFT or RSFC based on the outputs of the TDPD detector and the TDP predictor with rule:

• The QFT is selected (SSW = 0) once there is no packet loss and all delay components are less than their pre-defined threshold values: ($\lceil *\rceil$is ceiling function to return the nearest integer)

  ${{\displaystyle \overline{\tau }}}_{QFT}^{com}=\lceil {\tau }_k^{com}\rceil ,{{\displaystyle \overline{\tau }}}_{{}^{QFT}}^{ca}=\lceil {\tau }_k^{ca}\rceil ,{{\displaystyle \overline{\tau }}}_{{}^{QFT}}^{sc}=\lceil {\tau }_k^{sc}\rceil ,{{\displaystyle \overline{\tau }}}_{{}^{QFT}}^{}=\lceil {\tau }_k\rceil ,\forall k$E9

• Otherwise, the RSFC is chosen (SSW = 1).

Figure 9.

Configuration of networked LM controller using the RANC approach.

4.2. Design of RANC components

4.2.1. Design of VSPA

By using the TDPD and TDP, the sampling period for the coming step (k + 1) is defined as

$\begin{array}{l}{T}_{k+1}=\lceil T_{new}/T_0\rceil T_0=k_T{T}_0,k_T=\lceil T_{new}/T_0\rceil =1,2,\mathrm{\dots }\\ T_{new}=\max (T_0,{{\displaystyle \widehat{\tau }}}_{k+1}^{com}+{{\displaystyle \widehat{\tau }}}_{k+1}^{ca}+{{\displaystyle \widehat{\tau }}}_{k+1}^{sa})=\max (T_0,{{\displaystyle \widehat{\tau }}}_{k+1}),T_0>=0.1T_p,\left(T_0\text{is initial sampling period}\right)\end{array}$E10

where ${{\displaystyle \widehat{\tau }}}_{k+1}^{com},{{\displaystyle \widehat{\tau }}}_{k+1}^{ca}\text{and}{{\displaystyle \widehat{\tau }}}_{k+1}^{sa}$are the delays estimated by the TDP.

4.2.2. Design of TDPD

To measure accurately time delays and packet dropouts, the TDPD employs a micro-control unit (MCU) with proper logic. Here, PIC18F4620 MCU from microchip equipped with a 4 MHz oscillator is suggested to be used [19]. The real-time measurement accuracy of 50μs [19] which is suitable for this application. For each step k^th, the TDPD enhances the following tasks:

• Derive the real working time using the MCU, t_k.

• For a command u_k sent from the controller to the plant, a time stamp is encapsulated into this packet to detect the forward delay, ${\tau }_k^{ca}$, and packet dropouts if having, $p_k^{ca}$.

• For the signals sent from the sensors to the controller, they are combined with time stamps to detect the delay, ${\tau }_k^{sc}$, and packet dropouts if having, $p_k^{sc}$.

• The total number of continuous packet dropouts is then determined as:

  $p_k=S_k^{ca}{p}_k^{ca}+S_k^{sc}{p}_k^{sc}$E11

• Depend on the time stamps to detect the computation delay at the controller side, ${\tau }_k^{com}$.

4.2.3. Design of TDP

The TDP based on a so-called adaptive gray model with single-variable first-order, AGM (1,1) to estimate the system delays in the coming step, ${{\displaystyle \widehat{\tau }}}_{k+1}^{com},{{\displaystyle \widehat{\tau }}}_{k+1}^{ca},\text{and}{{\displaystyle \widehat{\tau }}}_{k+1}^{sa}$. The AGM (1,1) prediction procedure is as follows:

Step 1: For an object with a data sequence
$\left\{y_{Object}\left(t_{O1}\right),y_{Object}\left(t_{O2}\right),\mathrm{\dots },y_{Object}\left(t_{Om}\right)\right\}\left(m\ge 4\right)$, the raw input gray sequence representing for the object data is derived as

$y_{raw}^{\left(0\right)}=\left\{y_{raw}^{\left(0\right)}\left(t_1\right),y_{raw}^{\left(0\right)}\left(t_2\right),\mathrm{\dots },y_{raw}^{\left(0\right)}\left(t_n\right)\right\};\Delta t_k=t_k-t_{k-1};k=2,\mathrm{..},n\ge 4;$E12

Note: Eq. (12) is obtained from Eq. (14) if condition (13) satisfies; otherwise, it is obtained from Eq. (15).

$\begin{array}{l}\Delta t_{Oi}/T_{\text{TDP}}<2;\Delta t_{Oi}=t_{Oi}-t_{O\left(i-1\right)};\forall i\in \left[2,\mathrm{\dots },m\right];T_{\text{TDP}}\text{is the prediction}\text{sampling period}\\ \Delta t_{O1}=t_{O1}-t_{Olast};t_{Olast}:\text{time of previous value of }{y}_{Object}\left(t_{O1}\right)\end{array}$E13

$y_{raw}^{\left(0\right)}\left(t_k\right)=y_{Object}\left(t_i\right);t_k=t_{Oi};k=1,\mathrm{\dots },n;n=m$E14

$\begin{array}{l}{y}_{raw}^{\left(0\right)}\left(t_1\right)=y_{Object}\left(t_1\right);t_k=t_{k-1}+T_{\text{TDP}}\\ y_{raw}^{\left(0\right)}\left(t_k\right)=S{a}_i+S{b}_i{\left(t_k-t_{Oi}\right)}^1+S{c}_i{\left(t_k-t_{Oi}\right)}^2+S{d}_i{\left(t_k-t_{Oi}\right)}^3\\ t_{Oi}\le t_k\le t_{O\left(i+1\right)};k=2,\mathrm{\dots },n;n=\lfloor \left(t_{Om}-t_1\right)/T_{TDP}\rfloor ;i=1,\mathrm{\dots },m-1\end{array}$E15

where $\left\{S{a}_i,S{b}_i,S{c}_i,S{d}_i\right\}\left(i=1,\mathrm{\dots },m-1\right)$is a [4 × m – 1] coefficient matrix of the spline function going through the object data set $\left\{\left(t_1,y_{Object}\left(t_1\right)\right),\mathrm{\dots },\left(t_m,y_{Object}\left(t_m\right)\right)\right\}$.

Theorem 1: There always exist two non-negative additive factors, c₁ and c₂, to convert any raw sequence (12) to a gray sequence which satisfies both the gray checking conditions.

Proof: The proof of this theorem can be found in reference [19].

Thus from (12), the gray sequence is derived using Theorem 1:

$y^{\left(0\right)}=\left\{y^{\left(0\right)}\left(t_1\right),y^{\left(0\right)}\left(t_2\right),\mathrm{\dots },y^{\left(0\right)}\left(t_n\right)\right\}>0;y^{\left(0\right)}\left(t_k\right)=y_{raw}^{\left(0\right)}\left(t_k\right)+c_1+c_2;k=1,\mathrm{\dots },n$E16

Step 2: Use the 1-AGO to obtain a new series y⁽¹⁾ from y⁽⁰⁾:

$y^{\left(1\right)}\left(t_k\right)={\displaystyle {\sum }_{i=1}^k{y}^{\left(0\right)}\left(t_i\right)}\times \Delta t_k={\displaystyle {\sum }_{i=1}^k\left(y_{raw}\left(t_i\right)+c_1+c_2\right)}\times \Delta t_k$E17

Step 3: Create the background series z⁽¹⁾ from y⁽¹⁾ by the following general algorithm:

$z^{\left(1\right)}\left(t_k\right)=\alpha \left(t_k\right)y^{\left(1\right)}\left(t_k\right)+\left(1-\alpha \left(t_k\right)\right)y^{\left(1\right)}\left(t_{k-1}\right);k=2,\mathrm{\dots },n$E18

$\alpha \left(t_k\right)=\beta {\alpha }_{aver}+\left(1-\beta \right){\alpha }_{adapt}\left(t_k\right)$E19

${\alpha }_{adapt}\left(t_k\right)=[\begin{array}{l}1,\text{IF}:s\left(t_k\right)\ge {\displaystyle {\displaystyle \sum _{i=1}^{k+1}}\Delta t_i}/{\displaystyle {\displaystyle \sum _{i=1}^n}\Delta t_i}\\ s,\text{IF}:{\displaystyle {\displaystyle \sum _{i=1}^{k-1}}\Delta t_i}/{\displaystyle {\displaystyle \sum _{i=1}^n}\Delta t_i}<s\left(t_k\right)<{\displaystyle {\displaystyle \sum _{i=1}^{k+1}}\Delta t_i}/{\displaystyle {\displaystyle \sum _{i=1}^n}\Delta t_i}\\ 0,\text{IF}:s\left(t_k\right)\le {\displaystyle {\displaystyle \sum _{i=1}^{k-1}}\Delta t_i}/{\displaystyle {\displaystyle \sum _{i=1}^n}\Delta t_i}\end{array}$E20

$s\left(t_k\right)=\log \left(y^{\left(0\right)}\left(t_k\right)/y^{\left(0\right)}\left(t_1\right)\right)/\log \left(y^{\left(0\right)}\left(t_n\right)/y^{\left(0\right)}\left(t_1\right)\right)$E21

where ${\alpha }_{aver}$is the average weight and set as 0.5 [23, 24]; ^β is a momentum rate within range [0, 1] and tuned by a Lyaponov-based SISO fuzzy mechanism (LFM) in order to guarantee a robust prediction performance (see the proof of this theory in reference [20])

Step 4: Establish the gray differential equation:

$y^{\left(0\right)}\left(t_k\right)+a{z}^{\left(1\right)}\left(t_k\right)=b$E22

${{\displaystyle \widehat{\beta }}}_{ab}={\left[\begin{array}{cc}{\displaystyle \widehat{a}}& {\displaystyle \widehat{b}}\end{array}\right]}^T={\left(B^{T}B\right)}^{-1}{B}^{T}Y;B=\left[\begin{array}{cc}-z^{\left(1\right)}\left(t_2\right)& 1\\ -z^{\left(1\right)}\left(t_3\right)& 1\\ ⋮& ⋮\\ -z^{\left(1\right)}\left(t_n\right)& 1\end{array}\right],Y=\left[\begin{array}{c}{y}^{\left(0\right)}\left(t_2\right)\\ y^{\left(0\right)}\left(t_3\right)\\ ⋮\\ y^{\left(0\right)}\left(t_n\right)\end{array}\right]$E23

Step 5: Setup the AGM (1,1) prediction as follows:

${{\displaystyle \widehat{y}}}^{\left(0\right)}\left(t_k\right)=\left({\displaystyle \widehat{b}}-{\displaystyle \widehat{a}}{y}^{\left(1\right)}\left(t_{k-1}\right)\right)/\left(1+\alpha \left(t_k\right){\displaystyle \widehat{a}}\Delta t_k\right)=\frac{{\displaystyle \widehat{b}}-{\displaystyle \widehat{a}}{y}^{\left(0\right)}\left(t_1\right)\Delta t_1}{1+\alpha \left(t_2\right){\displaystyle \widehat{a}}\Delta t_2}{\displaystyle {\displaystyle \prod _{i=3}^k}\frac{1+\left(\alpha \left(t_{i-1}\right)-1\right){\displaystyle \widehat{a}}\Delta t_{i-1}}{1+\alpha \left(t_i\right){\displaystyle \widehat{a}}\Delta t_i}}$E24

Step 6: Perform the predicted value of y at step (n + p)^th:

${{\displaystyle \widehat{y}}}_{raw}^{\left(0\right)}\left(t_{n+p}\right)=\frac{{\displaystyle \widehat{b}}-{\displaystyle \widehat{a}}{y}^{\left(0\right)}\left(t_1\right)\Delta t_1}{1+\alpha \left(t_2\right){\displaystyle \widehat{a}}\Delta t_2}{\displaystyle {\displaystyle \prod _{i=3}^{n+p}}\frac{1+\left(\alpha \left(t_{i-1}\right)-1\right){\displaystyle \widehat{a}}\Delta t_{i-1}}{1+\alpha \left(t_i\right){\displaystyle \widehat{a}}\Delta t_i}}-c_1-c_2$E25

where p is the step size of the grey predictor. In this case, p = 1.

4.2.4. Design of QFT controller

Denotes the transfer functions of the plant and the controller in the NCS as G_p(s) and G_c(s), respectively. The closed-loop transfer function from input R(s) to output Y(s) including delays can be expressed as

$T^{\text{NCS}}\left(s\right)=\frac{Y\left(s\right)}{R\left(s\right)}=\frac{G_c\left(s\right)G_p\left(s\right)e^{-\left({\tau }_{com}+{\tau }_{ca}\right)s}}{1+G_c\left(s\right)G_p\left(s\right)e^{-\tau s}}$E26

And the open-loop transfer function is then derived as

$L^{\text{NCS}}\left(s\right)=G_c\left(s\right)G_p\left(s\right)e^{-\tau s}$E27

Next, the procedure to design this robust controller can be expressed as follows [25, 26]:

Step 1: The QFT controller should be designed on how the tracking signal meets the acceptable variation range with respect to a reference (for each value of ?i of interest)

$T_l^{\text{NCS}}\left(j{\omega }_i\right)\le \left|T^{\text{NCS}}\left(j{\omega }_i\right)\right|\le T_u^{\text{NCS}}\left(j{\omega }_i\right)$E28

Step 2: Establish bounds for robust stability of the closed-loop system

$\left|T^{\text{NCS}}\left(j\omega \right)\right|\le M=1.4,\omega \ge 0$E29

Step 3: A sensitivity function is defined as

$S\left(j\omega \right)={\left(1+L^{\text{NCS}}\left(j\omega \right)\right)}^{-1},\omega \ge 0$E30

Establish bounds for noise and disturbance rejection of the closed-loop system

$|1+L(j\omega ){|}_{\max }^{-1}\le M_D(\omega ),M_D>1(M_{\text{dB}}>0\text{dB}),\omega \ge 0$E31

Step 4: Define a working frequency range of the NCS.

Step 5: Bases on the nominal plant to design the controller, G_QFT(s), with initial poles and zeros.

Step 6: Use the loop-shaping method to refine the controller designed in Step 5. The principle to guarantee a robust control performance is the loop gain value is always on or above the boundaries defined by constraints (28)–(31) and, is to the right or on the robustness forbidden region for any critical frequency within the range defined in Step 4.

Step 7: The controller resulted from Step 6 could ensure that the variation in magnitude of T^NCS(s) (26) is satisfied at the desired constraints. However, it does not guarantee that the magnitude of T^NCS(s) actually lies between the given bounds $T_l^{\mathrm{NCS}}\left(j{\omega }_i\right)\text{and}{T}_u^{\text{NCS}}\left(j{\omega }_i\right)$for the whole frequency range. Therefore, a pre-filter F_QFT(s) is necessary to be designed and placed in front of the controller to compensate for this problem.

4.2.5. Design of RSFC

The RSFC is designed to deal with the NCS in cases that the delays are greater than the threshold values defined in Eq. (9), and/or there exists packet dropouts. To simplify the analysis, it can be assumed that r_k = 0 (Figure 9) during the system modeling and RSFC design. The system discrete form for step (k + 1)^th can be expressed through the following three cases:

Case 1: There is no packet loss in network transmission during both current period and previous period:

$x_{k+1}=A_k^0{x}_k+B_k^0{u}_k+B_{k-1}^1{u}_{k-1}+E{d}_k$E32

where $A_k^0=e^{A{T}_k},B_k^0={\displaystyle {\int }_0^{T_k-{\tau }_k}{e}^{At}B\text{d}t},B_k^1={\displaystyle {\int }_{T_k-{\tau }_k}^{T_k}{e}^{At}B\text{d}t}$.

Case 2: There exists i packet dropouts up to step k^th:

$x_{k+1}=A_k^0{x}_k+B_{k-i}^2{u}_{k-i}+E{d}_k$E33

where $B_k^2={\displaystyle {\int }_0^{T_k}{e}^{At}B\text{d}t}.$

Case 3: There were p_d continuous packet dropouts just passed up to step (k − 1)^th:

$x_{k+1}=A_k^0{x}_k+B_k^0{u}_k+B_{k-p_d-1}^1{u}_{k-p_d-1}+E{d}_k$E34

Since, the general discrete form of the system can be established as

$x_{k+1}=A_k^0{x}_k+B_k^0{\delta }_k^0{u}_k+{\displaystyle {\displaystyle \sum _{j=1}^2}{\displaystyle {\displaystyle \sum _{i=1}^{{{\displaystyle \overline{p}}}_{k+1}}}{B}_{k-i}^j{\delta }_{k-i}^j{u}_{k-i}}}+E{d}_k$E35

where ${\delta }^l$is an activation function:

${\delta }^l=\{\begin{array}{l}1,\text{If}\text{subsystem}l\text{is}\text{activated},l=\{0,1,2\}\\ 0,\text{Otherwise}.\end{array}$E36

In case of no delays, the control rule is designed as

$u_k=K{x}_k$E37

Then, Eq. (35) can be re-written in the following form:

$x_{k+1}=\left(A_k^0+B_k^0{\delta }_k^{0}K\right)x_k+{\displaystyle {\displaystyle \sum _{i=1}^{{{\displaystyle \overline{p}}}_{k+1}}}\left({\displaystyle {\displaystyle \sum _{j=1}^2}{B}_{k-i}^j{\delta }_{k-i}^j}\right)K{x}_{k-i}}+E{d}_k$E38

or

$X_{k+1}={\Phi }_k{X}_k+{\Phi }_E{d}_k$E39

where $X_k={\left[\begin{array}{cccc}{x}_k^T& x_{k-1}^T& \cdots & x_{k-{{\displaystyle \overline{p}}}_{k+1}}^T\end{array}\right]}^T,{\Phi }_k=\left[\begin{array}{cc}\left[\begin{array}{cccc}{A}_k^{*}& B_{1k}^{*}& \cdots & B_{\left({{\displaystyle \overline{p}}}_{k+1}-1\right)k}^{*}\end{array}\right]& B_{{{\displaystyle \overline{p}}}_{k+1}k}^{*}\\ I_{{{\displaystyle \overline{p}}}_{k+1}\times {{\displaystyle \overline{p}}}_{k+1}}& 0\end{array}\right],$

${\Phi }_E={\left[\begin{array}{cccc}E& 0& \cdots & 0\end{array}\right]}^T,A_k^{*}=A_k^0+B_k^0{\delta }_k^{0}K,B_{ik}^{*}=K{\displaystyle {\displaystyle \sum _{j=1}^2}{B}_{k-i}^j{\delta }_{k-i}^j}.$E0

Theorem 2: For given constants ${\xi }_i^j>0$, if there exist positive definite matrices $P_i^{*}>0,K_1>0,K_2>0,$ $i\in \left[0,1,\mathrm{\dots },{{\displaystyle \overline{p}}}_{k+1}\right],j=\{0,1,2\}$ such that following inequalities:

$\left[\begin{array}{cc}{\xi }_i^{-2}{P}_i^{*}& K_1^{*}{\Phi }_A^T+K_2^{*T}{\Phi }_B^T\\ {\Phi }_A{K}_1^{*T}+{\Phi }_B{K}_2^{*}& K_1^{*}+K_1^{*T}-P_i^{*}\end{array}\right]>0$E40

hold, then the NCS (39) is exponentially stable with a positive decay rate by using the RSFC control gain derived by

$K_{\text{RSFC}}=K_2{K}_1^{-T}$E41

with $K_1^{*}=diag\left[\left\{K_1\right\}\right],K_2^{*}=diag\left[\left\{K_2\right\}\right],K_{\text{RSFC}}^{*}=diag\left[\left\{K_{\text{RSFC}}\right\}\right]$

${\Phi }_A=\left[\begin{array}{cc}\left[\begin{array}{ccc}{A}_k^0& \cdots & 0\end{array}\right]& 0\\ I& 0\end{array}\right],{\Phi }_B=\left[\begin{array}{cc}\left[\begin{array}{ccc}{B}_k^0{\delta }_k^0& \cdots & {\displaystyle {\displaystyle \sum _{j=1}^2}{B}_{k-{{\displaystyle \overline{p}}}_{k+1}+1}^j{\delta }_{k-{{\displaystyle \overline{p}}}_{k+1}+1}^j}\end{array}\right]& {\displaystyle {\displaystyle \sum _{j=1}^2}{B}_{k-{{\displaystyle \overline{p}}}_{k+1}}^j{\delta }_{k-{{\displaystyle \overline{p}}}_{k+1}}^j}\\ 0& 0\end{array}\right].$E00

Proof: The proof of this theorem can be found in reference [20].

4.3. An illustrative example

4.3.1. Networked control system setup

To validate the RANC applicability, a simple networked control system was set up based on the configuration in Figure 9. Here, the main controller was built in an experimental PC equipped with the MCU (presented in Section 4.2.2) The network module includes one coordinator and one router employed the ZigBee protocol [19]. The multifunction card was used to perform the communications with the TDPD module and the network. The control objective is speed tracking control of a servomotor system via the setup network. Here, the servomotor system was RE-max series manufactured by Maxon Corporation. The transfer function from the input to output of this system can be expressed as [20]

$G_p\left(s\right)=\frac{32300}{0.001s^2+20.93s+493.4}$E42

To evaluate the capability of the proposed RANC, a comparative study of the RANC with three existing approaches, a QFT-based robust controller (QFTRC), a static state feedback controller (SSFC), and a hybrid QFT–SSFC as shown in Table 3, has been investigated. To make control challenges, disturbance loads and computation delays were randomly generated. Here, the loads were created by putting the system into a varied magnetic field which affected on the motor shaft. Meanwhile, an arbitrary matrix-based complex calculating procedure representing a complex application was added to the controller at the PC side.

Control; method	Main control modules					Functions
	QFT	SFC		SSW-based; TDPD–TDP	VSPA; based TDP	Delay; compensation	Packet loss compensation
		Fixed; gain	Dynamic; gain
QFTRC	√					√
SSFC		√				√	√
QFT–SSFC	√	√		√		√	√
RANC	√		√	√	√	√	√

Table 3.

Specifications of the compared controllers.

Based on Section 4.2.4, QFT delay bounds were defined as
${{\displaystyle \overline{\tau }}}_{QFT}^{com}=0.02s,{{\displaystyle \overline{\tau }}}_{{}^{QFT}}^{ca}=0.03,{{\displaystyle \overline{\tau }}}_{{}^{QFT}}^{sc}=0.03$and, the QFT controller was designed as

$\begin{array}{l}{G}_{QFT}\left(s\right)=\frac{30.5387s^2+56684.6s+3328880}{{10}^{-8}{s}^3+0.002852s^2+180.2s+1260000}\\ F_{QFT}\left(s\right)=426.65/\left(s+400\right)\end{array}$E43

Based on Section 4.2.5 and by setting the initial bounds for the total delay and packet dropouts as: ${{\displaystyle \overline{\tau }}}_0=0.1s>{{\displaystyle \overline{\tau }}}_{QFT},{{\displaystyle \overline{p}}}_0=2;$ the initial RSFC gain was computed as $K_{RSFC}^0=\left[\begin{array}{cc}0.3796& 0.2642\end{array}\right]$.

The initial sampling period of the RANC was selected as 10 ms, while that of the other controllers must be fixed to 0.15 s to cover all the delays.

4.3.2. Real-time experiments

In this section, real-time speed tracking of a servomotor system over the setup network has been investigated in which the reference speed was selected as 10 rad/s. The experiments using the comparative controllers were carried out and, consequently, the results were displaced in Figure 10. An analysis on the control performances using four evaluation criteria, PO (percent overshoot), ST (settling time), SSE (steady state error) and MSE (mean square error), is then performed as demonstrated in Table 4.

Controller	Step responses
	PO [%]	ST [s]	SSE [%]	MSE [rad/s]^2
QFTRC	2.6	1.0	13.9	1.23
SSFC	8.67	2.1	6.18	1.87
QFT–SSFC	2.07	1.35	5.88	1.35
RANC	0.2	0.8	1.58	0.63

Table 4.

Comparison of NCS performances using different controllers.

Figure 10.

Step speed performances using different controllers.

From Figure 10 and Table 4, it is clear that the worst-case performance was with the QFTRC. It is because the QFTRC was designed for an NCS in which the delays are bounded by constraint (9) and there is no packet dropout. Hence when facing with the networked servomotor including both the large delays and packet dropouts, the controller could not guarantee the robust tracking (SSE was 13.9%). In case of using the SSFC for which the control gain was designed for the worst network conditions (large delays and highest number of continuous packet dropouts), SSE was remarkably reduced to 6.18%. However due to this fixed control gain use, the system response was much slower than that of the QFTRC. Additionally, due to lack of disturbance rejection capability, the SSFC could not ensure a smooth tracking (PO = 8.67% and ST = 2.1s). The QFT-–SSFC, by taking advantages of both the QFT and SSFC, could improve the tracking result. Nonetheless, the QFT–SSFC with fixed control gains and fixed sampling period restricted the system adaptability to the sharp variations of network problems and disturbances (SSE was slightly reduced to 5.88%).

The best tracking result was achieved by using the RANC (see Figure 10 and Table 4). It comes as no surprise because the RANC possesses all the advantages of the QFT and state feedback designs and, the high adaptability to the system changes by using the adaptive control gains and adaptive sampling period. Figure 11 demonstrates the delay and packet dropout detection and prediction results of the TDPD and TDP modules, respectively. During the operation, these were supported by the VSPA and hybrid QFT–RSFC modules to regulate properly the sampling period and control gains.

Figure 11.

Step speed case: performances of TDPD and TDP.

5. Conclusions

In summary, an advanced bilateral teleoperation NCS has been introduced. In addition, the two important issues in designing the BTNCS system, FSFC and RANC, have been clearly addressed. The LVQNN-based FSFC shows the safe and cost-effective solution in controlling the FRM while the RANC-based LM controller is a feasible choice to drive the slave manipulator over the network with delay and packet loss problems.

The effectiveness of the LVQNN classifier as well as the RANC controller has been confirmed through the two case studies and analysis. One of our future research topics would be the full design of the proposed BTNCS to investigate the applicability of this method in a practical application such as remote construction equipment.