Toward Dynamic Manipulation of Flexible Objects by High-Speed Robot System: From Static to Dynamic

Abstract

This chapter explains dynamic manipulation of flexible objects, where the target objects to be manipulated include rope, ribbon, cloth, pizza dough, and so on. Previously, flexible object manipulation has been performed in a static or quasi-static state. Therefore, the manipulation time becomes long, and the efficiency of the manipulation is not considered to be sufficient. In order to solve these problems, we propose a novel control strategy and motion planning for achieving flexible object manipulation at high speed. The proposed strategy simplifies the flexible object dynamics. Moreover, we implemented a high-speed vision system and high-speed image processing to improve the success rate by manipulating the robot trajectory. By using this strategy, motion planning, and high-speed visual feedback, we demonstrated several tasks, including dynamic manipulation and knotting of a rope, generating a ribbon shape, dynamic folding of cloth, rope insertion, and pizza dough rotation, and we show experimental results obtained by using the high-speed robot system.

1. Introduction

Typical flexible object manipulation by robots has been performed in a static state or a quasi-static state [1, 2, 3, 4, 5, 6, 7]. In these methods, dynamical deformation of the flexible object is not considered, making it easier to manipulate the flexible object. However, the robots have to wait until the deformation converges to the steady state, which results in the manipulation time being very long, and the working efficiency is not good. Moreover, although it is desired to manipulate the flexible object while observing its deformation with cameras, the working efficiency becomes even worse because of the low speed of the cameras and image processing for recognizing the deformation. In addition, a deformation that changes momentarily cannot be recognized in real time by general-purpose cameras and image processing, and appropriate feedback control may not be carried out because of the latency from the recognition to the robot motion.

Recently, dynamic manipulations and nonprehensile manipulations of objects have been actively investigated with the goal of developing new manipulation techniques [8]. Our goal has been to achieve high-speed flexible object manipulation by handling the flexible object in a “dynamic,” nonprehensile state. As a result, we succeeded in simplifying the deformation model of flexible objects by using the high-speed motion of a high-speed robot, and we were able to perform appropriate real-time visual feedback control with a high-speed vision system. This dynamic manipulation technique may enable rapid realization of flexible object manipulation tasks.

In this chapter, we review the proposed method based on high-speed motion and visual feedback, and we demonstrate several tasks carried out by a high-speed robot system using the proposed method.

2. Concept of dynamic manipulation of flexible object

The key difficulties in the dynamic manipulation of a flexible object include:

1. Deformation of the flexible object during manipulation

2. Prediction of its deformation

In order to solve these problems as easily as possible and to enable dynamic manipulation of flexible objects, we proposed an entirely new method. In the proposed method, the robot moves with a tip velocity that decreases the effect of undesirable deformation of the flexible object; as a result, the model and motion planning can be simplified. Figure 1 shows the basic concept of dynamic rope manipulation. The proposed method can be understood by picturing the manipulation of a rhythmic gymnastics ribbon, in which the ribbon deforms according to the tip motion.

Figure 1.

Basic concept. (a) Low-speed motion and (b) High-speed motion.

From the above discussion, the rope deformation depends on the high-speed robot motion. In addition, we can assume that the rope deformation can be derived algebraically from the robot motion. Thus, the rope deformation model can be described as a simple deformation model derived from the robot motion. Also, since the rope deformation can be calculated algebraically from the robot motion in the model, the rope deformation model can be made more simple than typical models that use matrix differential equations as a multi-link model or partial differential equations as a continuous body model [9, 10].

Moreover, if the dynamic manipulation is performed in slow motion, gravity has a non-negligible effect on the rope. In that case, the algebraic equation does not hold, and we have to consider a differential equation, making dynamic manipulation in slow motion extremely difficult as a result. Thus, high-speed robot motion is required in order to achieve dynamic manipulation.

Dynamic manipulation enables faster manipulation, high efficiency of task realization, and shorter task completion time and working time.

3. Dynamic manipulation method using high-speed robot motion

First, this section describes a theoretical analysis of a dynamical deformation model of flexible objects. We deal with a flexible rope as one example of such flexible objects. Then, we obtain a condition for dynamic manipulation based on the analysis. Next, we propose a simple deformation model that can be expressed by an algebraic equation. We also suggest a robot motion planning method using this simple model [11].

In the analysis, we assume that gravity is basically ignored, but we briefly describe the handling of gravity in Appendix-A.2, and we also assume that the flexible object does not exhibit elasticity.

3.1 Theoretical analysis for dynamic manipulation

3.1.1 Equation of motion of rope

Let us consider the equation of motion of the rope when the rope deformation is restricted to a given curve and a constraint force acts on the rope, as shown in Figure 2 . The constraint force is R [N/m], the rope position is w [m], the position of the inside of the rope is σ [m], the line density of the rope is μ [kg/m], the tension is T [N], and the curvature is ρ [m]. Considering a length δσ [m] of the curve, the difference between the tensions in the tangential direction becomes δT [N]. Since the angle of the part δσ from the center of curvature is assumed to be δθ [rad], the equation δσ = ρδθ holds. Thus, the tension in the normal direction at both ends of this part can be described as:

E1

E2

Figure 2.

Rope mechanics.

Since the force R is the constraint force, the direction of the force is perpendicular to the curve. Then, the equations of motion in the tangential and normal directions of the part δσ become

E3

Eq. (3) leads to

E4

Eq. (4) represents the rope dynamics under the condition that the rope behavior is restricted to the given curve. Next, we derive conditions for the robot motion in order to simplify the rope dynamics.

3.1.2 Condition for restricting rope on reference trajectory

When the constraint force R is equal to zero, the rope can deform so as to track the given curve. Therefore, the condition that R = 0 be satisfied can be obtained from Eq. (4):

E5

This equation is a function of the rope velocity. Substituting Eq. (5) into Eq. (4) yields

E6

Since the part inside the brackets on the right-hand side is a function of the rope velocity, the right-hand size is equal to zero. This leads to

E7

As a result,

E8

holds. In addition, the tension T can be obtained from Eq. (5):

E9

And the tension T thus becomes constant. From this discussion, the necessary and sufficient condition that the rope can move along the reference trajectory in the absence of gravity and with the constraint force R = 0 can be summarized as

E10

This result means that when the rope moves along the rope reference trajectory, the velocity in the tangential direction of the rope is constant and a uniform force is applied to each joint of the rope. On the contrary, if the condition that the velocity and tension of the rope be constant is satisfied, the rope can move along the reference trajectory of the rope. Manipulating the rope at a constant velocity can be achieved by moving the robot arm in the tangential direction at a constant velocity. It is impossible to control the tension to be constant in the case of the free end of the rope. However, assuming that the rope is sufficiently long and that the rope tracks the reference configuration, the condition that the tension be constant approximately holds.

As a result, the robot motion conditions necessary to simplify the rope model are as follows:

1. Constant-velocity motion: the rope deformation can be restricted to the reference trajectory.

2. High-speed motion: the effects of gravity can be reduced under the gravity condition.

Thus, by manipulating the rope with this strategy, the rope can deform so as to track the robot motion. Since the robot is moved at a constant speed, each joint of the rope tracks the robot motion with a constant time delay. This time delay depends on the location of the joint.

In this analysis result, another solution (ρ = ∞) can also be obtained. The details are explained in Appendix-A.1.

3.2 Simple deformation model

3.2.1 Robot motion

First we consider the kinematics in order to derive the tip position of the robot arm. The joint angles and the tip position of the robot arm are defined by θ and r, respectively. In general, the relationship between the tip position and the joint angles can be obtained by the following equation:

E11

Although the details of the derivation are omitted, the tip position is derived by using the Denavit-Hartenberg description.

3.2.2 Algebraic deformation model of rope

In general, a rope model is described by a distributed parameter system represented by partial differential equations. As another model, the rope is approximated by a multi-link system, and an equation of motion expressed by matrix differential equations is derived. In this research, we apply the multi-link system to the rope model. Then, the equation of motion can be replaced by an algebraic equation under the condition of constant, high-speed motion of the robot.

Based on the analysis described in the previous section, the following facts can be introduced in the rope deformation model:

• The rope behavior tracks the robot motion.

• The distance between two joint coordinates of the rope is not variable.

• Twisting of the rope is not taken into account.

The first assumption holds by ensuring constant, high-speed motion. This means that the rope deformation model can be described by the robot motion. The second assumption is that the rope does not have elasticity. Thus, the link distance in the multi-link model does not change.

From the above discussion, we propose a simplified rope deformation described by the following equation:

E12

where t is time, i is the joint number of the rope (i = 0,1,···,N − 1), N is the number of particles in the multi-link rope model, s_i is the i-th joint coordinate of the rope, l is the distance between two joint coordinates (viz., the link length in the multi-link model), and e_i is a unit vector that represents the direction from the (i − 1)-th joint to the i-th joint (i = 1,···,N − 1). In the case of i = 0, s ₀(t) = r(t) holds. This means that the location of the end point of the rope is the same as the tip position of the robot. Figure 3 shows an overview of the proposed simple model. As shown in Figure 3 , the rope deforms so as to track the tip of the robot arm.

Figure 3.

Overview of simple model.

Since the proposed model does not include an inertia term, Coriolis and centrifugal force terms, or a spring term, we do not need to estimate the dynamic model parameters. The advantage of the proposed model is that the number of model parameters is lower than in typical models. Therefore, the proposed model itself is robust. Moreover, since the rope model can be algebraically calculated, the simulation time becomes much shorter.

3.3 Motion planning

This section explains the motion planning for deriving the trajectory of the robot arm from the rope configuration. The procedure for the motion planning of the robot is the following:

1. The desired rope configuration (s_r ) is defined by a user; that is, the user gives the control points, as shown in Figure 4 . Here, there exists a case where the link distance between two joint coordinates on the given rope configuration is not equal to l. Therefore, the rope configuration is corrected so that the link distance is equal to l.

2. The trajectory of the tip position (r) of the robot arm is calculated from the rope configuration (s_r ). The trajectory (r) of the robot arm can be obtained to track the given coordinate of each joint of the rope, as shown in Figure 4 . Namely, we have the following equations:

Figure 4.

Inverse problem.

E13

The trajectory is determined so as to linearly move from the N-th link to the first link during the motion time ∆T.

1. The joint angles (θ) of the robot arm can be obtained by solving the inverse kinematics.

3.4 Features of proposed method

The advantages of the proposed method can be summarized as follows:

• The rope can be controlled so as to trace the robot trajectory in the absence of gravity.

• In contrast, any reference shape of the rope can be obtained by moving the robot so as to trace the reference shape of the rope in the absence of gravity.

• The deformation model of the rope can be represented by an algebraic equation with the constant-velocity motion of the robot in the absence of gravity.

• A reference shape of the ribbon can be produced at a certain timing in the presence of gravity.

On the other hand, the disadvantages are as follows:

• There is a possibility that the deformation around the end of the rope may be out of alignment.

• The time during which a reference shape of the ribbon can be produced and maintained is limited to a short period. Moreover, since there is a possibility that the error between the reference shape and the actual shape may increase when the motion time is long, high-speed motion of the robot is required.

• The error in a rope with high rigidity or stretching properties may increase.

3.5 High-speed visual feedback control

The proposed method with the simple deformation model includes a modeling error, and the error may affect the dynamic manipulation of the flexible object. In order to correspond to the modeling error, we introduce a high-speed visual feedback control using a high-speed vision system and a high-speed image processing technique. A block diagram of the high-speed visual feedback control is shown in Figure 5 .

Figure 5.

Block diagram of high-speed visual feedback control.

By appropriately setting the extraction of the image feature and applying the self-window method, it is possible to speed up the image processing. As a result, a high-speed visual feedback is realized. In our method, the sampling rate of the visual feedback control is set at 1 kHz which is the same as the sampling rate of the robot control.

4. Task realization

In this section, we explain our high-speed robot system. Then, based on the proposed dynamic manipulation method, we describe proposed strategies for realizing various tasks such as dynamic knotting of a flexible rope, a generation of ribbon shape, and a dynamic folding of a cloth. The experimental results will be shown in Section 6.

4.1 High-speed robot system

The high-speed robot system consists of a high-speed robot arm (four degrees of freedom, manufactured by Barrett Technology Inc.), two high-speed robot hands (180°/0.1 s) [12] mounted on two sliders (2 m/s), a high-speed vision system (1000 frames per second), and a real-time controller (1 kHz). The high-speed vision system is used for high-speed visual feedback, including real-time image processing. The joint angles of the robot system (arm, hands, and sliders) are controlled by a proportional-derivative law to the reference joint angles every 1 ms (corresponding to a sampling rate of 1 kHz).

4.2 Dynamic knotting of a flexible rope

As a first example, we explain a dynamic knotting of a flexible rope. Figure 6 shows a strategy of the dynamic knotting. Firstly, we produce the circle shape on the rope as shown in Figure 6(a)–(d) . Then we perform the impact phenomenon on the intersection of the circle as shown in Figure 6(d)–(f) . After the impact, the end of the rope passes through the circle. As a result, the dynamic knotting can be completed.

Figure 6.

Strategy of dynamic knotting of flexible rope.

4.3 Generation of ribbon shape

This section describes circular shape control of a ribbon based on the proposed method [13]. Figure 7 shows a strategy of ribbon shape generation. Using constant, high-speed motion described above and air drag, we achieve the circle shape generation as shown in Figure 7 .

Figure 7.

Strategy of generation of ribbon shape.

For this task, we extended the simplified model of the rope to a belt-like flexible object, taking into account the effects of drag and gravity [13].

4.4 Dynamic folding of a cloth

Next, we explain a dynamic folding of a cloth, as shown in Figure 8 , which is an application of two-dimensional flexible object. In the initial state, two robot hands grasp a cloth. Then, based on the proposed method, we obtained the joint trajectories of the robot system in order to appropriately deform the cloth using the high-speed motion. Finally, the robot hands grasp the end of the cloth using the high-speed visual feedback.

Figure 8.

Strategy of dynamic folding of cloth.

In this task, we extended the simplified model of the rope to a sheet-like flexible object. We constructed a two-dimensional model [14].

5. Rotational motion system

Here we describe the realization of additional tasks with a rotational motion system, including robotic rope insertion and pizza dough spinning. First, we briefly discuss analysis of the rotational motion system using high-speed rotational motion. Then, based on the results, we describe proposed strategies of rope insertion and pizza dough spinning using real-time visual feedback. The experimental results will be shown in Section 6.

5.1 Discussion

In this section, we explain a simple theoretical analysis of the high-speed rotational motion system. Using high-speed rotational motion, the flexible characteristics of an object to be manipulated may be effectively decreased, allowing us to consider only the rigid characteristics of the object.

Considering the forces acting on a part of the object as shown in Figure 9 , the following equations can be obtained:

Figure 9.

Model of rotational motion (side view).

E14

where T is the tension, ∆m is the mass of a part of the object, r is the radius from the rotational center to the part, ω is the angular velocity, and θ is an approximate angle at the part, as shown in Figure 9 . From these equations, we can get

E15

From this result, we found that the angle θ decreases when the angular velocity ω increases. Thus, the deformation in the gravitational direction (z direction) can be ignored. In addition to this result, assuming that the elastic characteristics of the rope can be neglected, we can consider that the rotational system of the flexible object can be boiled down to the rotational system of a rigid body. This analysis result can be also applied to the dynamics of the flexible object in the radial direction.

5.2 Rope insertion

We briefly explain a rope insertion task as the first application [15]. Figure 10 shows a strategy of the rope insertion. The rope deformation is restricted to a linear shape by using high-speed rotation, and visual feedback positioning control, as shown in Figure 5 , between the tip position of the rope and the position of a hole is performed using the high-speed robot system. At the time when the tip position and the hole position are the same, the rope insertion is achieved by using the motion of a linear actuator (maximum speed is 2.4 m/s).

Figure 10.

Strategy of rope insertion.

5.3 Pizza dough spinning

Next, we describe spinning of pizza dough [16]. Figure 11 illustrates a strategy of the pizza dough spinning. In this task, the pizza deformation is restricted to a planar shape by using the proposed method. Angular acceleration of the rotation was achieved by high-speed finger motion. The contact control was carried out by using high-speed visual feedback.

Figure 11.

Strategy of pizza dough spinning.

6. Experimental results

Here we show experimental results of the dynamic knotting of a rope, the generation of ribbon shape, the dynamic folding of a cloth, the rope insertion, and the pizza dough spinning, respectively.

6.1 Result of dynamic knotting of a rope

Figure 12 shows the experimental result for dynamic knotting with the high-speed robot arm [17]. Also, Figure 13 show a desired rope configuration for producing the circle shape and the joint trajectories of the high-speed robot arm solved by the inverse kinematics. In this experiment, the diameter and the length of the rope are 3 mm and 0.5 m, respectively. It can be seen from Figure 12 that dynamic knotting of the flexible rope was achieved. Since the execution time is 0.5 s, the knotting can be carried out at high speed.

Figure 12.

Continuous sequence of photographs of dynamic knotting. (a) t = 0.00 [sec], (b) t = 0.16 [sec], (c) t = 0.32 [sec], (d) t = 0.48 [sec], (e) t = 0.64 [sec] and (f) t = 3.20 [sec] [17].

Figure 13.

Desired rope configuration and joint trajectories of the robot arm. (a) Desired rope configuration, (b) Rotation axis of the upper arm, (c) Circulation axis of the upper arm and (d) Rotation axis of the lower arm [17].

6.2 Result of generation of ribbon shape

The starting configuration was with the ribbon hanging straight down. Figure 14 shows a composite photograph in which the circle shape was produced. In Figure 14 , the dotted white line depicts the reference shape of the ribbon (i.e., the robot trajectory), and these pictures were taken at intervals of 0.067 s. Figure 15 shows an error between the reference circle shape and the actual ribbon shape. The error is calculated by a square root of sum of square error of each direction. As can be seen from these figures, the circle shape of the ribbon was successfully produced. We also produced a figure-eight shape, a wave shape, and a crank shape in the ribbon [18].

Figure 14.

Composite photograph of shape generation [13, 18].

Figure 15.

Error between reference and actual shapes [13, 18].

The error between the reference shape and the obtained shape was evaluated for each reference shape.

6.3 Result of dynamic folding of a cloth

Next, we show the experimental result for dynamic folding of a cloth [14] shown in Figure 16 . Figure 17 shows the trajectories of the slider and hand wrists, and Figure 18 shows the experimental data (visual information). Figure 16(a) represents the initial condition of the experiment, where the two hands grasp the cloth. Figure 16(b)–(c) shows the cloth being pulled toward the grasp position using the hand and slider motions. Figure 16(d) shows that when the hand and slider motions stop, the free end (the point far from the grasp position) of the cloth is folded by inertial force. Figure 16(e)–(f) shows grasping of the free end of the cloth using high-speed visual feedback [19]. As can be seen from the experimental results, dynamic folding of the cloth can be achieved by the two high-speed multi-fingered hands and two high-speed sliders. In addition, since the action time of the dynamic folding performed by the robot system is 0.4 s, high-speed folding can be achieved.

Figure 16.

Continuous sequence of photographs of dynamic folding. (a) Time = 0.0 [sec], (b) Time = 0.1 [sec], (c) Time = 0.2 [sec], (d) Time = 0.3 [sec], (e) Time = 0.4 [sec] and (f) Time = 0.5 [sec] [19].

Figure 17.

Trajectories of slider and hand wrist. (a) Trajectory of slider and (b) Trajectory of wrist of hand [19].

Figure 18.

Experimental data of dynamic folding. (a) Area of markers and (b) Y position of markers [19].

6.4 Result of rope insertion

Figures 19 and 20 show the experimental result and data during rope insertion experiment. From the experimental results, the rope is deformed by the effect of gravity in the initial state as shown in Figure 19(a) , but the rope deformation could be successfully controlled to a linear shape using the high-speed rotation motion. Consequently, we achieved the rope insertion task with the proposed method, as shown in Figure 19(b) .

Figure 19.

Continuous sequence of photographs of rope insertion task. (a) Initial state, (b) Finish state, (c) 2.0 [s], (d) 2.40 [s], (e) 2.75 [s], (f) 2.82 [s], (g) 2.90 [s] and (h) 2.95 [s] [15].

Figure 20.

Experimental data of rope insertion. (a) Angular feature, (b) Angular feature error and (c) Image coordinate x-y [15].

6.5 Result of pizza dough spinning

Figures 21 and 22 show the experimental result and data during pizza dough spinning experiment. It can be seen that the deformation of the pizza dough could be controlled to approximately a planar shape by the proposed method, and the angular acceleration of the rotation could be achieved using the high-speed finger motion with contact control by high-speed image processing.

Figure 21.

Continuous sequence of photographs of pizza dough spinning. Normal camera; (a) t = 0.00 [s], (b) t = 0.33 [s], (c) t = 0.67 [s], (d) t = 1.00 [s], High-speed camera; (e) t = 0.04 [s] and (f) t = 0.12 [s] [16].

Figure 22.

Experimental data of pizza dough spinning. (a) x-position of pizza dough and (b) y-position of pizza dough [16].

Summarizing this section, high-speed motion and real-time visual feedback allowed us to perform dynamic manipulation of flexible objects in a rotational motion system.

7. Conclusion

We have proposed a new method for dynamic and high-speed manipulation of flexible objects. The proposed method includes constant, high-speed robot motion and real-time visual feedback control. The findings described in this chapter can be summarized as follows:

1. The constant, high-speed robot motion contributed to the construction of a simple deformation model of flexible objects (expressed by an algebraic equation).

2. High-speed visual feedback also allowed for compensation of the modeling error, in order to increase the success rate and robustness for task realization.

3. Dynamic, high-speed manipulations of flexible objects, such as dynamic knotting, shape generation, dynamic folding, rope insertion, and pizza dough spinning, could be achieved successfully by using the proposed method and a high-speed robot system.

Our proposed method will bring about a paradigm shift in which the flexible object strategy changes from static or quasi-static to dynamic.

In the future, we plan to demonstrate dynamic manipulations of flexible objects other than the tasks we explained in this article.

Conflict of interest

The authors declare no conflict of interest.

A.1 Another solution in the theoretical analysis:

As another solution from Eq. (4), we can obtain the following condition so that R = 0 holds:

E16

This is also a necessary and sufficient condition that the rope can move along the reference trajectory in the absence of gravity and with the constraint force R = 0. This means that the reference shape of the rope is a straight line, and therefore, it is trivial that the rope moves with straight-line motion.

Thus, we do not adopt this solution

in this research.

A.2 Case in which effects of gravity are considered

Here, we briefly explain a case in which the effects of gravity are considered in the simple deformation model and the motion planning [17].

In the calculation of the rope deformation, since the robot moves at high speed, the effects of gravity (G(t)) can be approximated as

E17

and then this term is added to z direction in Eq. (12).

In the motion planning, compensation for the effects of gravity,

E18

is also considered, and this term is added to the given control points, as shown in Figure 23 .

Figure 23.

Inverse problem with consideration of effects of gravity.

After that, the inverse kinematics are calculated in order to obtain the reference joint angles of the robot.

Supplementary materials

The experimental results can be seen on our YouTube channels and web sites [20, 21, 22, 23].