Design of Oscillatory Neural Network for Locomotion Control of Humanoid Robots

Standing and walking are very important activities for daily living, so that their absence or any abnormality in their performance causes difficulties in doing regular task independently. Analysis of human motion has traditionally been accomplished by subjectively through visual observations. By combining advanced measurement technology and biomechanical modeling, the human gait is today objectively quantified in what is known as Gait analysis. Gait analysis research and development is an ongoing activity. New models and methods continue to evolve. Recently, humanoid robotics becomes widely developing world-wide technology and currently represents one of the main tools not only to investigate and study human gaits but also to acquire knowledge on how to assist paraplegic walking of patient (Acosta-M ́arquez and Bradley, 2000). Towards a better control of humanoid locomotion, much work can be found in the literature that has been focused on the dynamics of the robot using the Zero Moment Point (ZMP) approach (Vukobratovic and Borovac, 2004). More recently, biologically inspired control strategies such as Central Pattern Generators (CPG) have been proposed to generate autonomously adaptable rhythmic movement (Grillner, 1975, 1985; Taga, 1995; Taga et. al, 1991). Despite the extensive research focus in this area, suitable autonomous control system that can adapt and interact safely with the surrounding environment while delivering high robustness are yet to be discovered. In this chapter, we deal with the design of oscillatory neural network for bipedal motion pattern generator and locomotion controller. The learning part of the system will be built based on the combination of simplified models of the system with an extensive and efficient use of sensory feedback (sensor fusion) as the main engine to stabilize and adapt the system against parameters changes. All motions including reflexes will be generated by a neural network (NN) that represents the lower layer of the system. Indeed, we believe that the NN would be the most appropriate code when dealing, to a certain limit, with the system behavior, which can be described by a set of ordinary differential equations (ODEs) (Zaier and Nagashima, 2002, 2004). The neural network will be augmented by neural controllers with sensory connections to maintain the stability of the system. Hence, the proposed learning method is expected to be much faster than the conventional ones. To validate the theoretical results, we used the humanoid robot “HOAP-3” of Fujitsu. The structure of the chapter is as follows: the first section will present an introduction on the conventional CPG based locomotion control as well as the Van der Pol Based Oscillator;

. Matsuoka neural oscillator model. Black dots corresponds to inhibitory connections and the white dots to excitery. Using the notation in (Zaier and Nagashima, 2002), the symbol ">0" reprents a switcher that take the positive part of the input, for example the input from x1 becomes [x1]+  max(0, ) The extensor neuron 1 in Fig. 1 is governed by equations (2) and (3) and flexor neuron 2 by equations (4) and (5). The variable x i is the neuron potential or firing rate of the i th neuron, v i is the variable that represents the degree of adaptation or self-inhibition, c is the external tonic input with a constant rate, β specify the time constant for the adaptation, τ a and τ r are the adaptation and rising rates of the neuron potential, μ ij is the weight that represents the strength of inhibition connection between neurons, and g j is an external input which is usually the sensory feedback weighted by gain h j . Where the positive part of the input [g j ] + is applied to one neuron and the negative part [g j ] -=-min(g j ,0) is applied to the other neuron. The positive part of the output of each neuron is denoted by y i =[x 1 ] + and the final output of the oscillator y out is the difference between the two neurons' outputs. However, the parameters should be adjusted correctly to suite the application the oscillator will be used www.intechopen.com for. This can be done by experiments or by specifying the constraints. According to Williamson (Williamson, 1999), for stable oscillations, / ra ττ should be in the range 0.1-0.5.
Keeping the ratio / ra ττ constant makes the natural frequency of the oscillator n ω (the frequency of the oscillator without an input) proportional to r 1/τ . According to Matsuoka (Matsuoka, 1987), the parameters should be selected based on the following criteria to ensure a stable rhythm 1 (1 )   ji μ β and 12 21 .1  r a τ μμ τ . Some researchers used Genetic Algorithm (GA) technique to find and optimize the oscillator's parameters (Inada, 2004).

Van der Pol based oscillator
The Van der Pol Oscillator, described by second order nonlinear differential equation (13), can be regraded as a mass-spring-damper system. The circuit of this oscillator is shown in Figure 2, where the neuron notation in (Zaier and Nagashima, 2002) is used. It has been adopted by Van de Pol in 1920th to study the oscillations in vacum tube circuits. Recently researchers built CPG model based on Van der Pol oscillator (Senda K, Tanaka T, 2000). Equation (10) is for the forced oscillotor where the right hand side term is the periodic forcing term. The unforced Van der Pol oscillator is investigated using MATLAB/Simulink © block diagram as shown in Figure 3.
Where x and x  are the states of the system and μ is a control parameter that represents the degree of nonlinearity of the system or its damping strength. www.intechopen.com

Design of Oscillatory Neural Network for Locomotion Control of Humanoid Robots 45
The equation 9 has a periodic solution that attracts other solutions except the trivial one at the unique equlibrum point (origin of the phase plote).
In the case of 0  μ , the system will be unstable, oscillate forever and the phase plot will never converge to an equilibrium point, as shown in Figure 4 (a) and (b). While in the case of μ 0  , the system will converge to an equilibrium point as shown in Figure 4 (c) and (d).
In this case, the decaying is directly proportional to the negative value of μ , where the oscillation dies faster as the constant goes larger in the negative direction. However, if μ is set to zero, then the system has no damping and will oscillate sinusoidally forever as shown in Figure 4 (e). The phase plot draws an exact circle in this case. In conclusion, the Van der Pol oscillator has the ability to produce a periodic behavior which can represent a periodic locomotion pattern of robots (Veskos P., 2005).

Piecewise linear function based oscillator
The control strategy here is based on piecewise linear oscillators and inspired by the solution of the Van der Pol system in Figure 4b, and with few parameters that can be easily obtained. Moreover, the motivation is to provide much flexibility to the design of the motion pattern generator so that modulation of the output by other circuits such as circuits generating reflexes can be realized without complexity or re-design of the motion pattern generator (Zaier and Kanda, 2007). The method requires not to satisfy constraints on robot's foot or ZMP stability margin, it simply uses piecewise-linear functions and a first order lowpass filter generated by an original recurrent neural network (RNN), where the "integrate and fire" neuron model (Gerstner, 1995) has been used. Indeed, the piecewise linear control is much easier to analyze than the control based directly on non-linear equations. Moreover, this type of control provides much intuition about the system behavior. Also, our method for generating walking motion, although inspired by the inverted pendulum, it considers not strictly the system dynamics. The stability of the system is assured by a gyro feedback loop, while terrain irregularities are compensated by adjusting the pitching motion using a virtual spring-damper systems. The proposed method is straightforward with respect to www.intechopen.com The Future of Humanoid Robots -Research and Applications 46 three design parameters namely, the slope of the piecewise function, the time delay of the neuron, and the rolling amplitude of the hip and ankle joints. In this section, we investigate the stability of the system simplified as an inverted pendulum, the robustness of the locomotion controller against terrain irregularities.

Motivation
To control the robot locomotion, we consider the general form of neural network as; where x∈R n is the neurons' outputs vector, y∈R m depicts the motors' commands vector, D is an nxn d i a g o n a l m a t r i x w i t h s t r i c t l y p o s i t i v e r e a l n u m b e r s , T is an nxn neurons' interconnection matrix, g(x): R n R n is the nonlinear vector activation function, C is an mxn output matrix, v∈R m is a sensory input vector, and S is an nxm matrix. Note that the right side of equation (11) can be considered as decomposition into linear part "Sv" and a nonlinear one "Tg(x)". To investigate the stability of equation (11), we consider the behavior of a linearized model of system at the equilibrium points with Tg(x)=0. Indeed, a motion can be described topologically as a homotopy mapping at each equilibrium point as well as switching, or jump mapping between these points (Forti, 1996). In this research framework we consider locomotion control of a humanoid robot simplified as an inverted pendulum when standing on one leg. Notice that, while the inverted pendulum model considerably simplifies the control of the humanoid robot, the inertia effect of distributed masses such as the arms presents a limitation to that approach.

Problem formulation
The proposed control strategy is illustrated in figures 5 and 6, where the neural network modeled by equation (11) has to control the humanoid robot locomotion as a continuous commutation between the two equilibrium points defined at single support phases. Then, using the output of the switched system, we decide the rolling motion profile. The stability of the equilibrium points during locomotion is maintained using gyro feedback, while the pitching motion is adjusted by virtual dumper-spring system. To begin with, we investigate the stability of the robot simplified as an inverted pendulum when standing at one leg that can be expressed by; where θ∈R is the counterclockwise angle of the inverted pendulum from the vertical, ∈R is damping ratio, μ=g/l∈R, and u ∈R is the control input to the system. Let θ = α-α s , where α is the angular position of the hip joint and α s is its value when the projected center of mass (COM) is inside the support polygon. Then, equation (12) can be rewritten as follows; x 1 Equilibrium point (at standing pose) Moreover, according to figure 7, the relationship between α and the COM is Since α s is very small (practically it is between -0.14 and 0.14rad), the relationship can be approximated as; www.intechopen.com

Design of Oscillatory Neural Network for Locomotion Control of Humanoid Robots 49
Hence we can write α s =D/L in [rad], where D is the lateral component of the distance between the COM and the hip joint. On the other hand, to control the inverted pendulum as illustrated in figures 1 and 2, we propose the control input as follows; where u g (α) is the gyro feedback signal stabilizing the inverted pendulum around α=α s , and u sw (α) is a relay type feedback signal with a hysteresis property defined by; where tis the time just before commutation. Notice that the switching law of equation (16) depends not only on the output y(t) but also on the switching time constant τ. In our design we choose a fixed switching time, which reduces the impact map to a linear one as detailed in (Goncalves et al., 2000). The control system of equation (15) can be written in the state space form as; where x= (  ) T , A and B are matrices with compatible dimensions, and C=[r 0]. Let's further illustrate the stability of the control system by considering the phase space diagram in figure 8, where the x 1 -axis represents the lateral component of the COM position, and which divides the phase space into two regions S 0 and S 1 . The axis x 2 represents the velocity. The state space trajectory in each region is defined by a different linear differential equation according to the state of relay feedback of equation (16). It will be possible to define Poincare sections where the switching takes place. The origin "O" corresponds to the state of the robot at the double support phase. The motion starts by switching the state of the robot to section M 0 , i.e., standing on the left leg (figure 5). The periodic motion is defined as a continuous switching of state between M 0 and M 1 , in other words, the state that starts from section M 0 at x k reaches section M 1 at x k+1 following a trajectory described by the linear differential equation defined for region S 1 . It can be shown that the system of equation (17) with the feedback control of equation (16) can produce limit cycles (Goncalves et al., 2000). The stability analysis of the system can be conducted by considering the manifolds of its equilibrium points x * as follows With the feedback signal of equation (15), the trajectories of the system will follow the invariant manifolds of only two equilibrium points of equation (17). To investigate the stability of equation (13) for α close to α s , reconsider the system in the absence of control input. Then, at each equilibrium point the system will have the same eigenvalues as follows; Since <<μ, the equilibrium points of equation (18) are saddle ones. To generate periodic movement as explained in figure 5 using equation (16), we first stabilize the equilibrium points by a feedback control u g (α) = k 1 α + 2k 2 d(α)/dt using the angular velocity of the inverted pendulum measured by the gyro sensor. Therefore, combining equations (13) and (16) then the eigenvalues will be complex conjugate with a negative real part. Now, consider the trajectories in both regions S 0 and S 1 defined by 1 ) (    sw u , which are governed by the invariant manifolds of stable virtual equilibrium points. The state space trajectory will follow the invariant manifold of a stable equilibrium point but a switching will occurs before reaching this point, then the trajectory will be governed by the invariant manifold of the other stable virtual equilibrium point and so on. This consecutive switching will result in a limit cycle in the phase space. Without loss of generality we assume that the time origin is when the state space is at M 0 . The trajectory of the state space of system is given by The switching sections are defined by, and equation (23) can be re-written as follows, It is clear from equations (27) and (28) that for any u g (α) with given parameters k 1 and k 2 stabilizing the system at the switching point, the Poincare map of equation (26) is a contracting one. Moreover, it can be noticed that when there is no switching, the state trajectory obviously will converge to one of the equilibrium points of equation (18). Moreover, if the switching occurs outside the attraction region of equation (18), the robot will have an unstable walking behavior and will fall down after few steps. Therefore, the choice of the switching time is crucial for the stability of the system. In this paper we approximate the angular position solution of equation (25) for t∈[0,π/2] (during the commutation from M 0 to M 1 ) as a piecewise linear function modulated by a decaying exponential instead of sine function modulated by a decaying exponential as will be explained later on. The smallest time  the state trajectory may take when the robot is commutating from one leg to another is such that τ =π/2 and we will have; of the robot, the walking cycle time is assumed to be 2τ. Moreover, for given commutation duration τ calculated by equation (29) with the feedback controller parameters in equation (22), the initial linear velocity can be expressed from equation (25)

Approximate solution of rolling
According to equation (25), the position command to the hip is shown in figure 10, and the control law at switching times can be expressed as; where i is the number of half walking cycle τ, the tand t + are the times just before and after commutation, respectively. For  i t  the state trajectory will follow the desired limit cycle as described in figure 9, where the stability of the equilibrium points is guaranteed by the gyro feedback added to the system. Since the position command to the hip is the same as that to the ankle joint with opposed sign, the robot upper body orientation in the absence of disturbance remains unchanged, in other words the angular velocity around the rolling axis during locomotion is zero. Notice that the control law described by equation (31) is explicit of time. Instead, the state commutation can be constrained by angular velocity 0 /  dt d , and the control law becomes independent of the cycle time as a pure feedback law. However, the condition of having a fixed commutation time (Goncalves et al., 2000) will be no longer satisfied. For the sake of reducing the computing cost and ease the implementation of the controller, instead of considering the exact solution of the state trajectory of equation (25), we propose an approximate solution α(t) as shown in figures 11 and 12, using a unit piecewise linear function as follows; where u s (.) is the unit step function, ω i >0 is the slope of the function u(.) between t i and (t i +1/ω i ). The approximate solution α(t) can be formulated, therefore, as a function of time delay , joint angular velocity ω, walking period T, and the rolling amplitude α s .
where t r0 , t f are the times at the start and the end of the rolling motion, respectivel. n is number of walking steps. f 1 and f 2 are the relative times with respect to the gait. Let to be sent respectively to the hip and ankle joints are given by, where ) (t r fb  is the PD controller's output stabilizing the system of equation (17).

Numerical Example
Consider the inverted pendulum with the control law of equation (20) where we choose an arbitrary high gain controller satisfying equation (22) with k 1 = 32 and k 2 = 0.9, and does not trigger mechanical resonance. We consider the rolling profile equation (34) with = 0.15, and α s =6 deg. The phase portrait of the output signal is shown in figure  13, where case (a) is for non perturbed system, while case (b) is in the presence of perturbation. This result shows that the state trajectory is bounded, and therefore it is stable.

Robot is rolling
At t 1 the damper is used to land softly, while at t 2 the spring energy is used to lift

Swing and stepping motion
The dynamic of the swing leg can be considered the same as that of a pendulum, and hence it is inherently stable without any compensator. Due to the switching control that we already adopted in generating the rolling motion, the swing motion can be generated in a similar fashion and as illustrated in figure 11, which can be expressed as; where l  is the lifting motion, A l is the amplitude of lifting, and t l0 and t l2 are the switching time for the first and second lifting, t l1 and t lf are the switching times for the first and the last landing, and ω l is the joints' angular velocity generating the lifting motion.
To control the motion along the pitching direction, besides the gyro feedback control we consider the robustness against rough ground (ground with some surface irregularity) by controlling the time-varying parameters of the virtual spring damper system b s (t) and k s (t) in equation (38) during the gait as described in figure 14. That is, to minimize the force of collision at the landing time t 1 , the damping coefficient b s (t 1 ) is set at its maximum value while k s (t 1 ) is at its minimum value. At the lifting time t 2 , we inject spring energy into the system by setting b s (t 2 ) at its minimum while k s (t 2 ) is at its maximum. We consider linear time-parameter changes between lower/upper limits.
where y c (t)is the displacement of the mass m along the vertical axis, and g(t, F y ) is a piecewise function in t and locally Lipschitz in F y that depicts the external force acting on the supporting leg. We choose g as a saturation function. If we let y c =x 1 , and dy c /dt = x 2 , equation (38) can be re-written in the state space form as;  (40) where N is the number of state space, and using the neuron model given by (1) As a result, the right side of (41) represents the inputs to the neuron x i . The neural controller reflecting (41) is shown in figure 15 for N=2, k i =1+ i , i = i /a ii ,   is the angular position induced by the virtual damper spring system in equation (22). The L t is the length of the thigh. The ) (t l fb  is the feedback signal satisfying the stability of inverted pendulum in the sagittal plane. where V 0 = ω. Moreover, the dynamics of the robot during stepping motion is investigated using the phase portrait of the ZMP position in the lateral plane that is shown in figure 16, which is similar to the simulation result in figure 13. Notice that the more we increase the derivative gain the better convergence we obtain. However, we are limited by the high frequencies resonance that could be triggered at high feedback gain. Figure 17 demonstrates the effect of the virtual damper-spring system of equation (38), which is implemented to work as illustrated in figure 14. Hence, it demonstrates how the robot uses gravity for landing and the spring energy for lifting. Figure 17a shows that the lifting phase starts when the angular position of the rolling joints is almost zero. In other words, the actuators of the hip and ankle rolling joints do not contribute in moving the ZMP to the supporting leg. These joints, therefore, can be considered as locked joints. Figure 18 shows the rolling of the hip and the pitching motions of the knee, hip, and ankle. It demonstrates also how smooth the approximate solution is using the proposed pattern generator with sensory feedback. This figure shows also how the robot posture is controlled by changing the ankle position at the single support phase according to equation (42). To demonstrate the robustness against plant perturbations and disturbances, we conducted two experiments; one consists of making the robot walks on hard floor, which can be regarded as walking in the presence of small disturbance. The phase portrait in figure 18 demonstrates that the system exhibits stable limit cycle with three periods. The second experiment consists of locomotion in the presence of larger disturbance and plant perturbation by letting the robot walks on a carpet with surface irregularities. In this case, despite the high damping coefficient of the carpet, the proposed locomotion controller could robustly maintain the stability of the system as shown in figure 19. This result also demonstrates the efficiency of the proposed approach in designing a robust locomotion controller, which is simply based on few parameters, which are the robot mass, COG height, and the distance between hip pitching joints.

Conclusion
In this chapter, we first introduced the Matsuoka and Van der Pol Oscillators. Then we presented our proposed oscillator, which was inspired by the solution of the Van der Pol www.intechopen.com Design of Oscillatory Neural Network for Locomotion Control of Humanoid Robots 59 that leads to stable limit cycle. The oscillator was designed using linear piecewise linear functions and the design parameters were obtained based on the equation of motion of inverted pendulum. Moreover, sensory feedback namely gyro and force sensors feedbacks were considered by adding neural controller to the oscillator's neural network. Therefore, the neural network was augmented by neural controllers with sensory connections to maintain the stability of the system. In addition, the rolling profile parameters were analytically obtained and the approximate solution was implemented giving much modularity to the motion pattern generator to include cuircuts of reflexes and task planning. The locomotion controller became so adaptive that the robot is enabled to walk on floor, carpet, and slope. In order to demonstrate the effectiveness of the proposed system, we conducted experiment using Fujitsu's humanoid robot HOAP-3. It was shown that the proposed pattern generator is robust in the presence of disturbance.

Acknowledgements
The experimental work was conducted at Fujitsu Laboratories Limited, Japan. The author would like to acknowledge the help of Ms Asiya Al-Busaidi in summarizing part of the literature section.