This chapter focuses on path tracking of a wheeled mobile manipulator designed for manufacturing processes such as drilling, riveting, or line drawing, which demand high accuracy. This problem can be solved by combining two approaches: improved localization and improved calibration. In the first approach, a full-scale kinematic equation is derived for calibration of each individual wheel’s geometrical parameters, as opposed to traditionally treating them identical for all wheels. To avoid the singularity problem in computation, a predefined square path is used to quantify the errors used for calibration considering the movement in different directions. Both statistical method and interval analysis method are adopted and compared for estimation of the calibration parameters. In the second approach, a vision-based deviation rectification solution is presented to localize the system in the global frame through a number of artificial reflectors that are identified by an onboard laser scanner. An improved tracking and localization algorithm is developed to meet the high positional accuracy requirement, improve the system’s repeatability in the traditional trilateral algorithm, and solve the problem of pose loss in path following. The developed methods have been verified and implemented on the mobile manipulators developed by Shanghai University.
- mobile manipulator
- path following
Recently, mobile manipulators have been used in various industries including aerospace or indoor decoration engineering, which requires a large workspace [1, 2, 3, 4]. The said mobile manipulator consists of an industrial manipulator mounted on a mobile platform to perform various manufacturing tasks such as drilling/riveting in aerospace industry or baseline drawing in decoration engineering. Wheeled mobile platforms with Mecanum wheels that can easily navigate through crowded spaces due to their omnidirectionality with a zero turning radius are commonly used. Path tracking is one of the important issues for mobile manipulators, in particular for performing manufacturing tasks. This chapter addresses this issue from the aspect of localization and calibration.
Localization is a key functionality of mobile manipulator in order to track and determine its position around the environment . Many methods are proposed to address this issue . They can be divided into two categories: absolute localization and relative localization.
Absolute localization relies on detecting and recognizing different features in the environment to obtain the position and posture. The features can be normally divided into two types: artificial landmarks and natural landmarks. Compared with the natural landmarks, artificial landmarks have advantages of high recognition, which will lead to high accuracy. There is no cumulative error problem when a localization method based on artificial landmarks is used. The key challenge is to identify and extract the needed information from the raw data of landmarks. For the relative localization, dead reckoning and inertial navigation are commonly carried out to obtain the systems’ position. It does not have to perceive the external environment, but the drift error accumulates over time.
Researchers have proposed several solutions, such as fuzzy reflector-based localization  and color reflector-based self-localization . The main drawbacks of these two methods are that the anti-interference ability is poor and the computation is huge. To solve these problems pertaining to the localization method based on artificial landmarks, Madsen and Andersen  proposed a method using three reflectors and the triangulation principle. Betke and Gurvits  proposed a multichannel localization method with the three-dimensional localization principle and the least squares method. Because of the unavoidable errors in position and angle measurement of reflectors, the use of only a trilateral or triangular method will not achieve high accuracy . Nevertheless, there are still many challenges for mobile manipulator working in industrial environments such as aerospace manufacturing or decoration engineering, which requires high maneuverability and high accuracy at the same time. The stationary industrial manipulator has high repeated localization accuracy, which the mobile manipulator cannot achieve. This chapter focuses on the improvement of path tracking of a mobile manipulator through enhanced localization combined with calibration. Calibration is required for the system kinematic model established based on nominal geometry parameter to improve motion accuracy. Muir and Neuman  proposed a kinematic error model for Mecanum wheeled platform and applied actuated inverse and sensed forward solutions to the kinematic control. Wang and Chang  carried out error analysis in terms of distribution among Mecanum wheels. Shimada et al.  presented a position-corrective feedback control method with vision system on Mecanum wheeled mobile platform. Qian et al.  conducted a more detailed analysis on the installation angle of rollers. An improved calibration method is presented in this chapter to improve the tracking accuracy of a mobile manipulator.
2. System modeling
The wheeled mobile manipulator, as shown in Figure 1, is built with a manipulator onto a wheeled mobile platform with four Mecanum wheels. This system aims to carry out fuselage drilling/riveting tasks at assembly stations in an adaptive and flexible way in the aerospace industry.
This system needs to localize in real time during machining process. The global frame is attached to the ground in the environment. The platform frame is attached to the center of the mobile platform. The tool frame is attached to the tool of the manipulator. Two laser range finders are equipped and their frames and are attached to the left-front and right-rear of the mobile platform, respectively. The vision frame is attached to the industrial camera.
The position and posture of the system in the global frame can be defined as
where and are the positions in two directions, respectively, and is the azimuth angle, as shown in Figure 2.
3. Accuracy analysis of a wheeled mobile manipulator
3.1. Problem formulation
where , , .
is defined as the linear and angular speeds of the platform; L is the half distance from the front axle to the rear axle as shown in Figure 3; is the transverse distance from the wheel centers to the platform center line; is the radius of the Mecanum wheel; , , , and are angular velocities of the four wheels, respectively.
The Mecanum wheel and its roller are shown in Figure 4. The roller is fitted on the edge of the wheel at a certain angle (generally 45°) with its central axis. Each roller can rotate freely around the central axis. The wheel relies on the friction between the roller and the ground to move. The material of roller’s outer rim is usually rubber, which will deform under pressure between ground and the wheel.
Figure 5 shows the distribution of roller deformation. F and T are the force and driving torque on the roller, respectively. The radiuses of wheels reduce differently under different pressures. The deformation zone and the position of the wheel center change with the action of driving torque and the shifting . Furthermore, with such deformation, and will change.
where ; here is defined by , , and , which are individual radiuses of the four Mecanum wheels, respectively. In order to improve the system motion accuracy, the relative errors between and , and relative errors between and () should be obtained to revise matrix (Figure 6).
3.2. Error modeling
By multiplying time , Eq. (3) becomes a displacement equation as
where is . is . Individual geometric parameters including , , , , , , and are variables in Eq. (4). The relative errors can be determined from
where , , and
With the least squares method, the geometric errors can be solved as
is generally defined as the tolerance resulting from manufacturing or assembly. With the deformation of the roller and limited to and , ) respectively, and can be defined as:
where and are in and , respectively, and r is a random number between 0 and 1. The angular speeds of all wheels are set to , and the time t is set to 1 s.
The rank of matrix in Eq. (7) is generally 3, so is a full rank matrix and can be obtained. The following steps can be carried out. First, the displacement error measurement is analyzed. In order to obtain , it is needed to obtain the displacement error matrix first.
The system is moved in the direction and stopped every 2 s to obtain its position and posture . The principle of measurement is shown in Figure 7 and can be calculated as
According to Eq. (2), , , and are theoretical values which can be obtained as follows: , , . Finally, through the experiment, the displacement errors can be obtained.
The Monte Carlo analysis is applied to deal with 50 samples. While has been given in Eq. (11), the corresponding should also satisfy its own tolerance, i.e., and .
The results are given in Figure 8. These data are averaged to form a new result: A process capability index is then used to estimate the value of . Process capability refers to the ability of a process to meet the required quality. It is a measure of the minimum fluctuation of the internal consistency of the process itself in the most stable state. When the process is stable, the quality characteristic value of the product is in the range , where μ is the ensemble average of the quality characteristic value of the product, and σ is the standard deviation of the quality characteristic value of the product.
There are two kinds of situations for which the process capability index has to be solved. First, as shown in Figure 9: is the maximum error of quality and is the minimum error of quality; ; μ is the average value of process machining; and indicates the process capability index, and
Second, as shown in Figure 10: indicates the process capability index, and . Only process capability index greater than 1 is valid. As and , and , the following results can be obtained: , , , , , , , .
The values of , , , and are closer to the real values than the values of , , , and . Now, taking , in Eq. (5), it can be seen that , . According to the change rate relative to Eq. (10), the influence of ,, and is much bigger than that of , , and . Although the values of , , , and are not accurate enough for the real values, is valid.
The interval analysis is also carried out. In Eq. (6), is uncertain because of the change of and in and , respectively. Now, the increment of and is set to 1 mm; there are 7 values of , and 11 values of , so there are groups of combinations of matrix Take all the combinations of matrix to Eq. (6) to obtain and exclude the cases for which and are not in and , respectively. Thus, 277 groups of can be obtained. As shown in Figure 11, the average values of , , , , , , , and , lead to a new , which is close to .
The same method is used to solve the process capability index of . One obtains: , , , , , , , . The values of , , and in are closer to the real ones than the values of , , and in . As the influence of , , , is bigger than that of , , and , is valid.
The result is verified as well. is used to revise the parameters of in Eq. (2) as
By setting two sets of four wheel speeds and , the displacement errors can be computed as and . The results of the two correction methods are almost identical in theory.
The experiment with four movements is shown in Figure 12(a), while the actual movement is shown in Figure 12(b). It can be found that and are close to the measured displacement errors, so both and are satisfied to revise the matrix .
4.1. System configuration
The localization component of mobile manipulator includes two laser range finders and a number of reflectors that are of cylindrical shape placed in the environment. Each reflector is covered by a reflective tape with high reflectivity (Figure 13).
4.2. Dynamic tracking
To achieve accurate localization, the system should have the ability to perceive external information through extracting the reflector features. In this research, as shown in Figure 13, there are n reflectors, and the feature of each reflector is extracted from the raw data. The feature extraction algorithm consists of three steps: (i) filtering and clustering, (ii) identification, and (iii) feature extraction.
The first step is filtering and clustering. The raw data obtained by each finder are a set of discrete data sequences . γ is the distance from the target point to the finder. ∅ is the polar angle. is the intensity value of the data point. In the process of data analysis, the outlier points that are contaminated by noise will be filtered out.
The density of the collected data points is proportional to the distance from the target point to the laser range finder. To improve the efficiency of the feature extraction process, an adaptive clustering method is adopted as given by Eq. (12). Unless the distance between two data points is less than the threshold δ, these data points are clustered for one reflector.
where is the distance value of the data point, is the angle resolution of the laser range finder, β is an auxiliary constant parameter, and is the measurement error. The values of parameters and β are given as 0.01 m and respectively.
The second step is identification. After clustering, the data set can be correlated to each observable reflector. Each reflector data set is then used to calculate the position of the reflector in the laser range finder frame . Let be the reflected intensity threshold and be the diameter range of a reflector, where D is a nominal reflector diameter and is the tolerance. The values of and are selected based on the actual situation. Considering that represents a set after clustering, i.e., , for each reflector, the measured diameter is calculated as , where and are the beginning and end data of a reflector set, respectively. When the set satisfies the following two conditions
then the set is established for all reflectors.
The third step is feature extraction. The feature extraction algorithm of a reflector extracts its central position in the laser range finder frame , where and are the distance and azimuth of each reflector, respectively. In the process of extracting the feature of the circular reflector, only a small portion of the entire circle was scanned with noise, so the accuracy of the fitting result would be low if a general least square circle fitting method is used. Since the radius of the reflector is known, the known radius is used as a constraint to improve the accuracy.
First, the value of is obtained from consecutive data points of the reflector set
As shown in Figure 14, represents the ith data and is a line between the reflector center B and the laser range finder center ; the projected angle of the line in the laser range finder frame is , while the angle between the line and the line is . The distance from to B is approximately the radius of the reflector, i.e.,
Furthermore, as mentioned before, the position of the finder in the platform frame is . The position of reflector center B in the platform frame can be expressed as
The optimal triangulation localization algorithm based on angle measurement is carried out. A number of experiments were carried out on the reflector feature extraction algorithm before proposing a localization algorithm for the system. A single reflector was placed in the measuring range of the finder. The finder performed repeated scanning when it was placed at difference places. Two different feature extraction algorithms were tested to calculate the position of the reflector, and the results are shown in Figure 15. Algorithm A is the feature extraction algorithm proposed in this chapter, while algorithm B is the least square circle fitting method without a radius constraint. Apparently, the former one yields a better result.
After extracting the reflector center, the positions of all the reflectors can be obtained and used to calculate the distance measurement range and the angler measurement range of the reflector, as given below:
where , , , .
It can be seen from Figure 16 that the angle measurement accuracy is better than the distance measurement accuracy. Therefore, based on these results, this chapter proposes an optimal trilateral localization algorithm based on angle measurement. The idea is to use the azimuth angle of the reflector in the platform frame to find the optima by the cosine theorem. The global pose of the mobile manipulator is then calculated based on the triangular method. The algorithm details are given as follows.
First, it is assumed that the finder can cover at least three reflectors at each position. After feature extraction, the positions of three reflectors , and are calculated as ; ; and , respectively. In the global frame, these positions can also be obtained as ; ; and . Since they are measured from the same finder, three circles must intersect at the same point , and the value of represents the position of the mobile platform, as shown in Figure 17.
According to the cosine theorem, the relations between the above variables can be expressed by the following equations:
In Eq. (21), the known parameters , and are used to solve , and . Since the above equations are nonlinear, the steepest descent method is adopted. The three circle equations can be expressed as
The position of the system can be obtained in the global frame by solving the above equations. Since the actual position of the reflector in the global environment deviates from the theoretical position, these circles may not intersect at the same point. In order to minimize the difference between the calculated position of the system and the actual position, the least squares estimation principle is applied. Assuming that the coordinate of the reflector is , n is the number of reflectors detected by laser range finder. The position value of the system is calculated as
The posture of the system also includes an azimuth angle in the global frame. First, is obtained from the reflector as
The azimuth angle of the system is the averaged value from all the reflectors, as in
The dynamic tracking algorithm is then carried out. Localization of the system is based on landmarks. The system needs to meet the following two conditions to complete real-time localization:
The system requires real-time localization of the reflector in the environment.
The localization system correctly matches the real-time observed reflectors.
During the matching process, the reflectors observed in real time are made to correspond to all reflectors in the last moment in the environment one by one to extract the effective reflectors . The localization can be achieved.
During localization, owing to the fact that some of the reflectors are obscured by obstacles or confused with a highly reflective metal surface object, the loss of position of the system is observed. To address the above problems, a dynamic tracking algorithm is proposed as shown in Figure 18.
After placing n reflectors in the global environment, the system actually observes q reflectors at the tth moment, and the coordinate value of the ith reflector in the platform frame is . The sampling time of the laser range finder is 0.1 s. Therefore, the theoretical position value of the reflector in the platform frame can be deduced by the position of the system at the th moment and the position of the reflector in the global environment.
If the difference between the theoretical value and the observed value is less than η, then the observed reflector is an effective reflector and is considered matched with the reference reflector as follows:
Therefore, a set A of effective reflectors at the tth moment can be obtained, and . Taking the mobile manipulator’s moving speed into account, the value of η depends on the actual situation. Through the use of the optimal triangulation localization algorithm based on angle measurement, the pose of the mobile manipulator can be calculated at the tth moment.
4.3. Experimental verification
The experimental data are obtained by the LMS 100 laser range finder with a scanning range of 270° and an angular resolution of 0.25°. The experimental platform is shown in Figure 19. The outside of the reflector is wrapped by reflective tape. In the experimental environment, five reflectors are placed around the system. Their global coordinate values are (0,995); (0, 0); (0, 1774); (2905, −2449); and (3956, −2032), and the unit is mm, as shown in Figure 20.
The optimal triangulation method based on angle measurement is used for validation by the repeatability of the system. In the stationary state of the system, the environment is scanned by finders. Each result is indicated by a red dot in Figure 21. The repeatability obtained by the trilateral method is nearly 18 mm, while the repeatability of the optimal method is only 9 mm. It can be shown that the optimal method is better than the traditional method.
The mobile manipulator moves in the direction of the arrow in Figure 19, and each time the system moves a certain distance, the localization system will perform an experiment, i.e., it will use the left rear finder to calculate the current position. An average of 30 samples is taken for each experiment.
Figure 22 shows the results of static localization accuracy. The maximum distance error is 18 mm and the maximum angle error is 2°, which satisfies the localization requirement of the system.
The mobile manipulator moves in the designated route, and it needs to constantly calculate and record its own position in the moving process. As shown in Figure 23, the trajectory of the moving system based on the localization method is smoother.
This chapter demonstrates the feasibility of a tracking and localization algorithm for mobile manipulators. The following conclusions can be drawn from this study: (i) In the detection of a reflector in the laser range finder frame, the angle repeatability of the reflector is better than that of the distance repeatability based on the feature extraction algorithm; (ii) The repeatability localization accuracy using the optimal triangulation method based on the angle measurement is nearly 9 mm, which is better than that of the trilateral method; (iii) The localization error of the system is 18 mm, which satisfies the localization requirement of system. Improvements in the location method based on reflectors, such as optimizing the layout of reflectors and the map of reflectors selection strategy for localization, are still needed.
In this chapter, through analyzing the roller deformation of the Mecanum wheel, the changed parameters of the motion equation of mobile system are found. The relative variation of the parameters in the motion equation of the Mecanum motion platform is solved by Monte Carlo analysis and interval analysis. Using the relative variation of the parameters to revise the motion equation, the displacement errors in different spaces in theory are solved for and compared with the measured displacement errors. From the comparison, both the methods are found to satisfy the system’s requirement. Then, the feasibility of a tracking and locating algorithm for mobile manipulator is demonstrated. The following conclusions can be drawn from this study: (i) In the detection of a reflector in the laser range finder frame, the angle repeatability of the reflector is better than that of the distance repeatability based on the feature extraction algorithm; (ii) The repeatability localization accuracy using the optimal triangulation method based on the angle measurement is nearly 9 mm, which is better than that of the trilateral method; (iii) The localization error of the system is 18 mm, which satisfies the localization requirement of system. Improvements in the localization method based on reflectors, such as optimizing the layout of reflectors and the map of reflectors selection strategy for localization, are still needed.
The method in this chapter is also used in the research of MoMaCo (Mobile manipulator in Construction), which can draw baseline for architectural decoration engineering as shown in Figure 24. The application result also verified the effectiveness of the method.