Precise 6DOF Localization of Robot End Effectors Using 3D Vision and Registration without Referencing Targets

Abstract

A method for detecting the precise 6-degree-of-freedom (6DOF) localization of robotic arms end effectors without referencing any additional feature target during in-line robot operation is introduced to facilitate precise robot positioning and monitoring. In this work, a 3D vision probe with digital structured-light projection is integrated with a wafer handling robot to perform online 6DOF location monitoring in semiconductor production. Precise alignment of the robotic arms end effector moving in the 3D operation space is realized by robust point cloud object alignment using regional surface area descriptors and the variant iterative closest point algorithm. Verified and confirmed by experimental tests, the developed method can achieve online 6DOF location monitoring with micron-level accuracy. Moreover, the proposed method can completely avoid the disadvantages of existing methods, namely relying on planar 2D images and demanding an additional target to be embedded with the end effector for localization, which reduces practical application, especially in an in-line operation environment. The major technical breakthrough of the present work is the target-free precise 6DOF localization of moving objects.

1. Introduction

Robotic arms are usually used in automated production lines to perform repetitive or hazardous operations to ensure high manufacturing quality. For instance, a wafer handling robot is usually used in the automated production line of semiconductor components for sending or taking wafers between cassettes at etching, deposition, and photolithography stations with demanding accuracy, repeatability, and reliability [1]. However, potential induced stress or deformation during continuous robotic arm operation may cause the end effector position and orientation to deviate from the position specified, which would result in the collapse of the processing unit with the wafer scrapped or crushed, as shown in Figure 1, thus incurring significant production losses. To avoid such an undesired scenario, this study proposes a robust and precise monitoring method for performing the 6-degree-of-freedom (6DOF) localization of robotic arms end effectors.

Figure 1.

Wafer damage caused by positioning deviation of robot end effectors during manufacturing.

Recent studies have investigated the problems related to position uncertainties of robotic arms end effectors and analyzed the life cycle of the robotic arm to improve the reliability of the wafer handling robot [2, 3]. However, in general, planar translation of end effectors has been measured using only 2D imaging; with depth information missing, 6DOF localization of end effectors cannot be realized.

To obtain accurate variations in both position and orientation of the robotic arms end effector, this study developed a 3D machine vision probe with point clouds measured for achieving 6DOF localization of the end effector. Point cloud registration is crucial to determining precise object 6DOF transformation between different locations in the 3D space and involves the coarse alignment stage and the refined model alignment stage. During coarse alignment, the object point cloud aligns only approximately with the target (reference) point cloud. Methods used include spin image [4], point signature [5], and regional surface area descriptor [6], with different levels of accuracy, efficiency, and robustness achieved. This initial matching between the object and the target model is then followed by fine model alignment for more precise matching of the two. The most widely adopted algorithm is the iterative closest point (ICP) [7], which uses singular value decomposition (SVD) to find the set of closest point clouds between the object and the target model. It refines the deviation between the corresponding points until the least-squares error is minimized.

This chapter presents a novel 6DOF detection method that uses a developed structured-light 3D scanner [8] to obtain the 3D information of the robotic arms end effector and then performs point cloud alignment to detect the 6DOF variations of the robotic arms end effector. The process of the proposed method is illustrated in Figure 2 and described in detail in Section 2.

Figure 2.

Process of the proposed method.

2. Methods

2.1 Point cloud preprocessing

2.1.1 Denoise algorithm

Optical measurement is susceptible to noise and outliers caused by the external environment, object surface material, and reflectivity, and these noises and outliers must be eliminated before point cloud registration. In this study, Statistical Outlier Removal (SOR) [9] was applied to neighboring points of each point in the point cloud cluster. First, the average distance of each point to their K nearest neighbors is calculated. The average distance ${\mathrm{d}}_{\text{mean}}$ and standard deviation σ are then used as the threshold for determining noise or outlier. The distance threshold can be expressed as:

$$d_{\mathit{\text{thresh}}}=d_{\mathit{\text{mean}}}+K\ast \mathrm{\sigma }$$E1

where K is a scale factor and is usually set to equal 1.0. Figure 3 illustrates the outlier determination and removal. First, the distances of the nearest K points ${\mathrm{P}}_{\mathrm{j}}$ (j = 0,1,2,3,…,k) of the point ${\mathrm{P}}_i$ are calculated. The average distance between ${\mathrm{P}}_{\mathrm{o}}$ and its neighboring point ${\mathrm{P}}_{\mathrm{j}}$ is ${\mathrm{d}}_{\mathrm{i}}$; if ${\mathrm{d}}_{\mathrm{i}}$ is greater than the distance threshold, ${\mathrm{d}}_{\text{thresh}}$, Po is regarded as an outlier and is thus removed from the point set P.

Figure 3.

Illustration of outlier removal.

Figure 4a shows the original point cloud, while Figure 4b shows the point cloud with the outlier removed with a multiplicity factor of 1.0 after searching the neighboring points.

Figure 4.

(a) Original point cloud and (b) point cloud after outlier removal.

2.1.2 Downsampling strategy

The structured-light measurement probe uses a high-resolution sensor of up to two megapixels; hence, the amount of data for the reconstructed point cloud is significant and dense, often reaching hundreds of thousands of points. It is thus necessary to effectively reduce the number of data points while strictly preserving the morphological characteristics of the object for evaluating pose variation.

The widely used point-cloud downsampling algorithms are voxel grid filtering and Delaunay Triangulation. Pixel is the basic unit that constitutes an image, and voxel means a cubic grid, which is the unit cube that constitutes a 3D space. For data reduction, the voxel grid filter replaces all the points in the voxel with the mass center of all the points. All the points in the voxel are represented by the n points belonging to the unit voxel, as shown in Figure 5.

Figure 5.

Illustration of voxel-grid filtering: (a) unfiltered voxel; (b) unfiltered unit voxel; and (c) the mass center of the unit voxel.

Delaunay triangulation [10] allows all input points to form a convex polygon (convex hull) [11], as shown in Figure 6, and all triangles within the convex polygon must satisfy the property of maximizing the minimum interior angle and the noncircularity of any four points. If Delaunay triangulation is applied to the 3D space, the four adjacent points can form a tetrahedron and create an external sphere, the so-called Alpha-ball, as shown in Figure 7. The red circle denotes the external sphere of the tetrahedron, and the excess points can be removed using the Alpha-ball algorithm [12], thus achieving data reduction while preserving the object’s surface characteristics.

Figure 6.

Illustration of convex hull formed by Delaunay triangulation.

Figure 7.

Illustration of alpha-ball [12] formed by applying Delaunay triangulation to 3D space.

As shown in Figure 8a, a precision ceramic gauge block with sharp edges was used as the test object to compare the above two-point cloud downsampling methods. The reconstructed result after outlier removal is shown in Figure 8b, and the number of points is enormous, totaling 300,495.

Figure 8.

(a) Original and (b) reconstructed gauge block after outlier removal.

Such a large amount of data typically requires a long computation time. Therefore, voxel grid filtering and Delaunay triangulation are performed in the proposed method to reduce the amount of 3D reconstructed point cloud data. Their performances are compared in terms of the integrity of object features retained. Figure 9a shows the result of the original point cloud data after voxel grid filtering; the total number of points becomes 9239, reduced by 96.9%; and Figure 9b shows the registration result with an average registration error of 40.8 μm between the original and downsampled point cloud, and the maximum error of 520 μm at the object’s corner edge. Figure 9a shows the result of the original point cloud data after Delaunay triangulation; the total number of points becomes 9253, reduced by 96.9%; and the registration result, shown in Figure 10b, has an average registration error of 10 μm and the maximum error of 90 μm.

Figure 9.

(a) Original point cloud after voxel grid filtering and (b) registration result.

Figure 10.

(a) Original point cloud after Delaunay triangulation and (b) registration result.

Comparing the two-point cloud downsampling methods revealed that Delaunay triangulation can achieve a much smaller registration error than voxel grid filtering, especially in the edge area of the measured object. Voxel grid filtering uses average filtering, which smooths out the edge features of the object. In contrast, Delaunay triangulation removes only unnecessary points, thus preserving the edge features of the object. Hence, using Delaunay triangulation to reduce the number of data points as inputs for the subsequent algorithm can obtain better accuracy when calculating pose variation and avoid the error generated by the preprocessing of point clouds.

2.2 Coarse alignment between scanned data and model object

This study used the regional surface area descriptor (RSAD), proposed by Chen et al. [6], to determine the geometric relation between 6DOF of two sets of point clouds measured on an object of two different locations and orientations. The method assumes that the same surface area distribution defined in the oriented bounding box (OBB) [13] of a point cloud set can be employed to detect the 6DOF of the object in 3D space. Thus, to find the most matched pose, the RSAD of the measured point cloud is iteratively compared with the feature descriptor generated by the 3D virtual camera from the model point cloud, which represents the target model or the original pose of the object. Figure 11 illustrates the matching method using the RSAD.

Figure 11.

Matching using regional surface area descriptor (RSAD).

2.2.1 Database generation

First, a coordinate system is established at the center of the model point cloud, and the virtual camera is located on the z-axis of the model point cloud coordinate system, as shown in Figure 12. The template point cloud is generated by rotating along the x-axis and y-axis of the model point cloud coordinate system with a fixed incremental rotation angle. As shown in Figure 13a–f, the template point clouds recorded by the virtual camera are rotated along the y-axis by 0, 60, 120, 180, 240, and 270 degrees, respectively.

Figure 12.

Location of the virtual camera in model point cloud coordinate system.

Figure 13.

(a)–(h) are model point clouds captured by the virtual camera at different viewing angles.

After the virtual camera records the template point clouds with different viewing angles, the surface area feature descriptors can be calculated for each template point cloud. First, the OBB of the template point cloud is calculated; it is the smallest 3D rectangular bounding frame that can be covered by the point cloud, as shown in Figure 14. The OBB thus obtained can be divided into $k_1\times k_2\times k_3$ sub-oriented bounding boxes (Sub-OBB) along its three main directions, as shown in Figure 15. The total number of sub-OBB equal to $v=k_1\times k_2\times k_3$, and the total number of points covered by the objects in a single sub-OBB is $n_v^p$. If n is the total number of points in the whole template point cloud, then the number of points in a sub-OBB $f_v$ can be expressed as:

Figure 14.

OBB of an object point cloud.

Figure 15.

Sub-OBB of an object point cloud.

$$f_v=\frac{n_v^p}{n}$$E2

where $n_v^p$ is the total number of points in the ${\mathrm{v}}_{\mathrm{th}}$ sub-OBB and n is the total number of points in the template point cloud.

According to (2), the percentage of point clouds in each sub-OBB can be calculated to obtain the regional surface area distribution of the whole point cloud, which is called the surface area feature descriptor FV and expressed as:

$$\mathit{FV}=\left\{f_0^p,f_1^p,\dots ,f_v^p\right\}$$E3

The surface area distribution of the object point cloud in Figure 14 is shown in Figure 16.

Figure 16.

Regional surface area distribution histogram of the object point cloud.

2.2.2 Feature matching

Feature matching involves two steps, which aim to obtain the closest pose between the model point cloud and the object point cloud. First, the OBB parameters between the object point cloud and the model point cloud are compared. If the above matching conditions are met, the surface area feature descriptors of the object point cloud and the model point cloud are compared. The feature matching process is summarized in Figure 17.

Figure 17.

Flowchart of feature matching process.

Each OBB is represented by three principal vectors and a corner. The matching process involves finding a template point cloud with an OBB size similar to the object point cloud. These two OBBs should satisfy the following equation:

$$d_{\mathit{\text{corr}}}=\frac{1}{3}\left\{\frac{‖C{C}_{M1}-C{C}_{O1}‖}{‖C{C}_{M1}‖}+\frac{‖C{C}_{M2}-C{C}_{O2}‖}{‖C{C}_{M2}‖}+\frac{‖C{C}_{M3}-C{C}_{O3}‖}{‖C{C}_{M3}‖}\right\}<d_{\mathit{\text{thresh}}}$$E4

where $d_{\mathit{\text{thresh}}}$ is the given adequate threshold, $C{C}_{\mathit{Mi}}$ and $C{C}_{\mathit{Oi}}$ (i = 1,2,3) are the three principal vectors of the model OBB and object OBB, respectively.

If (Eq. (4)) is satisfied, matching is successful. On the contrary, if $d_{\mathit{\text{corr}}}>d_{\mathit{\text{thresh}}}$, matching fails, indicating a significant OBB size difference between the object point cloud and the model point cloud. Assume that Figure 18a and b depict, respectively, the OBB of the object and model point cloud. According to details of their three principal vectors listed in Table 1, the OBB matching coefficient $d_{\mathit{\text{corr}}}$ between the two point clouds is 0.593. Hence, if the matching coefficient threshold is set as 0.2, then the matching between Figure 18a and b fails because $d_{\mathit{\text{corr}}}$ exceeds the given threshold.

Figure 18.

(a) OBB of the model point cloud and (b) OBB of the object point cloud.

	Model point cloud (CCM)	Object point cloud (CC0)
Principal vector of X-axis	(112.4,0,0)	(69.3,0,0)
Principal vector of Y-axis	(0,35.1,0)	(0,68.6,0)
Principal vector of Z-axis	(0,0,21.7)	(0,0,12.1)

Table 1.

OBB parameters of model and object point clouds.

Following this, to determine the best match between the model point cloud and the object point cloud measured using the 3D scanner, the feature descriptors of the model point cloud and the object point clouds measured at different viewing angles are matched using conventional normalized cross-correlation (NCC) [14, 15].

If the OBB matching satisfies (Eq. (4)), then the regional surface area descriptors of the two-point clouds are compared. The regional surface area descriptor shows the surface area distribution of the point cloud within its OBB. The similarity of regional surface area descriptors between the two point clouds can be determined using the NCC method [14, 15]. Assume that the regional surface area descriptor of the object point cloud is expressed as $F_O=\left\{f_v,v=0,\dots ,n_v\right\}$, and the regional surface area descriptor of the model point cloud is $F_M=\left\{f_v,v=0,\dots ,n_v\right\}$, then the NCC coefficient between the two descriptors $F_O$ and $F_M$ can be expressed as:

$$C\left(F_O,F_M\right)=\frac{{\sum }_{v=0}^{n_v}\left(f_v-\overline{f}\right)\left(f_{\mathit{Mv}}-\overline{f_M}\right)}{\sqrt{{\sum }_{v=0}^{n_v}{\left(f_v-\overline{f}\right)}^2\bullet {\sum }_{v=0}^{n_v}{\left(f_{\mathit{Mv}}-\overline{f_M}\right)}^2}}$$E5

where

$$\overline{f}=\frac{1}{n_v+1}\sum _{v=0}^{n_v}{f}_v$$E6

$$\overline{f_M}=\frac{1}{n_v+1}\sum _{v=0}^{n_v}{f}_{\mathit{Mv}}$$E7

If the NCC C(F₀, F_M) between the RASD of the model point cloud and that of the object point cloud is greater than the preset threshold, the two-point clouds have similar poses in the 3D space. The relationship between the two point clouds can be calculated using OBB parameters. Assume that the initial conversion matrix T_initial is the relationship between the two point clouds, as shown in Figure 19. Their conversion relationship in the space T_initial can be expressed as:

Figure 19.

Schematic diagram of the relationship between object and model point cloud.

$$\left[\begin{array}{cccc}{x}_c& x_c+v_{11}& x_c+v_{21}& x_c+v_{31}\\ y_c& y_c+v_{12}& y_c+v_{22}& y_c+v_{32}\\ z_c& z_c+v_{13}& z_c+v_{23}& z_c+v_{33}\\ 1& 1& 1& 1\end{array}\right]=T_{\mathit{\text{initial}}}\left[\begin{array}{cccc}{x}_{\mathit{Mc}}& x_{\mathit{Mc}}+v_{M11}& x_{\mathit{Mc}}+v_{M21}& x_{\mathit{Mc}}+v_{M31}\\ y_{\mathit{Mc}}& y_{\mathit{Mc}}+v_{M12}& y_{\mathit{Mc}}+v_{M22}& y_{\mathit{Mc}}+v_{M32}\\ z_{\mathit{Mc}}& z_{\mathit{Mc}}+v_{M13}& z_{\mathit{Mc}}+v_{M23}& z_{\mathit{Mc}}+v_{M33}\\ 1& 1& 1& 1\end{array}\right]$$E8

where

$$T_{\mathit{\text{initial}}}=\left[\begin{array}{cccc}{r}_{11}& r_{21}& r_{31}& t_x\\ r_{12}& r_{22}& r_{23}& t_y\\ r_{13}& r_{23}& r_{33}& t_z\\ 0& 0& 0& 1\end{array}\right].$$

2.3 Fine alignment

The iterative closest point (ICP) algorithm proposed by Besl [7] is a reliable and widely used point-cloud fine alignment algorithm. The ICP algorithm starts with two point clouds and an initial guess for their relative rigid-body transform and refines iteratively the transform by repeatedly generating pairs of corresponding points on the point clouds and minimizing an error metric. The flowchart of the ICP algorithm is shown in Figure 20.

Figure 20.

Flow chart of the ICP algorithm.

To further improve the computational efficiency of the ICP algorithm and optimize the alignment results of the 3D point cloud, many variants have been developed from the basic ICP concept, such as NICP [16], GICP [17], and Point-to-Plane ICP [18]. Rusinkiewicz [19] classified these variants as six stages in the ICP algorithm, illustrated in Figure 21, and indicated that the most critical stages that affect the convergence speed and the accuracy of the ICP algorithm are correspondence calculation and error minimization.

Figure 21.

Six stages of the ICP algorithm.

In this work, the classic ICP algorithm is further improved with the normal shooting method used in the stage of correspondence calculation instead of finding the closest point. Moreover, in the stage of metric error selection, the point-to-plane distance is applied instead of the point-to-point distance. The concept of the normal shooting method and point-to-plane distance is shown in Figures 22 and 23, respectively.

Figure 22.

Correspondence calculation is achieved by (a) the conventional method by finding the closest point and (b) the proposed method using normal shooting.

Figure 23.

Error metric selection is achieved by (a) the conventional method using point-to-point distance and (b) the proposed method using point-to-plane distance.

In the proposed approach, test data make up the measured point cloud obtained by measuring a robotic arms end effector with the developed structured-light 3D scanner. In contrast, target data make up the measured point cloud transformed by a known transformation matrix, as shown in Figure 24a. The white point cloud is the original point cloud, and the green point cloud contains target data transformed by a known transformation matrix. An example of the end effector alignment result is shown in Figure 24b. It is important to indicate that no artifact calibration is deployed in the detection and alignment of the method. This is significantly different from the conventional method, which generally utilizes a known or pre-defined artifact for alignment and calibration purposes. The alignment object used in the method is only the robot arm itself.

Figure 24.

Fine point-cloud registration: (a) test data (white) and target data (green); (b) registration result after fine alignment.

Figure 25 compares the performance between the classic ICP algorithm and the proposed method. As can be seen, the proposed method takes only 0.116 seconds for the mean squared error (MSE) of point-to-point distance to converge to less than 1 μm, while the classic ICP takes a longer time of 0.659 seconds to reach convergence, yet a higher deviation of 230 μm. These results indicate that the proposed method has a faster convergence speed and higher accuracy than the classic ICP algorithm.

Figure 25.

Comparison of performance between conventional ICP and the proposed method.

3. 6DOF measuring system

3.1 Experimental setup

Figure 26 displays the setup of the developed system for detecting 6DOF variations of the robotic arms end effector, which comprises the developed structured-light 3D scanner, a wafer handling robot, and a computer. The specifications of these three modules are listed in Table 2.

Figure 26.

Experimental setup.

Wafer handling robot
Positioning accuracy	± 25 μm
Wafer sizes	75 to 300 mm
Structured light 3D scanner
Working distance	250 mm
Measurable depth range	100 mm
Volumetric measuring accuracy	60 μm
Computer
CPU	Intel Core i7–8700
RAM	32.0 GB

Table 2.

System specifications.

3.2 Design of 3D structured-light measurement probe

The 3D structured-light measurement probe, as shown in Figure 27, is designed according to the triangulation measuring principle. As can be seen, it comprises a structured-light projector and a camera, with the projector’s optical axis and the camera’s optical axis intersecting at a point and forming a triangular relationship with each other. Moreover, the greater the angle between the projector and the camera, the better the depth resolution is. Under a trade-off between the effect of light shielding and depth resolution, this system adopts an included angle of 30°. Moreover, to avoid blurring or poor contrast of the image taken by the sensor module, the measurement depth range is defined as the area where the contrast between the nearest focus surface and the farthest focus surface of the camera lens in the modulation transfer function (MTF) diagram, was maintained above 70%. As shown in Figure 28, the area is a trapezoid space, and the camera’s field of view is gradually enlarged from the front to the back focal plane.

Figure 27.

Schematic diagram of the 3D structured-light measurement probe.

Figure 28.

Schematic diagram of the structured-light probe measurement range.

Figure 29 is the engineering drawing of the structured-light measurement probe designed by Autodesk Inventor. As shown in the actual image (Figure 30), the internal structure includes a DLP projection module, an image sensor, a perspective lens, Arduino Mega 2560, and a breadboard. Arduino Mega 2560 is to synchronize the trigger signal between the camera and the projector. The measurement probe’s outer shell (Figure 31) is made of high-strength aluminum alloy to avoid deformation by external forces. The two side covers of the measurement probe are designed for heat dissipation so that the heat generated by the hardware device can be removed through the heat dissipation holes.

Figure 29.

Engineering drawing of the structured-light probe: (a) internal structure and (b) external view.

Figure 30.

Actual internal image of the probe.

Figure 31.

Actual appearance image of the probe.

4. Experimental results analysis and discussion

Usually, when the robotic arm moves to a specified position, it generally vibrates and shakes due to undesired external force or load. Therefore, upon reaching the working position, it often stops for a short time till the shaking ends and the arm becomes stable. In this study, the end effector of the robotic arm is measured in three dimensions during the standstill time, and the morphological information of the end effector is recorded and compared with the prebuilt database to find the point cloud closest to the pose and obtain the initial transformation matrix. The position and pose variation of the end effector are calculated and displayed as error diagrams in the measurement software’s graphic user interface (GUI), as shown in Figure 32.

Figure 32.

Pose variation detection during the robotic arm.

4.1 Database generation

Coarse registration requires database creation for prematched objects before measurement. In this experiment, the 3D point cloud information of the robot arm end effector at the working position was used as the model point cloud, the virtual camera recorded the template point clouds at different viewing angles, and the surface area feature descriptors were calculated and stored in the database as the basis for subsequent comparison.

Step 1: Move the robot arm to the working position, as shown in Figure 33.

Figure 33.

The robot arm is moved to the work position.

Step 2: Obtain the 3D constructed point cloud of the robotic arms ends effector by the scanning probe. Figure 34 shows the measurement range of the scanning probe. Figure 35 shows the image of the robotic arm captured by the camera, and Figure 36 shows the raw data of the 3D reconstructed point cloud.

Figure 34.

Measurement FOV of the probe.

Figure 35.

Image of robot arm end effector captured by the measurement probe.

Figure 36.

Reconstructed point cloud of robot arm end effector.

Step 3: Preprocess point cloud data. Figure 37 shows the point cloud after noise removal, and Figure 38 shows the downsampled point cloud.

Figure 37.

Point cloud after outlier removal.

Figure 38.

Point cloud after downsampling.

Step 4: Create a multiview template point cloud from the model point cloud, as shown in Figure 39a–h.

Figure 39.

Multiview template point clouds.

5: Calculate the regional surface area descriptor of the corresponding template point cloud, as shown in Figure 40a–h.

Figure 40.

RSAD of the corresponding template point cloud.

4.2 Experimental results analyses and discussion

To verify the actual performance of the developed method, a robotic arm with an end effector was repeatedly moved 100 times to observe its variations in position and orientation. As shown in Figure 41a and b, the robotic arm repeatedly moves between position A and position B. In the test, the developed 3D scanner was integrated with the system to measure the robotic arms end effector for its 3D point clouds whenever the robotic arm reaches position B.

Figure 41.

Different measured positions of the tested robotic arm with its end effector.

In the test, the 6DOF variations of the robotic arms end effector are compared with the measured point cloud whenever the arm moves from position A to B and stops at position B. Figure 42a shows the image captured using the developed 3D scanner when the robotic arm reaches position B and Figure 42b is the original reconstructed point cloud. Figure 43 shows point-cloud registration results before and after alignment without referencing any target.

Figure 42.

(a) Image captured using the developed 3D scanner when the robotic arm reaches position B. (b) Reconstructed point cloud at position B.

Figure 43.

Point-cloud registration before (left) and after (right) alignment without using any calibration artifact.

Figures 44 and 45 show the dynamic variations in position and angular orientation, respectively, obtained after 100 tests. As can be seen, for the robot end effector in the x-, y-, and z-axis, the averaged pose variations are 0.039 mm, 0.003 mm, and 0.005 mm, respectively. In contrast, the averaged orientation variations are 0.009°, 0.029°, and 0.009°, respectively. These results indicate that the tested robot end effector achieved positioning with micron accuracy in the worst scenario. The experimental results also show that the translational error of the x-axis and the rotation error of the y-axis may increase significantly with the operation time of the robotic arm. This discovery can lead to possible inspection and maintenance of the robot to be arranged to ensure robust manufacturing operation and avoid any potential catastrophic damage.

Figure 44.

Dynamic variations in positioning of robot end-effector along x-, y- and z axes.

Figure 45.

Dynamic variations in angular orientation of the robot end effector along x-, y- and z axes, defined as roll, pitch, and yaw angular errors.

4.3 Error analysis of robot arm pose variation

The sources of error in the analysis of the robot’s pose variation include the error of the structured-light measurement probe, pose detection algorithm, and the robotic arm itself. In addition to the positioning error of the robot arm itself, which is known from the technical specifications provided by the original manufacturer, the positioning repetition of the robot arm is 25 μm. In contrast, the reconstruction error of the measurement probe and the error of the pose detection algorithm must be found through experiments. The ideal point cloud data is used as the benchmark to quantify how error sources affect measurement results. The principle is to transform the known point cloud data through a known transformation matrix and to obtain the pose variation using the proposed pose detection algorithm. The method obtains the transformation matrix between the transformed point cloud data and the original point cloud data. It compares the known transformation matrix with the transformation matrix obtained using the proposed algorithm to quantify the error. The error of the structured-light measurement probe is calculated by repeatedly measuring the pose variation of the robot arm at a fixed position, and the remaining error after deduction from the algorithm error can be regarded as the error of the structured-light measurement probe.

The translation error $T_{\mathit{err}}$, defined as the absolute difference between the translation component of the known transformation matrix $T_{\mathit{\text{true}}}$ and the translation vector of the transformation matrix $T_{\mathit{alg}}$ obtained using the proposed pose detection algorithm [20], is expressed as:

$$T_{\mathit{err}}=‖T_{\mathit{alg}}-T_{\mathit{\text{true}}}‖$$E9

The measured point cloud is converted in the 3D space to quantify the algorithm error using a known transformation matrix, as shown in Figure 46. The object to be measured is a 3D printing hammer. The red point cloud is the original point cloud, while the yellow one is transformed from the original point cloud in 3D space using 30 different known transformation matrices. Figure 47 shows the translation errors obtained using the proposed pose detection algorithm. The average translation error of the 30 transformations is 0.007 mm.

Figure 46.

Original (red) and transformed (yellow) point clouds.

Figure 47.

Translation error was obtained using the proposed pose detection algorithm.

System uncertainty of the 3D structured-light measurement probe will cause the measurement results of the same object to deviate. The robotic arm was scanned by the measurement probe 30 times to detect pose variation. For a fixed robot arm, the pose variation should be zero. Otherwise, the measurement obtained after deducting the algorithm error can be regarded as an error attributed to the structured-light measurement probe. Figure 48 shows the translation error obtained for the 30 scannings. The average translation error contributed by the structured-light measurement probe was 46 μm.

Figure 48.

Translation error contributed by the structured-light measurement probe.

Table 3 summarizes the error source distribution of the robot arm pose variations. As can be seen, the error comes mainly from the structured-light measurement probe. Light perception by the image sensor in the measurement probe may vary for the same object even under the same conditions, thus causing differences in measurement results. Experiments detected an average pose variation of 46 μm from this source, accounting for 58.5% of the total error. The next major source of error is the robotic arm itself. System errors lead to the difference in the position specified by the controller and the actual position reached. According to the original technical specifications, the pose variation is 25 μm, which accounts for 31.8% of the total error. A minor source of error is the pose detection algorithm, with pose variation attributed to floating point and matrix calculation errors. An average pose variation of 7 μm was detected, accounting for 9.6% of the total error.

	Error (μm)	Percentage (%)
Measurement probe	46	58.5
Robotic arm	25	31.8
Pose detection algorithm	7.6	9.6
Total	78.6	100

Table 3.

Error budget analysis of pose variation of the robotic arm.

5. Conclusions

This chapter introduces a novel method for measuring 6DOF variations of the robotic arms end effector using regional surface area descriptors and variant ICP to monitor accurate positioning and orientation. Experimental test results confirmed that the proposed method can accurately position and detect three angular errors, namely pitch, yaw, and roll angle, without referencing any known artifact for system alignment. A comprehensive measurement uncertainty was also performed to identify possible measured sources. From the measurement results, it reveals that the 3D optical probe can achieve 60 μm measurement accuracy and 100 μm depth resolution within the entire measurement range of 100 mm. On the other hand, for the automatic detection of the position and orientation on the robot arm end-effector, the experimental results show that it achieves the positioning accuracy of 100 um in the X, Y, and Z positions within the entire measurement range. Most importantly, it was also verified that the angular detection accuracy can be kept within 0.01 degrees for the pitch, yaw, and roll angular motions. The developed method can be widely applied to precise 6DOF detection of arbitrary objects moving dynamically in 3D space. Its high-accuracy 6DOF monitoring with target-free 3D image registration capability is a significant technical breakthrough in automated optical inspection (AOI) and precision metrology in manufacturing.