Development of a Versatile Modular Platform for Aerial Manipulators

2.1 Drone electric power units

The developed octarotor drone has a take-off weight of 40 kg and a 30 min flight time. The drone’s frame was designed and fabricated in collaboration with Vulcan UAV^©. The authors’ input on this aspect is related with both extending the bare-bone design of Vulcan to accommodate for payload carriage, as well as fabricating the final prototype and mounting all the additional modalities mentioned in the sequel. The backbone structure consists of three ø 25 mm, 1200 mm length aluminum tubes in a triangular cross sectional configuration. Four 575 mm length aluminum rectangular arms attached at each end of this structure and carry two motors in a coaxial configuration. The arms are fixed to the main frame using a 5 mm thick carbon plate. The resulting “H-frame” configuration can be visualized in Figure 1.

Although the lower motor provides 25% less thrust [11] it offers some redundancy against single motor failure. The selected 135 KV KDE^© brushless motors coupled with ø71.12 cm custom designed carbon propellers, collectively provide 37.2 kg of thrust at 50% throttle input. The extra thrust can be used for rapid maneuvering of the drone and for exerting forces by the aerial manipulator shown in Section 5.

Power is provided by a 12S 22 Ah LiPo battery pair connected in parallel to the Power Distribution Board (PDB). At 50% thrust with full payload while hovering, the octarotor’s motors sink 11.7 A each, resulting in a flight time of 2×2211.7×8×60min=26min.

Two carbon rods of ø12 mm are fixed at the underside of the mainframe tubes for payload carriage. The maximum payload weight is 30 kg and can be easily dismantled from the main frame using quick release clamps. Similarly, the retractable landing gear assembly is attached with these clamps to the main frame tubes for enhanced modularity, as shown in Figure 2. The gear can retract within a 45°−80°angle window using a Pulse Width Modulation (PWM) signal, provided by the FCU’s rail pins, with a 50 Hz switching frequency. The landing gear operation is achieved via the MAVlink protocol command set [12].

Additional power for peripherals and sensing modalities can be supplied through a dedicated 750 W buck converter, mounted on the payload carrier assembly, as shown in Figure 3. The converter is contained within a custom 3D printed case and standard Unmanned Aerial Vehicle (UAV) XT30,XT60connectors protrude to provide 24, 19 and 12 V respectively to the end-user.

2.2 Flight command unit and related software

The PixHawk Cube FCU was selected [13] featuring triple redundant dampened Inertial Measurement Units (IMUs), with a modular design and industrial standard I/O connectors. Additional telemetry and R/C circuits are deployed to enable monitoring and intervention and comply with flying regulations.

The Here+Global Navigation Satellite System (GNSS) [14] with Real-time kinematic (RTK) capabilities was selected for outdoor navigation and placed on top of a carbon fiber pole at a height of 35 cm from the main frame’s top plane. For immunity to electromagnetic interference, the primary magnetometer of the flight controller is selected to be the build-in magnetometer module of the GNSS receiver.

A high processing power 8th generation Intel NUC i7-computing unit with 32 GB RAM and 1 TB SSD, shown in Figure 3, was mounted symmetrically to the buck converter on the underside of the main frame. This 90 W computing unit allows for online computations on demanding tasks such as the visual object tracking methods of Section 4, as well as the easy development of autonomous flying applications.

On the software side, the ArduCopter flight stack [15] was selected to run on the FCU. The pose estimation is carried through a sophisticated Extended Kalman Filter (EKF) at 400 Hz. The Intel NUC companion computer is serially connected to the FCU at a baud rate of 1 Mbps and the communication packages are following the MAVlink protocol. The NUC’s operating system was Ubuntu Linux 16.04 and all applications are developed through the Robot Operating System (ROS) and MAVROS [16] middleware with a 50 Hz refresh rate.

The developed drone without any payload can be visualized in Figure 4.

3.1 Drone outdoor localization using RTK GNSS

The RTK enhancement feature of GPS is used for outdoor localization purposes. This is due to the more precise positioning [17] because the of the GPS satellite measurements’ correction using feedback from an additional stationary GPS module. The disadvantage of such systems is that their use is bounded to a significant pre-flight setup time which is inversely proportional to the achieved accuracy (cm range).

Although the internal loop of the flight controller operates at 400 Hz, the GPS receiver streams data at a lower rate of 5 Hz. In popular flight software such as ArduPilot, the aforementioned rate needs to be taken into consideration by the underlying EKFs running by the FCU. A typical comparison of the achieved accuracy using a drone in a hovering state can be seen in Figure 5.

The drone was flown in a hovering position with the RTK GPS module injecting measurements to the flight controller and the output of the FCU’s EKF was compared with and without the presence of the injected RTK measurements. The red line represents the EKF’s output based solely on the GPS signal, whilst the blue line indicates the same output when RTK correction (using a 30 min warmup period) is injected on the FCU.

The standard deviation was computed equal to 0.74 m, 0.47 m and 0.27 m for X, Yand Zrespectively when no RTK correction was applied. Contrary to this, the same values with RTK injection were computed to equal 0.05 m, 0.02 m and 0.23 m respectively. It should be noted that there is no significant improvement in the Z-direction, indicating the need to use either a barometer or a laser sensor for ground clearance measurements.

3.2 Drone indoor localization

During indoor navigation: a) the lack of GPS guidance, b) pressure changes affecting the barometric sensor, and c) power lines affecting compass accuracy can severely affect the output of a FCU. With only the accelerometers and gyroscopes being unaffected, the injection of an external feedback source to the FCU is considered essential. Such feedback is usually based on visual techniques, such as those presented in [18, 19].

For experimentation purposes, the used Motion Capture System (MoCaS) [20] injects measurements in the ArduCopter flight stack. The system comprises of 24 Vicon cameras uniformly scattered within an orthogonal space of 15×5×8=L×W×Hm. The utilized system allows simultaneous tracking of 100 objects at 120 Hz with sub-millimeter accuracy. Despite MoCaS’s high refresh rate, the ArduCopter flight stack at the FCU accepts external positioning data at a 4 Hz streaming rate.

The utilized ROS software at the MoCaS operates at 25 Hz and can efficiently wirelessly stream the measurements to the drone’s FCU. The latency time td=tVC+tCFCU+tFCU, where tVCtCFCUis the delay of data streaming to the companion computer (FCU), and tFCUthe delay of processing the data on the FCU. In the developed system typical measured values are tVC≃5ms, tCFCU=20msand tFCU=40ms, resulting in td≃75ms.

Because of the MoCaS’s efficiency, its weighing to the EKF is ten times larger compared to the GPS’s weight when flying outdoors. Subsequently, the efficiency of the implementation is assessed by comparing EKF’s position output with and without MoCaS’s feedback injection. In Figure 6 the drone’s position error (in each axis) between the aforementioned two quantities is visualized, where the red, green and blue lines represent the error along the XYand Zaxes respectively.

Real time pose tracking is satisfactorily achieved and minor differences are attributed to the EKF’s weighting of the accelerometer and gyroscope measurements during calculations.

4.1 Environment awareness using a spherical camera

Rather than using several cameras with a limited Field of View (FoV) to observe the surrounding space, a 360° FoV camera [25] is used. The spherical camera records images in a “spherical format” which is comprised of two wide-angle frames stitched together to form a virtual sphere [25]. The image can be rectified to the classic distortionless rectilinear format of a pinhole camera [26]. However, due to the nature of the “spherical format,” it is preferable to split the image into smaller segments and rectify each one to achieve results closer to the pinhole camera model. Instead of splitting into equal sized square segments [27], each image is split into tiles based on orientation-independent circles. With every tile having a different a-priori known calibration, the rectification can be carried out for each one independently, without high computational cost. By applying the solution and rectifying the image in Figure 7, for a selection of N=12tiles, the resulting rectified partitions are visualized in Figure 8.

For the case of collaborating drones, it is assumed that each one carries passive markers for visual recognition. Subsequently, the rectified images are processed for identification of these markers [28, 29, 30, 31] thus estimating the neighboring drone’s pose. For improved pose extraction, the solution of a multi-marker rhombicuboctahedron formation arrangement [32] is assumed to be present in each target.

The experimental setup for evaluation consists of the spherical camera mounted in a 2.7 m protruding stick, which subsequently is mounted to the underside of the octarotor using the generic mount base discussed in Section 5. A rhombicuboctahedron arrangement with markers at its faces is attached to a DJI-Mavic drone. Both UAVs were located within the MoCaS test volume, as shown in Figure 9. The quadrotor drone was flown in a randomized trajectory near the vicinity of the octarotor.

In Figure 10 the relative 3D-flight path between the drones is presented. The results recorded from the MoCaS and the visual ones are shown, where for the cases of detecting the marker the relative accuracy these measurements was 2.2 cm respectively.

4.2 Visual object tracking using a pan-tilt-zoom camera

Having identified the adversary or collaborating drone, a PTZ-camera is utilized to track its motion. This Visual Object Tracking (VOT) problem is challenging when the drone is occluded, thus Long Term Efficient (LTE) algorithms are sought for moving objects. Despite the development of Short Term Efficient (STE) algorithms [33] using either correlation methods or deep learning ones, an initial bounding box containing the target is required. In the authors’ case, the developed VOT algorithm employs two methods relying on a comparison: (a) between the tracking of the points transformed based on the PTZ-parameters and those using an optical flow, and (b) between the homography matrix transformed points and the optical flow.

The first method is based on the PTZ known motion and IMU’s acceleration and gyroscope measurements (Figure 11), in order to estimate the motion of the pixels due to the motion of the camera in relation to the surroundings [34]. An IMU with triple accelerometers, gyroscopes, and magnetometers is attached to the PTZ-camera, as shown in Figure 11. While the enhancement provided by the PTZ camera allows for efficient VOT, the need to control its parameters (pan, tilt, and zoom) while placed on a floating base and at the presence of several occlusions needs to be addressed.

The objective is to provide the bounding box ptof the approaching drone from the attached camera to the “Tracking drone” as shown in Figure 12. The IMU’s sensors are sent to an embedded EKF to compute the camera’s pan and tilt angles in the global coordinate system (and their angular velocities) at a 100 Hz rate. The angular velocities are used to compute the optical flow, and the angles are used for VOT purposes.

A GPU-based background subtraction technique eliminates the background pixels leaving only the moving object pixels. The bounding box encapsulates all pixels of the moving drone and the pan and tilt angles are adjusted to position the centroid of the moving bounding box at the image’s center while the zoom is adjusted to enlarge this box. The communication between the i7-minicomputer and the PTZ-camera is shown in Figure 13, while the VOT algorithm is shown in Figure 14.

The feature points are recognized in each frame and the transformation matrix between successive frames follows along [35]; the formulas provide the transformation based on the PTZ-parameters and an augmentation is needed to account for the camera’s translation, as provided by the on-board accelerometers. The pixels that correspond to static background objects will follow the predicted motion by the camera motion and coincide to the positions predicted by an optical flow based estimation, while the rest will be classified as belonging to moving objects of interest (Figure 15). The computations for the optical flow parallels that of the Lucas-Kanade method using a pyramidal scheme with variable image resolutions [36]. The basic optical flow premise is to discover the positioning of an image feature in the previous frame, in the current frame captured by the camera.

The second method is relying only on visual feedback and homography calculations [37] between two successive frames and does not require either the PTZ or the IMU-measurements, as shown in Figure 16. Initially a set using “strong image features” is identified on the previous camera frame and an optical flow technique is used to estimate the position of the features in the current frame. The method involves the discovery of special image areas with specific characteristics.

The algorithm used for finding the strong corners image features relies on the GPU-enhanced “goodFeaturestoTrack” [38]. Under the assumption that the background is formed by the majority of the pixels, a homography is calculated that transforms the features positions from the previous to the current frame; these correspond to the background pixels. The previous frame features are then transformed using the homography to get their position in the current frame. Herein, it is assumed that the background points transformed with the homography will coincide with the estimated ones by the optical flow, while the moving objects’ features estimated by the optical flow will diverge from the homography transformed pixels.

One downside of the technique is that when the tracked object remains static and blends with the background it is unable to identify it. In this case, a fast correlation-based STE-tracker relying on the MOSSE algorithm [39], is also used in order to estimate the drone’s position until new measurements of a moving drone are available. Several more robust but slower tracking algorithms were evaluated, including the KCF [40], CSRT [41], MIL [42], MedianFlow [43], TLD [44], and the MOSSE-algorithm was selected because of its fast implementation (600 Frames-per-Second (FpS)). A Kalman prediction scheme [45] was used to predict the bounding box and the one obtained from the MOSSE in the presence of noisy measurements of the moving object center, using a 2D-constant acceleration model for the estimated tracking window.