Vision-Based Path Finding Strategy of Unmanned Aerial Vehicles for Electrical Infrastructure Purpose

Abstract

In this chapter we present the development of automated visual inspection systems for electrical infrastructure. The inspection is performed using images acquired with an unmanned aerial vehicle (UAV). Through automated inspection routes, the state of the infrastructure can be evaluated and then the appropriate correcting measures be taken. The monitoring of power lines can be done using passive sensors such as cameras or active sensors such as light detection and ranging (LIDAR) cameras, image processing techniques, computer vision and control systems can then be used. Additionally, a three-dimensional (3D) reconstruction process is possible using images either offline or during the monitoring. An UAV with an onboard embedded computer is used to execute the computer vision and path planning algorithms. The work done shows that the proposed strategy aids in the automation of power line inspection.

1. Introduction

There exist numerous applications for UAVs that include autonomous navigation, tracking and 3D reconstruction [1, 2, 3]. Electrical infrastructure inspection with aerial robots, especially transmission line inspection, is very important since it can minimize costs, risk and logistic problems that usually are associated with manual inspection [4]. In this context the use of new technologies for power line inspection such as UAVs can bring great benefits. The common methods for power line inspection are manual inspection, manned helicopters and UAVs as shown in Figure 1.

Figure 1.

Example of different methods for power line inspection.

In the work presented in [5], three aspects have been shown: (1) strategies for risk administration in high power line corridors, (2) selection of suitable platforms for sensor location and (3) data processing techniques for identifying vegetation.

One of the principal problems to resolve in this field is the image distortion/noise due to camera movement/vibration, which can be mitigated with the use of the gimbal stabilizers and a mechanical vibration isolator under the flight controller.

A path planning process for electrical infrastructure inspection requires to consider the detection of power lines and electrical towers because these elements can generate a corridor where an autonomous system can perform the inspection task [5].

An important aspect of autonomous navigation systems is the collision avoidance [6, 7]. In a path planning process for automatic inspection of electrical infrastructure, it is necessary to be able to avoid electrical towers and power lines. The first task for accomplishing this goal is an object detection process that will be discussed in this chapter.

The access throughout the corridor is very important since it is necessary in order to inspect the area surrounding power lines. It must be free of obstacles and vegetation.

As a part of the inspection process, an important task is the detection of elements of the electrical infrastructure; this is achieved by using computer vision techniques such as object detection. The common objects present in the electrical infrastructure scene are the power lines and electrical towers. An additional task is the 3D reconstruction of the elements of the scene by using the captured images.

2. Power line detection

In this section, we show different methods for the detection of electrical lines through image processing and computer vision, which include methods for detection of rectilinear segments and catenary. Also, the use of machine learning is presented.

2.1 Line detection process

There are different methods for line detection [8, 9, 10, 11]. Some of them are based on graphics processing unit (GPU) approaches and geometrical considerations [12, 13, 14, 15] that can be used in the context of power line detection. It is important to note that line detection methods based on monocular images present better results in uniform background sceneries.

For the detection of rectilinear long segments from images taken from a top-down view, the process can be composed of the stages shown in Figure 2.

Figure 2.

Stages of a rectilinear process detection.

As this process cannot differentiate the power lines from other lines presented in the scene, there exists the possibility of using machine learning to reduce detection errors or improving the power line detection.

2.1.1 Machine learning method

The recognition system has to be trained with real power lines; after that the system must be able to recognize or select the power lines in a scene. In the first stage, it is necessary to define what lines are electrical lines. This is done by using an application for labelling as shown in Figure 3.

Figure 3.

Power line labelling.

This system operates in two modes, training and detection, as shown in Figure 4.

Figure 4.

Process for line recognition system, training and detection.

The training mode begins with an edge detector such as Sobel, Prewitt, Canny or Edge drawing. After that, different line detection methods can be used for detecting a representative set of lines present in the scene. The dataset is obtained by labelling (tagging), manually, the power lines in each image in order to select only the lines that correspond to power lines in the training mode as a true example. For this reason, an application with a graphical user interface (GUI) for selecting lines over a copy of the real image can be used (Figure 3).

The overall detected lines are compared with the labeled lines in order to differentiate the positive and negative samples. The positive samples are power lines that overlap the previously labeled lines. The negative samples are other lines detected in the scene that are not power lines. This corresponds to lines that have not been tagged.

The overlapping between the tagged lines and detected lines that are not power lines must be zero.

After that, a feature extraction stage is performed by using HOG descriptors [16], which are computed for the selected lines on the labeled dataset. This is done in spaced squared windows centred in the lines. In Figure 5, the extraction of the HOG descriptor in windows across a labeled power line is shown.

Figure 5.

Extracting HOG descriptor in lines.

Finally, the obtained descriptor values in the previous stage are the input data for the classifier. The SVM classifier is trained with this input data using a sigmoid kernel.

The detection mode has to be used after a training mode. The objective of this stage is to detect the power lines using the previously trained classifier.

This begins with segmenting and detecting lines as in the previous mode. After that, HOG descriptors are computed across the detected lines using squared windows as were done in training mode. This information is gathered for the classifier.

In Figure 6, the detection of all linear elements using a conventional line detection method is shown. The results of the machine learning method are shown in Figure 7.

Figure 6.

Line detection in the scene.

Figure 7.

Result of a machine learning method.

Finally, the SVM is evaluated with the obtained descriptor data. Line segments which pass this evaluation are power line candidates. Another possibility is to use deep learning methods and contextual information in order to improve the detection [1].

2.2 Catenary detection

Most of the works on power line detection are focused in straight line detection. Nevertheless, the electrical infrastructure is composed of catenaries which are generated when a flexible cable is suspended between two poles or towers. This type of object appears in images of electrical infrastructure taken from non-top-down views, which could be obtained using manned aircraft or UAVs. Different methods for catenary detection that includes the use of matching filters, line segment pool and a graph-cut model as is shown in [17] exist, also using geometrical considerations and data structures of segment concatenation [18]. The results of catenary detection based on a segment concatenation are shown in Figure 8.

Figure 8.

Catenary detection based in a segment concatenation.

3. Tower detection process

The transmission towers are important elements of the electric transmission system. They require maintenance of its components such as the isolators and the power line connections.

The tower detection process is done mainly using computer vision techniques based in machine learning methods such as neural networks, SVM [19, 20] and recently deep learning [1]. The process requires a training stage. In this case a classifier is trained using a set of labelled images. A manual tool is used for labelling, this is for selecting the region of interest (ROI) where the tower is located as shown in Figure 9. In the selected ROI, a set of descriptors is extracted. The descriptors are the input to the classifier. After that, a detection stage operates with frames of videos. The linear information of the scene can be obtained using line detection methods. This can be useful to simplify the scene. The tower detection process is composed of two stages, training and detection, as shown in Figure 10. The result of the tower detection process is in Figure 11.

Figure 9.

The ROI selection of an electrical tower.

Figure 10.

Training and detection stages for tower detection.

Figure 11.

Tower detection examples using SVM and a grid of descriptors.

For an autonomous inspection system based on UAV, the tower can be a distinctive element for the navigation process. However, it is an element that may be at risk of collision. Autonomous systems must be prepared to use towers as a reference area and also to avoid collisions with them.

The recent advances in computer vision methods for object detection towards the use of deep learning methods. These algorithms can be implemented on onboard computers provided with GPU (graphics processing unit) for accelerating computation [1].

4. Autonomous navigation process

Vision-based autonomous navigation for UAVs is a complex process that requires short computing times and accurate measurements in order to provide suitable and safe control commands to the device. The UAV navigation requires real-time measurements to produce a response within a specified time (at least 100 ms); otherwise, severe consequences including failure may affect the device. The simulation of a control system for a fixed-wing UAV that uses vision-based navigation for power line tracking is presented in [21]. In a previous work, the simulation of a visual-based navigation process for power line following in a 3D environment using a closed loop control was presented in [22].

Two pictures of the simulator of autonomous navigation using power line detection are shown in Figure 12.

Figure 12.

Simulation of autonomous navigation.

This camera provided images that are suitable for the line detection process. The frame rate permits closed loop control. The UAV system consists of a set of related components that are shown in Figure 13. The main components of the system are the UAV flight platform; the flight controller; the sensors that include GPS and inertial measurement unit (IMU) that includes three-axis magnetometers, gyroscopes, accelerometers and compass; camera; and the onboard computer to run the developed software. In this system, the flight controller receives setpoints from the onboard computer and sends sensed information to it. The vision sensor (camera) sends frames of images to the onboard computer.

Figure 13.

UAV system.

Different kinds of missions for power line following and terrain inspection can be established. The main stages of a complete mission are shown in Figure 14.

Figure 14.

Autonomous mission process stages.

5. 3D reconstruction of electrical infrastructure

Effective and efficient generation of 3D models from a set of 2D images is a well-studied problem in the literature and the principle of numerous computer vision applications. The keypoint detection and the 2D descriptor extraction are the first steps in the reconstruction process followed by the matching. There are different 2D descriptors such as SIFT, ORB, BRISK and FREAK that can be used in the context of 3D reconstruction using structure from motion (SFM). From the study [3], it can be concluded that it is possible to use the aforementioned descriptors in electrical tower reconstruction context. Also, the results shown that the SIFT descriptor presents the best performance in the generated cloud of points, but it spends more time than using other descriptors. Another good option is the use of the ORB descriptor. In Figure 15, a result using SIFT is presented.

Figure 15.

Results of 3D reconstruction of electrical infrastructure.

Current developments tend towards the use of other types of sensors such as LIDAR whose information can be merged with information from cameras with different spectra.

Also it is important to develop an online process of object recognition by using simultaneous localization and mapping (SLAM). This can help to improve the object detection stage in order to obtain a more robust navigation system.

6. Conclusions

Real-time power line detection is a challenging problem that must be performed using different methods such as edge detectors, machine learning methods and 3D computer vision. The main problems inside this are the image correction due the abrupt movements of the UAV, the difference of backgrounds found while flying and illumination changes. The tower detection using deep learning methods is recommended for a robust detection. The proposed vision strategy could help monitor the environment of power lines in order to prepare preventive maintenance for reducing risk of tree branches that can affect the electrical infrastructure.

As future work, a technique based on SLAM could be useful to deal with complex scenes in order to improve the process and extract 3D information as an online process using an onboard computer.

It is mandatory to focus the future work in collision avoidance systems that allow to protect both UAV and electrical infrastructure in order to minimize the risk of damages during inspection process or autonomous navigation. Motion prediction is necessary for path planning in autonomous systems, and risk assessment for intelligent vehicles is fundamental to improve the safety.

The development of a system with a fixed wing platform could be useful for long-distance inspections. Finally, the study of the effects of oscillation of the detected angle between the UAV and the power lines can be considered in order to improve the control strategy using methods such as filtering.

Acknowledgments

This work was supported by the INV-ING-2995 project of the Vicerrectoria de Investigaciones—Universidad Militar Nueva Granada.