Automatic Inspection of Aircraft Components Using Thermographic and Ultrasonic Techniques

Safety in aeronautics could be improved if continuous checks were guaranteed during the in-service inspection of aircraft. However, until now, the maintenance costs of doing so have proved prohibitive. In particular, the analysis of the internal defects (not detectable by a visual inspection) of the aircraft’s composite materials is a challenging task: invasive techniques are counterproductive and, for this reason, there is a great interest in the development of non-destructive inspection techniques that can be applied during normal routine tests.


Introduction
Safety in aeronautics could be improved if continuous checks were guaranteed during the in-service inspection of aircraft. However, until now, the maintenance costs of doing so have proved prohibitive. In particular, the analysis of the internal defects (not detectable by a visual inspection) of the aircraft's composite materials is a challenging task: invasive techniques are counterproductive and, for this reason, there is a great interest in the development of non-destructive inspection techniques that can be applied during normal routine tests.
Non Destructive Testing & Evaluation (NDT & E) techniques consist of a data acquisition phase (based on any scanning method that does not permanently alter the article being inspected) followed by a data analysis phase carried out by qualified personnel. In particular, transient thermography and ultrasound analysis are two of the most promising techniques for the analysis of aircraft composite materials (Hellier, 2001).
Non-destructive evaluation requires an excessive amount of money and time and its reliability depends on a multitude of different factors. These range from physical aspects of the technology used (e.g., wavelength of ultrasound) to application issues (e.g. probe coupling or scanning coverage) and human factors (e.g. inspector training and stress or time pressure during inspection) (Kemppainen. & Virkkunen, 2011).
Most of the work in the literature concentrates on the study of data acquisition and manipulation processes in order to prove the relationship between data and structural defects or composition of the material (Chatterjee et al., 2011). Unfortunately only some of the work from the literature concentrates on the posterior analysis of the acquired data in order to (fully or partially) delegate, to some computational algorithm, the automatic recognition of material composition, operative conditions, presence of defects, and so on. This is undoubtedly a very attractive research field since it can reduce operational costs, save time and make the process independent from human factors. However, the development of proper algorithms and methodologies is in its infancy and their level of inspection reliability is still inadequate for those sectors (namely, transportation) where an error can have serious health and safety consequences.
The pioneering work on the a posteriori analysis of data dates back to the early 1990s: it suggested that solutions to the problem of automatic ultrasonic NDT data interpretation could be found by expert systems which embody the knowledge of human interpreters (McNab & Dunlop, 1995) (Hopgood et al., 1993) (Avdelidis et al., 2003) (Meola et al., 2006) (Silva et al., 2003). More effective approaches, based on advanced signal processing and artificial intelligence paradigms, have been proposed in the last decade (Benitez et al., 2009) (Wang et al., 2008).
In this chapter, we address the problem of developing an automatic system for the analysis of sequences of thermographic images and ultrasonic signals to help safety inspectors in the diagnosis of problems in aircraft components in all those cases where the defects or the internal damage are not detectable with a visual inspection. In particular thermographic analysis is proposed to automatically discover water insertions whereas ultrasonic inspection aims at revealing solid insertions of brass foil.
The proposed approach considers two main steps for interpreting thermographic and ultrasonic data: in the first step a pre-processing technique is introduced to clean data from noise and to emphasise embedded patterns and the classification techniques used to compare ultrasonic signals and to detect classes of similar points. In the second step two neural networks are trained to extract the information that characterises a range of internal defects starting from ultrasonic and thermographic signals extracted in correspondence to the defective areas. After that the same neural networks are applied to automatically inspect real aircraft components.
Section 2 gives an overview of the proposed approach whereas section 3 and 4 concentrate on the data pre-processing and classification respectively. Finally, section 5 presents the experimental results on real aircraft material and conclusions are derived in section 6.

Overview of the system
The proposed system for automatic inspection of aircraft components is schematized in figure 1. The system takes the data extracted by non destructive processes reported in the literature as transient thermography and ultrasound scanning as input.
Transient thermography is a non-contact technique, which uses the thermal gradient variation to inspect the internal properties of the investigated area. The materials are heated by an external source (lamps) and the resulting thermal transient is recorded using an infrared camera. Of course, this kind of analysis is only applicable to materials that have a good thermal conductivity such as metals and carbon composites. Different types of thermal excitation can be used according to the materials and the defects under investigation: for instance uniform heating, spot heating, and line heating.
Ultrasonic inspection uses instead sound signals at frequencies beyond human hearing (more than 20 kHz) to estimate some properties of the irradiated material by analyzing either the reflected (reflection working modality) or transmitted (transmission working modality) signals. A typical ultrasonic inspection system consists of several functional units: pulser, receiver, transducer, and display devices. A pulser is an electronic device that can produce a high-voltage electrical pulse. Driven by the pulser, the transducer generates a high-frequency ultrasonic wave which propagates through the material. In the transmission www.intechopen.com modality, the receiver is placed on the opposite side of the material from the pulser, whereas, in the reflection modality, the pulser and the receiver are placed on the same side of the material.
Ultrasonic data can be collected and displayed in a number of different formats. The three most common formats are known in the NDT community as A-scan, B-scan, and C-scan presentations. Each presentation mode provides a different way of looking at and evaluating the region of material being inspected. On the one hand, thermographic analysis is carried out to automatically discover water insertions whereas ultrasonic inspection aims at revealing solid insertions of brass foil.
For thermographic inspection we analyze mono-dimensional signals obtained by considering the time variation of each pixel in the sequence of thermographic images. For each point (i,j) of the material the mono-dimensional signal is generated from the gray levels of the same point in the sequence of images: this signal represents the temperature variation of the material during and after the heating process. This way it is possible to generate spatial-time variant images, the analysis of which allows for the evaluation of the thermal gradient during the heating process.
In Figure 2, the one-dimensional signals extracted from the thermographic sequence of aircraft fuselage are shown: one point belongs to an area affected by the presence of water (red line) whereas the other signal corresponds to non-defective areas (gray lines). From the graph it is clearly evident that a functional description of the intensity variations cannot be easily generalized and the behaviours of points corresponding to defective and nondefective areas are very similar.
For the analysis of ultrasonic data we analyze one-dimensional signals acquired from the reflection working modality and A-scan representation. This means that, for each point of the inspected material, we have a continuous signal that represents the amount of received ultrasonic energy as a function of time.
In figure 3 two ultrasound signals are shown. The signal on top is relative to a non-defective point. Observe that there are large extrema at the beginning and at the end. These changes in ultrasound energy are caused by the transmitted signals being reflected by the boundaries of the material. These boundary extrema are referred to as tool side and bag side peaks, respectively. The ultrasonic signal for an area of material that contains defects is given on the bottom of figure 3. In addition to the boundary extrema, the signals contain extrema at other time locations caused by defective components. The time localization of the additional extrema depends on the defect location in the inspected material.
The temporal evolution of the thermographic and ultrasound signals x(t) is the input to the core of the proposed approach that consists of two main steps: the pre-processing of the data, in order to emphasize the characteristics of the signals belonging to the same class, and the following neural classification.
Pre-processing step allows to discard noise and to enhance the most relevant information for flawed area detection purposes. Two Multi Layer Perceptron (MLP) neural architectures characterized by the presence of an input layer of source nodes, a hidden layer and an output layer, are then used to build an inspection framework that automatically label each signal as belonging to a flawed area or not.
A final connectivity analysis of all the points labelled as belonging to flawed areas is done in order to both discard isolated false positives and to deduce size and shape of the flawed area as a whole.

Data pre-processing
The automatic classification of acquired signals as flawed or unflawed is not trivial due to the huge number of intra-class variance: on the one hand ultrasonic and thermal signals relative to unflawed areas can shows different temporal behaviours depending on manufacturing variations in the underlying composite layers or specimen thickness variations. This is evident in figure 4 where different thermographic signals relative to unflawed areas are reported. On the other hand, signals relative to flawed areas can differ since insertions and infiltrations can occur at different locations.
In order to make the classification easier, a pre-processing technique step is then required: on the one hand, it has to increase signal to noise ratio and, on the other hand, to detect and enhance the information that could increase the probability of separating signals belonging to different classes. There are many effective signal pre-processing techniques in the literature. Most of them work in a specific domain (time or frequency) whereas a few of them affect both domains simultaneously. In the latter category lies the so-called Wavelet Transform, an extension of Fourier Transform generalized to any wideband transient. For its capability to give a multidomain representation of the data, the wavelet transform has been used in this work to analyse collected thermographic and ultrasonic data.
In figure 5 the wavelet decomposition (by using Daubechies 3 kernels ) at level 3 using a thermographic (top) and ultrasound (bottom) signal are reported.
The next subsection gives some additional theoretical information about the considered preprocessing technique based on Wavelet Transform.

Wavelet transform
Let us think about our input as a time-varying signal. To analyze signal structures of very different sizes, it is necessary to use time-frequency atoms with different time supports. The wavelet transform decomposes signals over dilated and translated wavelets Mallat (1999). The signal may be sampled at discrete wavelength values yielding a spectrum. In continuous wavelet transform the input signal is correlated with an analyzing continuous wavelet. The latter is a function of two parameters such as scale and position. The widely used Fourier transform (FT) maps the input data into a new space, the basis functions of which are sins and cosines. Such basis functions are defined in an infinite space and are periodic, this means that FT is best suited to signal with these same features. The Wavelet transform maps the input signal into a new space which basis functions are usually of compact support. The term wavelet comes from well-localized wave-like functions.
In fact, they are well-localized in space and frequency i.e. their rate of variations is restricted.
Fourier transform is only local in frequency not space. Furthermore, Fourier analysis is unique, but wavelet not, since there are many possible sets of wavelets which one can choose.
Our trade-off between different wavelet sets is compactness versus smoothness. Working with fixed windows as in the Short Term Fourier Transform (STFT) may bring about problems. If the signal details are much smaller than the width of the window they can be detected but the transform will not localize them. If the signal details are larger than the window size, then they will not be detected properly. The scale is defined by the width of a modulation function. To solve this problem we must define a transform independent from the scale. This means that the function should not have a fixed scale but should vary. To achieve this, we start from a function () t  as a candidate of a modulation function and we can obtain a family starting from it by varying the scale s as follows: If  has width T then the width of s  is sT . In terms of frequencies, the smaller the s the higher the frequencies s  and vice versa.
The continuous wavelet transform X  is the result of the scalar product of the original signal () xt with the shifted and scaled version of a prototype analysing function () t  called mother wavelet which has the characteristic of a band pass filter impulse response.
The coefficients of the transformed signal represent how closely correlated the mother wavelet is with the section of the signal being analyzed. The higher the coefficient, the more the similarity.
Calculating wavelet coefficients at every possible scale is a fair amount of work, and it generates a great amount of data. If we choose scales and positions based on the power of two (called dyadic scales and positions) then our analysis will be much more efficient. This analysis is called the discrete wavelet transform.
In the discrete case, WT is sampled at discrete mesh points and using smoother basis functions. This way a multiresolution representation of the signal () xt can be achieved.

www.intechopen.com
Notice that the wavelet transform can be written as a convolution product (it is a linear space-invariant filter): ,, This leads to a fast and efficient implementation of the wavelet transform for a discrete signal obtained using digital filtering techniques. The signal to be analyzed is passed through filters with different cut off frequencies at different scales. The wavelet transform for a discrete signal is computed by successive low-pass and high-pass filtering of the discrete time-domain signal. Many filter kernels can be used for this scope and the best choice depends on the features of the input signal that have to be exploited.
At each decomposition level, the half-band filters produce signals spanning only half the frequency band. This doubles the frequency resolution as the uncertainty in frequency is reduced by half. At the same time, the decimation by 2 doubles the scale. With this approach, the time resolution becomes arbitrarily good at high frequencies, whereas the frequency resolution becomes arbitrarily good at low frequencies.

Automatic learning and classification of defective and non-defective patterns
After the pre-processing step the new wavelet based data representations is given as input to an automatic classifier that, after a proper learning phase, is able to label each input stream as belonging to a flawed or unflawed area on the basis of the learned input/output mapping model. One of the most powerful data modelling tools that is able to capture and represent complex input/output relationships is neural network (NN).

Neural network paradigm
The motivation for the development of neural network technology stemmed from the desire to develop an artificial system that could perform "intelligent" tasks similar to those performed by the human brain. Neural networks resemble the human brain in the following two ways: 1. A neural network acquires knowledge through learning. 2. A neural network's knowledge is stored within inter-neuron connection strengths known as synaptic weights.
The true power and advantage of neural networks lies in their ability to represent both linear and non-linear relationships and in their ability to learn these relationships directly from the data being modelled. Traditional linear models are simply inadequate when it comes to modelling data that contains non-linear characteristics.
The most common neural network model is the multilayer perceptron (MLP), having an architecture as reported in figure 6. This type of neural network is known as a supervised network because it requires a desired output in order to learn. The goal of this type of network is to create a model that correctly maps the input to the output using historical data so that the model can then be used to produce the output when the desired output is unknown. Fig. 6. A feed forward neural network scheme.
The MLP and many other neural networks learn using an algorithm called back propagation. With back propagation, the input data is repeatedly presented to the neural network. With each presentation the output of the neural network is compared to the desired output and an error is computed. This error is then fed back (back propagated) to the neural network and used to adjust the weights such that the error decreases with each iteration and the neural model gets closer and closer to producing the desired output. This process is known as "training".
The hidden layers enable the network to extract higher-order statistics especially when the size of the input layer is large. There is no theoretical limit to the number of hidden layers but, typically, architectures with just one hidden layer are adequate to face the complexity of most of the practical problems . Most used neural architecture have only one hidden layer. Supervised learning involves applying a set of training examples to modify the synaptic weights connecting the neurons of the network. Each example consists of a unique input signal and the corresponding desired response. The network is presented with many examples many times and the synaptic weights are tuned so as to minimize the difference between the desired response and the actual response of the network. The network training is repeated until a steady state is reached, where there are no further significant changes in the synaptic weights.
The input layer has a number of neurons equal to the number of image features. In this work, the features are those extracted after the pre-processing phase. The number of nodes in the output layer depends on the number of classes that the network has to recognize. In our context the network has to recognize the sound point and the defect points (2 output nodes). The number of nodes in the hidden layer is determined by experiment.
There is no quantifiable best answer to the layout of the network for any particular application. There are only general rules picked up over time and followed by most researchers and engineers applying this architecture to their problems.
Rule One: As the complexity in the relationship between the input data and the desired output increases, the number of the processing elements in the hidden layer should also increase.
Rule Two: If the process being modelled is separable into multiple stages, then additional hidden layer(s) may be required. If the process is not separable into stages, then additional www.intechopen.com layers may simply enable memorization of the training set, and not a true general solution effective with other data.
Rule Three: The amount of training data available sets an upper bound for the number of processing elements in the hidden layer(s). To calculate this upper bound, use the number of cases in the training data set and divide that number by the sum of the number of nodes in the input and output layers in the network. Then divide that result again by a scaling factor between five and ten. Larger scaling factors are used for relatively less noisy data. If you use too many artificial neurons the training set will be memorized. If that happens, generalization of the data will not occur.

Experimental setup and results
The composite material used in the experimental tests has an alloy core with a periodic honeycomb internal structure of 128-ply thicknesses (each ply has a thickness of 0.19 mm).
The experiments were carried out on two specimens: the first one presents two water infiltrations whereas the second one presents three solid insertions of brass foil (0.02±0.01 mm thickness). One solid insertion was placed two plies from the tool side surface (TOP INSERTION), one at mid part thickness (MIDDLE INSERTION) and the remaining one two plies from the bag side surface (BOTTOM INSERTION). Brass inserts were introduced to represent voids and delamination. In all the cases the defects or the internal damage were not detectable with a visual inspection.  The thermographic image sequence was obtained by using a thermo camera sensitive to the infrared emissions. A quasi-uniform heating was used to guarantee a temperature variation of the composite materials around 20C/sec. In figure 8 one of the thermographic images is reported. Only liquid infiltrations become visible due to the larger thermal variation of the water with respect to solid insertions.
www.intechopen.com Ultrasonic data were obtained by an ultrasonic reflection technique that uses a single transducer serving as transmitter and receiver (5MHz).
In figure 9, the signal on the left is relative to a non-defective area whereas the signal on the right is relative to a brass insertion placed in the middle of the material thickness and for this reason the corresponding extrema is far from the boundary ones. The signal in the centre of figure 3 is relative to a brass insertion placed very close to the inspected material surface and then the corresponding extrema is mixed with the tool side one. This shows that defective and non-defective areas can have very similar temporal behaviours under ultrasound scanning and this causes traditional NDT techniques to fail. Fig. 9. three ultrasound signals relative to the non-defective area (on the left), a brass insertion placed in the middle of the material thickness (on the right) and very close to the inspected material surface (in the centre).
Acquired experimental data were then represented in the wavelet domain by using Daubechies 3 family of filters and the derived coefficients were given as input to two different neural networks in order to specialize each of them to recognize water infiltration and solid insertion respectively. The defect segmentation step is performed by using neural networks with two output neurons. Each available signal is fed into the net, which classifies it as either relative to defective areas or an unflawed area.
Preliminary experiments aimed at defining the best data model through the selected neural paradigm. In particular they allow the definition of the best number of neurons in the hidden layer and the most suited number of training points. To accomplish this fundamental task different set training examples were built. In particular, for each neural www.intechopen.com network 3 different training sets consisting of 40, 60 and 80 examples (50% corresponding to unflawed and 50% to flawed areas) were used. At the same time different test sets of points were built for each specimen. In particular, for the specimen with water infiltration two data were built: the first set contained 250 signals relative to unflawed points, the second set contained 250 signals relative to defective areas damaged by the water.
Similarly for the specimen with brass foil insertions 4 data sets were built: the first set contained 250 signals relative to unflawed points, the second set contained 250 signals relative to the defective area corresponding to the brass foil positioned two plies from the tool side surface (Top Insertion), the third set contained 250 signals relative to the defective area corresponding to the brass foil positioned at mid part thickness (Middle Insertion) and finally the fourth set contained 250 signals relative to the defective area corresponding to the brass foil positioned two plies from the bag side surface (Bottom Insertion).
In each experiment a training set was selected and the learned network was then used to classify the data in the corresponding test set. The set of training examples consisted of input-output couples (input signal, corresponding desired response). During the training phase the points of known examples were extracted from the considered materials and continuously fed into the net so that the synaptic weights were tuned to ensure the minimum distance between the actual and the desired output of the net.
Training continues until a steady state is reached, i.e., no further significant change in the synaptic weights could be made to improve net performance. This is repeated also using different configurations of the hidden layer. In particular a number of hidden neurons ranging from 20 to 100 were considered.
The results of this demanding experimental phase are summed up in figure 10 and figure  11.    Table II. Scatter matrix derived in the experiment 1 for brass fail insertion detection. Table I and II give a quantitative evaluation of the possibility to automatically detect both liquid and solid insertions in composite materials by using thermal and ultrasonic techniques in combination with neural approaches.
In particular, Table II illustrates that brass foil insertions at the mid-thickness level were always better classified than those located either at the top or at the bottom. The defect www.intechopen.com location is one of the most important factors in ultrasound inspection. The defects placed either at the top or at the bottom of the inspecting structure are in general the most difficult to detect since their echo is mixed with the tool face or the bag side echo. On the contrary, defective areas in the mid-part of the material thickness produce a distinct peak in the signal trend that is straightforward to identify.
In the second part of the experimental phase all the signals extracted by the thermographic and ultrasonic analysis were classified by using the neural networks previously learned.
According to the neural network outputs, a binary image is produced containing black points for defective areas and white points for sound areas. Fig. 12. graphical representation of the raw classification of all the signals extracted from specimens with water infiltration (on the left) and brass foil insertions (on the right).
In figure 12 the graphical representation of the raw classification of all the signals extracted from specimens with water infiltration (on the left) and brass foil insertions (on the right) is reported. Defective areas are correctly detected but there are also a lot of points in the unflawed areas erroneously classified as flawed. For this reason an additional processing step was introduced in order to analyse the output images considering the vicinity of flawed pixels (region analysis). In other words, considering that these false detections were isolated and did not form connected regions having a considerable area value, the elimination of these points was made more straightforward if some a priori knowledge about the minimum expected size of the defective areas is available.
In figure 13 the final outcome is reported after a filtering process based on the connectivity analysis of the detected defective regions and a selection criterion based on removing the regions having an area less than 20 pixels, are shown.
Most of the false flawed points were removed even if some areas in addition to the real defects were still considered flawed. They mainly occurred in correspondence with a variation of the inclination of the surface (see fig. 7): unfortunately, in this unflawed area both thermographic and ultrasound signals changed their slope more evidently with respect to the corresponding signals used to train the net. This problem could be faced by learning the net also on the points belonging to this particular areas. However, this way of proceeding was not considered in this work since it could be counterproductive: the net could miss some real defective areas (or parts of it) and, in our opinion, considering the applicative context, it is critically important to detect all defective points, even at the expense of generating extra false positives Fig. 13. the result of the cleaning based on the point connectivity analysis on the images reported in figure 8.

Conclusion
In this chapter, we address the problem of developing an automatic system for the analysis of sequences of thermographic images and ultrasonic signals to help safety inspectors in the diagnosis of problems in aircraft components.
In particular, thermographic analysis was carried out to automatically discover water insertions whereas ultrasonic inspection aimed at revealing solid insertions of brass foil. Experiments were carried out on real aircraft specimens and demonstrated the capability of the proposed framework to discover flawed areas. A tolerable number of false positive occurrences were also found in correspondence to the part of the specimens having a sloping surface since their points were not included in the learning phase in order to get the best true positive detection rate considering the critical operative context. Future work will focus on investigating the defect identification capability of the proposed approach. This will be achieved by extending the analysis to material with different thicknesses and different defective insertions. In the future, we will also investigate the possibility of using an unsupervised-learning approach in order to reduce human intervention.