Useful Feature Extraction and Machine Learning Techniques for Identifying Unique Pattern Signatures Present in Hyperspectral Image Data

Abstract

This chapter introduces several feature extraction techniques (FETs) and machine learning algorithms (MLA) that are useful for pattern recognition in hyperspectral data analysis (HDA). This chapter provides a handbook of the most popular FETs that have proven successful. Machine learning algorithms (MLA) for use with HDA are becoming prevalent in pattern recognition literature. Several of these algorithms are explained in detail to provide the user with insights into applying these for pattern recognition. Unsupervised learning applications are useful when the system is provided with the correct set of independent variables. Various forms of linear regression assay adequately solve hyperspectral pattern resolution for identifying phenotypes. K-means is an unsupervised learning algorithm that is used for systematically dividing a dataset into K number of pattern groups. Supervised and unsupervised neural networks (NNs) are used to discern patterns in hyperspectral data with features as inputs and in large datasets where little a priori knowledge is applied. Other supervised machine learning procedures derive valuable feature detectors and descriptors through support vector machine. Several methods using reduced sets for extracting patterns from hyperspectral data are shown by discretized numerical techniques and transformation processes. The accuracy of these methods and their usefulness is generally assessed.

1. Introduction

Hyperspectral imaging and data analysis have recently received considerable attention since the representative data is ultra-high resolution and informative [1, 2, 3, 4, 5, 6, 7]. Hyperspectral data is collected by using a precision imaging device which emits light energy at wavelengths below, within, and above the visible range. Other cameras such as RGB (Red-Green-Blue sensitive filters) and multispectral, have more limited sets of data. These passive sensors only collect one to five reflective values from ambient light that is present (Figure 1). It then scans each pixel location sequentially for reflectance values from each wavelength of light emitted. Each wavelength is spaced about 2–5 nms. Apart. The data collected for each pixel in an image then represents the complete spectrum returns for the hyperspectral bands presented (see Figures 2–4). Within this data, patterns of information exist that have never been detected before, thus allowing the explorer to glean relevant and new highlights from this collected data set.

Figure 1.

Various camera operational bands.

Figure 2.

Multispectral comparison with hyperspectral (Dr. Nicholas M. Short, Sr. – Public domain).

Figure 3.

Hyperspectral imaging process (NASA – Public domain).

Figure 4.

Hyperspectral imaging analysis.

Hyperspectral cameras use a line scanning sensor (mostly, push broom type), that emits light at varying frequencies and then collects the reflected signal through a narrow slit. The narrower the slit, the higher the resolution of the camera, until it begins interfering with the light wave signal itself. The reflected light enters the slit and coincides with a concave mirror (Figure 5, M1) where the light is collimated. M1 redirects the collimated light from the scan to the optical grating. Here the light is divided or dispersed into its component frequencies. M2 acts to expand the beams and redirect the light to a reimaging lens array in the sensory unit.

Figure 5.

Hyperspectral imaging hardware operation.

The hyperspectral camera can be embedded on UAVs (e.g., Figure 6) to enhance aerial views for many image pursuits for agriculture, marine studies, search and rescue, surveillance, military activities, and construction site safety and management. Once hyperspectral images are stored, the data can be acquired per pixel per wavelength to reconstruct the image or study the reflected signatures. For instance, in agriculture, crops in a region can be surveyed by studying the map of the pure signature spectrum (Figure 7). Other pattern recognition algorithms can be used to understand how normal spectrums represent specific species of plants. Detecting plant diseases and stress factors in early stages of disease development is essential for selective and effective management of crop production. Through other data analysis procedures, such as feature extraction, statistical prediction, and reduced signature spectrums, pinpointing where the spectrums differ between a range of normal signatures and abnormal feature spectrums will reveal variations in the species [3, 8, 9]. Realizing how these spectrums differ because of diseases, abiotic stressors, nutritional deficiencies, or other factors, will give more useful information to the farmers.

Figure 6.

Hyperspectral imaging device as payload on UAV.

Figure 7.

Hyperspectral imaging of agricultural landscape (image credit: Geospatialworld.net).

The focus of this chapter is on extrication of features and pattern recognition algorithms that can be used in hyperspectral data analysis to obtain useful information. Common preprocessing and analysis applications include normalization and derivative spectra enhancement using finite differencing [10], complex step derivative [11], and derivative spectral shape equation [9]. Wavelet Transform has been used and compared to derivative spectra enhancement and shown to be very successful in spectral regions of interest; it is becoming more commonly used as an alternative to spectral derivative methods [12]. Polynomial interpolations are also used to smooth the (spectral) data and better represent enhanced spectra. Multivariate analysis can be used to gain a better understanding of spectral variance between feature data [2, 4]. Recently, autonomous ground and unmanned aerial vehicles with hyperspectral camera payloads have been used to collect data for agricultural purposes [13, 14]. Along with this method of data collection, deep and transfer learning artificial intelligence applications have been developed for pest and plant stress detection [5, 6]. These techniques required a high-quality training dataset for accurate development of the prediction models [5].

Hyperspectral Imaging is gaining widespread use in drone applications for agriculture and water safety. Agricultural applications include landscaping crop regions, analysis of crop health, understanding nutritional status of plants, harvest studies, flowering index, growth cycles comparison, trait discrimination, breeding information, and soil performance. Associated AI and machine learning applications are the mainstay of these informational systems. In water quality analysis, various hyperspectral algorithms such as partial least-squares, fully connected neural networks with backpropagation (FCNN-BP), Support vector machine (SVM) and Random Forest (RF) procedures have been successfully used and compared for quantitative investigation [15]. Other assessments have been implemented using FETs and AI for detecting water contamination in rivers [16], forest fire assessment, and automated drone team hyperspectral fusion.

Machine learning algorithms usually require some preprocessing of the data. Segmentation and feature extraction often use spatial filters, Laplacian of Gaussian with orientation filters [17], and other traditional methods that detect spectral discontinuities or similarities to adequately obtain prediction models for patterns. Gradient magnitude algorithms form ridges at high valued pixels, noted as watersheds, that are used to segment regions. Adaptive thresholding is applied to image data that has nonuniform background. This thresholding approach calculates local thresholds based on specified properties of pixel neighborhoods to segment regions of interest.

Active contours are another avenue of interest in obtaining features. Snakes were developed [18] as parametric curves that uncover boundary regions by minimizing an energy function. This optimization works well with hyperspectral data since it is relational to the spectra and energy reflected. Level sets use iterative solutions to find intersecting boundaries between features by optimizing a formulated level set equation. Using active contours, segmented bounded regions of interest can be brought to the forefront and presented as features in ML algorithms.

Features can be categorized as being primarily applicable to boundaries, regions, coded areas, spectral features or whole images. These are not mutually exclusive and can be used as feature map sections for convolutional neural networks. Hyperspectral signatures of various region features are often used to train neural networks and as inputs to MLA. Features in unsupervised learning environments are realized by the system.

A branched diagram of popular MLAs is given by Figure 8. Supervised and unsupervised methods have their own distinct advantages and are dependent on the context of the application. This chapter reviews some useful methods in both categories and clarifies some of the subtle differences between these two types of algorithms. Other useful techniques for MLAs, such as fuzzy logic and quadratic nonlinear methods are depicted in the diagram of Figure 8. The reader is encouraged to explore these other methods and compare and contrast how these techniques can be used efficiently to enhance machine learning purposes.

Figure 8.

Machine learning algorithms: Categories of methods to be considered when applying ML to hyperspectral data enhancement of soil data intensity range.

Detecting plant diseases and stress factors in early stages of disease development is essential for selective and effective management of crop production. Laboratory analysis of plant samples for disease detection is time-consuming and labor-intensive. For that reason, several disease detection methods have been developed utilizing advanced and sophisticated hyperspectral data analysis approaches [3, 8, 9] and MLAs. These unique applications will be reviewed.

The rest of the chapter is divided into sections for:

1. Preprocessing methods that are used in hyperspectral data analysis

2. FETs for supervised MLAs

3. FETs for unsupervised machine learning

4. Innovative methods for hyperspectral signature analysis in agriculture

5. Best practices for working with hyperspectral data and machine learning

2. Preprocessing methods for hyperspectral data analysis

Preprocessing of hyperspectral data involves numerical and statistical methods to filter noise and vibration more generally, as well as radiative transfer and empirical models used for airborne applications [19]. Some preprocessing methods include normalization of data, data smoothing, intensity transformation, histogram matching and histogram equalization, adaptive histogram equalization, correlation and convolution using spectral or spatial filters, and the use of fuzzy sets. This section reviews the most common and useful preprocessing methods for general purpose applications in hyperspectral data processing.

2.1 Normalization of hyperspectral data

Normalizing data taken from using similar sensors and similar methods is a common practice for purposes of understanding and clarifying data. This ensures the integrity of data presented for analysis, prediction and classification. It also provides some smoothing of the data using a standard normal variate transformation. The Standard Normal Variate (SNV) transformation is counter the effects of skewness of the data related to the reflectance spectra. The SNV was found to reduce error in the approximation which could be due to interferences caused by scattering and particle size differences. The probability distribution function that can be used for this standard normal variance transformation is given by

$$f\left(x\right)=\frac{1}{\sigma \sqrt{2\pi }}{e}^{-\zeta }$$E1

Where

$$\zeta =\frac{{\left(x-\mu \right)}^2}{2{\sigma }^2}$$

represents the SNV with mean, μ = 0, and standard deviation, σ = 1.

Other distributions can be used for normalizing the data, if the camera manufacturer recommends or if there are other anomalies in the data that need to be taken into consideration. For example, if the data is non-Gaussian, the Z-score standardization can be used to force the data into standard normal distribution. Z-score standardization involves using the data mean and standard deviation to adjust all level points in the data such that

$$x_{\mathit{ii}}=\frac{x_i-\mu }{\sigma }$$E2

Then the$x_{\mathit{ii}}$ values can be used as normally distributed data.

2.2 Hyperspectral data smoothing

After transforming the data into the SNV domain, the next steps might include resolving the data by curve fitting to smooth the spectral data enough so as to be able to reliably calculate the finite difference approximations or other numerical analysis of the data. A convolution method, such as the Savitzky–Golay Filter (SGF, [20]) can be used to approximate the spectra of the data.

For each pixel location, $i,$ of the data, a signal spectrum of the data can be smoothed by:

$$S_i=\sum _N{c}_N{f}_i+n_C$$E3

Where.

$S_i$ = smoothed pixel value per wavelength.

N = pixel neighborhood.

$c_N=\text{Coefficient of curve}\mathrm{fit}.$

$f_i=\text{pixel value}\mathrm{at}\text{each wavelength}$

$n_C=\text{center pixel value}\mathrm{per}\text{wavelength}$

Determining the coefficients can be done by a box filter, applying Gaussian smoothing for the data set, or other filtering methods. For the SGF, the data is point transformed by polynomial approximation and the coefficients are found by a least square fit.

SGF takes into consideration the order (n) of the polynomial to which the data is being fitted, and the size of the window (m) inscribing the real data points which are being incorporated for the smoothing at each data point.

The SGF conversion process at the pixel level can be described by:

$$y\left(i\right)=\sum _{n=0}^k{a}_n{i}^n\left|i\right|\le m$$E4

Where

y(i) = transformed output of Savitzky–Golay filter

n = term number of polynomials

$a_n=$ coefficients of polynomial

$k=$order of polynomial fit

To obtain the values of the $n+1$coefficients, a least squares criterion is used for solving Eq. (4). By taking the partial derivative of the wrt $a_n$ and setting the result equal to zero to minimize the error:

$${\frac{\partial }{\partial }}_{a_n}\left[\sum _{i=-m}^m{\left(y\left(i\right)-s_i\right)}^2\right]=0$$E5

Using Eq. (4) at i = 0 the first term can be found. Then using back-substitution and the criterion Eq. (5), the higher coefficients can be found

An example (Figure 9) shows the smoothing effect of the SGF applied to a sample hyperspectral signal in real-time data.

Figure 9.

Filter smoothing of hyperspectral Spectrum via Savitzky-Golay filter.

2.3 Intensity transformations

Intensity transformations can be used for data that is skewed because of background or foreground attenuations or ambient light interference. Stretching or compressing the value range of pixels that are converted to data values can be done with point transformation functions. The curves used for transformation can be gamma, exponential, power law, adaptive, piecewise linear, etc. The domain transform technique uses an operator for each pixel location:

$$f\left(x,y\right)=\mathrm{T}\left[I\left(x,y\right)\right]$$E6

$$s=\mathrm{T}\left(r\right)$$E7

Where

$s=\text{Intensity of}\mathrm{f}\left(\mathrm{x},\mathrm{y}\right)$

$r=\text{Intensity of Image data}$

These types of image transformations are sometimes called “mappings” since they use a point to point mapping of the data to express hidden quality features that misrepresented the original data. Common transforms are the gamma, power, logarithmic, contrast stretching, and exponential mapping.

2.4 Histogram equalization

For hyperspectral images that are digitized by special mapping of the spectral to intensity domains, histogram matching functions can be applied to obtain image data that is uniformly distributed on an interval [0,I]. A histogram of image data is defined by Eq. (8).

$$h\left(r_k\right)=n_k$$E8

where

$r_k=\mathit{\text{the}}\mathit{kth}\mathit{\text{intensity represented}}\mathit{by}\mathit{\text{the mapping}}$

$n_k=\#\mathit{\text{of pixels in image whose intensity is}}{r}_k$

For an image that is M x N pixels:

$$\sum _{k=0}^{I}h\left(r_k\right)=\mathit{MN}$$E9

We can then obtain an expression for the probability of the occurrence of a pixel of intensity level $r_k$ by dividing the histogram by the number of pixels in the image:

$$p\left(r_k\right)=\frac{h\left(r_k\right)}{\mathit{MN}}=\frac{n_k}{\mathit{MN}}$$E10

Then the sum of (10), which is one.

$$\sum _{k=0}^{I}p\left(r_k\right)=1$$E11

The cumulative distribution function can be found for an intensity value, $r_k,$ as:

$$\mathbb{C}\left(r_k\right)=\sum _{i=0}^{k}p\left(r_i\right)$$E12

By using the transformation expression given in Eq. (7) to remap the intensity value of a pixel of $r_i$ intensity to $s_i$ intensity by a scalar:

$$s_k=T\left(r_k\right)=\kappa \sum _{i=0}^{k}p\left(r_i\right)$$E13

Where $\kappa$ represents the maximum range of intensities (for integer values, Max(range)-1).

An example of the usefulness of hyperspectral data histogram equalization is shown in Figure 10. A hyperspectral landscape image of a section of the bison basin is shown in Figure 10(left). The image shows the hyperspectral mapped data before histogram equalization and afterwards (Figure 10-right). The spread of the image forested area is washed out since the degree of green is saturated by the equalization. However, the biocrust map is more enhanced by the allowance of the greater spread in the lower and higher spectra pixel intensity region (refer to Figure 11).

Figure 10.

Biocrust data (left) before histogram equalization; (right) after histogram equalization; note enhancement of soil.

Figure 11.

Biocrust data (left) before histogram equalization; (right) after histogram equalization; note the broadening of the intensity range after equalization.

3. Feature extraction techniques for supervised machine learning

In order to find regions of interests or embedded patterns in the data, feature extraction methods for hyperspectral data is used to reduce the learning time and amount of data necessary for MLAs. When there is a priori knowledge in data, it is useful to extract this information so that the MLA used for pattern recognition is built on worthwhile information. In ordinary image data, finding lines, edges and corners is usually an advantageous effort since locating areas with sharp transitions is quite often associated with a pattern feature descriptor. With hyperspectral data, a more common application designed by Lowe [17] is scale-invariant feature transforms (SIFT). There have been many adaptions of this SIFT transform over the years, but the robustness of this algorithm to find patterns in embedded frame data, real-time advancing frame data (such as moving target or moving platform), and for maximally stable extremal regions has become the supreme standard method used for keypoints feature extraction.

The first item that this transform addresses is scale invariance. By applying Gaussian filters to a stack of image(s) and increasing the smoothing by$k^n\sigma$ for each image that is stacked up to an octave (n = 4), these stacks of images are transformed by

$$\sigma ={\sigma }_1$$

$${\sigma }_2=2{\sigma }_1$$

$${\sigma }_n=2{\sigma }_{n-1}$$E14

To find the keypoints, a difference operation is performed as given by Eq. (15):

$$\mathcal{L}\left(x,y,\sigma \right)=\left[ℊ\left(x,y,\mathit{k\sigma }\right)-ℊ\left(x,y,\mathit{k\sigma }\right)\right]\divideontimes \text{I}(\text{x},\text{y})$$E15

Where

$\mathcal{L}\left(x,y,\sigma \right)=\mathit{\text{Laplacian operator}}$

$ℊ\left(x,y,\sigma \right)=\mathit{\text{Gaussian transformed images}}$

I(x,y) = original image data

After this Laplacian of Gaussian operation is performed on the octave stacked smoothed image data, the extrema regions of the data begin to emerge. This scale-invariant region become the keypoint features of the data.

To build more robustness into this algorithm, invariance to rotation and other affine transformations are accounted for by applying orientation invariant gradient directional operators at the keypoints extracted from Eq. (13). These operators are $4^2$ directional histogram matrices where each rotational element is 22.5 degrees differenced and weighted about bins that are multiples of 45^o. After correlating this directional filter at the keypoints, keypoint descriptors are indicated and also used as feature directives for the keypoint features. With this collection of extrema data labeled as features, any number of machine learning methods can now be applied with the feature keypoints and descriptors provided as inputs.

The SIFT method was applied for soil biocrust data taken from US geological society biocrust data. Three band Electro-Optical (EO) imaging system - collected on June 2, 2018 using a Ricoh GR II camera (18.3 mm lens) mounted on 3DR Solo quad-rotor aerial vehicle (9:45 AM MDT) were collected [21].

The data before and after the SIFT procedure was applied is given in Figure 12. After applying a fully connected neural network with two layers, backpropagation and labeling the data, the supervised MLA was able to locate nine distinct areas of the terrain, including the two areas of biocrust.

Figure 12.

Hyperspectral image data with features extracted by SIFT.

4. A feature extraction technique for unsupervised machine learning

While many variations for linear discriminant analysis (LDA) exist, the focus on K-means and K-medoids has gotten less attention. The main emphasis of these methods is to use clustering of data traits to classify features. Clustering algorithms are vital knowledge acquisition tools [22]. Numerical clustering algorithms generally use a Euclidean distance measure or geometric distance such as derived by cosine angle to classify new data that is provided to the system. K-medoids rather than calculate cluster centers by distance, it places data points or exemplars as data centroids and classifies by maximizing similarities (or by contrast minimizing dissimilarities) in data point features. The tested classification is given a goodness of fit parameter to test the choice for the number of clusters (mostly uses a “silhouette” function). If this figure of merit is best for a particular data set, the data is placed in the least dissimilar cluster until all data points are accounted for with the least change in the cost (or minimal dissimilarity). The algorithm consists of the following two layers – the “build” and the “swap”

In BUILD:

1. Apply a priori points to be the exemplars for the optimal number of clusters given by the silhouette function

2. Associate the rest of the data set to its closest cluster medoid

In SWAP: (while Cost is decreasing)

1. For each cluster, Swap the medoid, m, with data point, d

   1. Calculate the cost function

   2. When the cost function is minimal, identify the m_min and d_min

2. Swap m_min and d_min for each cluster until overall cost is minimized.

An application using k-medoids for selecting a set of data features from a hyperspectral image to be associated with decreased nitrogen (Figure 13) in an avocado plant was used to observe the classification pattern that would occur. The cluster associated with nitrogen deficient plants form a specific spectral signature in the hyperspectral data cube. Using this algorithm provided nearly 97% accuracy for random selected leaves of this signature compared to healthy and less nitrogen-deprived avocado plants as can be seen in Figures 14–17.

Figure 13.

Healthy avocado (left); nitrogen deficient (right).

Figure 14.

Model convergence.

Figure 15.

K-medoid model metrics.

Figure 16.

K-medoid cluster plot.

Figure 17.

K-medoid validation confusion matrix.

The model “build” mode is given by the first five iterations in Figure 14 and shows how the model converges to the minimal cost analysis (minimum error hyper parameters). Swap mode continues in iterations 3–30. Figure 15 shows the overall metrics for this algorithm. It took ∼166 K swap configurations for convergence. The distance metric used for identifying data point similarity was a Chebyshev distance metric. It also used a weighted function for decision by squared inverse. Figure 16 shows the cluster classification plot of the signature data points. The validation confusion matrix of Figure 17 resolves the classification accuracy [23].

5. Innovative methods for hyperspectral analysis in agriculture

The method of finite differencing has been found to work well with hyperspectral data to find dissimilarities in first and second derivative metrics. Using this deterministic method, it is reasonable to find regions-of-interest where the derivative max-min and inflections differ between spectral signature data (Figure 18). These are key features that can be shown on a parallel coordinates plot. These 2-D patterns that emerge in high dimensional data can help discern features and provide useful predictors for classification purposes.

Figure 18.

Parallel coordinates plot – Shows areas of discrimination in finite differences.

Other methods such as Karhounen-Loeve expansion of the data will provide discernment as to where the data has the highest variance [24]. If these points are used as input features to neural networks, supervised learning will enhance the prediction model convergence and accuracy.

6. Best practices for working with hyperspectral data and machine learning

When working with hyperspectral data for machine learning, minimizing the amount of data in the signature content is the first order of business. Doing so without losing important feature data is the goal of precision feature extraction techniques. Preprocessing of the data enhances the features to gain a clear understanding of pattern properties.

There are many areas of machine learning to explore (Figure 8), to discover the best solution for the context of the problem at hand. Incorporating several models and contrasting and comparing them will bring the best comprehension to what the data is revealing.

Decide what type of information is at the forefront of the problem presented and if unsupervised or supervised learning with feature extraction methods are appropriate.

Designing an accurate set of predictors, features, classifier methods and training data are the most important areas to consider when using machine learning with hyperspectral data. Determining which machine learning technique provides the most accurate solution for classifying data will help build a solution database that can be used for diagnostic purposes. Data fusion in the post processing area can be used with the classified features to acquire exclusive signatures. These unique pattern identifiers can then be stored in a database and used for identification and diagnostic purposes. A flowchart of best practices of data preprocessing and feature extraction procedures is given in Figure 19.

Figure 19.

Best practices for Hyperspectral Data Preprocessing & Feature Extraction Procedures for use in machine learning (ML).

7. Conclusions

This chapter presents an overview of preprocessing and feature extraction methods that are useful when working with hyperspectral data. Examples are shown for applications of these methods using supervised, unsupervised machine learning techniques and neural networks. Emphasis is placed on the context of the problem, development of accurate features and training sets, enhancement to features using weighting functions and decision parameters, and realizing reduced data signatures through preprocessing and feature map expansion.

A branched diagram of the various supervised and unsupervised methods that are popular in machine learning was given in Figure 8. This chapter provided a summary of selected techniques given in Figure 8 as well as provided insights on preprocessing for enhanced machine learning success. Through correct use of feature extraction in building the training and test data sets, machine learning algorithms can provide more accurate results. Machine learning is usually part of an embedded systems as shown in Figure 19. This chapter has provided insights into the aspects of feature extraction for enhanced machine learning success, and has examined some of the best algorithms to produce reliable machine learning results for use in diagnostic databases, robotics, factory automation, and other applications where decision and classification are necessary processes.