Sequential Classification of Hyperspectral Images

Abstract

Hyperspectral imaging has become increasingly popular in applications such as agriculture, food, and environment. Rich spectral information of hyperspectral images leads to new possibilities and new challenges in data processing. In this chapter, we consider the hyperspectral classification problems in consideration of sequential data collection, which is a frequent setting in industrial pushboom imaging systems. We present related techniques including data normalization, dimension reduction, classification, and spatial information integration and the way to accommodate these techniques to the context of sequential data collecting and processing. The propose scheme is validated with real data collected in our laboratory. The methodology of result assessment is also presented.

1. Introduction

Hyperspectral imaging is a continuously growing area and has received considerable attention in the last decade. Hyperspectral data provide a wide spectral range, coupled with a high-spectral resolution. These characteristics are suitable for detection and classification of surfaces and chemical elements in the observed images. Rich information in spectral dimension provides solutions to many problems that cannot be solved by traditional RGB imaging or multispectral imaging.

Applications include land use analysis, pollution monitoring, wide-area reconnaissance, and field surveillance, to cite a few. Typical cases related to food quality, agriculture, and environment include as follows:

1. Food safety plays an important role in our daily life. We often use a combination of appearance, hand-feel, and smell of the product to make a judgment of the quality of fruits or vegetables. But it is not enough to judge if there are abnormalities, deformations, or even visible defects in the fruit or vegetable. Awareness about food safety has exemplified the requirement for a rapid and accurate hyperspectral detection system [1].

2. Precision agriculture is a farming management concept based on observing, measuring, and inter and intrafield variability in crops. Precision agriculture using hyperspectral remote sensing is acquired and processed to derive maps of crop biophysical parameters, to measure the amount of plant cover, and to distinguish between crops and weeds [2].

3. Due to the pressures of over consumption, population, and technology, the biophysical environment is being degraded, sometimes permanently. Many of the earth’s resources are on the verge of exhaustion because they are influenced by human impacts across many countries [3]. Many attempts are made to prevent damage or manage the impacts of human activity on natural resources. Hyperspectral classification used in resource recovery can make it rapid and efficient.

One of the most important tasks of hyperspectral image processing is image classification. Rich spectral information of hyperspectral image provides the possibility to classify materials that are difficult to be distinguished by other imagery techniques. In the past decades, different kinds of hyperspectral classification methods have been proposed [4, 5, 6, 7, 8, 9]. However, the existed methods may not be suitable for a real-time material sorting system. Pushboom imaging systems are frequently used in industry sorting, such a system collects columns of an image one after another in a sequential manner (see Figure 1). It is thus necessary to design a framework for online classification tasks and accommodate convectional algorithms to the sequential processing setting.

Figure 1.

Sequential hyperspectral data collecting and processing by a pushboom system. Hyperspectral camera captures data$x_k$at time instant$k$, which is one of the sequential columns of the entire image,$y_k$is the result after processing (classification label in this case).

In this chapter, we present a scheme of sequential classification for hyperspectral sorting systems. This scheme can be used in various fields, such as measuring food quality and resource recovery. We present the main techniques in this sorting and processing, including data normalization, dimension reduction, classification, and spatial information integration and the way to accommodate these techniques to the context of sequential data collecting and processing.

The rest of this chapter is organized as follows. In Section 2, we propose the main steps of sequential hyperspectral classification processing system. In Section 3, detailed methods are presented for sequential hyperspectral image processing and sorting. Experiment results are then discussed in Section 4. Section 5 concludes the chapter.

2. System overview

Before proceeding to elaborate the proposed sequential hyperspectral image classification method, we first present the notation and the data model used in this work. We consider that the hyperspectral image under study has $h$pixels in column and $w$pixels in row, where $h$is a fixed size that is determined by the spatial resolution of the camera, and $w$actually increases toward infinity with the moving of the pushboom system. Each pixel consists of a reflectance vector consisting of $p$contiguous spectral bands. Then, let

• $N=h\times w$be the total number of pixels.

• $X={\left\{X_1,X_2,X_3,\cdots ,X_p\right\}}^T$be the $\left(p\times N\right)$hyperspectral images.

• $X_k\left(i,j\right)$represents a pixel, where the subscript $k$denotes the index of the spectral band, $i$and $j$represent the location of pixel in the spatial domain.

The data collecting and processing of a real-time hyperspectral sorting system consist of the following major steps.

1. Sequential image acquisition.

2. Data preprocessing.

3. Material classification.

The hyperspectral data used in this work set are collected by the system of GaiaField in our laboratory. The parameters of the used system are provided in Table 1. Our online processing is based on windowed columns. After collecting each column, we use this column with several previous ones to form a window and perform data processing steps within this window. Black-white normalization is used for basic data normalization. Techniques of PCA and hyperspectral decorrelation of fuzzy sets are used for dimension reduction [10]. Typical techniques such as GML and SVM are presented for material classification. Considering the positive effect of spatial information on processing results [11], we also propose to integrate spatial dimension and spectral dimension to achieve an enhanced classification accuracy. Finally, classification accuracy is characterized by metrics such as confusion matrix and κ coefficient. Details of the used techniques and results will be provided later.

Equipment type	GaiaField and GaiaSorter
Moving speed of loading platform	4.1 cm/s
Spectral resolution	128
Spatial resolution	$650\times 348$
Distance between lens and samples	24 cm
Exposure time	3 ms

Table 1.

Equipment parameters.

3. Processing methods

3.1. Data preprocessing

Data preprocessing steps include basic data normalization and spectral decorrelation. They are performed one after another as described later.

3.1.1. Basic data normalization

An important preprocessing is the so-called black-white calibration. This calibration is carried out by recording an image for black and another for white, as described below, to remove the effect of dark current of the camera sensor and avoid the uneven light intensity of each band. At an offline phase, the black image ($B$) is acquired by turning off the light source and covering the camera lens with its cap. The white image ($W$) is acquired by adopting a standard white ceramic tile under the same conditions as the raw image. Then, image correction is performed by [12],

$$I=\frac{I_0-B}{W-B}$$E1

where $I$is the hyperspectral image after normalization, $I_0$is the original hyperspectral image that is captured in our laboratory, $B$is the black reference image, and $W$is the white reference image.

3.1.2. Data dimension reduction

The high-spectral resolution of hyperspectral data enables us to classify materials that are undistinguishable with conventional methods. However, a large number of spectral channels result in difficulties in processing in terms of classifier training (Hughes phenomenon) and computational burdens. Data dimension reduction can be performed due to the above facts and existence of information redundancies across bands.

3.1.2.1. PCA

PCA is one of the most popular methods for data dimension reduction. PCA computes a linear transformation for high-dimensional input vectors, and this transformation maps the data into a low-dimensional orthogonal subspace. For simplicity, we assume that the data samples have zero mean. Otherwise, we can centralize the data by subtracting the mean

$$Y=X-E\left(X\right)$$E2

The principle analysis is based on the eigenstructure of the data. We therefore calculate the covariance matrix of $Y$and perform the eigendecomposition on this matrix. The ith eigenvector of matrix $Y$is denoted by $a_i$with associated eigenvalue denoted by ${\lambda }_i$.

To reduce the dimension of data, we select an appropriate number of eigenvectors $a_i$corresponding to the value of eigenvalues ${\lambda }_i$from large to small, to form the representation coefficient matrix $A$[13].

$$Z=A^{T}X$$E3

where $Z$is the hyperspectral image data after decorrelation.

3.1.2.2. Fuzzy sets

Using fuzzy sets to decorrelate the hyperspectral data is based on a priori knowledge that the adjacent wavelengths of the spectrum are more correlated than the distant pairs, as the spectral information varies smoothly and successively. We consider sampling spectral characteristics by a group of adjacent spectral bands, which can be obtained by dividing the spectra in separate groups to attain the desired spectral selectivity. We propose separating the hyperspectral data into a number of $M$fuzzy groups where each group covers a range of wavelengths [14]. The contribution of each wavelength is modeled by a membership function ${\mathit{Mf}}_i\left(\lambda \right)$. We use a triangular function as the membership function, shown in Figure 2.

$${\mathit{Mf}}_i=\left\{\begin{array}{l}1-\frac{\lambda -{\lambda }_i}{D},{\lambda }_i-D<\lambda <{\lambda }_i+D\\ 0\begin{array}{lll}\begin{array}{ll}\begin{array}{ll}& \end{array}& \end{array}& & \text{otherwise}\end{array}\end{array}\right.$$E4

where ${\lambda }_i$is the central wavelength value of the fuzzy set $i$, and $D$is the distance of central wavelengths of two adjacent fuzzy sets.

Figure 2.

Triangular function.

The spectral wavelengths of all points have different membership degrees in different fuzzy sets. Each wavelength has different degree of membership in two adjacent fuzzy sets, while the membership degree in the remaining fuzzy sets is 0 (Figure 3).

Figure 3.

Triangular function weighted process.

The energy of each fuzzy set is calculated by weighting the intensity of each spectrum element using membership functions associated with each fuzzy set, i.e.,

$$X_i={\int }_{\lambda =0}^{\lambda =K}{\mathit{Mf}}_i\cdot L\left(\lambda \right)\mathit{d\lambda }$$E5

where $X_i$is the energy of each fuzzy set, and $L\left(\lambda \right)$is the intensity of each spectrum element.

Based on the energy values of each fuzzy set, we can obtain useful information about the spectral characteristics. In this way, each hyperspectral image pixel can be defined by a vector containing the energy values of the $M$fuzzy sets as

$$X={\left\{X_1,X_2,\cdots ,X_M\right\}}^T$$E6

3.2. Material classification

In this section, we present the algorithm to classifier/sort the captured data using features (data of reduced dimension) extracted by PCA or fuzzy set method. We first review these two popular classification methods in a general manner. Then we introduce how to incorporate spatial information into the classification. Finally, sequential processing with window-based method will be discussed.

3.2.1. Gaussian maximum likelihood classification

Spectra of distinct material of hyperspectral data form data clusters in a space with the dimension of the feature, and we assume that the data features of each material approximately follow a multivariate normal distribution. To be specific, data features of a material $i$and the $p$dimension probability density function in form of:

$$p\left.(,X\left|{\omega }_i\right.\right)={\left(\frac{1}{2\pi }\right)}^{\frac{p}{2}}{\left|{\mathrm{\Sigma }}_i\right|}^{-\frac{1}{2}}\text{exp}\left[-\frac{1}{2}{\left(X-{\mu }_i\right)}^T{\mathrm{\Sigma }}_i^{-1}\left(X-{\mu }_i\right)\right]$$E7

where $\mu$and $\mathrm{\Sigma }$are the mean vector and the covariance matrix, respectively. $i$denotes the label of class [15]. Each pixel in the hyperspectral image is labeled as the class that achieves maximum probability.

3.2.2. Support vector machine

SVM is one of the most effectively and widely used methods in statistical learning. SVM aims to find the best tradeoff of model complexity and learning ability with limited sample information. SVM can effectively solve the Hughes phenomenon caused by insufficient samples in hyperspectral classification.

The goal of training algorithm is to design an optimal hyperplane. The training principle of SVM is to find a linear optimal separating hyperplane [16]. Let $x$be the input pixel vectors satisfying

$$g\left(x\right)={\omega }^{T}x+{\omega }_0=0$$E8

This method constructs a hyperplane that maximizes the margins between classes, specified by a (usually small) subset of the data that define the position of the separator. These points are referred to as the support vectors [17]. The decision function is as follow:

$$f\left(x\right)=\text{sgn}\left[{\omega }^{T}x+b\right]=\text{sgn}\left(\sum _{i=1}^l{\alpha }_i{y}_i{x_i}^{T}x+b\right)$$E9

where ${\alpha }_i$is the ith Lagrange coefficient, $y_i$is the corresponding classification label, $x_i$is the ith support vector, $x$is the input pixel vector, $N$is the number of support vector, and $b$is the decision offset coefficient. For two-class hyperspectral classification, $f\left(x\right)$takes value of either 1 or 0, suggesting the class that the current pixel belongs. For multiclass classification, we can use one versus one, one versus rest, hierarchical support vector machine or other strategy to obtain the multiclass label.

Sometimes, data cannot be separated by a linear classifier. Therefore, kernel methods are used to map data from the original input space to a higher dimension space. Thanks to the kernel trick, we only need to know the form of the inner product in that space instead of using the explicit map [16]. Popular kernel functions include as follows:

Linear kernel:

$$K\left(x,x_j\right)=x\cdot x_j.$$E10

Polynomial kernel:

$$K\left(x,x_i\right)={\left[\left(x^T{x}_i\right)+1\right]}^q,\begin{array}{ll}& \end{array}q>0$$E11

where $q$is the polynomial order.

Radial basis function kernel:

$$K\left(x,x_i\right)=\text{exp}\left(-\frac{-{‖x-x_i‖}^2}{{\sigma }^2}\right)$$E12

where ${\sigma }^2$is kernel bandwidth.

Sigmoid kernel:

$$K\left(x,x_i\right)=\text{tanh}\left[v\left(x^T{x}_i\right)+c\right]$$E13

for appropriate values of $v$and $c$, so that Mercer’s conditions are satisfied [16].

3.2.3. Incorporating spatial information

Conventionally, hyperspectral data classification algorithms are proposed based on spectroscopic viewpoint, and they ignore the spatial information that embeds in neighboring pixels [18]. Integration of spatial and spectral information may improve the processing performance. We propose to combine spatial dimension and spectral dimension information to improve the classification accuracy. The proposed method investigates the spatial information based on the connection component labeling in the following. We generate the mean image by averaging data after dimension reduction over spectra bands. A component labeling algorithms then applied to the binarized mean image. In our system, if an object is marked by connected component labeling and over 60% pixels are labeled as a class, we consider that all pixels within this connected region actually belong to the associated material. The classification accuracy will be improved using this strategy.

3.2.4. Sequential processing

We use a sliding window to assemble the acquired hyperspectral data, whose columns are collected sequentially one after another. The use of a sliding window facilitates to incorporate the spatial information in processing. The width of the sliding window should be determined by considering the data acquisition rate, data processing speed, and spatial correlation of the observed scenario. In our system, the width of the sliding window ($L$) is set to 15. Our hyperspectral images are captured by a pushboom system where columns of images are collected sequentially one after another. After collecting each column, we use this column with several previous ones to form a sliding window and perform data processing steps within this window. Let $L$be the width of the sliding window, and we set $L=15$in our experimental (Figure 4).

Figure 4.

Sliding window.

4. Experimental results

We collect the hyperspectral data with our pushboom system of Gaia. The images are acquired in the 400–1000 nm wavelength range, with a spectral resolution of 7 nm, for a total of 128 wavelengths ($p=128$). Their image resolutions $h$and $w$are 650 and 348 ($650\times 348$), respectively. The hyperspectral data include four kinds of fruits: tomato, jujube, lemon, and orange. In this study, we use a sliding window of size 15 for online processing of data. Twenty-three sequential hyperspectral images are extracted for classification. The datasets captured are divided into training and testing sets, where 300 pixels of each material are used for training and 30,603 pixels are used for testing.

After data preprocessing, we select 300 pixels of each material from the training set as sample points to form a hyperspectral image. The pixels of the image are converted into row vectors by row or column to form a two-dimensional matrix, which is used for data reduction. The operation of the test set is the same as that of the training set.

After the PCA transformation, the eigenvalue distribution is shown in Figure 5. This scree plot shows that the first eight factors explain most of the variability. The remaining factors explain a very small proportion of the variability and are likely unimportant. We select the principal component, which takes 99% of the eigenvalues, as the data after dimensionality reduction. For fuzzy-set data reduction, we fold 128 bands with a triangular window of length 32, and then we sample the data using at each 16 points, so that the data dimension also reduces to 8. We use eight-connected component labeling method to remove the background of data after dimension reduction.

Figure 5.

Eigenvalue distribution.

We then study the classification results of GML principle and SVM. We classify the data obtained from dimension reduction and background material removal (see Figures 6 and 7). The result of classification with spatial information (connected region labeling) is shown in Figure 8.

Figure 6.

(a) GML classification with PCA and (b) GML classification with fuzzy sets.

Figure 7.

(a) SVM classification with PCA and (b) SVM classification with fuzzy sets.

Figure 8.

Classification with spatial information (connected region labeling) achieves almost 100% accuracy.

4.1. Performance evaluation of results

4.1.1. Confusion matrix

A confusion matrix is a table that is often used to describe the performance of a classifier on a set of test data for which the true values are known. It compares the classification result with the reference image, and we need to determine the labels of each point in the reference image in the classified image. The confusion matrixes of our experiment are shown in Table 2.

$$A=\left[\begin{array}{llll}{m}_{11}& m_{12}& \cdots & m_{1k}\\ m_{21}& m_{22}& \cdots & m_{2k}\\ ⋮& ⋮& & ⋮\\ m_{k1}& m_{k2}& \cdots & m_{\mathit{kk}}\end{array}\right]$$E14

Class		Actual class
		Chinese date	Lemon	Orange	Tomato	Row sum
(1) GML classification with PCA
Predict class	Chinese date	1745	0	0	0	1745
	Lemon	4	9618	0	0	9622
	Orange	0	183	11,654	0	11,837
	Tomato	26	14	0	7359	7399
	Column sum	1775	9815	11,654	7359	30,603
(2) GML classification with fuzzy sets
Predict class	Chinese date	1745	0	0	0	1745
	Lemon	3	9619	0	0	9622
	Orange	258	2	11,577	0	11,837
	Tomato	36	7	0	7356	7399
	Column sum	2042	9628	11,577	7356	30,603
(3) SVM classification with PCA
Predict class	Chinese date	1744	1	0	0	1745
	Lemon	6	9616	0	0	9622
	Orange	466	1089	10,282	0	11,837
	Tomato	29	0	0	7370	7399
	Column sum	2245	10,706	10,282	7370	30,603
(4) SVM classification with fuzzy sets
Predict class	Chinese date	1738	0	0	7	1745
	Lemon	5	9617	0	0	9622
	Orange	1489	2826	7522	0	11,837
	Tomato	28	1144	0	6227	7399
	Column sum	3260	13,587	7522	6234	30,603

Table 2.

Confusion matrix of classification results: (1) GML classification with PCA, (2) GML classification with fuzzy sets, (3) SVM classification with PCA, and (4) SVM classification with fuzzy sets.

where $m_{\mathit{ij}}$shows pixels should belong to class $i$, which is wrongly assigned to class $j$, and $k$is the class number of the classification results (Figures 9 and 10).

Figure 9.

(a) Confusion matrix of GML classification with PCA and (b) confusion matrix of GML classification with fuzzy sets.

Figure 10.

(a) Confusion matrix of SVM classification with fuzzy sets and (b) confusion matrix of SVM classification with fuzzy sets.

4.1.2. κ coefficient

κ can reflect the classification error of the whole image and solve the problem that the classification accuracy depends too much on the number of classes and the number of samples. κ is performed by adopting the following equation:

$$\kappa =\frac{N\sum _{i=0}^k{m}_{i,i}-\sum _{i=0}^k{m}_{i+}\times m_{+i}}{N^2-\sum _{i=0}^k{m}_{i+}\times m_{+i}}$$E15

where $m_{i+}$is the sum of the line $i$in the confusion matrix, and $m_{+i}$is the sum of the column $i$in the confusion matrix.

κ of GML based on PCA dimensionality reduction is 98.93%, and κ of SVM is 92.55%. κ of GML based on fuzzy-set reduction technique is 98.56%, and κ of SVM is 74.69%. From the results of κ, we can see that the classification based on PCA is better than fuzzy sets, GML is better than SVM, and GML based on PCA is the best method for sequential classification of hyperspectral images.

4.1.3. Other metrics

Other metrics include classification accuracy, product’s accuracy (PA), and omission errors (OEs).

Classification accuracy indicates the correct rate of the classifier, as illustrated in Eq. (16).

$$A_i=\frac{\sum _{i=0}^k{m}_{i,i}}{\sum _{i=0}^k{m}_{+i}}$$E16

PA is used to indicate the rate of the classification result that is correctly classified, as illustrated in Eq. (17). User’s accuracy is used to indicate the rate of the pixels that are correctly divided into class $I$to the total number of pixels that are divided into $I$classes, as shown in Eq. (18).

$${\mathit{PA}}_i=\frac{m_{i,i}}{m_{+i}}$$E17

$${\mathit{UA}}_i=\frac{m_{i,i}}{m_{i+}}$$E18

OEs represent the number of pixels in class $I$that is incorrectly assigned to other class, as shown in Eq. (19). Commission errors (CEs) indicate the percentage of other class pixels that are incorrectly divided into class $I$, as illustrated in Eq. (20).

$${\mathit{OE}}_i=1-\frac{m_{i,i}}{m_{+i}}$$E19

$${\mathit{CE}}_i=1-\frac{m_{i,i}}{m_{i+}}$$E20

Classification accuracy of GML based on PCA dimensionality reduction is 99.26%, and classification accuracy of SVM is 94.80%. Classification accuracy of GML based on fuzzy-set reduction technique is 99.00%, and classification accuracy of SVM is 82.03%. From this evaluation and Table 3, GML based on PCA dimensionality reduction is the proposed solution for sequential classification of hyperspectral images.

Other metrics	Chinese date	Lemon	Orange	Tomato
(1) GML classification with PCA
${\mathit{PA}}_i$	0.9831	0.9799	1	1
${\mathit{OE}}_i$	0.0169	0.0201	0	0
${\mathit{UA}}_i$	1	0.9996	0.9845	0.9946
${\mathit{CE}}_i$	0	0.0004	0.0155	0.054
(2) GML classification with fuzzy sets
${\mathit{PA}}_i$	0.8546	0.9991	1	1
${\mathit{OE}}_i$	0.1454	0.0009	0	0
${\mathit{UA}}_i$	1	0.9997	0.9780	0.9941
${\mathit{CE}}_i$	0	0.0003	0.0220	0.0059
(3) SVM classification with PCA
${\mathit{PA}}_i$	0.7768	0.8982	1	1
${\mathit{OE}}_i$	0.2232	0.1018	0	0
${\mathit{UA}}_i$	1	0.9994	0.8686	0.9961
${\mathit{CE}}_i$	0	0.0006	0.1314	0.0039
(4) SVM classification with fuzzy sets
${\mathit{PA}}_i$	0.5331	0.7078	1	0.9989
${\mathit{OE}}_i$	0.4669	0.2922	0	0.0011
${\mathit{UA}}_i$	0.9960	0.9995	0.6355	0.8416
${\mathit{CE}}_i$	0.0040	0.0005	0.3645	0.1584

Table 3.

Other metrics: (1) GML classification with PCA, (2) GML classification with fuzzy sets, (3) SVM classification with PCA, and (4) SVM classification with fuzzy sets.

5. Conclusion

The major objective of this chapter is to build a sequential hyperspectral classification method for an industrial material sorting system. We propose hyperspectral images captured by a pushboom system where columns of images are collected sequentially one after another to get sequential hyperspectral images. PCA and fuzzy sets are used for data decorrelation. We study the GML and SVM classification with the data obtained from dimension reduction and background material removal and carry out the performance analysis. The results show that the accuracy rate of GML based on PCA dimensionality reduction is 99.26%, and the accuracy rate of SVM is 94.80%. The accuracy of GML based on fuzzy-set reduction technique is 99.00%, and the accuracy rate of SVM is 82.03%. After combing the spatial and spectral information, the accuracy of classification of hyperspectral images can be 100%.

The designed framework shows several advantages in terms of processing speed, efficiency, and accuracy. It may play an important role in industrial material sorting for agricultural products, food, and industrial waste sorting.

Acknowledgments

This work was supported in part by National Natural Science Foundation of China under grant 61671382 and in part by National Natural Science Foundation of Shenzhen under grant JCYJ2017030155315873.