Open Access is an initiative that aims to make scientific research freely available to all. To date our community has made over 100 million downloads. It’s based on principles of collaboration, unobstructed discovery, and, most importantly, scientific progression. As PhD students, we found it difficult to access the research we needed, so we decided to create a new Open Access publisher that levels the playing field for scientists across the world. How? By making research easy to access, and puts the academic needs of the researchers before the business interests of publishers.
We are a community of more than 103,000 authors and editors from 3,291 institutions spanning 160 countries, including Nobel Prize winners and some of the world’s most-cited researchers. Publishing on IntechOpen allows authors to earn citations and find new collaborators, meaning more people see your work not only from your own field of study, but from other related fields too.
To purchase hard copies of this book, please contact the representative in India:
CBS Publishers & Distributors Pvt. Ltd.
www.cbspd.com
|
customercare@cbspd.com
Hyperspectral imaging (HSI) is a technology able to measure information about the spectral reflectance or transmission of light from the surface. The spectral data, usually within the ultraviolet and infrared regions of the electromagnetic spectrum, provide information about the interaction between light and different materials within the image. This fact enables the identification of different materials based on such spectral information. In recent years, this technology is being actively explored for clinical applications. One of the most relevant challenges in medical HSI is the information extraction, where image processing methods are used to extract useful information for disease detection and diagnosis. In this chapter, we provide an overview of the information extraction techniques for HSI. First, we introduce the background of HSI, and the main motivations of its usage for medical applications. Second, we present information extraction techniques based on both light propagation models within tissue and machine learning approaches. Then, we survey the usage of such information extraction techniques in HSI biomedical research applications. Finally, we discuss the main advantages and disadvantages of the most commonly used image processing approaches and the current challenges in HSI information extraction techniques in clinical applications.
Research Institute for Applied Microelectronics, University of Las Palmas de Gran Canaria, Spain
Martin Halicek
Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, USA
Medical College of Georgia, Augusta University, USA
Himar Fabelo
Research Institute for Applied Microelectronics, University of Las Palmas de Gran Canaria, Spain
Eduardo Quevedo
Research Institute for Applied Microelectronics, University of Las Palmas de Gran Canaria, Spain
Baowei Fei
Department of Bioengineering, University of Texas at Dallas, USA
Advanced Imaging Research Center, University of Texas Southwestern Medical Center, USA
Department of Radiology, University of Texas Southwestern Medical Center, USA
Gustavo Marrero Callico
Research Institute for Applied Microelectronics, University of Las Palmas de Gran Canaria, Spain
*Address all correspondence to: sortega@iuma.ulpgc.es
1. Introduction
Hyperspectral imaging (HSI), also known as imaging spectroscopy, is a technology capable of sampling hundreds of narrow spectral bands across the electromagnetic spectrum through the use of an optical element that disperses the incoming radiation into certain wavelengths [1]. This technology combines the main features of two existing technologies: imaging and spectroscopy, making possible to exploit both the morphological features and the chemical composition of objects captured by a camera. The interaction between electromagnetic radiation and matter is distinctive for each material, therefore by using this technology it is possible to discriminate among different materials [2]. The characteristic spectral curve associated with a certain material is called spectral signature or spectral fingerprint, and through its analysis it is possible to differentiate among different materials or substances. The data structure used in HSI comprises both the spectral and spatial features from a given scene, and is referred to as hyperspectral (HS) cube. Figure 1 shows a graphical representation of an HS cube with an example of a spectral signature for the top-right pixel.
Figure 1.
Example of a HS cube with an example of a spectral signature.
Although historically HSI has been applied to remote sensing [3], in recent years this technology has become a trending topic in different research fields such as food quality analysis [4, 5], military and security applications [6] or agriculture [7, 8], among many others [9]. HSI is also an emerging imaging modality in the medical field. It has been proven that the interaction between the electromagnetic radiation and matter carries useful information for diagnostic proposes [10]. As an alternative diagnostic tool, one of the strengths offered by HSI is being completely non-invasive and label-free. In medical research applications, this technology has been employed for more than twenty years in different areas such as the analysis of cancerous tissues in in-vivo and ex-vivo samples [11], digital and computational pathology [12], melanoma detection [13] or several gastroenterology diseases [14].
In this chapter, a survey of the most common processing frameworks employed in the literature for information extraction in medical HSI will be presented. First, a brief introduction of the optical properties of biological tissues is provided. Second, the most common information extraction methods employed for HSI medical processing are described and discussed, including optical inverse modeling and machine learning methods. The last section summarizes the conclusions reached in this literature analysis.
The interaction between light and biological tissues has been proven to be a useful tool to identify and classify several diseases. Absorption, refraction and scattering are the three different types of interaction that can be measured in biological tissues [15]. Light absorption measures the amount of light absorbed and transformed to energy by tissue molecules. Specific wavelengths of the spectrum will present absorption peaks related to the transitions between two energy levels in a molecule, which can provide tissue diagnostic information. Absorption is the inverse measurement of reflectance using HSI systems. The measurement of refraction and reflection of light is based on changes in speed and direction of the incident light into tissue. Particularly, hemoglobin (Hb) is the major component of the spectral signature between 450 and 600 nm of biological tissues, and spectral differences can be observed in the absorption/reflectance between oxygenated and deoxygenated Hb states [16]. A single absorbance peak is found at 560 nm in deoxygenated Hb, while two absorbance peaks are found at 540 and 580 nm in oxygenated Hb [17]. Figure 2 shows an example of these Hb signatures published in [18].
Figure 2.
Oxy-Hb (a) and deoxy-Hb (b) normalized absorption spectra, with Hb concentrations of 50 g/L and 68 g/L, respectively. The solid lines are experimentally measured, and the dotted black lines are the ideal. Oxy-Hb (c) and deoxy-Hb (d) measured and theoretical attenuation coefficients [18].
Regarding the measurement of light scattering, it is achieved when there is a spatial variation of the reflective index in the illuminated tissue. Scattering properties can be highly useful in diagnostic applications, since they provide different variations in tissue affected by a certain disease [19]. For example, the spectral range between 700 and 900 nm is related with the scattering dominant optical properties of collagen [20]. Also, the near-infrared spectral region is the scattering dominant region of fat, lipids, collagen, and water. Moreover, several tissues have fluorescence properties that can be revealed when such tissue is excited with certain wavelengths. As an example, ultraviolet light can be used to excite tissues, revealing the fluorescence emission of proteins and nucleic acids [21]. More details about biological tissue optical properties can be found in [19].
There are two main types of medical HSI processing: optical inverse modeling and machine learning approaches. In this section, both methods will be presented in detail, showing their main characteristics, as well as their advantages and disadvantages.
3.1 Optical inverse modeling
In optical inverse modeling techniques, a mathematical equation which models the interaction between the light and tissue is proposed, and the collected HS data is used to extract optical properties, such as the absorption or scattering of tissue. First, a physics-based model is proposed for the light propagation in tissues. Second, the HS data are used to extract optical properties from the proposed light propagation model. Although the number of studies which make use of this type of approach is limited, some researchers have used HS and light transport models in tissue to extract useful information for the detection of different diseases or conditions. Milanic et al. used Monte Carlo simulations of a light transport model in skin to extract information about the contents of melanin and blood saturation, with the goal of measuring cholesterol levels in human skin [22]. The same authors performed a similar processing analysis to skin HS data, but with the goal of detecting arthritis [23]. Claridge et al. demonstrated the utility of optical inverse modeling techniques for the estimation of the blood volume fraction of ex-vivo colon samples, showing statistically significant differences between the blood volume fraction of tumor and healthy conditions [24].
The use of optical inverse modeling for information extraction in medical HSI presents some advantages and challenges. The main advantage is to count with an established physical-based model for correlating measured data, which are theoretically strong and contain tissue optical parameters that can be used for diagnostics. The main disadvantage of this approach is the possibility of bias in the model development and over-simplification of complex physical processes, which could result in suboptimal performance for information extraction.
3.2 Machine learning methods
Machine Learning (ML) methods are algorithms able to learn from data. ML algorithms enable solutions to difficult tasks which usually cannot be performed by a traditionally designed computer program [25]. There are different ML algorithms depending on the task they perform. In regression problems, a numerical variable is estimated from the data. In the context of medical HSI, Arimoto et al. used regression techniques to estimate the oxygen saturation map from human retina [26]. In classification problems, the objective is to assign a data sample to a fixed category. For example, Fabelo et al. used classification to identify normal tissue, tumor tissue, hypervascularized tissue and background in HS images from in-vivo human brain tissue [27]. The results of the classification of a medical HS image are usually represented as a classification map or heat map, where different colors are used for each class (Figure 3).
Figure 3.
Example of classification and heat maps obtained through ML classification from (A) in-vivo brain tissue HS images [27] and (B) in-vitro H&E brain tissue HS images [28].
ML algorithms can be classified as supervised and unsupervised. In unsupervised algorithms, the goal is to cluster similar data samples in groups, extracting the information from data features. In supervised algorithms, the data is comprised of the data features and associated labels [29]. For example, in the example of Figure 3A, the data features consist of the spectra of each pixel of the HS image, and the labels are the different categories into which each pixel can be categorized, i.e. normal tissue, tumor tissue, hypervascularized tissue and background. The main goal of supervised algorithms is to use data and their labels to train a model which can be used to perform predictions about new data. ML techniques can be categorized as Feature Learning (FL) or Deep Learning (DL) methods. In FL approaches, the inputs of a supervised classifier are given by features extracted from the data. For example, in an image processing framework, such features may be related to shape, texture or color. On the contrary, DL approaches are devoted to use all the data as input to a supervised classifier, and the important features to perform the classification task are learned by the supervised classifier.
There are challenges related with both types of ML approaches. On the one hand, in FL methods, the classification may be biased by which features are selected from the data for the classification, while the identification of features is performed automatically in a DL algorithm. On the other hand, DL methods usually require large amounts of data to succeed in the feature extraction and classification, while FL approaches may provide good performance with a limited dataset. Next, we provide a survey about the different ML approaches which are commonly used for HSI processing in medical applications.
3.2.1 Feature learning
In this section, we describe the most common FL approaches which have been employed for processing medical HS data. This section is categorized in three main categories, namely pixel-wise classification, feature extraction and selection methods, and the usage of both the spatial and spectral information.
3.2.1.1 Pixel-wise classifiers
In the HS literature, the concept of pixel-wise processing refers to the exclusive usage of the spectral information within an HS cube for extracting information from HS data. Recently, Ghamisi et al. performed a survey between the most commonly used classifiers in pixel-wise classification of HS images [30]. The most common classifiers used for the classification of HS images from a feature learning perspective are Support Vector Machines (SVMs), Random Forest (RF) and Multinomial Logistic Regression (MLR) based approaches.
SVM is a binary classification algorithm proposed by Vapnik [31]. The algorithm finds the optimal hyperplane that maximizes the margin between samples belonging to different classes. Although it was originally designed for linear classification, an SVM classifier can be used for nonlinear classification problems by using different kernels to map the data into a higher dimensional space. SVM has been shown to provide competitive classification performance on HS data even with a limited training sample size [32].
RF was firstly proposed by Breiman [33]. This algorithm consists of an ensemble of decision trees, where, in each decision tree, the training data are hierarchically partitioned into smaller homogeneous groups. In RF, different decision trees are generated from the training data, and the different classification results are combined by a voting process. The main advantage of RF is a reduced training time. RF has been successfully used for the classification of HS images [34].
Finally, MLR [35] approaches exploit the posterior class distributions of the training data for making predictions, and these methods have been successfully applied for the classification of HS images. The main advantages of MLR are fast computation for training and customizability, which allows modifications to the original algorithm to provide better generalization, e.g. sparsity constraints or multiple feature learning.
In the context of medical HS classification, several authors have utilized the spectral information for the diagnosis of different diseases in a pixel-wise manner. The most commonly used pixel-wise classifier in medical HSI is SVM. In the context of surgical guidance, Akbari et al. processed HS images from the abdomen to detect intestinal ischemia [36]. For cancer detection, SVM and HSI have been used for the identification of gastric cancer [37], prostate cancer [38], tongue cancer [39], and skin cancer [40]. Although RF and MLR have been widely used for HS information extraction, their usage in medical HSI is limited. RF has been used for the detection of in-vivo oral cancer [41], while MLR has been considered for identification of ulcerative colitis in histological slides [42]. The main challenge in this field is to determine which pixel-wise classifier is more suitable for the classification of certain HS data. In this sense, some authors have performed comparisons of performance of different pixel-wise classifiers for the detection of brain cancer in histological slides [43], or the detection of the tumor margins in head and neck ex-vivo tissue [44]. Although SVM has been shown to outperform other classifiers, a deeper comparison between different classifiers should be urgently performed to definitively demonstrate which pixel-wise classifier performs better with HSI across multiple applications.
3.2.1.2 Feature extraction and feature selection
HS data are characterized by a high dimensionality. For this reason, instead of exploiting the complete spectral signature for image analysis, one trend in HSI processing is the use of Dimensionality Reduction (DR) methods. These methods are devoted to reduce dimensionality of the original data while preserving the most relevant information [45]. DR methods have been extensively used for HS image processing. There are two main types of DR approaches: feature extraction and feature selection methods.
On the one hand, in feature extraction methods, a transformation is applied to the data to generate a new representation with lower dimensionality, but similar information content. The most studied DR algorithm for HSI is Principal Component Analysis (PCA). The goal of PCA [46] is searching for a linear transformation of the data by using orthogonal projections which minimize the covariance matrix of the original data. On the other hand, several data transformation approaches have been proposed for dimensionality reduction, such as wavelet transformations [47], different orthogonal projection approaches, or the exploitation of manifold embedding [48].
Nevertheless, in feature extraction methods the data are transformed, and thus the physical information about specific wavelengths is lost, which means that the provided interaction between light and tissue cannot be analyzed, which may affect certain applications. For this reason, feature selection methods are devoted to find the most relevant features from the original data by keeping the most relevant information. In the context of HSI, feature selection methods are also known as band selection methods, which also seek to identify the most relevant spectral features for a certain application. There are several types of band selection methods. In this chapter, we only describe the most prominent methods used in medical HSI. In a large-dissimilarity criteria approach, the goal is to select the most dissimilar spectral bands. Conversely, in a low-correlation criterion, the spectral bands showing low correlation between each other are selected. An example of this kind of algorithm is Maximum Relevance Minimum Redundancy (mRMR). In search-based methods, the band selection is performed by solving an optimization problem driven by a given optimization function. These algorithms search for the best bands to solve such optimization problem. Some search-based methods used in HSI are Genetic Algorithm (GA) [49] or Particle Swarm Optimization (PSO) [50]. Further details about more sophisticated band selection techniques can be found in [51].
In the context of medical HSI, feature extraction methods are used both as standalone methods and as a preprocessing stage before further data analysis. The former approach is to enhance the visualization of data, while the latter reduces the complexity of the data for being processed by other machine learning approaches. As an example of the direct application of PCA for tissue visualization enhancement, Zuzak et al. applied PCA to abdominal HS images in order to enhance the visualization of biliary trees using in-vivo samples [52]. Also, Wilson et al. demonstrated the ability of HSI for melanin detection in histological unstained specimens of melanocytic lesions in the skin and the eye using PCA and false-color representations of data [53]. PCA has been used for extracting the most important features of HS data prior to classification in different applications, such as the detection of in-vivo oral cancer [54], prostate cancer in histological slides [55], the identification of white blood cells in blood smear slides [56] or the intraoperative delineation of brain tumors [57]. Another example of the utility of feature extraction methods was demonstrated by Hadoux et al., where relevant differences between the retinal spectral data from patients with Alzheimer and healthy patients were found after applying an orthogonal projection of data [58]. Such differences in the spectral signature from different disease states were not possible using the raw spectral signature of tissue. Beyond PCA and orthogonal projection methods, Ravi et al. proposed a modification of the t-Distributed Stochastic Neighbor Embedding feature extraction algorithm, a non-linear dimensionality reduction technique, prior to the identification of tumor tissue within in-vivo brain samples [59]. Other feature extraction methods used in medical HSI prior to classification are the use of wavelet transformation for the detection of prostate cancer in mice models [60], or the use of Fourier Series coefficients for breast cancer detection [61].
The use of band selection methods for medical HSI applications is not as extended as in other fields, such as remote sensing. However, some researchers have successfully exploited different band selection methods in HSI. Goto et al. used the Mahalanobis distance to determine the optimal wavelengths for gastric cancer, correctly identifying normal and tumor mucosa [62]. Additionally, mRMR has been used for the identification of the most relevant bands for ex-vivo breast cancer detection [61], and for in-vivo head and neck cancer [63]. Finally, Martinez-Vega et al. proposed a search-based method based on different optimization algorithms for the identification of the most relevant wavelengths for brain tumor detection within in-vivo HS images [64]. The optimization function was the pixel-wise classification performance metrics obtained by an SVM classifier. The results demonstrated that a GA optimization slightly improves tumor identification compared to the full-spectra counterpart.
Both feature selection and feature extraction methods aim to reduce the dimensionality of HS data while retaining the most important information. Successful application of these techniques leads to reduced computational time, which is required in applications such as surgical guidance. Nevertheless, for biomedical HS applications, there are some relevant advantages of using band selection methods instead of feature extraction methods. The first advantage is that the information about the concrete wavelengths that are used is retained. This fact allows further analysis about the physical response of different tissues to specific wavelengths. The second advantage of band selection methods is the possibility of developing custom HS cameras which only captures the most relevant spectral channels for a given application. Such reduced-band cameras would be able to acquire HS video, which would be also convenient for some surgical guidance applications.
3.2.1.3 Spatial-spectral information
Although the aforementioned data processing methods rely on the spectral information, a HS cube is a 3D data structure containing both the spatial and the spectral information of a scene. In a recent review manuscript, He et al. provided a survey about different spatial-spectral techniques which have been used for the classification of HSI [65]. The inclusion of both spectral and spatial information is motivated by the limitations found in the spectral data. First, the high dimensionality of spectral data together with a limited dataset can lead to the curse of dimensionality. This phenomenon offers more detailed information about the captured scene, but it also contains redundant information and increases the computational time required to process the data [4]. Second, the high variability shown in the spectral data due to different lighting conditions, instrumentation noise, or other phenomena, makes the classification based only on the spectral information a challenge. In addition, high intra-class and low inter-class variability of the spectral signatures produces difficulties in the differentiation between classes. This problem is particularly challenging in biomedical data, where data originate from multiple patients. For these reasons, researchers within the HSI processing community have successfully improved the classification of pixel-wise approaches by the utilization of spatial and spectral features from HS images.
In [65], the authors proposed a classification of spatial-spectral approaches in three main types, depending on how the spatial information is integrated in the processing framework. In pre-processing approaches, spatial and spectral features are extracted from the HS cube, and then such features are used for the classification. In integrated classification, both spatial and spectral features are used to train the classifier. Finally, in post-processing approaches, the spatial information is employed to refine the results of a pixel-wise processing of the HS cube.
In the context of medical HSI processing, most of the spatial-spectral approaches have been focused in pre-processing and post-processing schemes. Some pre-processing approaches are the following. In leukemia detection in blood smear slides, Wang et al. evaluated the usage of three types of inputs for a supervised classifier: spatial features, spectral features, and spatial-spectral features. The results of this study suggest that the exploitation of both the spatial and the spectral features significantly improves the quality of the classification [66]. Similarly, Li et al. evaluated the feasibility of utilizing HSI for Red Blood Cell (RBC) counting. After conducting the RBC counting using uniquely spatial or spectral features of blood cells, the authors found an improvement in the under-counting and over-counting rates when they performed the image analysis using both types of features together [67]. Ortega et al. make use of the spatial information of the HS data by performing superpixel segmentation [68]. In post-processing approaches, Fabelo et al. proposed the incorporation of the spatial information to the SVM pixel-wise classification by using a K-nearest neighbors spatial filter which makes use of a one-dimensional representation of the HS cube extracted using PCA for the identification of in-vivo brain tumor [57].
3.2.2 Deep learning methods
Deep Learning is a family of machine learning algorithms that learn abstract features to best represent and make predictions about new data that is presented. More specifically, neural networks (NNs) consist of consecutive layers of neurons that have non-linear activations that connect the input data, extract features, and connect to logical outputs representing the classes of labels to provide prediction probabilities. Neural networks can have various dimensionalities, which largely depends on the size and dimensions of the input data. For example, utilizing only spectral signature information, a 1-D NN can extract features with fully-connected layers or 1-D convolutions. However, HS cameras acquire spatial information and spectral signatures simultaneously. Therefore, to exploit both sets of features, pseudo 3-D HS data can be input directly into a 2D-CNN and extract spatial features with learned convolutional kernels in the spatial domain, and these filters are connected across the entire spectral domain of the HS data. Lastly, 3D-CNN can utilize the full pseudo 3D HS data as input and extract spatial-spectral features with 3D convolutional kernels. There are numerous approaches, but these methods require more computational processing as more features and dimensions are involved.
The most widely used approach is 2D-CNNs. Aggressive brain tumors, such as glioblastoma, often require surgical resection for treatment, and surgeons often implement multiple imaging modalities, including fluorescence, to aid in this very challenging task. In a pilot study to aid brain surgeons with label-free HSI, Fabelo et al. compared both 2D-CNN and 1D-DNN, considering spectral-only and spectral-spatial classification using DL [69]. In HSI digital histology, Ortega et al. detected glioblastoma brain cancer in digital slides using a patch-based 2D-CNN approach [70]. Additionally, Halicek et al. has employed very deep 2D-CNNs for classification, specifically the widely-used Inception v4 model (Figure 4) implemented in a sliding patch-based approach for head and neck squamous cancer [71] and thyroid and salivary gland cancers [72]. For comparing 2D-CNN and 3D-CNNs, in [73] Halicek et al. explored spatial-spectral convolutions in 3D CNNs with 3D convolutional kernels to 2D approaches. Although data were limited to only 12 patients, the preliminary results suggest 3D convolutions outperformed 2D convolutions for CNN design at the cost of computational power and speed.
Figure 4.
Schematic diagram of the modified inception v4 CNN architecture. The CNN was customized to operate on the 25 × 25 × 91 patch-size selected. The receptive field size and number of convolutional filters is shown at bottom of each inception block. The convolutional kernel size used for convolutions is shown in italics inside each convolution box. Squeeze-and-excitation modules were added to the CNN to increase performance [72].
Another desired application of DL for HSI is semantic segmentation, which allows the entire scene to be classified altogether from spectral-spatial features in the entire scene. Semantic segmentation does not require image reconstruction like patch-based 2D-CNN approaches. The most commonly used method is the U-Net, as first used in HSI by Trajanovski et al. for tongue cancer detection with a 2D input data using all HS channels for semantic segmentation of ex-vivo specimens [74]. Additionally, Kho et al. used ex-vivo specimens from patients with breast cancer and applied a standard U-net with 2D input HS data using all spectral channels for semantic segmentation [75].
More recently, several modern DL approaches with origins in computer-vision have been applied to medical HSI experimentally. In [76], a generative adversarial network (GAN) was applied to use DL to learn the association of RGB images and HS images to learn the ability to generate HS digital histology images from standard RGB digital histology images of breast cancer. Another modern approach is long-short-term-memory (LSTM) and recurrent neural networks (RNN) which can utilize spatial-spectral and time-based inputs to operate in real-time video approaches. In [77], RNNs are compared to and outperform 2D- and 3D-CNN methods for in-vivo cancer detection with the goal of real-time video endoscopy.
The use of DL for HS processing is currently a hot topic in the research community in different fields. The main advantage of DL approaches in HSI is their capability to exploit jointly the spatial and the spectral information for image processing tasks. Currently, researchers are experimenting with different DL architectures in order to find the most appropriate DL model for HSI [78]. In the context of medical HSI, the use of DL in medical HS have shown good performance in different applications, but its usage is still limited compared to other ML approaches. The main reason is the limited number of data due to the novelty of the technology. More publicly available datasets with a large number of patients are required in order to definitively establish an adequate comparative of DL and traditional ML techniques.
In this book chapter, we provide a survey about the most common processing frameworks for information extraction in medical HSI. First, we show the main motivations on the usage of HS technology for biomedical data: the interaction between the light and tissue provides useful information for diagnostic applications. Second, we survey the most common approaches for HSI processing in the medical field: inverse optical modeling and machine learning approaches.
Within the ML approaches, we show there is a big variety in the methods which are used, mainly in two different types: traditional machine learning approaches based handcrafted features and recent DL techniques. Even within each subfield, the variety of options to extract information in medical HSI is still wide.
In Table 1 we provide a summary of the applications of the different methods which have been described in this book chapter. Such table relates the main information extraction methods and the biomedical applications of HSI. Further literature revision about the different biomedical HSI applications are out of the scope of this chapter. However, we recommend readers who are interested in further information about the usage of HSI for different biomedical applications to refer the different literature reviews in this context mentioned in the introduction section.
Information extraction method
Algorithm
Application
Ref.
Optical inverse modeling
Light transport models and Monte Carlo Simulations
Summary of information extraction methods for medical HSI.
Publicly available datasets are marked with ‡.
The main challenge in HS medical image processing is to determine which processing framework is the most appropriate for clinical applications. Nowadays, the current trend for researchers working with medical HS data is to collect their own data, and then propose a processing framework to address a certain problem. Normally such processing frameworks are customized for their particular applications. In order to reach an agreement by the research community on the most successful information extraction methods for HSI, there is the need of further investigations with comparisons among the most promising processing approaches. To this end, the availability of large public datasets would help. However, although there is no general processing framework, the different information extraction techniques together with HS medical data have demonstrated several advantages for biomedical applications.
This research was supported in part by the Canary Islands Government through the ACIISI (Canarian Agency for Research, Innovation and the Information Society), ITHACA project under Grant Agreement ProID2017010164 and by the Spanish Government through PLATINO project (TEC2017-86722-C4-4-R). This research was supported in part by the Cancer Prevention and Research Institute of Texas (CPRIT) grant RP190588 and the U.S. National Institutes of Health (NIH) grants (R01CA156775, R01CA204254, R01HL140325, and R21CA231911). This work was completed while Samuel Ortega was beneficiary of a pre-doctoral grant given by the “Agencia Canaria de Investigacion, Innovacion y Sociedad de la Información (ACIISI)” of the “Conserjería de Economía, Industria, Comercio y Conocimiento” of the “Gobierno de Canarias,” which is part-financed by the European Social Fund (FSE) (POC 2014-2020, Eje 3 Tema Prioritario 74 (85%)).
The authors declare that there are no conflicts of interest related to this chapter.
References
1.Kamruzzaman M, Sun D-W. Introduction to Hyperspectral Imaging Technology. In: Computer Vision Technology for Food Quality Evaluation [Internet]. Elsevier; 2016 [cited 2018 Sep 11]. p. 111-39. Available from: https://www.sciencedirect.com/science/article/pii/B9780128022320000050
2.Soares JANT. Introduction to Optical Characterization of Materials. In: Practical Materials Characterization. Springer Nature; 2014. p. 43-92
3.Sudharsan S, Hemalatha R, Radha S. A survey on hyperspectral imaging for mineral exploration using machine learning algorithms. In: 2019 International Conference on Wireless Communications, Signal Processing and Networking, WiSPNET 2019. Institute of Electrical and Electronics Engineers Inc.; 2019. p. 206-12
4.Lei T, Sun DW. Developments of nondestructive techniques for evaluating quality attributes of cheeses: A review. Vol. 88, Trends in Food Science and Technology. Elsevier Ltd; 2019. p. 527-42
5.Vejarano R, Siche R, Tesfaye W. Evaluation of biological contaminants in foods by hyperspectral imaging: A review. Int J Food Prop [Internet]. 2017 Jun 8 [cited 2020 Apr 22];20:1-34. Available from: https://www.tandfonline.com/doi/full/10.1080/10942912.2017.1338729
6.Shimoni M, Haelterman R, Perneel C. Hypersectral imaging for military and security applications: Combining Myriad processing and sensing techniques. IEEE Geosci Remote Sens Mag. 2019 Jun 1;7(2):101-17
7.Mishra P, Asaari MSM, Herrero-Langreo A, Lohumi S, Diezma B, Scheunders P. Close range hyperspectral imaging of plants: A review. Vol. 164, Biosystems Engineering. Academic Press; 2017. p. 49-67
8.Adão T, Hruška J, Pádua L, Bessa J, Peres E, Morais R, et al. Hyperspectral imaging: A review on UAV-based sensors, data processing and applications for agriculture and forestry. Remote Sens. 2017;9(11)
9.Khan MJ, Khan HS, Yousaf A, Khurshid K, Abbas A. Modern Trends in Hyperspectral Image Analysis: A Review. Vol. 6, IEEE Access. Institute of Electrical and Electronics Engineers Inc.; 2018. p. 14118-29
10.Lu G, Fei B. Medical hyperspectral imaging: a review. J Biomed Opt [Internet]. 2014;19(1):10901. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3895860&tool=pmcentrez&rendertype=abstract
11.Halicek M, Fabelo H, Ortega S, Callico GM, Fei B. In-Vivo and Ex-Vivo Tissue Analysis through Hyperspectral Imaging Techniques: Revealing the Invisible Features of Cancer. Cancers (Basel) [Internet]. 2019 May 30;11(6):756. Available from: https://www.mdpi.com/2072-6694/11/6/756
12.Ortega S, Halicek M, Fabelo H, Callico GM, Fei B. Hyperspectral and multispectral imaging in digital and computational pathology: a systematic review [Invited]. Biomed Opt Express [Internet]. 2020 Jun 1 [cited 2020 May 22];11(6):3195. Available from: https://www.osapublishing.org/abstract.cfm?URI=boe-11-6-3195
13.Johansen TH, Møllersen K, Ortega S, Fabelo H, Garcia A, Callico GM, et al. Recent advances in hyperspectral imaging for melanoma detection. Wiley Interdiscip Rev Comput Stat [Internet]. 2019 Apr 22 [cited 2019 May 17];e1465. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1002/wics.1465
14.Ortega S, Fabelo H, Iakovidis D, Koulaouzidis A, Callico G, Ortega S, et al. Use of Hyperspectral/Multispectral Imaging in Gastroenterology. Shedding Some–Different–Light into the Dark. J Clin Med [Internet]. 2019 Jan 1 [cited 2019 Jan 12];8(1):36. Available from: http://www.mdpi.com/2077-0383/8/1/36
15.Tuchin V V. Tissue optics: Light scattering methods and instruments for medical diagnosis: Third edition [Internet]. Tuchin V, editor. Tissue Optics: Light Scattering Methods and Instruments for Medical Diagnosis: Third Edition. SPIE; 2015 [cited 2020 Mar 27]. 1-935 p. Available from: file:///C:/Users/Cooper/Downloads/PM254_ch2.pdf
16.Chen P-C, Lin W-C. Spectral-profile-based algorithm for hemoglobin oxygen saturation determination from diffuse reflectance spectra. Biomed Opt Express [Internet]. 2011 May 1 [cited 2019 Mar 20];2(5):1082. Available from: https://www.osapublishing.org/boe/abstract.cfm?uri=boe-2-5-1082
17.Eaton WA, Hanson LK, Stephens PJ, Sutherland JC, Dunn JBR. Optical spectra of oxy- and deoxyhemoglobin. J Am Chem Soc [Internet]. 1978 Aug [cited 2019 Mar 20];100(16):4991-5003. Available from: http://pubs.acs.org/doi/abs/10.1021/ja00484a013
18.Robles FE, Chowdhury S, Wax A. Assessing hemoglobin concentration using spectroscopic optical coherence tomography for feasibility of tissue diagnostics. Biomed Opt Express [Internet]. 2010 Jul 27 [cited 2018 Dec 31];1(1):310-7. Available from: http://www.ncbi.nlm.nih.gov/pubmed/21258468
19.Jacques SL. Optical properties of biological tissues: a review. Phys Med Biol [Internet]. 2013 May 7 [cited 2019 Mar 20];58(11):R37-61. Available from: http://stacks.iop.org/0031-9155/58/i=11/a=R37?key=crossref.e58b67eb50f9f6507dd5e75b7744fa07
20.Sekar SKV, Bargigia I, Mora AD, Taroni P, Ruggeri A, Tosi A, et al. Diffuse optical characterization of collagen absorption from 500 to 1700 nm. J Biomed Opt [Internet]. 2017 Jan 31 [cited 2019 Mar 20];22(1):015006. Available from: http://www.ncbi.nlm.nih.gov/pubmed/28138693
21.Excited States of Proteins and Nucleic Acids. Excited States of Proteins and Nucleic Acids. Springer US; 1971
22.Milanic M, Bjorgan A, Larsson M, Strömberg T, Randeberg LL. Detection of hypercholesterolemia using hyperspectral imaging of human skin. In: Brown JQ , Deckert V, editors. ProcSPIE - European Conference on Biomedical Optics [Internet]. SPIE-Intl Soc Optical Eng; 2015. p. 95370C. Available from: http://proceedings.spiedigitallibrary.org/proceeding.aspx?doi=10.1117/12.2183880
23.Milanic M, Paluchowski LA, Randeberg LL. Hyperspectral imaging for detection of arthritis: feasibility and prospects. J Biomed Opt [Internet]. 2015 Sep;20(9):096011. Available from: http://biomedicaloptics.spiedigitallibrary.org/article.aspx?doi=10.1117/1.JBO.20.9.096011
24.Claridge E, Hidović-Rowe D, Taniere P, Ismail T. Quantifying mucosal blood volume fraction from multispectral images of the colon. In: Manduca A, Hu XP, editors. Medical Imaging 2007: Physiology, Function, and Structure from Medical Images. SPIE; 2007
25.Goodfellow I, Bengio Y, Courville A. Deep Learning. The MIT Press; 2016
26.Arimoto H, Furukawa H. Retinal blood oxygen saturation mapping by multispectral imaging and morphological angiography. In: Annual International Conference of the IEEE Engineering in Medicine and Biology - Proceedings. 2007. p. 1627-30
27.Fabelo H, Ortega S, Ravi D, Kiran BR, Sosa C, Bulters D, et al. Spatio-spectral classification of hyperspectral images for brain cancer detection during surgical operations. Fred AL, editor. PLoS One [Internet]. 2018 Mar 19;13(3):e0193721. Available from: https://doi.org/10.1371/journal.pone.0193721
28.Ortega S, Fabelo H, Halicek M, Camacho R, Plaza M de la L, Callicó GM, et al. Hyperspectral Superpixel-Wise Glioblastoma Tumor Detection in Histological Samples. Appl Sci [Internet]. 2020 Jun 28 [cited 2020 Jul 21];10(13):4448. Available from: https://www.mdpi.com/2076-3417/10/13/4448
29.Bishop CM. Pattern Recognition and Machine Learning (Information Science and Statistics). 1st ed. Springer; 2007
30.Ghamisi P, Plaza J, Chen Y, Li J, Plaza AJ. Advanced Spectral Classifiers for Hyperspectral Images: A review. IEEE Geoscience and Remote Sensing Magazine Institute of Electrical and Electronics Engineers Inc.; Mar 1, 2017 p. 8-32
31.Vapnik V. Support vector machine. Mach Learn. 1995;20(3):273-97
32.Melgani F, Bruzzone L. Classification of hyperspectral remote sensing images with support vector machines. IEEE Trans Geosci Remote Sens. 2004;42(8):1778-90
33.Breiman L. Random forests. Mach Learn. 2001;45(1):5-32
34.Raczko E, Zagajewski B. Comparison of support vector machine, random forest and neural network classifiers for tree species classification on airborne hyperspectral APEX images. Eur J Remote Sens. 2017 Jan;50(1):144-54
35.Böhning D. Multinomial logistic regression algorithm. Ann Inst Stat Math [Internet]. 1992 Mar [cited 2020 Jul 21];44(1):197-200. Available from: https://link.springer.com/article/10.1007/BF00048682
36.Akbari H, Kosugi Y, Kojima K, Tanaka N. Detection and Analysis of the Intestinal Ischemia Using Visible and Invisible Hyperspectral Imaging. IEEE Trans Biomed Eng. 2010;57(8):2011-7
37.Akbari H, Uto K, Kosugi Y, Kojima K, Tanaka N. Cancer detection using infrared hyperspectral imaging. Cancer Sci. 2011 Feb;102(4):852-7
38.Akbari H, Halig L V., Schuster DM, Osunkoya A, Master V, Nieh PT, et al. Hyperspectral imaging and quantitative analysis for prostate cancer detection. J Biomed Opt [Internet]. 2012 Jul;17(7):0760051. Available from: http://biomedicaloptics.spiedigitallibrary.org/article.aspx?doi=10.1117/1.JBO.17.7.076005
39.Liu Z, Wang H, Li Q . Tongue tumor detection in medical hyperspectral images. Sensors [Internet]. 2011;12(1):162-74. Available from: https://www.scopus.com/inward/record.uri?eid=2-s2.0-84863076803&partnerID=40&md5=af7557174562a8e50e3243bcc91cb839
40.Leon R, Martinez-Vega B, Fabelo H, Ortega S, Melian V, Castaño I, et al. Non-Invasive Skin Cancer Diagnosis Using Hyperspectral Imaging for In-Situ Clinical Support. J Clin Med [Internet]. 2020 Jun 1 [cited 2020 Jul 21];9(6):1662. Available from: https://www.mdpi.com/2077-0383/9/6/1662
41.Regeling B, Laffers W, Gerstner AOHH, Westermann S, Müller NA, Schmidt K, et al. Development of an image pre-processor for operational hyperspectral laryngeal cancer detection. J Biophotonics [Internet]. 2016 Mar [cited 2017 May 12];9(3):235-45. Available from: http://dx.doi.org/10.1002/jbio.201500151
42.Nakaya D, Tomiyama Y, Satori S, Saegusa M, Yoshida T, Yokoi A, et al. Development of high-performance pathological diagnosis software using a hyperspectral camera. In: 2018 IEEE EMBS Conference on Biomedical Engineering and Sciences, IECBES 2018 - Proceedings. Institute of Electrical and Electronics Engineers Inc.; 2019. p. 217-20
43.Ortega S, Fabelo H, Camacho R, de la Luz Plaza M, Callicó GMGM, Sarmiento R, et al. Detecting brain tumor in pathological slides using hyperspectral imaging. Biomed Opt Express [Internet]. 2018 Feb 1;9(2):818-31. Available from: https://www.osapublishing.org/abstract.cfm?URI=boe-9-2-818
44.Lu G, Little J V., Wang X, Zhang H, Patel MR, Griffith CC, et al. Detection of head and neck cancer in surgical specimens using quantitative hyperspectral imaging. Clin Cancer Res. 2017;23(18):5426-36
45.Van Der Maaten LJP, Postma EO, Van Den Herik HJ. Dimensionality Reduction: A Comparative Review. J Mach Learn Res. 2009;10:1-41
46.Francis PJ, Wills BJ. Introduction to Principal Components Analysis. PM R [Internet]. 1999 May 6 [cited 2020 Jul 21];6(3):275-8. Available from: http://arxiv.org/abs/astro-ph/9905079
47.Bruce LM, Koger CH, Li J. Dimensionality reduction of hyperspectral data using discrete wavelet transform feature extraction. IEEE Trans Geosci Remote Sens. 2002 Oct;40(10):2331-8
48.Van Der Maaten LJP, Hinton GE. Visualizing high-dimensional data using t-sne. J Mach Learn Res [Internet]. 2008;9:2579-605. Available from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=pubmed&cmd=Retrieve&dopt=AbstractPlus&list_uids=7911431479148734548related:VOiAgwMNy20J
49.Deepa SN, Sivanandam SN. Introduction to Genetic Algorithms. In Berlin, Heidelberg: Springer Berlin Heidelberg; 2008. p. 15-37
51.Sun W, Du Q . Hyperspectral band selection: A review. Vol. 7, IEEE Geoscience and Remote Sensing Magazine. Institute of Electrical and Electronics Engineers Inc.; 2019. p. 118-39
52.Zuzak KJ, Naik SC, Alexandrakis G, Hawkins D, Behbehani K, Livingston E. Intraoperative bile duct visualization using near-infrared hyperspectral video imaging. Am J Surg. 2008;195(4):491-7
53.Wilson JW, Robles FE, Deb S, Warren WS, Fischer MC. Comparison of pump-probe and hyperspectral imaging in unstained histology sections of pigmented lesions. Biomed Opt Express [Internet]. 2017 Aug 1 [cited 2020 Jul 21];8(8):3882. Available from: https://doi.org/10.1364/BOE.8.003882
54.Halicek M, Fei B, Little J V., Wang X, Patel M, Griffith CC, et al. Optical biopsy of head and neck cancer using hyperspectral imaging and convolutional neural networks. In: Wong BJF, Ilgner JF, Witjes MJ, editors. Optical Imaging, Therapeutics, and Advanced Technology in Head and Neck Surgery and Otolaryngology 2018 [Internet]. SPIE; 2019 [cited 2019 Apr 1]. p. 33. Available from: http://dx.doi.org/10.1117/1.jbo.24.3.036007
55.Zarei N, Bakhtiari A, Gallagher P, Keys M, Macaulay C. Automated prostate glandular and nuclei detection using hyperspectral imaging. In: Proceedings - International Symposium on Biomedical Imaging. IEEE Computer Society; 2017. p. 1028-31
56.Li Q , Wang Y, Liu H, He X, Xu D, Wang J, et al. Leukocyte cells identification and quantitative morphometry based on molecular hyperspectral imaging technology. Comput Med Imaging Graph. 2014;38(3):171-8
57.Fabelo H, Ortega S, Lazcano R, Madroñal D, M. Callicó G, Juárez E, et al. An Intraoperative Visualization System Using Hyperspectral Imaging to Aid in Brain Tumor Delineation. Sensors [Internet]. 2018 Feb 1;18(2):430. Available from: http://www.mdpi.com/1424-8220/18/2/430
58.Hadoux X, Hui F, Lim JKH, Masters CL, Pébay A, Chevalier S, et al. Non-invasive in vivo hyperspectral imaging of the retina for potential biomarker use in Alzheimer’s disease. Nat Commun. 2019 Dec;10(1):1-12
59.Ravi D, Fabelo H, Callico GM, Yang GG-Z, Callic GM, Yang GG-Z. Manifold Embedding and Semantic Segmentation for Intraoperative Guidance with Hyperspectral Brain Imaging. IEEE Trans Med Imaging. 2017;36(9)
60.Ma L, Halicek M, Fei B. In vivo cancer detection in animal model using hyperspectral image classification with wavelet feature extraction. In: Gimi BS, Krol A, editors. Medical Imaging 2020: Biomedical Applications in Molecular, Structural, and Functional Imaging [Internet]. SPIE; 2020 [cited 2020 Jul 21]. p. 48. Available from: https://www.spiedigitallibrary.org/conference-proceedings-of-spie/11317/2549397/In-vivo-cancer-detection-in-animal-model-using-hyperspectral-image/10.1117/12.2549397.full
61.Pourreza-Shahri R, Saki F, Kehtarnavaz N, Leboulluec P, Liu H. Classification of ex-vivo breast cancer positive margins measured by hyperspectral imaging. In: 2013 IEEE International Conference on Image Processing, ICIP 2013 - Proceedings. 2013. p. 1408-12
62.Goto A, Nishikawa J, Kiyotoki S, Nakamura M, Nishimura J, Okamoto T, et al. Use of hyperspectral imaging technology to develop a diagnostic support system for gastric cancer. J Biomed Opt [Internet]. 2015;20(1):016017. Available from: http://biomedicaloptics.spiedigitallibrary.org/article.aspx?doi=10.1117/1.JBO.20.1.016017
63.Lu G, Wang D, Qin X, Halig L, Muller S, Zhang H, et al. Framework for hyperspectral image processing and quantification for cancer detection during animal tumor surgery. J Biomed Opt [Internet]. 2015;20(12):126012. Available from: http://biomedicaloptics.spiedigitallibrary.org/article.aspx?doi=10.1117/1.JBO.20.12.126012
64.Martinez B, Leon R,Fabelo H, Ortega S, Piñeiro JF, Szolna A, et al. Most Relevant Spectral Bands Identification for Brain Cancer Detection Using Hyperspectral Imaging. Sensors. 2019 Dec;19(24):5481
65.He L, Li J, Liu C, Li S. Recent Advances on Spectral-Spatial Hyperspectral Image Classification: An Overview and New Guidelines. IEEE Trans Geosci Remote Sens. 2018 Mar 1;56(3):1579-97
66.Wang Q , Wang J, Zhou M, Li Q , Wang Y, Ang QIANW, et al. Spectral-spatial feature-based neural network method for acute lymphoblastic leukemia cell identification via microscopic hyperspectral imaging technology. Biomed Opt Express. 2017 Jun;8(6)
67.Li Q , Zhou M, Liu H, Wang Y, Guo F. Red Blood Cell Count Automation Using Microscopic Hyperspectral Imaging Technology. Appl Spectrosc. 2015 Dec;69(12):1372-80
68.Lin L, Zhang S. Superpixel Segmentation of Hyperspectral Images Based on Entropy and Mutual Information. Appl Sci [Internet]. 2020 Feb 13 [cited 2020 Jul 22];10(4):1261. Available from: https://www.mdpi.com/2076-3417/10/4/1261
69.Fabelo H, Halicek M, Ortega S, Shahedi M, Szolna A, Piñeiro JF, et al. Deep Learning-Based Framework for In Vivo Identification of Glioblastoma Tumor using Hyperspectral Images of Human Brain. Sensors [Internet]. 2019 Feb 22 [cited 2019 Mar 4];19(4):920. Available from: http://www.mdpi.com/1424-8220/19/4/920
70.Ortega S, Halicek M, Fabelo H, Camacho R, Plaza M de la L, Godtliebsen F, et al. Hyperspectral Imaging for the Detection of Glioblastoma Tumor Cells in H&E Slides Using Convolutional Neural Networks. Sensors. 2020 Mar;20(7):1911
71.Halicek M, Dormer JD, Little J V., Chen AY, Myers L, Sumer BD, et al. Hyperspectral Imaging of Head and Neck Squamous Cell Carcinoma for Cancer Margin Detection in Surgical Specimens from 102 Patients Using Deep Learning. Cancers (Basel) [Internet]. 2019 Sep 14 [cited 2019 Nov 17];11(9):1367. Available from: https://www.mdpi.com/2072-6694/11/9/1367
72.Halicek M, Dormer J, Little J, Chen A, Fei B. Tumor detection of the thyroid and salivary glands using hyperspectral imaging and deep learning. Biomed Opt Express. 2020 Jan;
73.Halicek M, Little J V., Wang X, Patel MR, Griffith CC, Chen AY, et al. Tumor margin classification of head and neck cancer using hyperspectral imaging and convolutional neural networks. In: Webster RJ, Fei B, editors. Medical Imaging 2018: Image-Guided Procedures, Robotic Interventions, and Modeling [Internet]. SPIE; 2018 [cited 2018 Sep 14]. p. 4. Available from: https://www.spiedigitallibrary.org/conference-proceedings-of-spie/10576/2293167/Tumor-margin-classification-of-head-and-neck-cancer-using-hyperspectral/10.1117/12.2293167.full
74.Trajanovski S, Shan C, Weijtmans PJC, de Koning SGB, Ruers TJM. Tumor Semantic Segmentation in Hyperspectral Images using Deep Learning. In: International Conference on Medical Imaging with Deep Learning -- Extended Abstract Track. London, United Kingdom; 2019
75.Kho E, Dashtbozorg B, de Boer LL, Van de Vijver KK, Sterenborg HJCM, Ruers TJM. Broadband hyperspectral imaging for breast tumor detection using spectral and spatial information. Biomed Opt Express. 2019 Sep;10(9):4496
76.Halicek M, Ortega S, Fabelo H, Lopez C, Lejaune M, Callico GM, et al. Conditional generative adversarial network for synthesizing hyperspectral images of breast cancer cells from digitized histology. In: Tomaszewski JE, Ward AD, editors. Medical Imaging 2020: Digital Pathology [Internet]. SPIE; 2020 [cited 2020 May 22]. p. 29. Available from: https://www.spiedigitallibrary.org/conference-proceedings-of-spie/11320/2549994/Conditional-generative-adversarial-network-for-synthesizing-hyperspectral-images-of-breast/10.1117/12.2549994.full
77.Bengs M, Gessert N, Laffers W, Eggert D, Westermann S, Mueller NA, et al. Spectral-Spatial Recurrent-Convolutional Networks for In-Vivo Hyperspectral Tumor Type Classification. 2020 Jul;
78.Li S, Song W, Fang L, Chen Y, Ghamisi P, Benediktsson JA. Deep Learning for Hyperspectral Image Classification: An Overview. IEEE Trans Geosci Remote Sens. 2019;1-20
Written By
Samuel Ortega, Martin Halicek, Himar Fabelo, Eduardo Quevedo, Baowei Fei and Gustavo Marrero Callico
Submitted: 22 May 2020Reviewed: 10 September 2020Published: 10 October 2020
Open access peer-reviewed chapter
