Processing hyperspectral images allows you to decode images and recognize objects in the scene on the base of analysis of spectrums. In some problems, information about the spectra may not be sufficient. In this case, visualization of data sets may use, for object recognition, by use additional non-formalized external attributes (for example, indicating the relative position of objects). Target visualization is a visualization adapted to a specific task of application. The method discussed in this chapter uses a way to visualize a measure of similarity to the sample. As a result of the transformation, the hyperspectral (multichannel) image is converted into a single-channel synthesized image in grayscale, on which the objects of interest for the problem under consideration are selected. By changing the brightness and contrast of the synthesized image, it is possible to interactively adjust the results of automatic processing.
- hyperspectral image
- interactive participation
Hyperspectral image (HSI) is a set of images of one object (scene, slide, field of view, etc.), that match different narrow wavelength subranges, provides powerful tool for detection and recognition of objects under surveying.
Hyperspectral data use is a serious problem caused by need of processing of huge amount of information (tens and hundreds gigabytes). Most of hyperspectral image recognition algorithms are highly resource-intensive, especially when striving to provide full classification of hyperspectral information. Information about spectra may be insufficient for some tasks. In those cases acceptable interpretation cannot be realized without human assistance. The data visualization facilitates making decision for an expert-user allowing him to use unobvious signs obtained from experience. Watching all of the band images from HSI (HSI can consist of more than 300 spectral bands) is a labor-intensive and inefficient task. Segmentation methods are used to represent visual information in more compact view. These methods mark out homogeneous regions in the image that allows representing hyperspectral image in view of some synthesized image, for example, in pseudo-color, which facilitates visual perception. However, only in some cases a criterion of true segmentation exists. Visualization only allows us to reveal errors. And for the result correction, we need to process all over again using altered parameters of the algorithm. When solving a certain task, not all of the objects are of equal interest. Obviously, the visualization should be done so that the objects of interest were represented in the result image in the best way, but the details that prevent from good visual perception were eliminated. I.e. visualization should be targeted. Such adaptation of data processing to a specific practical task can be realized by means of target visualization method. As a result of such transformation the information contained in HSI is represented in the synthesized image in grayscale where the brightness range is concentrated on the details of the image that are of interest for user. The method combines mathematical computation speed with visual assessment and possibility to correct interpretation process interactively. In addition, the useful information is appreciably compressed, that is very important for remote sensing tasks.
2. Target visualization of HSI
Target visualization converts a set of images that constitute a hyperspectral image into a synthesized single-channel image that provides a visual representation of the data for a particular application. To perform the target visualization of HSI the image transform method is used. The method visualizes measure of similarity to a sample within a space of attributes [1, 2, 7].
Hyperspectral image, or so-called hypercube, is a collection of pictures of a single object that correspond to different narrow subranges of wavelengths,
where k – the spectral subrange number; (m, n) – the spatial coordinates of an image pixel; M – the number of lines in the image; N – the number of columns; Bkmn – the brightness value at point (m, n) in the
As the attributes, that distinguish the objects of interest, the spectral characteristics are used. Moreover, can be used other characteristics that are used in tasks on image classification [3, 4], such as: features of brightness, texture, gradient and characteristics of adjacency. The values of the characteristics used for each pixel, can be obtained a priori or via image pre-processing.
Thus, each pixel of the HSI with certain spatial coordinates has its coordinates within a feature space. In the visualization method for each task on HSI processing a decision function is formed within a set of feature values. The decision function quantitatively determines belonging of the pixel to the object of interest. Then the result image is synthesized. Pixel brightness of this image conforms to the decision function value. Changing brightness range in the obtained image and using his notion about the object being studied, user can interactively set threshold to mark out the object of interest.
The main problem of the method is to choose a decision function that corresponds to a specific applied task. Some of tasks and the choice of the decision function for them are listed as an example below:
To pick out objects on the HSI that have spectral characteristics that are similar to characteristics Ok from a spectral library.
To pick out objects on the HSI that are similar to the characteristics of an object-sample with specified spatial coordinates—i.e., sample Ok is specified by spatial coordinates (m, n): Ok = Bkmn.
To pick out objects which have the most widely represented spectra on the HSI (endmembers).
To pick out objects which have the most uncommon spectra.
To pick out boundaries of homogeneous regions on the HSI.
To pick out objects on thematic images—i.e., on images obtained by using mathematical operations with spectral bands .
The suitable decision function for the tasks 1–5 is the measure of similarity to the sample [1, 2]. Similarity can be specified by one of the measures of closeness of the vectors used for HSI classification, such as:
the Euclidean distanceE2
the distance measured by the spectral angle;E3
the maximum distanceE4
As a sample, multidimensional feature vector is chosen from set of points of the attribute space. A sample can be chosen either by specifying of a priori known feature values or by pointing to one of the discriminated objects.
Some set of vectors over region E of the feature space can be chosen as a sample, too. In this case it is possible to use the Mahalanobis distance—i.e., the distance given by the expression:
where —the mean of set E, and C—a covariant matrix of set E.
For task 6, the suitable decision function is
where —some function of the spectral characteristics, used when synthesizing index images, and
If an information on probability distribution of attributes of the sample is available, a measure based on conditional probabilities can be used (Bayesian approach). Any of a priori estimated characteristics of studied objects can be participant in forming of a measure.
By choosing of a sample and a measure of similarity we can adapt visual representation to a specified task on interpretation. Choice of a sample can be based on:
visual analysis of some of the presented pictures;
a priori information about the object of interest location;
an information about the spectral features of the required object from a specially created database.
The value of the decision function (similarity measure value) is brightness of the pixel on the synthesized image. The largest brightness value is assigned to the pixels which are closest to the sample within the attribute space. The other pixels have brightness value according to their distance from the sample. By representing the matrix of the similarity measure in gradations of gray, we obtain a visualization of the HSI adapted to the applied task. Since the brightness of each image pixel characterizes its similarity to the object of interest, by visually choosing the brightness threshold and by varying the contrast, a specialist can discriminate the object that interests him by using his knowledge of the object and accumulated experience.
Various versions of the visualization of the same scene can be obtained by selecting various samples and similarity measures. This allows the observer to analyze the imaged scene by changing samples and similarity measures. Such a tool can be compared to a magnifying glass in feature space (on synthesized images various parts of the
Target visualization used in various application areas of image processing, has a specificity that is determined by the area of the study and conditions of registration of images.
3. Target visualization of biomedical images
Computer image analysis is one of the relatively new medical technologies that are of great importance in medical research and diagnosing diseases. Due to the complexity of biomedical images, it is almost impossible to completely replace the human brain in processing them. At the same time, the method of target visualization of images allows, with the participation of a qualified user, to quickly identify and measure objects of interest.
An example of the visualization of hyperspectral images with different samples is shown in Figures 1 and 2. The source data was obtained using autofluorescent microscopy in the spectral range from 420 to 750 nm with an interval of 6 nm. (The image represents
The capabilities of visualizing multispectral images are demonstrated in Figure 2. The results obtained by transforming the 50 initial images using various samples are shown in this figure. The samples are represented in the form of graphs of spectra under the transformed images.
Another example of using visualization of hyperspectral images of biomedical theme. This example demonstrates a non-invasive method of isolation of tumor cells using a hyperspectral image. In this paper we used the HSI represented on the website of the Australian National Agency CSIRO—www.cmis.csiro.au/iap.
In Figure 3 the multispectral image of a mouse lung with a tumor, marked with fluorescent protein, obtained in 16 spectral bands is shown. It is assumed that the tumor has uncommon spectrums relating to the attribute space. The initial multispectral image is difficult for visual analysis (it is not easy to notice differences), but on the resulting synthesized images, shown in Figure 4, various regions of the studied object are clearly distinguished. For distinguishing these regions all of the initial spectral band images are involved into processing.
Under each synthesized image in Figure 4, there is the histogram which corresponds to the distribution of distances (d) in the attribute space from the spectrum of the sample to the spectrum of each pixel in the image. The histogram provides information on the number of pixels with a spectrum similar to the spectrum of the sample. Using the histogram, it is possible to set the start threshold value for the image binarization and distinguishing the objects like the sample.
Figure 5 shows the result of selecting objects corresponding to the three samples.
The thresholds were chosen at the first minimum of the histograms.
By altering the threshold value, it is possible to vary result of the visualization in according to the contextual information and experience of the user.
The method also allows us to distinguish pixels with spectral characteristics which are rare relatively most spectra of this image. For this purpose the Mahalanobis distance for some region E can be used—i.e., the distance from the vector
where is a covariant matrix of set
In our case, the set
All pixels which are close by their spectral characteristics to have light hue. Pixels, which are appreciably different from , have dark hue. Nevertheless, the dark pixels may have different spectral characteristics. To find regions of the image, which are similar by uncommon characteristics, one should choose a dark pixel within the interesting region and use its spectral characteristics as a sample. In Figure 6c. the result of distinguishing one of the clusters of pixels with uncommon spectral characteristics is represented.
4. The target visualization of HSI using contextual information
The use of contextual information when processing hyperspectral images will be shown in the example of the detection of tampering with a financial document. The HSI of the document was presented to us by ZAO “Scientific Devices”.
Figure 7 represents 4 of 125 initial pictures used for processing of the HSI of the financial document, obtained within spectral range 401–998 nm.
In these pictures, as in the others, not shown in Figure 7, it is difficult to separate the details of the text written with, though similar by spectrum, but still different ink. To detect the falsification the target visualization of the HSI consisting of 125 was used. To detect the falsification the target-visualization of the HSI consisting of 125 pictures was used. The goal of the visualization initially was to distinguish objects that have a spectrum similar to that of the main handwritten text.
The method  was used; the letter
A section of the text was then selected that did not appear on the synthesized image (indicated in Figure 8a by an arrow). Choice of this specimen was based on the fact that the spectrum of this object was different from the main handwritten text and was handwriting, too (another sort of inks might be used). The spectrum of this section, differed from the spectrum of the main text, was taken as a sample of an forgery. Differences between spectrum of ink of the main text and one’s spectrum of the forgery are shown in Figure 9.
Visualization of the text that corresponds to this spectrum is shown in Figure 10b and after correcting the brightness and contrast—in Figure 10c. The letters “TY” and the numeral “0”belong to the forgery.
Thus, forgery detection was performed with an interactive user participation, which used information of an indirect nature (“the main text” and “handwritten text different from the main one”).
Detection of objects of interest can be performed with a greater degree of automation by combining automatic segmentation and target visualization. At the first stage, automatic segmentation of the spectra of the hyperspectral image is performed. As a result of automatic segmentation, the main spectra are determined. After such preliminary segmentation, an interactive correction can be made using the target visualization of the measure of similarity with the samples-representatives of the corrected areas.
As an example, we performed HSI processing with the joint use of automatic segmentation and the target visualization of the main spectra [6, 7, 8]. The main spectra are determined by automatic segmentation. The visualization of the similarity of the spectra of current pixels to the main spectra is shown by the synthesized images in Figure 11a. Interactive correction of these synthesized images, performed with a change in the brightness range, is shown in Figure 11b.
5. Automatic high-accurate coordinate superposition of images obtained in different conditions
Efficient interpretation of images of objects and scenes may be often achieved only with combined processing of video data obtained at different time, by different surveying devices, within different spectral ranges. For combined processing of such images automatic high-accurate coordinate superposition of them is required. Such images, even being obtained with the same aspects and at the same scale, considerably differs one from other. Thus usual correlation methods of image superposition may be inadequate . There are two appreciably different approaches to image coordinate superposition: methods based on areas and methods based on image details. Each of these approaches has various advantages and disadvantages. These two approaches may be combined by means of the method of image processing by similarity to a sample .
To carry out the correlation coordinate superposition of these images, each of them is suggested to be preliminary transformed by the method of similarity to a sample involving, as an attribute space, textural, gradient features and features of adjacency.
The method does not use reference points of a scene directly. Transforming each of the superposed images by the method of similarity to a sample increase reliability of the superposition methods based on areas. Indeed, this transforming allows us to increase similarity of the images when true superposing of them and to increase their differences when relative shifting. Such transformations can be performed quite a lot, but as much as different samples exist in the fixed image.
For each sample, the correlation method determines the coordinates of the shift. The samples are chosen automatically (without the assistance of user). All of the available samples may be used. But if the images have a lot of samples, they can be selected randomly. For the selected samples a distribution is plotted.
Example of defining a coordinate shift the image obtained in the visible range with the image obtained in the infrared range, is presented in Figure 12. Presented in these shots images of the scene in the visible and IR ranges (Figure 12a and b), have parallel shift. The proposed method determined the shift; the displacement was estimated in fractions of a pixel. Figure 12c represents distribution of shifts obtained for various samples. Quality of the superposing may be estimated by statistical characteristics. For example, for this distribution of shifts the following characteristics were calculated: mode of the distribution corresponds to shifts: −0.25 pixels along OX axis, −0.5 pixels along OY axis; the mean shift along OX axis is equal to −0.2 pixels and the mean shift along OY axis is equal to −0.6 pixels; the standard deviation along OX axis is equal to 0.52 pixels and the standard deviation along OY axis is equal to 0.55 pixels.
6. The target visualization of hyperspectral images in the sphere of environmental remote sensing
Up-to-date equipment allows you to collect and analyze information in very narrow spectral bands and is used in various applications. In particular, for water monitoring, hyperspectral technology makes it possible to identify the distribution of algae by the concentrations of chlorophyll, and also to detect impurities of mineral substances in water and to determine the coefficients of cloudiness. Hyperspectral data can also be used to identify threatened vegetation types, some chemical toxins, changes in the chemical composition of soils, etc.
Processing hyperspectral images in remote sensing is related with the difficulties arising from the need to take into account the various distortions that appear in the process of formation of the HSI.
In known methods of processing remote sensing data before using hyperspectral information, the data pre-processing is performed. The main stages, of the pre-processing are:
radiometric correction (correcting the distortion due to the uneven sensitivity of the elements of the detectors, accounting for the effects of the atmosphere),
geometric correction (georeferencing);
combining images obtained in different spectral bands,
recalculation of data of distant shooting in values of coefficients of spectral brightness .
Such pretreatment is complex and requires additional information, time and computational resources.
Using the target visualization allows us to facilitate the processing of remote sensing data. Really, the target visualization allows us to significantly reduce influence on results of decryption of conditions of shooting (a condition of the atmosphere, exposition parameters), as a sample can be chosen on the initial images. In this case, the values of attributes of a the sample are obtained under the same conditions of shooting, as the values of attributes of the visualized scene.
Using the developed method of the target visualization the work on creation of the information layer “Swamp” on the territory of St.-Petersburg has been done . Remote sensing data, 435 micro landscapes of 11 types on total area of about 40 sq. km were mapped.
Majority of the swamps in the city is under strong anthropogenic influence due to which the images of the wetlands on the territory of St.-Petersburg are very diverse (by type) and variegated (by the spectral and spatial characteristics) structure, which creates a big problem in their interpretation by space images. As input data, the following materials were used:
the multispectral images of the territory of St. Petersburg obtained by the Quickbird satellite equipment with the resolution 2.4 m on per pixel;
the materials of field researches;
the available landscape and thematic maps.
Here, the images of the territory of town Lomonosov (near by the seaport) in four spectral bands, obtained in August by means of the Quickbird satellite equipment, are presented (Figure 13).
Using the linked archival map and the results of field researches, the necessary samples area on the image was allocated. As the sample area in this case, understand the territory which characteristics precisely match the description of this type of a micro landscape in the accepted scheme of classification.
For an example the polygon corresponding to the coastal open swamp was chosen. Due to the complexity of the research object, additional information was used in the form of additional images (Figure 14):
images derived on the base of spatial-spectral, textural, gradient and morphological characteristics of the original images;
images obtained at other season.
7. Search for the boundaries of homogeneous regions on hyperspectral image
The target visualization method is also applicable to the problem of finding out the boundaries of homogeneous regions on hyperspectral image.
Methods of search the boundaries are well developed for grayscale images [3, 10]. Grayscale image is considered as a function of two variables (x, y) and it is assumed that the boundaries of the regions correspond to maxima of the gradient of this function. The main problem of the search of boundary is instability their results to noise. Due to some redundancy in the joint use of multi-channel data, the selection of boundaries on the hyperspectral image is more stable to noise. The target visualization can be used to find out the boundaries of homogeneous regions. For this purpose, the hyperspectral image is transformed in the following way. For each pixel of the hyperspectral image with coordinates (m, n) the maximum spectral angle in a local region of size 3 × 3 pixels with center at (m, n) is calculated. The hyperspectral image is converted to a single-band image in accordance with the value of the spectral angle . To the obtained synthesized image in tones of gray that contains information of all channels, the known methods of search for the boundaries on halftone images can be applied. In addition, visualization of the gradient makes it possible to interactively edit the result of the boundary outlining.
Figure 16 demonstrates this method of selection of boundaries of homogeneous regions on the hyperspectral image of an urban area.
The hyperspectral image was provided by the Norwegian company Norsk Elektro Opitkk (330 channels). Figure 16a shows some shots from this hyperspectral cube. Note that the images of the individual channels are fuzzy, and some channels are highly noisy. In Figure 16b the synthesized image, visualizing the maximal spectral angles within local regions of 3 × 3, is presented. The result of the interactive selection of the boundaries on the basis of this synthesized image is shown in Figure 16c.
Use of the target visualization of hyperspectral images has a number of useful properties. Image processing by using this method is easy and quite intelligible to applied specialists. Visual data representation makes it easy for user to take a decision, allowing him to use unformalized signs according to his experience and to interactively control process of recognition of objects of interest. The transform performs rapid and compact representation of data. Indeed, measure of similarity is computed quickly and is a scalar value, that allows representing the synthesized image in one channel, and allows us to transmit it over one communication channel. It is easy enough to implement both interactive and automatic selection of objects of interest by choosing a threshold for brightness on a synthesized image. The method allows us to considerably diminish the influence of survey conditions (state of atmosphere, exposure parameters) on results of visual interpretation, because a sample can be chosen from the initial images. In this case the spectral attribute values are obtained under the same survey conditions as the feature values of the visualized scene.
The method is implemented in the basic software , which allows the target-visualization of hyperspectral images both automatically and with interactive user participation.
The program performs the following operations.
Shows the source images in each channel.
Implements a method for visualizing hyperspectral image using the method of similarity to the reference spectrum.
Changes the brightness and contrast of the visualization result.
Synthesizes a color image in a pseudocolor using different reference spectra and different similarity measures.
The program includes tools that allow you to:
Zoom the image. To do making a selection of region of an arbitrary shape on the image
Cut an arbitrary fragment of a hyperspectral image and perform the same actions with it as with the whole image.
Write the results of the processing to the file.
View the spectral curves in different, user-selected, spatial points of the image.
To store the spectra measured in the process of operation in the spectral library and use them later.
Sheremetyeva TA. Method of representation of remote sensing data that facilitates visual interpretation. In: Warmbein B, editor. Proceedings of the 5th International Conference on Space Optics (ICSO 2004); 30 March–2 April 2004; Toulouse, France. ESTEC, Noordwijk, The Netherlands: ESA Publication Division; 2004. pp. 199-201
Sheremet'eva TA, Filippov GN. Method of transforming images. Russian Federation Patent No. 2267232. Invention Bulletin. 2005; 12(36):265
Gonzalez R, Woods R. Digital Image Processing. 2nd ed. Prentice Hall, Upper Saddle River, N.J.: Pearson Education, Inc.; 2002. 1072 pp
Jahne B. Digital Image Processing. 6th ed. Berlin, Heidelberg: Springer-Verlag; 2005. 584 pp
Wiegant CL, Richardson AJ, Escobar DE, et al. Vegetation indices in crop assessments. Remote Sensing of Environment. 1991; 35:105-119
Sheremet’eva TA, Filippov GN, Malov AM. Using the target-visualization method to process hyperspectral iamges. Opticheskii Zhurnal. 2015; 82(1):32-36
Sheremet’eva TA, Filippov GN, Malov AM. Software for visualization of hyperspectral images. Certificate of State Registration of Software No. 2013615622. Copyrighter-OAO “RI OED”. 2013
Sheremet’eva TA, Filippov GN. Method of determination of spatial shift of image. Russian Federation Patent No. 246013; 2012
Sheremet’eva TA, Osipov VM, Filippov GN. Interpretation of swampy microlandscapes by space images using spectral and textural attributes of sample ground. In: International Symposium of CIS “Atmospheric Radiation and Dynamics (ISARD - 2009); June 22-26; St.-Petersburg, Russia. 2009. pp. 19-20
Sheremet’eva TA, Filippov GN, Malov AM. Method of visualization of hyperspectral images in remote sensing system. Scientific and Technical Bulletin of the S.-Petersburg State University of ITMO. 2007; 7(43):293-300