Open access peer-reviewed chapter

Resolution Enhancement of Hyperspectral Data Exploiting Real Multi-Platform Data

By Rocco Restaino, Gemine Vivone, Paolo Addesso, Daniele Picone and Jocelyn Chanussot

Submitted: January 9th 2020Reviewed: May 11th 2020Published: November 4th 2020

DOI: 10.5772/intechopen.92795

Downloaded: 79

Abstract

Multi-platform data introduce new possibilities in the context of data fusion, as they allow to exploit several remotely sensed images acquired by different combinations of sensors. This scenario is particularly interesting for the sharpening of hyperspectral (HS) images, due to the limited availability of high-resolution (HR) sensors mounted onboard of the same platform as that of the HS device. However, the differences in the acquisition geometry and the nonsimultaneity of this kind of observations introduce further difficulties whose effects have to be taken into account in the design of data fusion algorithms. In this study, we present the most widespread HS image sharpening techniques and assess their performances by testing them over real acquisitions taken by the Earth Observing-1 (EO-1) and the WorldView-3 (WV3) satellites. We also highlight the difficulties arising from the use of multi-platform data and, at the same time, the benefits achievable through this approach.

Keywords

  • hyperspectral image sharpening
  • Hyperion data
  • WorldView-3 images
  • data fusion
  • remote sensing

1. Introduction

Hyperspectral (HS) data often provide great insights in the field of Earth Observing (EO) for the analysis and monitoring of the planet surface [1, 2]. As they embed a very detailed spectral information of the observed scene, their employment has become necessary in many applications, including natural vegetation classification and monitoring, geological map construction, chemical properties detection, land cover observation, and water resources management [1, 2]. The widespread use of hyperspectral data pushed toward the development of acquisition devices with increasing capabilities, the most recent of which are characterized by a ground spatial interval (GSI) even below 10 m [2].

However, this spatial resolution is still insufficient in many fields, as, for instance, geology [3], agriculture [4], and land cover classification [5]. Data fusion techniques provide a possible solution to this issue that has been validated in several studies performed on both on real and simulated datasets [6, 7, 8]. In principle, high spatial resolution improvement factors can be attained for hyperspectral data, but the scarcity of exploitable companion high-resolution (HR) data represents a major issue. In fact, it is just possible to find very few examples of hyperspectral sensors co-located onboard of the same platform with high spatial resolution devices, such as panchromatic (PAN) and/or multispectral (MS) sensors. Since the Earth Observing-1, which mounted both a panchromatic and a multispectral camera onboard, is currently dismissed, the only remaining satellites to assure the availability of companion panchromatic sensors are the new Prisma and HypXIM, which are characterized by a six and four times higher spatial resolution with respect to the HS instrument, respectively.

The presence of a high-resolution sensor mounted on the same platform represents the ideal setting for the data fusion problem since the two images to combine are almost simultaneously acquired from the same point of view. However, in addition to the cited difficulty in finding platforms with this feature, the resolution ratio between the HS images and the companion high-resolution image is constrained to be very small, ranging from a value of 3 (EO-1 case) to 6 (Prisma case). Further resolution enhancement would require an additional upsampling procedure at one point in the algorithmic stack, thus strongly compromising the quality of the final fused product.

An alternative is constituted by the fusion of data acquired by multiple platforms, which, on the other hand, implies further difficulties related to the different observation geometry and the unavoidable lack of simultaneity between the acquisitions. Although this approach has been deeply investigated in the literature, the studies have almost always utilized simulated data [9, 10], thus ignoring the two cited issues that affect real data. A previous study based on real acquisitions was performed in [11] with temporally aligned images acquired by drones and aircrafts.

The current study focuses on multi-platform real data and aims at illustrating the state of the art of the, both classical and recent, low-level data fusion algorithms applicable to these data. Classical algorithms were adapted from the pansharpening literature, namely, from studies concerning the fusion of a panchromatic and a multispectral image [12]. They can be straightforwardly applied to the HS/PAN fusion problem [9, 12, 13], but they require a preliminary assignation phase when the high-resolution image is constituted by a multispectral image [14]. Indeed, a specific channel of the MS image has to be assigned to each hyperspectral band to complete the fusion process by means of classical techniques. The assignation algorithm (AA) significantly impacts the final results, and, for this reason, several algorithms have been proposed for completing this task [14, 15]. The latter fusion algorithms have been properly developed for the fusion of the HS and MS data and thus can be straightforwardly applied to the problem at hand. They include proper modifications of classical algorithms (hypersharpening) [16, 17] and applications of more general statistical approaches, as, for instance, the Bayesian framework [18], which is employed with naive [18, 19] and sparse Gaussian priors [20] and with alternative regularization terms [21, 22].

Three different datasets collected form the Earth Observing-1 and the WorldView-3 (WV3) satellites were employed in this study to evaluate the performance of the fusion algorithms. The tests were conducted according to the reduced resolution (RR) assessment procedure, based on Wald’s protocol [23]. Specifically, the available HS image is employed as reference (or ground truth (GT)), and the images to fuse are constituted by properly degraded versions of the available data. This facilitates the use of accurate indexes for evaluating the quality of the final products thanks to the presence of a reference image. The availability of real data allowed to draw conclusions about the behavior of the different types of fusion algorithms and, in the case of classical pansharpening, about the assignation approaches.

The work is organized as follows. Section 2 describes the problem under consideration, including some details on the main fusion techniques employed in hyperspectral image sharpening. The conducted experimental analysis is detailed in Section 3, whereas the outcomes are reported in Section 4. Finally, conclusions are drawn in Section 5.

2. The hyperspectral sharpening framework

The data fusion procedure to sharp hyperspectral images consists in augmenting the spatial information contained in a low-resolution (LR) hyperspectral image, by injecting information from high-resolution data.

In the following, we will denote a generic acquisition composed by NAchannels as a set of bidimensional matrices, as follows: A=Akk=1,,NA. More in detail, the HS datacube will be denoted by H=Hkk=1,,NH, an MS acquisition by M=Mkk=1,,NM, and a PAN image by P=Pkk=1. The enhancement ratio, namely, the ratio between the spatial resolution of the original HS image and the desired spatial resolution, is indicated by R. We restrict the analysis of the fusion problem to the combination of two images, i.e., the details to be injected are extracted by a single image.

2.1 Classical pansharpening approaches

Classical pansharpening algorithms are designed to operate with a monochromatic image, which acts as source to extract details to be injected into the LR image. Consequently, as long as the HR image is still monochromatic, the framework of classical pansharpening can be directly applied to this scenario, with the straightforward adjustment of using the HS image as the LR source image to fuse. Conversely, when the details are extracted from a multichannel image, the application of pansharpening approaches requires an assignment procedure between each HS band and a specific channel of the high-resolution MS image.

Data fusion through classical pansharpening approaches can be formalized by the following equation [24]:

Ĥk=H˜k+GkYkYkLP,E1

which represents the sharpening procedure of a generic k-th channel of the HS image. In Eq. 1, the estimated HR hyperspectral image is indicated by Ĥ, while H˜denotes an upsampled (interpolated) version of the original image Hto match the scale of Ĥ. The details, represented by the difference between the HR image Yand its low pass version YLP, are additively injected in the latter image by properly weighting them through an element-by-element matrix product (indicated by the operator) by the injection coefficient matrix G. It is worth to remind that both the details and the matrix Gin (1) are band-dependent, since some methods require a preliminary equalization of the HR image and Gis often optimized for each channel.

2.1.1 Component substitution and multi-resolution analysis algorithms

It is possible to specify different techniques of classical pansharpening methods according to the particular definition of the injection gain matrixGand the method used for calculating the low-resolution image YLP. In the literature, the key taxonomy for the macro-categorization is related to the techniques to YLP, as two separate classes of methods arise with very distinguished properties. In particular, YLPcan be obtained either by properly combining the channels of H˜or by spatially degrading the HR image Y. The first approach defines the so-called component substitution (CS), or spectral, methods, whose name is to underline that the fusion is obtained by substituting the HS intensity component with the HR image [25]. This class includes both archetypical methods, such as the Brovey transform (BT) [26], the intensity-hue-saturation [27, 28], the principal component decomposition [29, 30, 31], or the Gram-Schmidt (GS) expansion [32], and more recent approaches, such as the Gram-Schmidt adaptive (GSA) method [33], which is able to achieve state-of-the-art performance [24].

The second class of approaches is known in the literature as multi-resolution analysis (MRA), or spatial, methods, since they operate directly in the spatial domain to obtain YLPthrough a multi-scale decomposition. The MRA class includes a wide plethora of methods, which exploit a variety linear filters (box filters [34, 35], Gaussian filters [36], and à trous wavelet filters [37]) or nonlinear decompositions (morphological filters) [38].

The two classes have different characteristics, both in terms of visual aspect of sharpened images and in terms of robustness against nonideal working conditions. Specifically, methods belonging to the CS class usually yield final products featuring an accurate reproduction of the spatial details with an intrinsic robustness to limited spatial misalignments between the two images to fuse [39]. Images produced by MRA approaches are instead characterized by a higher spectral coherence with the original LR image, possibly even reducing temporal misalignments among data to be combined [40].

2.1.2 Assignation algorithms

As seen in the previous section, in the case of HS/PAN fusion, the only possible choice for HR data Yin (1) is represented by the PAN image. Conversely, for the HS/MS fusion, any of the MS channels can act as HR data, demanding an assignation algorithm to couple a specific MS band with a given HS channel. This problem was addressed in previous papers by defining a series of criteria for selecting the most suitable MS channel [15, 41]. The possible approaches can be either data-independent, exclusively utilizing the characteristics of the sensors, or data-dependent, for which the assignment depends on the particular datasets. The analysis reported in [14] highlights the superior performances of the second approach but at the cost of requiring an additional computational effort to evaluate the new assignation for each new dataset.

Among the data-independent approaches, acceptable performance can be obtained by minimizing the distance between the centroid of the relative spectral response (RSR) of the sensor acquiring the Hkchannel and the centroids of the RSRs of the HR sensor. This method, nicknamed CEN-AA, assigns to Hkthe channel Mnthat verifies the condition:

n=argminjμHkμMj,E2

wherein

μAi=fRAifRAifdfdfE3

defines the centroid of the generic relative spectral response (RSR) RAifof a given channel Ai.

For the AA step, the overall best results in terms of data fidelity of the reconstructed fused image are obtained by employing the algorithms proposed in [15, 41]. The first consists in maximizing the cross correlation (CC) between Hkand the MS channels and is thus denoted in the following as CC-AA. Formally, it consists in coupling Hkwith the HR image Y=Mnsuch that:

n=argmaxjCCHkMjR=argmaxjHkMjRHkHkMjRMjRE4

where MjRindicates the image obtained by degrading the resolution of Mjby means of a filter matched to the modulation transfer function (MTF) of the j-th MS channel and a downsampling by a factor R; AjAkrepresents the scalar product among the vectorized version of two generic channels Ajand Ak.

The alternative approach, defined in [41] and assessed in [14], aims at evaluating the spectral coherence of each available HR channel if it acts as a substitute of Hk. In order to quantify this criterion, let us build the supporting images Rj,kby substituting the Mjbands at the place of Hk, which are compared to the original image H. Formally, Rj,kis defined as:

Rij,k=Mjk,i=k,Hi,i=k,E5

where Mjkis obtained by equalizing the first two statistical moments of Mjrw.r.t. Hk. The AA rule is defined by setting Yequal to the channel Mnthat satisfies the equation:

n=argminjSAMHRj,k,E6

in which SAMABdenotes the spectral angle mapper (SAM) between Aand B[42]. Accordingly, this approach is named SAM-AA by the authors.

2.2 Methods designed for hyperspectral image sharpening

Several different option have been recently developed ad hoc for the sharpening of HS data by using complementary images of different nature. A first option is to modify the existing pansharpening algorithms to account for the specific characteristics of the HS data. A different approach consists in developing a completely novel method by resorting to a suitable mathematical framework, as the widely exploited statistical Bayesian formalization.

2.2.1 Hypersharpening

A very effective method for sharpening HS images relies upon the construction of a simulated HR image assigned for each channel and obtained as a certain combination the available HR channels.

This approach proposed in [16] under the name of hypersharpening consists in defining the synthetic HR image Ykto use in (1) for a given Hkthrough the expression:

Yk=m=1NMwk,mMm,E7

in which the weights wk,mare optimized through linear regression as described in [16]. Equalizing the mean and variance of Ykwith respect to Hkyields an improved version of hypersharpening, as proposed in [17].

The term YkLPin (1) is suggested to be obtained with the same strategies proposed by MRA methods, by degrading Ykvia an appropriate filter such as the MTF-matched generalized Laplacian pyramid. The fusion formula (1) is completed by defining the injection gain matrix that is derived through the regression-based model. Namely, Gkis a constant matrix with entries:

gk=covH˜kYkLPcovYkLPYkLP,E8

where covdenotes the covariance operator.

2.2.2 Bayesian approaches

Most novel methods for sharpening HS images exploit the Bayesian statistical formalization of the fusion problem. In this approach, both the LR and HR available data are modeled as transformations, operating, respectively, in the spatial and in the spectral domains of an unknown ideal HR hyperspectral image denoted as Z[9].

Accordingly, the equation relating the target HR and the available LR image is written as:

h=zBS+nH,E9

where the lowercase letters denote the version of the matrices in lexicographic order (obtained by concatenating the columns of each channel), Bis the blurring matrix, Sis the downsampling matrix, and nHis the noise accounting for the unmodeled effects corrupting the relationship. (9) is coupled either to:

p=RPz+nP,E10

or to

m=RMz+nM,E11

in the HS/PAN and HS/MS cases, respectively. They express the functional models relating zto por mand include the factors RPand RMthat model the RSRs of the HR sensors and the noise addends nPand nM, accounting for the inaccuracy of the first terms.

The Bayesian approach based on the maximum a posteriori probability (MAP) consists in estimating the target vector zthrough the formula:

ẑ=argmaxzpzhv.E12

in which we denote by vthe available HR image (mor p). A reliable solution of (12) can be found by regularizing the problem by adding a penalization term to the quantity pzhv. Examples of widely employed regularization terms include Gaussian priors [43, 44] or vector total variation (VTV) [22].

3. Quality assessment of fusion products

In this section, we present the performance assessment setup procedure for sharpening the HS data. The objective is to test the viability of fusion algorithms to reach a resolution enhancement factor Rthat goes beyond the limitations of single-platform setups. In the specific testbed, the HS data are constituted by acquisitions taken by the Hyperion sensor, which is characterized by a GSI of 30 m. The satellite platform also features the a PAN sensor, called ALI, whose GSI is 10 m, which corresponds to a nominal enhancement factor R=3. For more ambitious factors, two extra scenarios are considered; in particular a very interesting comparison can be taken at R=6and R=12by analyzing different behaviors for single- and multi-platform with a selection of 12 state-of-the-art fusion algorithms. Specifically the single-platform case requires a preliminary interpolation of the ALI images, here performed via a convolution with a 45-tap interpolation kernel. The multi-platform case will employ, as companion source image, the MS imagery acquired by the WorldView-3, which instead have to be downsized to the target resolution, as it is characterized by a smaller GSI than the target one reached by all the considered R. The decimation procedure is performed by employing a filter, mimicking the modulation transfer function of the MS sensor and a downsampling.

We want to remark here that this study will ignore the contribution of the ALI MS and WV3 PAN sensors. The former has the same GSI of the Hyperion sensor, making its information mostly redundant. Regarding the latter, we want to remark that the native GSI of the MS WV3 sensors already exceeds that of the target resolution characterized, for all Runder examination. Consequently, it is preferable to employ the MS sensor, as it is already characterized by a better spectral resolution, as shown in previous studies [14, 41].

3.1 Assessment procedure

The assessment procedure has been carried out at reduced resolution, namely, the original HS image is used as reference, and the images to fuse are obtained by degrading the available images by a factor equal to the resolution enhancement factor R. The adopted Wald’s protocol [23] requires the reproduction of the characteristics of the fusion problem at a lower resolution. Accordingly, all the available images are degraded by using an MTF-shaped filter matched to the specific sensor and a downsampling system with factor R.

The reduced resolution assessment protocol allows the use of many accurate quality indexes, since the ground truth image is available. In this work we consider the spectral angle mapper [42] for evaluating the spectral distortion and the erreur relative globale adimensionnelle de synthèse (ERGAS) [45] for assessing the radiometric distortion. The vectorial Q2n-index [46] index is used for obtaining a comprehensive measure of the overall image quality. Finally, we employed the universal image quality index (UIQI) or Q-index, proposed by Wang and Bovik [47], for performing a band-by-band comparison of the final product with the reference image.

3.2 Datasets

Three datasets are used for illustrating the capabilities of data fusion algorithms in producing very high-resolution hyperspectral images. The images have been acquired by the Earth Observing-1 and WorldView-3 sensors. The different settings allow to examine the features of the sharpening algorithms in the presence of the most common issues implied by multi-platform data fusion, namely, the different points of view and the temporal changes in the illuminated scenes between the two acquisitions. In this study we employ the visible near-infrared (VNIR) bands B09-B57, acquired by the sensor Hyperion, as HS data. The single-platform companion data are constituted by the PAN images collected by the ALI sensor, having a 10 m spatial resolution. All EO-1 data share a radiometric resolution of 15 bits. The multi-platform data have been acquired by the WV3 satellite. They are represented by an MS image composed of eight channels (coastal, blue, green, yellow, red, red edge, NIR1, and NIR2) with a radiometric resolution of 11 bits and an original spatial resolution of 1.2 m.

The employed datasets are briefly described below:

  • Harlem dataset: the images have been collected in New York, USA, in the neighborhoods of the Harlem River. The size of the Hyperion and PAN ALI data, acquired on July 21, 2016, is 144 × 144 pixels and 432 × 432 pixels, respectively, while the native dimension of the WV3 MS image, acquired on June 9, 2016, is 4320 × 4320 pixels.

  • Agnano dataset: the images refer to the area of the Agnano Racecourse, next to the city of Naples, Italy. The size of the Hyperion data is 144 × 72 pixels, and thus the corresponding ALI and WV3 images are composed by 432 × 216 pixels and 4320 × 1660 pixels, respectively. The acquisition dates of the EO-1 and WV3 sensors are May 20, 2015, and June 8, 2015, respectively.

  • Capodichino dataset: the images refer to the east surrounding Naples, Italy, around Capodichino Airport. The images are composed of the same number of pixels of the Agnano dataset and were acquired on May 20, 2015, and on February 4, 2002, by the EO-1 and WV3 satellites, respectively.

3.3 Fusion algorithms

We compare several fusion algorithms to fully assess the quality of HS products achievable through data acquired by a single or multiple platforms. We firstly focus on the use of classical pansharpening approaches, which constitute an almost ready-to-use solution and then present the purposely designed methods. Among the wide plethora of available pansharpening methods [24], we employed the following CS and MRA methods: Brovey transform [26], Gram-Schmidt spectral sharpening [32], and the Gram-Schmidt adaptive [33] belonging to the CS class, the additive wavelet luminance proportional (AWLP) [37], the generalized Laplacian pyramid [48] with MTF-matched filter [49] using both the high-pass modulation scheme [50] (GLP-HPM), and the regression-based injection model (GLP-CBD) [51] belonging to the MRA class.

Among the second group of approaches, we consider the hypersharpening (Hyper) method, developed in [16, 17], and four Bayesian techniques, namely, the coupled nonnegative matrix factorization (CNMF) [21], the naive Gaussian prior (Bay-N) [43], the sparsity promoted Gaussian prior (Bay-S) [44], and the hyperspectral superresolution (HySure) [22].

Finally, we report the results related to a method for upscaling the original image at the target scale by a simple interpolation of the original HS image. We denote as EXP this method that is carried out through a 45-tap interpolation filter and that constitutes also the baseline for more complex sharpening methods presented here.

4. Experimental results

The performance of the fusion algorithms are evaluated both by calculating the numerical values of the chosen quality indexes and by assessing the final products by visual inspection.

The first proposed dataset is about Harlem that has the purpose of illustrating the capabilities of producing a significant improvement of the spatial quality of the original HS images through the compared algorithms. To this aim, we report in Figure 1 the results related to all the tested enhancement factors (R= 3,6,12), using one exemplary algorithm of each class. The RGB images are built by averaging a group of channels in the red, green, and blue frequency ranges (B29–B33, B17–B22, and B11–B15, respectively) to construct the required channels. Naturally, all the reference images (or ground truth) coincide, since they are represented by the original HS image; see Figure 1(a), (m), and (y). On the contrary, the simulated LR HS images, whose upsampled versions (EXP) are reported in Figure 1(g), (s), and (ae), get more and more degraded as the enhancement factor increases.

Figure 1.

Close-ups of the products achieved by applying the pansharpening algorithms to the RR Harlem dataset (red, green, and blue bands). The resolution enhancement factor is R = 3 (first two rows), 6 (3rd and 4th rows), 12 (5th and 6th rows). (WorldView-3 satellite images courtesy of the DigitalGlobe Foundation). The figure labels indicate the fusion algorithm (see Section 3.3) and the high resolution data utilized.

Some introductory considerations can be drawn from the images in Figure 1 obtained by using the SAM-AA for coupling the MS bands to the HS channels. In fact, the first remarkable result is the high quality of the final products achievable also at very high enhancement factors. More in detail, the images obtained by using classical GLP-CBD and the Hyper approach (which constitutes a generalization of the former approach, since both employ a regression-based injection scheme) produce the most appealing sharpened image. They are able to greatly enhance the spatial content of the original HS image, preserving an appreciable coherence of the colors.

On the other hand, the images achievable by using the ALI PAN have a satisfactory aspect only for R=3, as it was arguable by the 10 m resolution of the employed HR sensor. The effect of the interpolation is clearly visible in Figure 1(n) and (z), and, thus, this approach could be preferable only when spatial or temporal misalignments among the multi-platform data cannot be avoided.

Those results are perfectly matched to the index values contained in Table 1. Actually, the numerical values point out that in most cases, the use of perfectly aligned images coming from a different satellite can produce images with superior quality also in the case of R=3. In this case, the closer correspondence between the details extracted in the MS channel and the missing spatial information of the HS image can justify the outcome.

R = 3R = 6R = 12
SAMERGASQ2nSAMERGASQ2nSAMERGASQ2n
Optimum001001001
EXP3.52655.83260.71305.13193.88360.45046.59242.30970.1948
BTPAN3.52656.51020.79665.13193.90270.67226.59242.27750.4588
MS-CEN3.58035.28280.82264.29432.86640.78814.97751.61750.7234
MS-CC3.30635.11590.82283.93162.79880.78934.68011.60360.7325
MS-SAM3.23605.10280.82103.93162.79880.78934.68011.60360.7325
GSPAN5.38627.04840.80876.64164.13690.66807.71762.37420.4410
MS-CEN3.63285.29390.82304.32612.87230.78864.98421.61890.7235
MS-CC3.27525.12240.82653.86072.76790.79294.52711.57340.7273
MS-SAM3.25245.11750.82643.86072.76790.79294.52711.57340.7273
GSAPAN3.22144.71240.87384.89343.36520.68946.60722.20220.4087
MS-CEN3.52235.36520.84093.40182.58220.84283.89591.32160.8294
MS-CC3.13174.99600.85602.91162.39780.85843.47081.29320.8209
MS-SAM3.17434.99750.85612.91162.39780.85843.47081.29320.8209
AWLPPAN3.76165.42250.86765.27603.46020.68786.67352.21210.3316
MS-CEN2.82624.31520.89173.44762.50870.84534.63961.54230.7435
MS-CC2.59184.08020.89643.16292.40250.85004.41461.51990.7475
MS-SAM2.58584.06370.89573.16292.40250.85004.41461.51990.7475
GLP-HPMPAN3.81365.46350.85865.48843.73300.68556.95042.29290.4577
MS-CEN2.38433.80740.90403.15782.30750.85353.97341.35810.7813
MS-CC2.12853.65780.90832.78862.20140.85923.53831.30000.7867
MS-SAM2.13073.65860.90822.78862.20140.85923.53831.30000.7867
GLP-CBDPAN3.19414.51030.88094.90753.36530.70916.72632.21000.4833
MS-CEN2.44424.07780.89553.03022.38540.86533.30371.26030.8559
MS-CC2.19613.91090.90072.63602.25240.87282.77721.17300.8643
MS-SAM2.19493.91030.90072.63602.25240.87282.77721.17300.8643
HyperPAN3.19254.51030.88094.89483.36460.70916.57712.20620.4840
MS2.18104.12960.89012.51452.33310.86312.46831.16850.8584
CNMFPAN3.61664.95170.84865.16293.48780.67676.51402.17780.4502
MS2.74345.03130.82443.01442.56660.81613.15481.29580.8152
Bay-NPAN3.07264.17250.88325.56483.66320.63917.31892.28280.4263
MS2.44124.27390.87413.77822.80950.81224.17161.42520.8083
Bay-SPAN2.97264.12640.88575.45153.64790.63287.35402.27660.4241
MS2.43814.27400.87413.78002.81040.81224.12951.42330.8085
HySurePAN2.97584.24840.88646.61414.34980.60347.70522.33140.4497
MS2.33044.53470.86343.77362.86940.80683.49661.36470.8187

Table 1.

Values of the quality indexes, related to the reduced resolution assessment procedure, using the Harlem dataset, for resolution enhancement ratio R=3,6,12.

For each algorithm, the best result among the HR options is in boldface.

Finally, we note that the comparison among the assignation algorithms mainly underlines that the two methods optimizing the assignation according to the specific dataset get almost the same results.

The other two scenes allow to gain more insight about the comparison of the sharpening algorithms and of the assignation algorithms. The EO-1 data have been extracted by the same images, while the multi-platform data have very different characteristics. In fact, while the WV3 image of the Agnano dataset has been acquired within a few days from the EO-1 data, the WV3 image of the Capodichino was collected more than 10 years earlier. Accordingly, the layout of the object present in the Capodichino scene is very different among the two passages, also because the area contains rapidly changing objects. A comparison of the two images can be achieved by having a look at Figure 2(a) and (b). The latter scene refers to Naples Airport and contains a plane on the runway that is not detectable in the corresponding WV3 MS image shown in Figure 3(d). Furthermore, different man-made objects are present in the illuminated area at the two acquisition times.

Figure 2.

Close-ups of the products achieved by applying the pansharpening algorithms to the RR Agnano dataset (red, green, and blue bands). The resolution enhancement factor is R=6 (WorldView-3 satellite images courtesy of the DigitalGlobe Foundation). The figure labels indicate the fusion method (see Section 3.3), the high resolution data and the assignation algorithm (see Section 2.1.2), utilized.

Figure 3.

Close-ups of the products achieved by applying the pansharpening algorithms to the RR Capodichino dataset (red, green, and blue bands). The resolution enhancement factor is R=6 (WorldView-3 satellite images courtesy of the DigitalGlobe Foundation). The figure labels indicate the fusion method (see Section 3.3), the high resolution data and the assignation algorithm (see Section 2.1.2), utilized.

The results related to the Agnano dataset (see Table 2) confirm the conclusions drawn from the analysis of the Harlem dataset. They correspond to the most typical situation in which the images to fuse are ordered to a data provider, minimizing as much as possible the difference between the passage times of the two satellites. Accordingly, in both cases, the illuminated areas contain very similar features that make the multi-temporal data particularly valuable. However, both also represent almost ideal cases, since the presence of rapidly changing objects, for example, the aircrafts present in the Capodichino dataset, can vary also within very close passages. Accordingly, particularly interesting is the case of the Capodichino dataset, which gives rise to somewhat unalike results, which are reported in Table 3. In fact, in most cases, the single-platform setting almost always yields better results, even if the visual appearance of the images related to the multi-platform approach is often preferable in terms of quantity of injected details (see Figure 3). Actually, a more accurate analysis evidences that multi-platform products yield a sharpened image in which the plane is absent (especially in CS methods). Moreover, the spectral quality of the final products is significantly compromised if the WV3 MS images are used.

R = 3R = 6R = 12
SAMERGASQ2nSAMERGASQ2nSAMERGASQ2n
Optimum001001001
EXP2.61732.55630.80683.76011.78900.55234.64961.08070.2831
BTPAN2.61733.05480.80583.76011.87230.60094.64961.09840.3604
MS-CEN3.22952.84060.82283.67381.54250.76244.28020.91260.6314
MS-CC2.87722.74820.82593.30371.47600.77783.98210.88290.6574
MS-SAM2.87852.75450.82463.31671.47780.77713.97990.88310.6573
GSPAN2.55862.99060.82453.58261.81540.63504.49931.07520.3833
MS-CEN3.35662.90270.81643.79371.58640.75514.37470.93020.6257
MS-CC2.87252.75520.82893.27661.47470.77953.94700.88040.6553
MS-SAM2.87512.76270.82793.28271.47560.77923.94230.88030.6554
GSAPAN2.03861.91460.91312.98471.44900.73894.07610.97540.4544
MS-CEN3.51623.36260.83863.45061.51130.84473.58180.75760.8114
MS-CC3.16283.21250.84672.67541.35710.86312.78240.67870.8335
MS-SAM3.12053.23410.84342.65821.35790.86252.78820.68010.8343
AWLPPAN2.83382.58140.88133.86461.68420.68044.70831.05240.3699
MS-CEN2.74082.37750.90463.39021.39750.84614.26340.86380.7295
MS-CC2.06211.96610.92772.43361.14920.87883.17550.74100.7617
MS-SAM2.03831.97840.92572.43481.15120.87793.17810.74140.7613
HPMPAN2.62582.36370.88013.75001.66950.69464.67451.03140.4644
MS-CEN2.07321.86830.92792.73821.17540.86453.44790.73990.7318
MS-CC1.61811.62590.94042.21471.05220.88083.01510.69260.7478
MS-SAM1.60251.62840.94012.22111.05280.88063.01260.69270.7478
GLP-CBDPAN1.95341.85860.91402.96611.44340.75283.76560.93120.5485
MS-CEN2.11621.95900.92512.89191.24440.88073.52860.74180.8378
MS-CC1.75461.81290.93402.19601.10430.90032.49930.62790.8699
MS-SAM1.73441.82560.93232.19171.10670.89942.50420.62760.8699
HyperPAN1.95431.85860.91402.96881.44190.75303.77030.91710.5490
MS1.58971.72140.93822.02291.04440.90802.64300.68150.8437
CNMFPAN2.77202.60750.85843.88591.73370.71384.63291.04050.5231
MS2.50882.75250.84132.68651.38430.83023.01050.76850.7988
Bay-NPAN2.25602.11310.90173.88321.73130.67945.03461.13190.5224
MS2.21262.34480.89343.47571.64220.80133.96940.92660.7793
Bay-SPAN1.98581.84160.91903.76041.68620.68074.94241.10490.5233
MS2.21302.34510.89343.48181.64330.80104.17450.94680.7705
HySurePAN2.35092.21740.89224.52352.06950.61285.37421.22500.5362
MS2.56492.77760.85133.19891.58630.77703.29560.84370.7707

Table 2.

Values of the quality indexes, related to the reduced resolution assessment procedure, using the Agnano dataset, for resolution enhancement ratio R=3,6,12.

For each algorithm, the best result among the HR options is in boldface.

R = 3R = 6R = 12
SAMERGASQ2nSAMERGASQ2nSAMERGASQ2n
Optimum001001001
EXP2.90413.46400.76074.08092.35560.49305.12711.44880.2097
BTPAN2.90414.16280.78044.08092.47500.63105.12711.46430.3702
MS-CEN4.67925.73170.59384.90252.88100.55765.24321.51540.4746
MS-CC4.43435.54890.58754.62562.81730.56084.88581.49600.4877
MS-SAM4.45095.63040.59264.59512.83220.56564.89691.49700.4884
GSPAN3.22864.21330.77584.65582.59120.59606.06831.58210.3326
MS-CEN4.77005.74760.59425.00222.89970.55595.32041.52800.4738
MS-CC4.48915.63630.60084.58622.83200.56744.86631.49730.4873
MS-SAM4.47725.67130.60024.57102.84010.56884.86251.49770.4875
GSAPAN2.48072.59760.88333.44501.86870.69984.75401.29700.3636
MS-CEN7.06198.22740.53576.46583.60620.56506.28111.67610.5705
MS-CC6.09766.87630.60015.12922.92850.62364.53741.33040.6124
MS-SAM6.32867.33630.57185.19703.06030.60814.53661.33110.6116
AWLPPAN3.14913.37320.85984.36512.12540.68995.36841.37890.3466
MS-CEN3.53413.96060.79204.41682.40480.67995.36961.42230.5509
MS-CC3.26803.61630.80903.96602.22910.70164.63781.32930.5765
MS-SAM3.25953.67030.80953.92152.23810.70444.63131.33000.5770
HPMPAN3.09653.16050.85264.43712.21550.68235.63021.40590.4234
MS-CEN2.71983.19640.83733.52132.04070.72124.26371.22160.5688
MS-CC2.53623.05480.84623.21281.97350.73233.86681.19260.5824
MS-SAM2.51333.06720.84693.18411.97420.73433.86401.19270.5827
CBDPAN2.41262.50370.88323.36881.81930.72614.52751.22460.4657
MS-CEN2.63553.12710.83853.62602.12480.72344.74701.36810.6321
MS-CC2.55133.06240.84323.44882.07900.73224.29921.32010.6488
MS-SAM2.54213.08070.84363.43062.08950.73364.28281.32200.6487
HyperPAN2.41262.50370.88323.36841.81990.72594.51521.22540.4652
MS2.71383.25700.82243.82932.21940.69664.61471.31810.6051
CNMFPAN3.03093.21740.83614.15972.11990.70355.08921.30820.4882
MS4.26955.46480.59843.78042.63070.60274.33181.46440.5163
Bay-NPAN2.48092.54120.88974.28762.14780.66005.73061.46190.4276
MS2.82663.39960.81734.31092.56380.63544.91581.51270.5777
Bay-SPAN2.30802.37050.89814.17322.11200.65965.75971.45220.4304
MS2.82673.39890.81734.31142.56370.63544.91261.51150.5780
HySurePAN2.49282.79470.87545.32292.77320.61015.78681.47470.4592
MS3.27594.39630.73674.19312.84710.59064.54971.50790.5723

Table 3.

Values of the quality indexes, related to the reduced resolution assessment procedure, using the Capodichino dataset, for resolution enhancement ratio R=3,6,12.

For each algorithm, the best result among the HR options is in boldface.

The spatial quality differences among the various algorithms can be further investigated by resorting to a quality index that allows a band-by-band analysis of the quality of the algorithms output. To this aim, we report in Figures 4 and 5 the behavior of the Q-index as a function of the HS band. The two images reveal both similarities and discrepancies in the algorithms’ performance. In particular, we can note that for the HS channels with support contained in the frequency range covered by the ALI PAN, the use of single-platform data is always preferable, except for the case of the Hyper algorithm applied to the Agnano dataset with R= 6. Clearly, this consideration is all the more true in the experiment related to the Capodichino dataset. Instead, different trends are experienced for the near-infrared (NIR) bands. All the algorithms (except the GSA algorithm with R= 3) obtain better performance by using multi-platform data working on the Agnano dataset. On the contrary, using the Capodichino dataset, the GSA algorithms always obtain superior results by using the ALI PAN image, while the other two methods obtain a slightly better performance in the NIR region that is not able to balance the scarce quality in the visible range, thus resulting in an inferior overall performance of the multi-platform approach. Finally, it is very clear from both Figures 4 and 5 that the CC-AA and the SAM-AA algorithms are able to obtain significant improvements with respect to CEN-CC, especially in the NIR frequencies.

Figure 4.

Q-index as a function of the HS band for the Agnano dataset. The curves refer to the data fusion of the Hyperion images with the ALI PAN (black continuous), the MS-CEN (red dashed curve), MS-CC (green dotted curve), and MS-SAM (blue continuous curve with circle marks).

Figure 5.

Q-index as a function of the HS band for the Capodichino dataset. The curves refer to the data fusion of the Hyperion images with the ALI PAN (black continuous), the MS-CEN (red dashed curve), MS-CC (green dotted curve), and MS-SAM (blue continuous curve with circle marks).

5. Conclusions

The aim of this work was to illustrate the recent advances in the field of hyperspectral image sharpening through single-platform and multi-platform data. The study was conducted on real data acquired by the Earth Observing-1 and the WorldView-3 satellites in order to highlight the practical issues to be addressed when fusing images acquired by different platforms. We focused on well-known algorithms based on classical approaches borrowed from the pansharpening literature and on techniques developed on purpose. We evaluated the possibility of completing the fusion process, both in the absence and presence of temporal misalignments between the scenes illuminated by the sensors mounted on the two satellites. The study highlighted the suitability of the employment of multi-platform data especially in the presence of high-resolution enhancement factors. Actually, in some cases, the use of multispectral images was also proven to be useful at low-resolution enhancement factors, and this result can be easily justified by taking into consideration that the details contained in the MS channels are able to provide more specific spatial information for a given HS channel.

© 2020 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 License, which permits use, distribution and reproduction for non-commercial purposes, provided the original is properly cited.

How to cite and reference

Link to this chapter Copy to clipboard

Cite this chapter Copy to clipboard

Rocco Restaino, Gemine Vivone, Paolo Addesso, Daniele Picone and Jocelyn Chanussot (November 4th 2020). Resolution Enhancement of Hyperspectral Data Exploiting Real Multi-Platform Data, Recent Advances in Image Restoration with Applications to Real World Problems, Chiman Kwan, IntechOpen, DOI: 10.5772/intechopen.92795. Available from:

chapter statistics

79total chapter downloads

More statistics for editors and authors

Login to your personal dashboard for more detailed statistics on your publications.

Access personal reporting

Related Content

This Book

Recent Advances in Image Restoration with Applications to Real World Problems

Edited by Chiman Kwan

Next chapter

Application of Deep Learning Approaches for Enhancing Mastcam Images

By Ying Qu, Hairong Qi and Chiman Kwan

Related Book

First chapter

A Survey of Image Segmentation by the Classical Method and Resonance Algorithm

By Fengzhi Dai, Masanori Sugisaka and Baolong Zhang

We are IntechOpen, the world's leading publisher of Open Access books. Built by scientists, for scientists. Our readership spans scientists, professors, researchers, librarians, and students, as well as business professionals. We share our knowledge and peer-reveiwed research papers with libraries, scientific and engineering societies, and also work with corporate R&D departments and government entities.

More About Us