Open access peer-reviewed chapter

Resolution Enhancement of Hyperspectral Data Exploiting Real Multi-Platform Data

Written By

Rocco Restaino, Gemine Vivone, Paolo Addesso, Daniele Picone and Jocelyn Chanussot

Submitted: January 9th, 2020 Reviewed: May 11th, 2020 Published: November 4th, 2020

DOI: 10.5772/intechopen.92795

Chapter metrics overview

494 Chapter Downloads

View Full Metrics


Multi-platform data introduce new possibilities in the context of data fusion, as they allow to exploit several remotely sensed images acquired by different combinations of sensors. This scenario is particularly interesting for the sharpening of hyperspectral (HS) images, due to the limited availability of high-resolution (HR) sensors mounted onboard of the same platform as that of the HS device. However, the differences in the acquisition geometry and the nonsimultaneity of this kind of observations introduce further difficulties whose effects have to be taken into account in the design of data fusion algorithms. In this study, we present the most widespread HS image sharpening techniques and assess their performances by testing them over real acquisitions taken by the Earth Observing-1 (EO-1) and the WorldView-3 (WV3) satellites. We also highlight the difficulties arising from the use of multi-platform data and, at the same time, the benefits achievable through this approach.


  • hyperspectral image sharpening
  • Hyperion data
  • WorldView-3 images
  • data fusion
  • remote sensing

1. Introduction

Hyperspectral (HS) data often provide great insights in the field of Earth Observing (EO) for the analysis and monitoring of the planet surface [1, 2]. As they embed a very detailed spectral information of the observed scene, their employment has become necessary in many applications, including natural vegetation classification and monitoring, geological map construction, chemical properties detection, land cover observation, and water resources management [1, 2]. The widespread use of hyperspectral data pushed toward the development of acquisition devices with increasing capabilities, the most recent of which are characterized by a ground spatial interval (GSI) even below 10 m [2].

However, this spatial resolution is still insufficient in many fields, as, for instance, geology [3], agriculture [4], and land cover classification [5]. Data fusion techniques provide a possible solution to this issue that has been validated in several studies performed on both on real and simulated datasets [6, 7, 8]. In principle, high spatial resolution improvement factors can be attained for hyperspectral data, but the scarcity of exploitable companion high-resolution (HR) data represents a major issue. In fact, it is just possible to find very few examples of hyperspectral sensors co-located onboard of the same platform with high spatial resolution devices, such as panchromatic (PAN) and/or multispectral (MS) sensors. Since the Earth Observing-1, which mounted both a panchromatic and a multispectral camera onboard, is currently dismissed, the only remaining satellites to assure the availability of companion panchromatic sensors are the new Prisma and HypXIM, which are characterized by a six and four times higher spatial resolution with respect to the HS instrument, respectively.

The presence of a high-resolution sensor mounted on the same platform represents the ideal setting for the data fusion problem since the two images to combine are almost simultaneously acquired from the same point of view. However, in addition to the cited difficulty in finding platforms with this feature, the resolution ratio between the HS images and the companion high-resolution image is constrained to be very small, ranging from a value of 3 (EO-1 case) to 6 (Prisma case). Further resolution enhancement would require an additional upsampling procedure at one point in the algorithmic stack, thus strongly compromising the quality of the final fused product.

An alternative is constituted by the fusion of data acquired by multiple platforms, which, on the other hand, implies further difficulties related to the different observation geometry and the unavoidable lack of simultaneity between the acquisitions. Although this approach has been deeply investigated in the literature, the studies have almost always utilized simulated data [9, 10], thus ignoring the two cited issues that affect real data. A previous study based on real acquisitions was performed in [11] with temporally aligned images acquired by drones and aircrafts.

The current study focuses on multi-platform real data and aims at illustrating the state of the art of the, both classical and recent, low-level data fusion algorithms applicable to these data. Classical algorithms were adapted from the pansharpening literature, namely, from studies concerning the fusion of a panchromatic and a multispectral image [12]. They can be straightforwardly applied to the HS/PAN fusion problem [9, 12, 13], but they require a preliminary assignation phase when the high-resolution image is constituted by a multispectral image [14]. Indeed, a specific channel of the MS image has to be assigned to each hyperspectral band to complete the fusion process by means of classical techniques. The assignation algorithm (AA) significantly impacts the final results, and, for this reason, several algorithms have been proposed for completing this task [14, 15]. The latter fusion algorithms have been properly developed for the fusion of the HS and MS data and thus can be straightforwardly applied to the problem at hand. They include proper modifications of classical algorithms (hypersharpening) [16, 17] and applications of more general statistical approaches, as, for instance, the Bayesian framework [18], which is employed with naive [18, 19] and sparse Gaussian priors [20] and with alternative regularization terms [21, 22].

Three different datasets collected form the Earth Observing-1 and the WorldView-3 (WV3) satellites were employed in this study to evaluate the performance of the fusion algorithms. The tests were conducted according to the reduced resolution (RR) assessment procedure, based on Wald’s protocol [23]. Specifically, the available HS image is employed as reference (or ground truth (GT)), and the images to fuse are constituted by properly degraded versions of the available data. This facilitates the use of accurate indexes for evaluating the quality of the final products thanks to the presence of a reference image. The availability of real data allowed to draw conclusions about the behavior of the different types of fusion algorithms and, in the case of classical pansharpening, about the assignation approaches.

The work is organized as follows. Section 2 describes the problem under consideration, including some details on the main fusion techniques employed in hyperspectral image sharpening. The conducted experimental analysis is detailed in Section 3, whereas the outcomes are reported in Section 4. Finally, conclusions are drawn in Section 5.


2. The hyperspectral sharpening framework

The data fusion procedure to sharp hyperspectral images consists in augmenting the spatial information contained in a low-resolution (LR) hyperspectral image, by injecting information from high-resolution data.

In the following, we will denote a generic acquisition composed by NA channels as a set of bidimensional matrices, as follows: A=Akk=1,,NA. More in detail, the HS datacube will be denoted by H=Hkk=1,,NH, an MS acquisition by M=Mkk=1,,NM, and a PAN image by P=Pkk=1. The enhancement ratio, namely, the ratio between the spatial resolution of the original HS image and the desired spatial resolution, is indicated by R. We restrict the analysis of the fusion problem to the combination of two images, i.e., the details to be injected are extracted by a single image.

2.1 Classical pansharpening approaches

Classical pansharpening algorithms are designed to operate with a monochromatic image, which acts as source to extract details to be injected into the LR image. Consequently, as long as the HR image is still monochromatic, the framework of classical pansharpening can be directly applied to this scenario, with the straightforward adjustment of using the HS image as the LR source image to fuse. Conversely, when the details are extracted from a multichannel image, the application of pansharpening approaches requires an assignment procedure between each HS band and a specific channel of the high-resolution MS image.

Data fusion through classical pansharpening approaches can be formalized by the following equation [24]:


which represents the sharpening procedure of a generic k-th channel of the HS image. In Eq. 1, the estimated HR hyperspectral image is indicated by Ĥ, while H˜ denotes an upsampled (interpolated) version of the original image H to match the scale of Ĥ. The details, represented by the difference between the HR image Y and its low pass version YLP, are additively injected in the latter image by properly weighting them through an element-by-element matrix product (indicated by the operator) by the injection coefficient matrix G. It is worth to remind that both the details and the matrix G in (1) are band-dependent, since some methods require a preliminary equalization of the HR image and G is often optimized for each channel.

2.1.1 Component substitution and multi-resolution analysis algorithms

It is possible to specify different techniques of classical pansharpening methods according to the particular definition of the injection gain matrixG and the method used for calculating the low-resolution image YLP. In the literature, the key taxonomy for the macro-categorization is related to the techniques to YLP, as two separate classes of methods arise with very distinguished properties. In particular, YLP can be obtained either by properly combining the channels of H˜ or by spatially degrading the HR image Y. The first approach defines the so-called component substitution (CS), or spectral, methods, whose name is to underline that the fusion is obtained by substituting the HS intensity component with the HR image [25]. This class includes both archetypical methods, such as the Brovey transform (BT) [26], the intensity-hue-saturation [27, 28], the principal component decomposition [29, 30, 31], or the Gram-Schmidt (GS) expansion [32], and more recent approaches, such as the Gram-Schmidt adaptive (GSA) method [33], which is able to achieve state-of-the-art performance [24].

The second class of approaches is known in the literature as multi-resolution analysis (MRA), or spatial, methods, since they operate directly in the spatial domain to obtain YLP through a multi-scale decomposition. The MRA class includes a wide plethora of methods, which exploit a variety linear filters (box filters [34, 35], Gaussian filters [36], and à trous wavelet filters [37]) or nonlinear decompositions (morphological filters) [38].

The two classes have different characteristics, both in terms of visual aspect of sharpened images and in terms of robustness against nonideal working conditions. Specifically, methods belonging to the CS class usually yield final products featuring an accurate reproduction of the spatial details with an intrinsic robustness to limited spatial misalignments between the two images to fuse [39]. Images produced by MRA approaches are instead characterized by a higher spectral coherence with the original LR image, possibly even reducing temporal misalignments among data to be combined [40].

2.1.2 Assignation algorithms

As seen in the previous section, in the case of HS/PAN fusion, the only possible choice for HR data Y in (1) is represented by the PAN image. Conversely, for the HS/MS fusion, any of the MS channels can act as HR data, demanding an assignation algorithm to couple a specific MS band with a given HS channel. This problem was addressed in previous papers by defining a series of criteria for selecting the most suitable MS channel [15, 41]. The possible approaches can be either data-independent, exclusively utilizing the characteristics of the sensors, or data-dependent, for which the assignment depends on the particular datasets. The analysis reported in [14] highlights the superior performances of the second approach but at the cost of requiring an additional computational effort to evaluate the new assignation for each new dataset.

Among the data-independent approaches, acceptable performance can be obtained by minimizing the distance between the centroid of the relative spectral response (RSR) of the sensor acquiring the Hk channel and the centroids of the RSRs of the HR sensor. This method, nicknamed CEN-AA, assigns to Hk the channel Mn that verifies the condition:




defines the centroid of the generic relative spectral response (RSR) RAif of a given channel Ai.

For the AA step, the overall best results in terms of data fidelity of the reconstructed fused image are obtained by employing the algorithms proposed in [15, 41]. The first consists in maximizing the cross correlation (CC) between Hk and the MS channels and is thus denoted in the following as CC-AA. Formally, it consists in coupling Hk with the HR image Y=Mn such that:


where MjR indicates the image obtained by degrading the resolution of Mj by means of a filter matched to the modulation transfer function (MTF) of the j-th MS channel and a downsampling by a factor R; AjAk represents the scalar product among the vectorized version of two generic channels Aj and Ak.

The alternative approach, defined in [41] and assessed in [14], aims at evaluating the spectral coherence of each available HR channel if it acts as a substitute of Hk. In order to quantify this criterion, let us build the supporting images Rj,k by substituting the Mj bands at the place of Hk, which are compared to the original image H. Formally, Rj,k is defined as:


where Mjk is obtained by equalizing the first two statistical moments of Mjr w.r.t. Hk. The AA rule is defined by setting Y equal to the channel Mn that satisfies the equation:


in which SAMAB denotes the spectral angle mapper (SAM) between A and B [42]. Accordingly, this approach is named SAM-AA by the authors.

2.2 Methods designed for hyperspectral image sharpening

Several different option have been recently developed ad hoc for the sharpening of HS data by using complementary images of different nature. A first option is to modify the existing pansharpening algorithms to account for the specific characteristics of the HS data. A different approach consists in developing a completely novel method by resorting to a suitable mathematical framework, as the widely exploited statistical Bayesian formalization.

2.2.1 Hypersharpening

A very effective method for sharpening HS images relies upon the construction of a simulated HR image assigned for each channel and obtained as a certain combination the available HR channels.

This approach proposed in [16] under the name of hypersharpening consists in defining the synthetic HR image Yk to use in (1) for a given Hk through the expression:


in which the weights wk,m are optimized through linear regression as described in [16]. Equalizing the mean and variance of Yk with respect to Hk yields an improved version of hypersharpening, as proposed in [17].

The term YkLP in (1) is suggested to be obtained with the same strategies proposed by MRA methods, by degrading Yk via an appropriate filter such as the MTF-matched generalized Laplacian pyramid. The fusion formula (1) is completed by defining the injection gain matrix that is derived through the regression-based model. Namely, Gk is a constant matrix with entries:


where cov denotes the covariance operator.

2.2.2 Bayesian approaches

Most novel methods for sharpening HS images exploit the Bayesian statistical formalization of the fusion problem. In this approach, both the LR and HR available data are modeled as transformations, operating, respectively, in the spatial and in the spectral domains of an unknown ideal HR hyperspectral image denoted as Z [9].

Accordingly, the equation relating the target HR and the available LR image is written as:


where the lowercase letters denote the version of the matrices in lexicographic order (obtained by concatenating the columns of each channel), B is the blurring matrix, S is the downsampling matrix, and nH is the noise accounting for the unmodeled effects corrupting the relationship. (9) is coupled either to:


or to


in the HS/PAN and HS/MS cases, respectively. They express the functional models relating z to p or m and include the factors RP and RM that model the RSRs of the HR sensors and the noise addends nP and nM, accounting for the inaccuracy of the first terms.

The Bayesian approach based on the maximum a posteriori probability (MAP) consists in estimating the target vector z through the formula:


in which we denote by v the available HR image (m or p). A reliable solution of (12) can be found by regularizing the problem by adding a penalization term to the quantity pzhv. Examples of widely employed regularization terms include Gaussian priors [43, 44] or vector total variation (VTV) [22].


3. Quality assessment of fusion products

In this section, we present the performance assessment setup procedure for sharpening the HS data. The objective is to test the viability of fusion algorithms to reach a resolution enhancement factor R that goes beyond the limitations of single-platform setups. In the specific testbed, the HS data are constituted by acquisitions taken by the Hyperion sensor, which is characterized by a GSI of 30 m. The satellite platform also features the a PAN sensor, called ALI, whose GSI is 10 m, which corresponds to a nominal enhancement factor R=3. For more ambitious factors, two extra scenarios are considered; in particular a very interesting comparison can be taken at R=6 and R=12 by analyzing different behaviors for single- and multi-platform with a selection of 12 state-of-the-art fusion algorithms. Specifically the single-platform case requires a preliminary interpolation of the ALI images, here performed via a convolution with a 45-tap interpolation kernel. The multi-platform case will employ, as companion source image, the MS imagery acquired by the WorldView-3, which instead have to be downsized to the target resolution, as it is characterized by a smaller GSI than the target one reached by all the considered R. The decimation procedure is performed by employing a filter, mimicking the modulation transfer function of the MS sensor and a downsampling.

We want to remark here that this study will ignore the contribution of the ALI MS and WV3 PAN sensors. The former has the same GSI of the Hyperion sensor, making its information mostly redundant. Regarding the latter, we want to remark that the native GSI of the MS WV3 sensors already exceeds that of the target resolution characterized, for all R under examination. Consequently, it is preferable to employ the MS sensor, as it is already characterized by a better spectral resolution, as shown in previous studies [14, 41].

3.1 Assessment procedure

The assessment procedure has been carried out at reduced resolution, namely, the original HS image is used as reference, and the images to fuse are obtained by degrading the available images by a factor equal to the resolution enhancement factor R. The adopted Wald’s protocol [23] requires the reproduction of the characteristics of the fusion problem at a lower resolution. Accordingly, all the available images are degraded by using an MTF-shaped filter matched to the specific sensor and a downsampling system with factor R.

The reduced resolution assessment protocol allows the use of many accurate quality indexes, since the ground truth image is available. In this work we consider the spectral angle mapper [42] for evaluating the spectral distortion and the erreur relative globale adimensionnelle de synthèse (ERGAS) [45] for assessing the radiometric distortion. The vectorial Q2n-index [46] index is used for obtaining a comprehensive measure of the overall image quality. Finally, we employed the universal image quality index (UIQI) or Q-index, proposed by Wang and Bovik [47], for performing a band-by-band comparison of the final product with the reference image.

3.2 Datasets

Three datasets are used for illustrating the capabilities of data fusion algorithms in producing very high-resolution hyperspectral images. The images have been acquired by the Earth Observing-1 and WorldView-3 sensors. The different settings allow to examine the features of the sharpening algorithms in the presence of the most common issues implied by multi-platform data fusion, namely, the different points of view and the temporal changes in the illuminated scenes between the two acquisitions. In this study we employ the visible near-infrared (VNIR) bands B09-B57, acquired by the sensor Hyperion, as HS data. The single-platform companion data are constituted by the PAN images collected by the ALI sensor, having a 10 m spatial resolution. All EO-1 data share a radiometric resolution of 15 bits. The multi-platform data have been acquired by the WV3 satellite. They are represented by an MS image composed of eight channels (coastal, blue, green, yellow, red, red edge, NIR1, and NIR2) with a radiometric resolution of 11 bits and an original spatial resolution of 1.2 m.

The employed datasets are briefly described below:

  • Harlem dataset: the images have been collected in New York, USA, in the neighborhoods of the Harlem River. The size of the Hyperion and PAN ALI data, acquired on July 21, 2016, is 144 × 144 pixels and 432 × 432 pixels, respectively, while the native dimension of the WV3 MS image, acquired on June 9, 2016, is 4320 × 4320 pixels.

  • Agnano dataset: the images refer to the area of the Agnano Racecourse, next to the city of Naples, Italy. The size of the Hyperion data is 144 × 72 pixels, and thus the corresponding ALI and WV3 images are composed by 432 × 216 pixels and 4320 × 1660 pixels, respectively. The acquisition dates of the EO-1 and WV3 sensors are May 20, 2015, and June 8, 2015, respectively.

  • Capodichino dataset: the images refer to the east surrounding Naples, Italy, around Capodichino Airport. The images are composed of the same number of pixels of the Agnano dataset and were acquired on May 20, 2015, and on February 4, 2002, by the EO-1 and WV3 satellites, respectively.

3.3 Fusion algorithms

We compare several fusion algorithms to fully assess the quality of HS products achievable through data acquired by a single or multiple platforms. We firstly focus on the use of classical pansharpening approaches, which constitute an almost ready-to-use solution and then present the purposely designed methods. Among the wide plethora of available pansharpening methods [24], we employed the following CS and MRA methods: Brovey transform [26], Gram-Schmidt spectral sharpening [32], and the Gram-Schmidt adaptive [33] belonging to the CS class, the additive wavelet luminance proportional (AWLP) [37], the generalized Laplacian pyramid [48] with MTF-matched filter [49] using both the high-pass modulation scheme [50] (GLP-HPM), and the regression-based injection model (GLP-CBD) [51] belonging to the MRA class.

Among the second group of approaches, we consider the hypersharpening (Hyper) method, developed in [16, 17], and four Bayesian techniques, namely, the coupled nonnegative matrix factorization (CNMF) [21], the naive Gaussian prior (Bay-N) [43], the sparsity promoted Gaussian prior (Bay-S) [44], and the hyperspectral superresolution (HySure) [22].

Finally, we report the results related to a method for upscaling the original image at the target scale by a simple interpolation of the original HS image. We denote as EXP this method that is carried out through a 45-tap interpolation filter and that constitutes also the baseline for more complex sharpening methods presented here.


4. Experimental results

The performance of the fusion algorithms are evaluated both by calculating the numerical values of the chosen quality indexes and by assessing the final products by visual inspection.

The first proposed dataset is about Harlem that has the purpose of illustrating the capabilities of producing a significant improvement of the spatial quality of the original HS images through the compared algorithms. To this aim, we report in Figure 1 the results related to all the tested enhancement factors (R = 3,6,12), using one exemplary algorithm of each class. The RGB images are built by averaging a group of channels in the red, green, and blue frequency ranges (B29–B33, B17–B22, and B11–B15, respectively) to construct the required channels. Naturally, all the reference images (or ground truth) coincide, since they are represented by the original HS image; see Figure 1(a), (m), and (y). On the contrary, the simulated LR HS images, whose upsampled versions (EXP) are reported in Figure 1(g), (s), and (ae), get more and more degraded as the enhancement factor increases.

Figure 1.

Close-ups of the products achieved by applying the pansharpening algorithms to the RR Harlem dataset (red, green, and blue bands). The resolution enhancement factor is R = 3 (first two rows), 6 (3rd and 4th rows), 12 (5th and 6th rows). (WorldView-3 satellite images courtesy of the DigitalGlobe Foundation). The figure labels indicate the fusion algorithm (see Section 3.3) and the high resolution data utilized.

Some introductory considerations can be drawn from the images in Figure 1 obtained by using the SAM-AA for coupling the MS bands to the HS channels. In fact, the first remarkable result is the high quality of the final products achievable also at very high enhancement factors. More in detail, the images obtained by using classical GLP-CBD and the Hyper approach (which constitutes a generalization of the former approach, since both employ a regression-based injection scheme) produce the most appealing sharpened image. They are able to greatly enhance the spatial content of the original HS image, preserving an appreciable coherence of the colors.

On the other hand, the images achievable by using the ALI PAN have a satisfactory aspect only for R=3, as it was arguable by the 10 m resolution of the employed HR sensor. The effect of the interpolation is clearly visible in Figure 1(n) and (z), and, thus, this approach could be preferable only when spatial or temporal misalignments among the multi-platform data cannot be avoided.

Those results are perfectly matched to the index values contained in Table 1. Actually, the numerical values point out that in most cases, the use of perfectly aligned images coming from a different satellite can produce images with superior quality also in the case of R=3. In this case, the closer correspondence between the details extracted in the MS channel and the missing spatial information of the HS image can justify the outcome.

R = 3R = 6R = 12

Table 1.

Values of the quality indexes, related to the reduced resolution assessment procedure, using the Harlem dataset, for resolution enhancement ratio R=3,6,12.

For each algorithm, the best result among the HR options is in boldface.

Finally, we note that the comparison among the assignation algorithms mainly underlines that the two methods optimizing the assignation according to the specific dataset get almost the same results.

The other two scenes allow to gain more insight about the comparison of the sharpening algorithms and of the assignation algorithms. The EO-1 data have been extracted by the same images, while the multi-platform data have very different characteristics. In fact, while the WV3 image of the Agnano dataset has been acquired within a few days from the EO-1 data, the WV3 image of the Capodichino was collected more than 10 years earlier. Accordingly, the layout of the object present in the Capodichino scene is very different among the two passages, also because the area contains rapidly changing objects. A comparison of the two images can be achieved by having a look at Figure 2(a) and (b). The latter scene refers to Naples Airport and contains a plane on the runway that is not detectable in the corresponding WV3 MS image shown in Figure 3(d). Furthermore, different man-made objects are present in the illuminated area at the two acquisition times.

Figure 2.

Close-ups of the products achieved by applying the pansharpening algorithms to the RR Agnano dataset (red, green, and blue bands). The resolution enhancement factor is R=6 (WorldView-3 satellite images courtesy of the DigitalGlobe Foundation). The figure labels indicate the fusion method (see Section 3.3), the high resolution data and the assignation algorithm (see Section 2.1.2), utilized.

Figure 3.

Close-ups of the products achieved by applying the pansharpening algorithms to the RR Capodichino dataset (red, green, and blue bands). The resolution enhancement factor is R=6 (WorldView-3 satellite images courtesy of the DigitalGlobe Foundation). The figure labels indicate the fusion method (see Section 3.3), the high resolution data and the assignation algorithm (see Section 2.1.2), utilized.

The results related to the Agnano dataset (see Table 2) confirm the conclusions drawn from the analysis of the Harlem dataset. They correspond to the most typical situation in which the images to fuse are ordered to a data provider, minimizing as much as possible the difference between the passage times of the two satellites. Accordingly, in both cases, the illuminated areas contain very similar features that make the multi-temporal data particularly valuable. However, both also represent almost ideal cases, since the presence of rapidly changing objects, for example, the aircrafts present in the Capodichino dataset, can vary also within very close passages. Accordingly, particularly interesting is the case of the Capodichino dataset, which gives rise to somewhat unalike results, which are reported in Table 3. In fact, in most cases, the single-platform setting almost always yields better results, even if the visual appearance of the images related to the multi-platform approach is often preferable in terms of quantity of injected details (see Figure 3). Actually, a more accurate analysis evidences that multi-platform products yield a sharpened image in which the plane is absent (especially in CS methods). Moreover, the spectral quality of the final products is significantly compromised if the WV3 MS images are used.

R = 3R = 6R = 12

Table 2.

Values of the quality indexes, related to the reduced resolution assessment procedure, using the Agnano dataset, for resolution enhancement ratio R=3,6,12.

For each algorithm, the best result among the HR options is in boldface.

R = 3R = 6R = 12

Table 3.

Values of the quality indexes, related to the reduced resolution assessment procedure, using the Capodichino dataset, for resolution enhancement ratio R=3,6,12.

For each algorithm, the best result among the HR options is in boldface.

The spatial quality differences among the various algorithms can be further investigated by resorting to a quality index that allows a band-by-band analysis of the quality of the algorithms output. To this aim, we report in Figures 4 and 5 the behavior of the Q-index as a function of the HS band. The two images reveal both similarities and discrepancies in the algorithms’ performance. In particular, we can note that for the HS channels with support contained in the frequency range covered by the ALI PAN, the use of single-platform data is always preferable, except for the case of the Hyper algorithm applied to the Agnano dataset with R = 6. Clearly, this consideration is all the more true in the experiment related to the Capodichino dataset. Instead, different trends are experienced for the near-infrared (NIR) bands. All the algorithms (except the GSA algorithm with R = 3) obtain better performance by using multi-platform data working on the Agnano dataset. On the contrary, using the Capodichino dataset, the GSA algorithms always obtain superior results by using the ALI PAN image, while the other two methods obtain a slightly better performance in the NIR region that is not able to balance the scarce quality in the visible range, thus resulting in an inferior overall performance of the multi-platform approach. Finally, it is very clear from both Figures 4 and 5 that the CC-AA and the SAM-AA algorithms are able to obtain significant improvements with respect to CEN-CC, especially in the NIR frequencies.

Figure 4.

Q-index as a function of the HS band for the Agnano dataset. The curves refer to the data fusion of the Hyperion images with the ALI PAN (black continuous), the MS-CEN (red dashed curve), MS-CC (green dotted curve), and MS-SAM (blue continuous curve with circle marks).

Figure 5.

Q-index as a function of the HS band for the Capodichino dataset. The curves refer to the data fusion of the Hyperion images with the ALI PAN (black continuous), the MS-CEN (red dashed curve), MS-CC (green dotted curve), and MS-SAM (blue continuous curve with circle marks).


5. Conclusions

The aim of this work was to illustrate the recent advances in the field of hyperspectral image sharpening through single-platform and multi-platform data. The study was conducted on real data acquired by the Earth Observing-1 and the WorldView-3 satellites in order to highlight the practical issues to be addressed when fusing images acquired by different platforms. We focused on well-known algorithms based on classical approaches borrowed from the pansharpening literature and on techniques developed on purpose. We evaluated the possibility of completing the fusion process, both in the absence and presence of temporal misalignments between the scenes illuminated by the sensors mounted on the two satellites. The study highlighted the suitability of the employment of multi-platform data especially in the presence of high-resolution enhancement factors. Actually, in some cases, the use of multispectral images was also proven to be useful at low-resolution enhancement factors, and this result can be easily justified by taking into consideration that the details contained in the MS channels are able to provide more specific spatial information for a given HS channel.


  1. 1. Camps-Valls G, Tuia D, Bruzzone L, Benediktsson JA. Advances in hyperspectral image classification: Earth monitoring with statistical learning methods. IEEE Signal Processing Magazine. 2014;31(1):45-54
  2. 2. Yang J, Zhao Y-Q, Chan JC-W. Survey of hyperspectral earth observation applications from space in the Sentinel-2 context. Remote Sensing. 2018;10(2):157
  3. 3. Gomez C, Viscarra Rossel RA, McBratney AB. Soil organic carbon prediction by hyperspectral remote sensing and field vis-NIR spectroscopy: An Australian case study. Geoderma. 2008;146(3):403-411
  4. 4. Carter G, Lucas K, Blossom G, Lassitter C, Holiday D, Mooneyhan D, et al. Remote sensing and mapping of tamarisk along the Colorado river, USA: A comparative use of summer-acquired Hyperion, Thematic Mapper and QuickBird data. Remote Sensing. 2009;1(3):318-329
  5. 5. Okujeni A, van der Linden S, Hostert P. Extending the vegetation-impervious-soil model using simulated EnMAP data and machine learning. Remote Sensing of Environment. 2015;158:69-80
  6. 6. Walsh SJ, McCleary AL, Mena CF, Shao Y, Tuttle JP, González A, et al. Quickbird and Hyperion data analysis of an invasive plant species in the Galapagos islands of Ecuador: Implications for control and land use management. Remote Sensing of Environment. 2008;112(5):1927-1941
  7. 7. Siegmann B, Jarmer T, Beyer F, Ehlers M. The potential of pan-sharpened EnMAP data for the assessment of wheat LAI. Remote Sensing. 2015;7(10):12737-12762
  8. 8. Yokoya N, Chan JC-W, Segl K. Potential of resolution-enhanced hyperspectral data for mineral mapping using simulated EnMAP and Sentinel-2 images. Remote Sensing. 2016;8(3):172
  9. 9. Loncan L, Fabre S, Almeida LB, Bioucas-Dias JM, Wenzhi L, Briottet X, et al. Hyperspectral pansharpening: A review. IEEE Geoscience and Remote Sensing Magazine. 2015;3(3):27-46
  10. 10. Yokoya N, Grohnfeldt C, Chanussot J. Hyperspectral and multispectral data fusion: A comparative review of the recent literature. IEEE Geoscience and Remote Sensing Magazine. 2017;5(2):29-56
  11. 11. Marcello J, Ibarrola-Ulzurrun E, Gonzalo-Martn C, Chanussot J, Vivone G. Assessment of hyperspectral sharpening methods for the monitoring of natural areas using multiplatform remote sensing imagery. IEEE Transactions on Geoscience and Remote Sensing. 2019;57(10):8208-8222
  12. 12. Vivone G, Restaino R, Licciardi G, Dalla Mura M, Chanussot J. Multiresolution analysis and component substitution techniques for hyperspectral pansharpening. In: Proc. IEEE IGARSS. 2014. pp. 2649-2652
  13. 13. Licciardi GA, Khan MM, Chanussot J. Fusion of hyperspectral and panchromatic images: A hybrid use of indusion and nonlinear PCA. In: Proc. 2012 IEEE International Conference on Image Processing (ICIP). 2012. pp. 2133-2136
  14. 14. Picone D, Restaino R, Vivone G, Addesso P, Dalla Mura M, Chanussot J. Band assignment approaches for hyperspectral sharpening. IEEE Geoscience and Remote Sensing Letters. 2017;14(5):739-743
  15. 15. Aiazzi B, Alparone L, Baronti S, Santurri L, Selva M. Spatial resolution enhancement of ASTER thermal bands. Proceedings of SPIE. 2005;5982:59821G-59821G–10
  16. 16. Selva M, Aiazzi B, Butera F, Chiarantini L, Baronti S. Hyper-sharpening: A first approach on SIM-GA data. IEEE Journal on Selected Topics in Signal Processing. 2015;8(6):3008-3024
  17. 17. Selva M, Santurri L, Baronti S. “Improving hypersharpening for WorldView-3 data.” IEEE Geoscience and Remote Sensing Letters. 2019;16(6):987-991
  18. 18. Hardie RC, Eismann MT, Wilson GL. MAP estimation for hyperspectral image resolution enhancement using an auxiliary sensor. IEEE Transactions on Image Processing. 2004;13(9):1174-1184
  19. 19. Zhang Y, De Backer S, Scheunders P. Noise-resistant wavelet-based Bayesian fusion of multispectral and hyperspectral images. IEEE Transactions on Geoscience and Remote Sensing. 2009;47(11):3834-3843
  20. 20. Wei Q, Bioucas-Dias J, Dobigeon N, Tourneret J. Hyperspectral and multispectral image fusion based on a sparse representation. IEEE Geoscience and Remote Sensing Letters. 2015;53(7):3658-3668
  21. 21. Yokoya N, Yairi T, Iwasaki A. Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Transactions on Geoscience and Remote Sensing. 2012;50(2):528-537
  22. 22. Simões M, Bioucas-Dias JM, Almeida LB, Chanussot J. A convex formulation for hyperspectral image superresolution via subspace-based regularization. IEEE Transactions on Geoscience and Remote Sensing. 2015;53(6):3373-3388
  23. 23. Wald L, Ranchin T, Mangolini M. Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogrammetric Engineering and Remote Sensing. 1997;63(6):691-699
  24. 24. Vivone G, Alparone L, Chanussot J, Dalla Mura M, Garzelli A, Licciardi G, et al. A critical comparison among pansharpening algorithms. IEEE Transactions on Geoscience and Remote Sensing. 2015;53(5):2565-2586
  25. 25. Tu T-M, Su S-C, Shyu H-C, Huang PS. A new look at IHS-like image fusion methods. Information Fusion. 2001;2(3):177-186
  26. 26. Gillespie AR, Kahle AB, Walker RE. Color enhancement of highly correlated images-II. Channel ratio and “chromaticity” transform techniques. Remote Sensing of Environment. 1987;22(3):343-365
  27. 27. Carper W, Lillesand T, Kiefer R. The use of intensity-hue-saturation transformations for merging SPOT panchromatic and multispectral image data. Photogrammetric Engineering and Remote Sensing. 1990;56(4):459-467
  28. 28. Chavez PS Jr, Sides SC, Anderson JA. Comparison of three different methods to merge multiresolution and multispectral data: Landsat TM and SPOT panchromatic. Photogrammetric Engineering and Remote Sensing. 1991;57(3):295-303
  29. 29. Chavez PS Jr, Kwarteng AW. Extracting spectral contrast in Landsat Thematic Mapper image data using selective principal component analysis. Photogrammetric Engineering and Remote Sensing. 1989;55(3):339-348
  30. 30. Shettigara VK. A generalized component substitution technique for spatial enhancement of multispectral images using a higher resolution data set. Photogrammetric Engineering and Remote Sensing. 1992;58(5):561-567
  31. 31. Shah VP, Younan NH, King RL. An efficient pan-sharpening method via a combined adaptive-PCA approach and contourlets. IEEE Transactions on Geoscience and Remote Sensing. 2008;46(5):1323-1335
  32. 32. Laben CA, Brower BV. Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. 2000, U.S. Patent # 6011875
  33. 33. Aiazzi B, Baronti S, Selva M. Improving component substitution pansharpening through multivariate regression of MS + Pan data. IEEE Transactions on Geoscience and Remote Sensing. 2007;45(10):3230-3239
  34. 34. Liu JG. Smoothing filter based intensity modulation: A spectral preserve image fusion technique for improving spatial details. International Journal of Remote Sensing. 2000;21(18):3461-3472
  35. 35. Wald L, Ranchin T. Comments on the paper by Liu “smoothing filter based intensity modulation: A spectral preserve image fusion technique for improving spatial details”. International Journal of Remote Sensing. 2002;23(3):593-597
  36. 36. Burt PJ, Adelson EH. The Laplacian pyramid as a compact image code. IEEE Transactions on Communications. 1983;31(4):532-540
  37. 37. Otazu X, González-Audícana M, Fors O, Núñez J. Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods. IEEE Transactions on Geoscience and Remote Sensing. 2005;43(10):2376-2385
  38. 38. Restaino R, Vivone G, Dalla Mura M, Chanussot J. Fusion of multispectral and panchromatic images based on morphological operators. IEEE Transactions on Image Processing. 2016;25(6):2882-2895
  39. 39. Baronti S, Aiazzi B, Selva M, Garzelli A, Alparone L. A theoretical analysis of the effects of aliasing and misregistration on pansharpened imagery. IEEE Journal on Selected Topics in Signal Processing. 2011;5(3):446-453
  40. 40. Aiazzi B, Alparone L, Baronti S, Carlà R, Garzelli A, Santurri L. Sensitivity of pansharpening methods to temporal and instrumental changes between multispectral and panchromatic data sets. IEEE Transactions on Geoscience and Remote Sensing. 2017;55(1):308-319
  41. 41. Picone D, Restaino R, Vivone G, Addesso P, Chanussot J. Pansharpening of hyperspectral images: Exploiting data acquired by multiple platforms. Proc. IEEE IGARSS. 2016:7220-7223
  42. 42. Yuhas RH, Goetz AFH, Boardman JW. Discrimination among semi-arid landscape endmembers using the Spectral Angle Mapper (SAM) algorithm. In: Proc. Summaries of the Third Annual JPL Airborne Geoscience Workshop. 1992. pp. 147-149
  43. 43. Wei Q, Dobigeon N, Tourneret J. Fast fusion of multi-band images based on solving a Sylvester equation. IEEE Transactions on Image Processing. 2015;24(11):4109-4121
  44. 44. Wei Q, Dobigeon N, Tourneret J. Bayesian fusion of multi1193 band images. IEEE Journal on Selected Topics in Signal Processing. 2015;9(6):1117-1127
  45. 45. Wald L. Data Fusion: Definitions and Architectures—Fusion of Images of Different Spatial Resolutions. Paris, France: Les Presses de l’École des Mines; 2002
  46. 46. Garzelli A, Nencini F. Hypercomplex quality assessment of multi-/hyper-spectral images. IEEE Transactions on Geoscience and Remote Sensing. 2009;6(4):662-665
  47. 47. Wang Z, Bovik AC. A universal image quality index. IEEE Signal Processing Letters. 2002;9(3):81-84
  48. 48. Aiazzi B, Alparone L, Baronti S, Garzelli A. Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis. IEEE Transactions on Geoscience and Remote Sensing. 2002;40(10):2300-2312
  49. 49. Aiazzi B, Alparone L, Baronti S, Garzelli A, Selva M. MTF-tailored multiscale fusion of high-resolution MS and Pan imagery. Photogrammetric Engineering and Remote Sensing. 2006;72(5):591-596
  50. 50. Aiazzi B, Alparone L, Baronti S, Garzelli A, Selva M. An MTF-based spectral distortion minimizing model for Pan-sharpening of very high resolution multispectral images of urban areas. In: Proc. 2nd GRSS/ISPRS Joint Workshop on Remote Sensing and Data Fusion over Urban Areas. 2003. pp. 90-94
  51. 51. Alparone L, Wald L, Chanussot J, Thomas C, Gamba P, Bruce LM. Comparison of pansharpening algorithms: Outcome of the 2006 GRS-S data fusion contest. IEEE Transactions on Geoscience and Remote Sensing. 2007;45(10):3012-3021

Written By

Rocco Restaino, Gemine Vivone, Paolo Addesso, Daniele Picone and Jocelyn Chanussot

Submitted: January 9th, 2020 Reviewed: May 11th, 2020 Published: November 4th, 2020