Resolution Enhancement of Hyperspectral Data Exploiting Real Multi-Platform Data

Abstract

Multi-platform data introduce new possibilities in the context of data fusion, as they allow to exploit several remotely sensed images acquired by different combinations of sensors. This scenario is particularly interesting for the sharpening of hyperspectral (HS) images, due to the limited availability of high-resolution (HR) sensors mounted onboard of the same platform as that of the HS device. However, the differences in the acquisition geometry and the nonsimultaneity of this kind of observations introduce further difficulties whose effects have to be taken into account in the design of data fusion algorithms. In this study, we present the most widespread HS image sharpening techniques and assess their performances by testing them over real acquisitions taken by the Earth Observing-1 (EO-1) and the WorldView-3 (WV3) satellites. We also highlight the difficulties arising from the use of multi-platform data and, at the same time, the benefits achievable through this approach.

1. Introduction

Hyperspectral (HS) data often provide great insights in the field of Earth Observing (EO) for the analysis and monitoring of the planet surface [1, 2]. As they embed a very detailed spectral information of the observed scene, their employment has become necessary in many applications, including natural vegetation classification and monitoring, geological map construction, chemical properties detection, land cover observation, and water resources management [1, 2]. The widespread use of hyperspectral data pushed toward the development of acquisition devices with increasing capabilities, the most recent of which are characterized by a ground spatial interval (GSI) even below 10 m [2].

However, this spatial resolution is still insufficient in many fields, as, for instance, geology [3], agriculture [4], and land cover classification [5]. Data fusion techniques provide a possible solution to this issue that has been validated in several studies performed on both on real and simulated datasets [6, 7, 8]. In principle, high spatial resolution improvement factors can be attained for hyperspectral data, but the scarcity of exploitable companion high-resolution (HR) data represents a major issue. In fact, it is just possible to find very few examples of hyperspectral sensors co-located onboard of the same platform with high spatial resolution devices, such as panchromatic (PAN) and/or multispectral (MS) sensors. Since the Earth Observing-1, which mounted both a panchromatic and a multispectral camera onboard, is currently dismissed, the only remaining satellites to assure the availability of companion panchromatic sensors are the new Prisma and HypXIM, which are characterized by a six and four times higher spatial resolution with respect to the HS instrument, respectively.

The presence of a high-resolution sensor mounted on the same platform represents the ideal setting for the data fusion problem since the two images to combine are almost simultaneously acquired from the same point of view. However, in addition to the cited difficulty in finding platforms with this feature, the resolution ratio between the HS images and the companion high-resolution image is constrained to be very small, ranging from a value of 3 (EO-1 case) to 6 (Prisma case). Further resolution enhancement would require an additional upsampling procedure at one point in the algorithmic stack, thus strongly compromising the quality of the final fused product.

An alternative is constituted by the fusion of data acquired by multiple platforms, which, on the other hand, implies further difficulties related to the different observation geometry and the unavoidable lack of simultaneity between the acquisitions. Although this approach has been deeply investigated in the literature, the studies have almost always utilized simulated data [9, 10], thus ignoring the two cited issues that affect real data. A previous study based on real acquisitions was performed in [11] with temporally aligned images acquired by drones and aircrafts.

The current study focuses on multi-platform real data and aims at illustrating the state of the art of the, both classical and recent, low-level data fusion algorithms applicable to these data. Classical algorithms were adapted from the pansharpening literature, namely, from studies concerning the fusion of a panchromatic and a multispectral image [12]. They can be straightforwardly applied to the HS/PAN fusion problem [9, 12, 13], but they require a preliminary assignation phase when the high-resolution image is constituted by a multispectral image [14]. Indeed, a specific channel of the MS image has to be assigned to each hyperspectral band to complete the fusion process by means of classical techniques. The assignation algorithm (AA) significantly impacts the final results, and, for this reason, several algorithms have been proposed for completing this task [14, 15]. The latter fusion algorithms have been properly developed for the fusion of the HS and MS data and thus can be straightforwardly applied to the problem at hand. They include proper modifications of classical algorithms (hypersharpening) [16, 17] and applications of more general statistical approaches, as, for instance, the Bayesian framework [18], which is employed with naive [18, 19] and sparse Gaussian priors [20] and with alternative regularization terms [21, 22].

Three different datasets collected form the Earth Observing-1 and the WorldView-3 (WV3) satellites were employed in this study to evaluate the performance of the fusion algorithms. The tests were conducted according to the reduced resolution (RR) assessment procedure, based on Wald’s protocol [23]. Specifically, the available HS image is employed as reference (or ground truth (GT)), and the images to fuse are constituted by properly degraded versions of the available data. This facilitates the use of accurate indexes for evaluating the quality of the final products thanks to the presence of a reference image. The availability of real data allowed to draw conclusions about the behavior of the different types of fusion algorithms and, in the case of classical pansharpening, about the assignation approaches.

The work is organized as follows. Section 2 describes the problem under consideration, including some details on the main fusion techniques employed in hyperspectral image sharpening. The conducted experimental analysis is detailed in Section 3, whereas the outcomes are reported in Section 4. Finally, conclusions are drawn in Section 5.

2. The hyperspectral sharpening framework

The data fusion procedure to sharp hyperspectral images consists in augmenting the spatial information contained in a low-resolution (LR) hyperspectral image, by injecting information from high-resolution data.

In the following, we will denote a generic acquisition composed by $N_A$ channels as a set of bidimensional matrices, as follows: $\mathbf{A}={\left\{{\mathbf{A}}_k\right\}}_{k=1,\dots ,N_A}$. More in detail, the HS datacube will be denoted by $\mathbf{H}={\left\{{\mathbf{H}}_k\right\}}_{k=1,\dots ,N_H}$, an MS acquisition by $\mathbf{M}={\left\{{\mathbf{M}}_k\right\}}_{k=1,\dots ,N_M}$, and a PAN image by $\mathbf{P}={\left\{{\mathbf{P}}_k\right\}}_{k=1}$. The enhancement ratio, namely, the ratio between the spatial resolution of the original HS image and the desired spatial resolution, is indicated by $R$. We restrict the analysis of the fusion problem to the combination of two images, i.e., the details to be injected are extracted by a single image.

2.1 Classical pansharpening approaches

Classical pansharpening algorithms are designed to operate with a monochromatic image, which acts as source to extract details to be injected into the LR image. Consequently, as long as the HR image is still monochromatic, the framework of classical pansharpening can be directly applied to this scenario, with the straightforward adjustment of using the HS image as the LR source image to fuse. Conversely, when the details are extracted from a multichannel image, the application of pansharpening approaches requires an assignment procedure between each HS band and a specific channel of the high-resolution MS image.

Data fusion through classical pansharpening approaches can be formalized by the following equation [24]:

$${\widehat{\mathbf{H}}}_k={\tilde{\mathbf{H}}}_k+{\mathbf{G}}_k\circ \left({\mathbf{Y}}_k-{\mathbf{Y}}_k^{\mathit{LP}}\right),$$E1

which represents the sharpening procedure of a generic $k$-th channel of the HS image. In Eq. 1, the estimated HR hyperspectral image is indicated by $\widehat{\mathbf{H}}$, while $\tilde{\mathbf{H}}$ denotes an upsampled (interpolated) version of the original image $\mathbf{H}$ to match the scale of $\widehat{\mathbf{H}}$. The details, represented by the difference between the HR image $\mathbf{Y}$ and its low pass version ${\mathbf{Y}}^{\mathit{LP}}$, are additively injected in the latter image by properly weighting them through an element-by-element matrix product (indicated by the $\circ$ operator) by the injection coefficient matrix $\mathbf{G}$. It is worth to remind that both the details and the matrix $\mathbf{G}$ in (1) are band-dependent, since some methods require a preliminary equalization of the HR image and $\mathbf{G}$ is often optimized for each channel.

2.1.1 Component substitution and multi-resolution analysis algorithms

It is possible to specify different techniques of classical pansharpening methods according to the particular definition of the injection gain matrix$\mathbf{G}$ and the method used for calculating the low-resolution image ${\mathbf{Y}}^{\mathit{LP}}$. In the literature, the key taxonomy for the macro-categorization is related to the techniques to ${\mathbf{Y}}^{\mathit{LP}}$, as two separate classes of methods arise with very distinguished properties. In particular, ${\mathbf{Y}}^{\mathit{LP}}$ can be obtained either by properly combining the channels of $\tilde{\mathbf{H}}$ or by spatially degrading the HR image $\mathbf{Y}$. The first approach defines the so-called component substitution (CS), or spectral, methods, whose name is to underline that the fusion is obtained by substituting the HS intensity component with the HR image [25]. This class includes both archetypical methods, such as the Brovey transform (BT) [26], the intensity-hue-saturation [27, 28], the principal component decomposition [29, 30, 31], or the Gram-Schmidt (GS) expansion [32], and more recent approaches, such as the Gram-Schmidt adaptive (GSA) method [33], which is able to achieve state-of-the-art performance [24].

The second class of approaches is known in the literature as multi-resolution analysis (MRA), or spatial, methods, since they operate directly in the spatial domain to obtain ${\mathbf{Y}}^{\mathit{LP}}$ through a multi-scale decomposition. The MRA class includes a wide plethora of methods, which exploit a variety linear filters (box filters [34, 35], Gaussian filters [36], and à trous wavelet filters [37]) or nonlinear decompositions (morphological filters) [38].

The two classes have different characteristics, both in terms of visual aspect of sharpened images and in terms of robustness against nonideal working conditions. Specifically, methods belonging to the CS class usually yield final products featuring an accurate reproduction of the spatial details with an intrinsic robustness to limited spatial misalignments between the two images to fuse [39]. Images produced by MRA approaches are instead characterized by a higher spectral coherence with the original LR image, possibly even reducing temporal misalignments among data to be combined [40].

2.1.2 Assignation algorithms

As seen in the previous section, in the case of HS/PAN fusion, the only possible choice for HR data $\mathbf{Y}$ in (1) is represented by the PAN image. Conversely, for the HS/MS fusion, any of the MS channels can act as HR data, demanding an assignation algorithm to couple a specific MS band with a given HS channel. This problem was addressed in previous papers by defining a series of criteria for selecting the most suitable MS channel [15, 41]. The possible approaches can be either data-independent, exclusively utilizing the characteristics of the sensors, or data-dependent, for which the assignment depends on the particular datasets. The analysis reported in [14] highlights the superior performances of the second approach but at the cost of requiring an additional computational effort to evaluate the new assignation for each new dataset.

Among the data-independent approaches, acceptable performance can be obtained by minimizing the distance between the centroid of the relative spectral response (RSR) of the sensor acquiring the ${\mathbf{H}}_k$ channel and the centroids of the RSRs of the HR sensor. This method, nicknamed CEN-AA, assigns to ${\mathbf{H}}_k$ the channel ${\mathbf{M}}_n$ that verifies the condition:

$$n=arg\underset{j}{min}\mid {\mu }_{{\mathbf{H}}_k}-{\mu }_{{\mathbf{M}}_j}\mid ,$$E2

wherein

$${\mu }_{{\mathbf{A}}_i}=\int f\frac{R_{{\mathbf{A}}_i}\left(f\right)}{\int R_{{\mathbf{A}}_i}\left(f\right)\mathit{df}}\mathit{df}$$E3

defines the centroid of the generic relative spectral response (RSR) $R_{{\mathbf{A}}_i}\left(f\right)$ of a given channel ${\mathbf{A}}_i$.

For the AA step, the overall best results in terms of data fidelity of the reconstructed fused image are obtained by employing the algorithms proposed in [15, 41]. The first consists in maximizing the cross correlation (CC) between ${\mathbf{H}}_k$ and the MS channels and is thus denoted in the following as CC-AA. Formally, it consists in coupling ${\mathbf{H}}_k$ with the HR image $\mathbf{Y}={\mathbf{M}}_n$ such that:

$$n=arg\underset{j}{max}\mathit{CC}\left({\mathbf{H}}_k,{\mathbf{M}}_j^{\downarrow R}\right)=arg\underset{j}{max}\frac{⟨{\mathbf{H}}_k,{\mathbf{M}}_j^{\downarrow R}⟩}{\sqrt{⟨{\mathbf{H}}_k,{\mathbf{H}}_k⟩⟨{\mathbf{M}}_j^{\downarrow R},{\mathbf{M}}_j^{\downarrow R}⟩}}$$E4

where ${\mathbf{M}}_j^{\downarrow R}$ indicates the image obtained by degrading the resolution of ${\mathbf{M}}_j$ by means of a filter matched to the modulation transfer function (MTF) of the $j$-th MS channel and a downsampling by a factor $R$; $⟨{\mathbf{A}}_j,{\mathbf{A}}_k⟩$ represents the scalar product among the vectorized version of two generic channels ${\mathbf{A}}_j$ and ${\mathbf{A}}_k$.

The alternative approach, defined in [41] and assessed in [14], aims at evaluating the spectral coherence of each available HR channel if it acts as a substitute of ${\mathbf{H}}_k$. In order to quantify this criterion, let us build the supporting images ${\mathbf{R}}^{j,k}$ by substituting the ${\mathbf{M}}_j$ bands at the place of ${\mathbf{H}}_k$, which are compared to the original image $\mathbf{H}$. Formally, ${\mathbf{R}}^{j,k}$ is defined as:

$${\mathbf{R}}_i^{j,k}=\left\{\begin{array}{ll}{\mathbf{M}}_j^{\left(k\right)},& i=k,\\ {\mathbf{H}}_i,& i\cancel{=}k,\end{array}\right.$$E5

where ${\mathbf{M}}_j^{\left(k\right)}$ is obtained by equalizing the first two statistical moments of ${\mathbf{M}}_j^{\downarrow r}$ w.r.t. ${\mathbf{H}}_k$. The AA rule is defined by setting $\mathbf{Y}$ equal to the channel ${\mathbf{M}}_n$ that satisfies the equation:

$$n=arg\underset{j}{min}\mathit{SAM}\left[\mathbf{H},{\mathbf{R}}^{j,k}\right],$$E6

in which $\mathit{SAM}\left[\mathbf{A},\mathbf{B}\right]$ denotes the spectral angle mapper (SAM) between $\mathbf{A}$ and $\mathbf{B}$ [42]. Accordingly, this approach is named SAM-AA by the authors.

2.2 Methods designed for hyperspectral image sharpening

Several different option have been recently developed ad hoc for the sharpening of HS data by using complementary images of different nature. A first option is to modify the existing pansharpening algorithms to account for the specific characteristics of the HS data. A different approach consists in developing a completely novel method by resorting to a suitable mathematical framework, as the widely exploited statistical Bayesian formalization.

2.2.1 Hypersharpening

A very effective method for sharpening HS images relies upon the construction of a simulated HR image assigned for each channel and obtained as a certain combination the available HR channels.

This approach proposed in [16] under the name of hypersharpening consists in defining the synthetic HR image ${\mathbf{Y}}_k$ to use in (1) for a given ${\mathbf{H}}_k$ through the expression:

$${\mathbf{Y}}_k=\sum _{m=1}^{N_M}{w}_{k,m}{\mathbf{M}}_m,$$E7

in which the weights $w_{k,m}$ are optimized through linear regression as described in [16]. Equalizing the mean and variance of ${\mathbf{Y}}_k$ with respect to ${\mathbf{H}}_k$ yields an improved version of hypersharpening, as proposed in [17].

The term ${\mathbf{Y}}_k^{\mathit{LP}}$ in (1) is suggested to be obtained with the same strategies proposed by MRA methods, by degrading ${\mathbf{Y}}_k$ via an appropriate filter such as the MTF-matched generalized Laplacian pyramid. The fusion formula (1) is completed by defining the injection gain matrix that is derived through the regression-based model. Namely, ${\mathbf{G}}_k$ is a constant matrix with entries:

$$g_k=\frac{cov\left({\tilde{\mathbf{H}}}_k,{\mathbf{Y}}_k^{\mathit{LP}}\right)}{cov\left({\mathbf{Y}}_k^{\mathit{LP}},{\mathbf{Y}}_k^{\mathit{LP}}\right)},$$E8

where $\mathit{cov}\left(\cdot ,\cdot \right)$ denotes the covariance operator.

2.2.2 Bayesian approaches

Most novel methods for sharpening HS images exploit the Bayesian statistical formalization of the fusion problem. In this approach, both the LR and HR available data are modeled as transformations, operating, respectively, in the spatial and in the spectral domains of an unknown ideal HR hyperspectral image denoted as $\mathbf{Z}$ [9].

Accordingly, the equation relating the target HR and the available LR image is written as:

$$\mathbf{h}=\mathbf{zBS}+{\mathbf{n}}_H,$$E9

where the lowercase letters denote the version of the matrices in lexicographic order (obtained by concatenating the columns of each channel), $\mathbf{B}$ is the blurring matrix, $\mathbf{S}$ is the downsampling matrix, and ${\mathbf{n}}_H$ is the noise accounting for the unmodeled effects corrupting the relationship. (9) is coupled either to:

$$\mathbf{p}={\mathbf{R}}_P\mathbf{z}+{\mathbf{n}}_P,$$E10

or to

$$\mathbf{m}={\mathbf{R}}_M\mathbf{z}+{\mathbf{n}}_M,$$E11

in the HS/PAN and HS/MS cases, respectively. They express the functional models relating $\mathbf{z}$ to $\mathbf{p}$ or $\mathbf{m}$ and include the factors ${\mathbf{R}}_P$ and ${\mathbf{R}}_M$ that model the RSRs of the HR sensors and the noise addends ${\mathbf{n}}_P$ and ${\mathbf{n}}_M$, accounting for the inaccuracy of the first terms.

The Bayesian approach based on the maximum a posteriori probability (MAP) consists in estimating the target vector $\mathbf{z}$ through the formula:

$$\widehat{\mathbf{z}}={\text{argmax}}_{\mathbf{z}}p\left(\mathbf{z}|\mathbf{h},\mathbf{v}\right).$$E12

in which we denote by $\mathbf{v}$ the available HR image ($\mathbf{m}$ or $\mathbf{p}$). A reliable solution of (12) can be found by regularizing the problem by adding a penalization term to the quantity $p\left(\mathbf{z}|\mathbf{h},\mathbf{v}\right)$. Examples of widely employed regularization terms include Gaussian priors [43, 44] or vector total variation (VTV) [22].

3. Quality assessment of fusion products

In this section, we present the performance assessment setup procedure for sharpening the HS data. The objective is to test the viability of fusion algorithms to reach a resolution enhancement factor $R$ that goes beyond the limitations of single-platform setups. In the specific testbed, the HS data are constituted by acquisitions taken by the Hyperion sensor, which is characterized by a GSI of 30 m. The satellite platform also features the a PAN sensor, called ALI, whose GSI is 10 m, which corresponds to a nominal enhancement factor $R=3$. For more ambitious factors, two extra scenarios are considered; in particular a very interesting comparison can be taken at $R=6$ and $R=12$ by analyzing different behaviors for single- and multi-platform with a selection of 12 state-of-the-art fusion algorithms. Specifically the single-platform case requires a preliminary interpolation of the ALI images, here performed via a convolution with a 45-tap interpolation kernel. The multi-platform case will employ, as companion source image, the MS imagery acquired by the WorldView-3, which instead have to be downsized to the target resolution, as it is characterized by a smaller GSI than the target one reached by all the considered $R$. The decimation procedure is performed by employing a filter, mimicking the modulation transfer function of the MS sensor and a downsampling.

We want to remark here that this study will ignore the contribution of the ALI MS and WV3 PAN sensors. The former has the same GSI of the Hyperion sensor, making its information mostly redundant. Regarding the latter, we want to remark that the native GSI of the MS WV3 sensors already exceeds that of the target resolution characterized, for all $R$ under examination. Consequently, it is preferable to employ the MS sensor, as it is already characterized by a better spectral resolution, as shown in previous studies [14, 41].

3.1 Assessment procedure

The assessment procedure has been carried out at reduced resolution, namely, the original HS image is used as reference, and the images to fuse are obtained by degrading the available images by a factor equal to the resolution enhancement factor $R$. The adopted Wald’s protocol [23] requires the reproduction of the characteristics of the fusion problem at a lower resolution. Accordingly, all the available images are degraded by using an MTF-shaped filter matched to the specific sensor and a downsampling system with factor $R$.

The reduced resolution assessment protocol allows the use of many accurate quality indexes, since the ground truth image is available. In this work we consider the spectral angle mapper [42] for evaluating the spectral distortion and the erreur relative globale adimensionnelle de synthèse (ERGAS) [45] for assessing the radiometric distortion. The vectorial Q2${}^n$-index [46] index is used for obtaining a comprehensive measure of the overall image quality. Finally, we employed the universal image quality index (UIQI) or Q-index, proposed by Wang and Bovik [47], for performing a band-by-band comparison of the final product with the reference image.

3.2 Datasets

Three datasets are used for illustrating the capabilities of data fusion algorithms in producing very high-resolution hyperspectral images. The images have been acquired by the Earth Observing-1 and WorldView-3 sensors. The different settings allow to examine the features of the sharpening algorithms in the presence of the most common issues implied by multi-platform data fusion, namely, the different points of view and the temporal changes in the illuminated scenes between the two acquisitions. In this study we employ the visible near-infrared (VNIR) bands B09-B57, acquired by the sensor Hyperion, as HS data. The single-platform companion data are constituted by the PAN images collected by the ALI sensor, having a 10 m spatial resolution. All EO-1 data share a radiometric resolution of 15 bits. The multi-platform data have been acquired by the WV3 satellite. They are represented by an MS image composed of eight channels (coastal, blue, green, yellow, red, red edge, NIR1, and NIR2) with a radiometric resolution of 11 bits and an original spatial resolution of 1.2 m.

The employed datasets are briefly described below:

• Harlem dataset: the images have been collected in New York, USA, in the neighborhoods of the Harlem River. The size of the Hyperion and PAN ALI data, acquired on July 21, 2016, is 144 × 144 pixels and 432 × 432 pixels, respectively, while the native dimension of the WV3 MS image, acquired on June 9, 2016, is 4320 × 4320 pixels.

• Agnano dataset: the images refer to the area of the Agnano Racecourse, next to the city of Naples, Italy. The size of the Hyperion data is 144 × 72 pixels, and thus the corresponding ALI and WV3 images are composed by 432 × 216 pixels and 4320 × 1660 pixels, respectively. The acquisition dates of the EO-1 and WV3 sensors are May 20, 2015, and June 8, 2015, respectively.

• Capodichino dataset: the images refer to the east surrounding Naples, Italy, around Capodichino Airport. The images are composed of the same number of pixels of the Agnano dataset and were acquired on May 20, 2015, and on February 4, 2002, by the EO-1 and WV3 satellites, respectively.

3.3 Fusion algorithms

We compare several fusion algorithms to fully assess the quality of HS products achievable through data acquired by a single or multiple platforms. We firstly focus on the use of classical pansharpening approaches, which constitute an almost ready-to-use solution and then present the purposely designed methods. Among the wide plethora of available pansharpening methods [24], we employed the following CS and MRA methods: Brovey transform [26], Gram-Schmidt spectral sharpening [32], and the Gram-Schmidt adaptive [33] belonging to the CS class, the additive wavelet luminance proportional (AWLP) [37], the generalized Laplacian pyramid [48] with MTF-matched filter [49] using both the high-pass modulation scheme [50] (GLP-HPM), and the regression-based injection model (GLP-CBD) [51] belonging to the MRA class.

Among the second group of approaches, we consider the hypersharpening (Hyper) method, developed in [16, 17], and four Bayesian techniques, namely, the coupled nonnegative matrix factorization (CNMF) [21], the naive Gaussian prior (Bay-N) [43], the sparsity promoted Gaussian prior (Bay-S) [44], and the hyperspectral superresolution (HySure) [22].

Finally, we report the results related to a method for upscaling the original image at the target scale by a simple interpolation of the original HS image. We denote as EXP this method that is carried out through a 45-tap interpolation filter and that constitutes also the baseline for more complex sharpening methods presented here.

4. Experimental results

The performance of the fusion algorithms are evaluated both by calculating the numerical values of the chosen quality indexes and by assessing the final products by visual inspection.

The first proposed dataset is about Harlem that has the purpose of illustrating the capabilities of producing a significant improvement of the spatial quality of the original HS images through the compared algorithms. To this aim, we report in Figure 1 the results related to all the tested enhancement factors ($R$ = 3,6,12), using one exemplary algorithm of each class. The RGB images are built by averaging a group of channels in the red, green, and blue frequency ranges (B29–B33, B17–B22, and B11–B15, respectively) to construct the required channels. Naturally, all the reference images (or ground truth) coincide, since they are represented by the original HS image; see Figure 1(a), (m), and (y). On the contrary, the simulated LR HS images, whose upsampled versions (EXP) are reported in Figure 1(g), (s), and (ae), get more and more degraded as the enhancement factor increases.

Figure 1.

Close-ups of the products achieved by applying the pansharpening algorithms to the RR Harlem dataset (red, green, and blue bands). The resolution enhancement factor is R = 3 (first two rows), 6 (3rd and 4th rows), 12 (5th and 6th rows). (WorldView-3 satellite images courtesy of the DigitalGlobe Foundation). The figure labels indicate the fusion algorithm (see Section 3.3) and the high resolution data utilized.

Some introductory considerations can be drawn from the images in Figure 1 obtained by using the SAM-AA for coupling the MS bands to the HS channels. In fact, the first remarkable result is the high quality of the final products achievable also at very high enhancement factors. More in detail, the images obtained by using classical GLP-CBD and the Hyper approach (which constitutes a generalization of the former approach, since both employ a regression-based injection scheme) produce the most appealing sharpened image. They are able to greatly enhance the spatial content of the original HS image, preserving an appreciable coherence of the colors.

On the other hand, the images achievable by using the ALI PAN have a satisfactory aspect only for $R$=3, as it was arguable by the 10 m resolution of the employed HR sensor. The effect of the interpolation is clearly visible in Figure 1(n) and (z), and, thus, this approach could be preferable only when spatial or temporal misalignments among the multi-platform data cannot be avoided.

Those results are perfectly matched to the index values contained in Table 1. Actually, the numerical values point out that in most cases, the use of perfectly aligned images coming from a different satellite can produce images with superior quality also in the case of $R$=3. In this case, the closer correspondence between the details extracted in the MS channel and the missing spatial information of the HS image can justify the outcome.

		R = 3			R = 6			R = 12
		SAM	ERGAS	Q2${}^n$	SAM	ERGAS	Q2${}^n$	SAM	ERGAS	Q2${}^n$
Optimum		0	0	1	0	0	1	0	0	1
EXP		3.5265	5.8326	0.7130	5.1319	3.8836	0.4504	6.5924	2.3097	0.1948
BT	PAN	3.5265	6.5102	0.7966	5.1319	3.9027	0.6722	6.5924	2.2775	0.4588
	MS-CEN	3.5803	5.2828	0.8226	4.2943	2.8664	0.7881	4.9775	1.6175	0.7234
	MS-CC	3.3063	5.1159	0.8228	3.9316	2.7988	0.7893	4.6801	1.6036	0.7325
	MS-SAM	3.2360	5.1028	0.8210	3.9316	2.7988	0.7893	4.6801	1.6036	0.7325
GS	PAN	5.3862	7.0484	0.8087	6.6416	4.1369	0.6680	7.7176	2.3742	0.4410
	MS-CEN	3.6328	5.2939	0.8230	4.3261	2.8723	0.7886	4.9842	1.6189	0.7235
	MS-CC	3.2752	5.1224	0.8265	3.8607	2.7679	0.7929	4.5271	1.5734	0.7273
	MS-SAM	3.2524	5.1175	0.8264	3.8607	2.7679	0.7929	4.5271	1.5734	0.7273
GSA	PAN	3.2214	4.7124	0.8738	4.8934	3.3652	0.6894	6.6072	2.2022	0.4087
	MS-CEN	3.5223	5.3652	0.8409	3.4018	2.5822	0.8428	3.8959	1.3216	0.8294
	MS-CC	3.1317	4.9960	0.8560	2.9116	2.3978	0.8584	3.4708	1.2932	0.8209
	MS-SAM	3.1743	4.9975	0.8561	2.9116	2.3978	0.8584	3.4708	1.2932	0.8209
AWLP	PAN	3.7616	5.4225	0.8676	5.2760	3.4602	0.6878	6.6735	2.2121	0.3316
	MS-CEN	2.8262	4.3152	0.8917	3.4476	2.5087	0.8453	4.6396	1.5423	0.7435
	MS-CC	2.5918	4.0802	0.8964	3.1629	2.4025	0.8500	4.4146	1.5199	0.7475
	MS-SAM	2.5858	4.0637	0.8957	3.1629	2.4025	0.8500	4.4146	1.5199	0.7475
GLP-HPM	PAN	3.8136	5.4635	0.8586	5.4884	3.7330	0.6855	6.9504	2.2929	0.4577
	MS-CEN	2.3843	3.8074	0.9040	3.1578	2.3075	0.8535	3.9734	1.3581	0.7813
	MS-CC	2.1285	3.6578	0.9083	2.7886	2.2014	0.8592	3.5383	1.3000	0.7867
	MS-SAM	2.1307	3.6586	0.9082	2.7886	2.2014	0.8592	3.5383	1.3000	0.7867
GLP-CBD	PAN	3.1941	4.5103	0.8809	4.9075	3.3653	0.7091	6.7263	2.2100	0.4833
	MS-CEN	2.4442	4.0778	0.8955	3.0302	2.3854	0.8653	3.3037	1.2603	0.8559
	MS-CC	2.1961	3.9109	0.9007	2.6360	2.2524	0.8728	2.7772	1.1730	0.8643
	MS-SAM	2.1949	3.9103	0.9007	2.6360	2.2524	0.8728	2.7772	1.1730	0.8643
Hyper	PAN	3.1925	4.5103	0.8809	4.8948	3.3646	0.7091	6.5771	2.2062	0.4840
	MS	2.1810	4.1296	0.8901	2.5145	2.3331	0.8631	2.4683	1.1685	0.8584
CNMF	PAN	3.6166	4.9517	0.8486	5.1629	3.4878	0.6767	6.5140	2.1778	0.4502
	MS	2.7434	5.0313	0.8244	3.0144	2.5666	0.8161	3.1548	1.2958	0.8152
Bay-N	PAN	3.0726	4.1725	0.8832	5.5648	3.6632	0.6391	7.3189	2.2828	0.4263
	MS	2.4412	4.2739	0.8741	3.7782	2.8095	0.8122	4.1716	1.4252	0.8083
Bay-S	PAN	2.9726	4.1264	0.8857	5.4515	3.6479	0.6328	7.3540	2.2766	0.4241
	MS	2.4381	4.2740	0.8741	3.7800	2.8104	0.8122	4.1295	1.4233	0.8085
HySure	PAN	2.9758	4.2484	0.8864	6.6141	4.3498	0.6034	7.7052	2.3314	0.4497
	MS	2.3304	4.5347	0.8634	3.7736	2.8694	0.8068	3.4966	1.3647	0.8187

Table 1.

Values of the quality indexes, related to the reduced resolution assessment procedure, using the Harlem dataset, for resolution enhancement ratio $R=\mathrm{3,6,12}$.

For each algorithm, the best result among the HR options is in boldface.

Finally, we note that the comparison among the assignation algorithms mainly underlines that the two methods optimizing the assignation according to the specific dataset get almost the same results.

The other two scenes allow to gain more insight about the comparison of the sharpening algorithms and of the assignation algorithms. The EO-1 data have been extracted by the same images, while the multi-platform data have very different characteristics. In fact, while the WV3 image of the Agnano dataset has been acquired within a few days from the EO-1 data, the WV3 image of the Capodichino was collected more than 10 years earlier. Accordingly, the layout of the object present in the Capodichino scene is very different among the two passages, also because the area contains rapidly changing objects. A comparison of the two images can be achieved by having a look at Figure 2(a) and (b). The latter scene refers to Naples Airport and contains a plane on the runway that is not detectable in the corresponding WV3 MS image shown in Figure 3(d). Furthermore, different man-made objects are present in the illuminated area at the two acquisition times.

Figure 2.

Close-ups of the products achieved by applying the pansharpening algorithms to the RR Agnano dataset (red, green, and blue bands). The resolution enhancement factor is $R=6$ (WorldView-3 satellite images courtesy of the DigitalGlobe Foundation). The figure labels indicate the fusion method (see Section 3.3), the high resolution data and the assignation algorithm (see Section 2.1.2), utilized.

Figure 3.

Close-ups of the products achieved by applying the pansharpening algorithms to the RR Capodichino dataset (red, green, and blue bands). The resolution enhancement factor is $R=6$ (WorldView-3 satellite images courtesy of the DigitalGlobe Foundation). The figure labels indicate the fusion method (see Section 3.3), the high resolution data and the assignation algorithm (see Section 2.1.2), utilized.

The results related to the Agnano dataset (see Table 2) confirm the conclusions drawn from the analysis of the Harlem dataset. They correspond to the most typical situation in which the images to fuse are ordered to a data provider, minimizing as much as possible the difference between the passage times of the two satellites. Accordingly, in both cases, the illuminated areas contain very similar features that make the multi-temporal data particularly valuable. However, both also represent almost ideal cases, since the presence of rapidly changing objects, for example, the aircrafts present in the Capodichino dataset, can vary also within very close passages. Accordingly, particularly interesting is the case of the Capodichino dataset, which gives rise to somewhat unalike results, which are reported in Table 3. In fact, in most cases, the single-platform setting almost always yields better results, even if the visual appearance of the images related to the multi-platform approach is often preferable in terms of quantity of injected details (see Figure 3). Actually, a more accurate analysis evidences that multi-platform products yield a sharpened image in which the plane is absent (especially in CS methods). Moreover, the spectral quality of the final products is significantly compromised if the WV3 MS images are used.

		R = 3			R = 6			R = 12
		SAM	ERGAS	Q2${}^n$	SAM	ERGAS	Q2${}^n$	SAM	ERGAS	Q2${}^n$
Optimum		0	0	1	0	0	1	0	0	1
EXP		2.6173	2.5563	0.8068	3.7601	1.7890	0.5523	4.6496	1.0807	0.2831
BT	PAN	2.6173	3.0548	0.8058	3.7601	1.8723	0.6009	4.6496	1.0984	0.3604
	MS-CEN	3.2295	2.8406	0.8228	3.6738	1.5425	0.7624	4.2802	0.9126	0.6314
	MS-CC	2.8772	2.7482	0.8259	3.3037	1.4760	0.7778	3.9821	0.8829	0.6574
	MS-SAM	2.8785	2.7545	0.8246	3.3167	1.4778	0.7771	3.9799	0.8831	0.6573
GS	PAN	2.5586	2.9906	0.8245	3.5826	1.8154	0.6350	4.4993	1.0752	0.3833
	MS-CEN	3.3566	2.9027	0.8164	3.7937	1.5864	0.7551	4.3747	0.9302	0.6257
	MS-CC	2.8725	2.7552	0.8289	3.2766	1.4747	0.7795	3.9470	0.8804	0.6553
	MS-SAM	2.8751	2.7627	0.8279	3.2827	1.4756	0.7792	3.9423	0.8803	0.6554
GSA	PAN	2.0386	1.9146	0.9131	2.9847	1.4490	0.7389	4.0761	0.9754	0.4544
	MS-CEN	3.5162	3.3626	0.8386	3.4506	1.5113	0.8447	3.5818	0.7576	0.8114
	MS-CC	3.1628	3.2125	0.8467	2.6754	1.3571	0.8631	2.7824	0.6787	0.8335
	MS-SAM	3.1205	3.2341	0.8434	2.6582	1.3579	0.8625	2.7882	0.6801	0.8343
AWLP	PAN	2.8338	2.5814	0.8813	3.8646	1.6842	0.6804	4.7083	1.0524	0.3699
	MS-CEN	2.7408	2.3775	0.9046	3.3902	1.3975	0.8461	4.2634	0.8638	0.7295
	MS-CC	2.0621	1.9661	0.9277	2.4336	1.1492	0.8788	3.1755	0.7410	0.7617
	MS-SAM	2.0383	1.9784	0.9257	2.4348	1.1512	0.8779	3.1781	0.7414	0.7613
HPM	PAN	2.6258	2.3637	0.8801	3.7500	1.6695	0.6946	4.6745	1.0314	0.4644
	MS-CEN	2.0732	1.8683	0.9279	2.7382	1.1754	0.8645	3.4479	0.7399	0.7318
	MS-CC	1.6181	1.6259	0.9404	2.2147	1.0522	0.8808	3.0151	0.6926	0.7478
	MS-SAM	1.6025	1.6284	0.9401	2.2211	1.0528	0.8806	3.0126	0.6927	0.7478
GLP-CBD	PAN	1.9534	1.8586	0.9140	2.9661	1.4434	0.7528	3.7656	0.9312	0.5485
	MS-CEN	2.1162	1.9590	0.9251	2.8919	1.2444	0.8807	3.5286	0.7418	0.8378
	MS-CC	1.7546	1.8129	0.9340	2.1960	1.1043	0.9003	2.4993	0.6279	0.8699
	MS-SAM	1.7344	1.8256	0.9323	2.1917	1.1067	0.8994	2.5042	0.6276	0.8699
Hyper	PAN	1.9543	1.8586	0.9140	2.9688	1.4419	0.7530	3.7703	0.9171	0.5490
	MS	1.5897	1.7214	0.9382	2.0229	1.0444	0.9080	2.6430	0.6815	0.8437
CNMF	PAN	2.7720	2.6075	0.8584	3.8859	1.7337	0.7138	4.6329	1.0405	0.5231
	MS	2.5088	2.7525	0.8413	2.6865	1.3843	0.8302	3.0105	0.7685	0.7988
Bay-N	PAN	2.2560	2.1131	0.9017	3.8832	1.7313	0.6794	5.0346	1.1319	0.5224
	MS	2.2126	2.3448	0.8934	3.4757	1.6422	0.8013	3.9694	0.9266	0.7793
Bay-S	PAN	1.9858	1.8416	0.9190	3.7604	1.6862	0.6807	4.9424	1.1049	0.5233
	MS	2.2130	2.3451	0.8934	3.4818	1.6433	0.8010	4.1745	0.9468	0.7705
HySure	PAN	2.3509	2.2174	0.8922	4.5235	2.0695	0.6128	5.3742	1.2250	0.5362
	MS	2.5649	2.7776	0.8513	3.1989	1.5863	0.7770	3.2956	0.8437	0.7707

Table 2.

Values of the quality indexes, related to the reduced resolution assessment procedure, using the Agnano dataset, for resolution enhancement ratio $R=\mathrm{3,6,12}$.

For each algorithm, the best result among the HR options is in boldface.

		R = 3			R = 6			R = 12
		SAM	ERGAS	Q2${}^n$	SAM	ERGAS	Q2${}^n$	SAM	ERGAS	Q2${}^n$
Optimum		0	0	1	0	0	1	0	0	1
EXP		2.9041	3.4640	0.7607	4.0809	2.3556	0.4930	5.1271	1.4488	0.2097
BT	PAN	2.9041	4.1628	0.7804	4.0809	2.4750	0.6310	5.1271	1.4643	0.3702
	MS-CEN	4.6792	5.7317	0.5938	4.9025	2.8810	0.5576	5.2432	1.5154	0.4746
	MS-CC	4.4343	5.5489	0.5875	4.6256	2.8173	0.5608	4.8858	1.4960	0.4877
	MS-SAM	4.4509	5.6304	0.5926	4.5951	2.8322	0.5656	4.8969	1.4970	0.4884
GS	PAN	3.2286	4.2133	0.7758	4.6558	2.5912	0.5960	6.0683	1.5821	0.3326
	MS-CEN	4.7700	5.7476	0.5942	5.0022	2.8997	0.5559	5.3204	1.5280	0.4738
	MS-CC	4.4891	5.6363	0.6008	4.5862	2.8320	0.5674	4.8663	1.4973	0.4873
	MS-SAM	4.4772	5.6713	0.6002	4.5710	2.8401	0.5688	4.8625	1.4977	0.4875
GSA	PAN	2.4807	2.5976	0.8833	3.4450	1.8687	0.6998	4.7540	1.2970	0.3636
	MS-CEN	7.0619	8.2274	0.5357	6.4658	3.6062	0.5650	6.2811	1.6761	0.5705
	MS-CC	6.0976	6.8763	0.6001	5.1292	2.9285	0.6236	4.5374	1.3304	0.6124
	MS-SAM	6.3286	7.3363	0.5718	5.1970	3.0603	0.6081	4.5366	1.3311	0.6116
AWLP	PAN	3.1491	3.3732	0.8598	4.3651	2.1254	0.6899	5.3684	1.3789	0.3466
	MS-CEN	3.5341	3.9606	0.7920	4.4168	2.4048	0.6799	5.3696	1.4223	0.5509
	MS-CC	3.2680	3.6163	0.8090	3.9660	2.2291	0.7016	4.6378	1.3293	0.5765
	MS-SAM	3.2595	3.6703	0.8095	3.9215	2.2381	0.7044	4.6313	1.3300	0.5770
HPM	PAN	3.0965	3.1605	0.8526	4.4371	2.2155	0.6823	5.6302	1.4059	0.4234
	MS-CEN	2.7198	3.1964	0.8373	3.5213	2.0407	0.7212	4.2637	1.2216	0.5688
	MS-CC	2.5362	3.0548	0.8462	3.2128	1.9735	0.7323	3.8668	1.1926	0.5824
	MS-SAM	2.5133	3.0672	0.8469	3.1841	1.9742	0.7343	3.8640	1.1927	0.5827
CBD	PAN	2.4126	2.5037	0.8832	3.3688	1.8193	0.7261	4.5275	1.2246	0.4657
	MS-CEN	2.6355	3.1271	0.8385	3.6260	2.1248	0.7234	4.7470	1.3681	0.6321
	MS-CC	2.5513	3.0624	0.8432	3.4488	2.0790	0.7322	4.2992	1.3201	0.6488
	MS-SAM	2.5421	3.0807	0.8436	3.4306	2.0895	0.7336	4.2828	1.3220	0.6487
Hyper	PAN	2.4126	2.5037	0.8832	3.3684	1.8199	0.7259	4.5152	1.2254	0.4652
	MS	2.7138	3.2570	0.8224	3.8293	2.2194	0.6966	4.6147	1.3181	0.6051
CNMF	PAN	3.0309	3.2174	0.8361	4.1597	2.1199	0.7035	5.0892	1.3082	0.4882
	MS	4.2695	5.4648	0.5984	3.7804	2.6307	0.6027	4.3318	1.4644	0.5163
Bay-N	PAN	2.4809	2.5412	0.8897	4.2876	2.1478	0.6600	5.7306	1.4619	0.4276
	MS	2.8266	3.3996	0.8173	4.3109	2.5638	0.6354	4.9158	1.5127	0.5777
Bay-S	PAN	2.3080	2.3705	0.8981	4.1732	2.1120	0.6596	5.7597	1.4522	0.4304
	MS	2.8267	3.3989	0.8173	4.3114	2.5637	0.6354	4.9126	1.5115	0.5780
HySure	PAN	2.4928	2.7947	0.8754	5.3229	2.7732	0.6101	5.7868	1.4747	0.4592
	MS	3.2759	4.3963	0.7367	4.1931	2.8471	0.5906	4.5497	1.5079	0.5723

Table 3.

Values of the quality indexes, related to the reduced resolution assessment procedure, using the Capodichino dataset, for resolution enhancement ratio $R=\mathrm{3,6,12}$.

For each algorithm, the best result among the HR options is in boldface.

The spatial quality differences among the various algorithms can be further investigated by resorting to a quality index that allows a band-by-band analysis of the quality of the algorithms output. To this aim, we report in Figures 4 and 5 the behavior of the Q-index as a function of the HS band. The two images reveal both similarities and discrepancies in the algorithms’ performance. In particular, we can note that for the HS channels with support contained in the frequency range covered by the ALI PAN, the use of single-platform data is always preferable, except for the case of the Hyper algorithm applied to the Agnano dataset with $R$ = 6. Clearly, this consideration is all the more true in the experiment related to the Capodichino dataset. Instead, different trends are experienced for the near-infrared (NIR) bands. All the algorithms (except the GSA algorithm with $R$ = 3) obtain better performance by using multi-platform data working on the Agnano dataset. On the contrary, using the Capodichino dataset, the GSA algorithms always obtain superior results by using the ALI PAN image, while the other two methods obtain a slightly better performance in the NIR region that is not able to balance the scarce quality in the visible range, thus resulting in an inferior overall performance of the multi-platform approach. Finally, it is very clear from both Figures 4 and 5 that the CC-AA and the SAM-AA algorithms are able to obtain significant improvements with respect to CEN-CC, especially in the NIR frequencies.

Figure 4.

Q-index as a function of the HS band for the Agnano dataset. The curves refer to the data fusion of the Hyperion images with the ALI PAN (black continuous), the MS-CEN (red dashed curve), MS-CC (green dotted curve), and MS-SAM (blue continuous curve with circle marks).

Figure 5.

Q-index as a function of the HS band for the Capodichino dataset. The curves refer to the data fusion of the Hyperion images with the ALI PAN (black continuous), the MS-CEN (red dashed curve), MS-CC (green dotted curve), and MS-SAM (blue continuous curve with circle marks).

5. Conclusions

The aim of this work was to illustrate the recent advances in the field of hyperspectral image sharpening through single-platform and multi-platform data. The study was conducted on real data acquired by the Earth Observing-1 and the WorldView-3 satellites in order to highlight the practical issues to be addressed when fusing images acquired by different platforms. We focused on well-known algorithms based on classical approaches borrowed from the pansharpening literature and on techniques developed on purpose. We evaluated the possibility of completing the fusion process, both in the absence and presence of temporal misalignments between the scenes illuminated by the sensors mounted on the two satellites. The study highlighted the suitability of the employment of multi-platform data especially in the presence of high-resolution enhancement factors. Actually, in some cases, the use of multispectral images was also proven to be useful at low-resolution enhancement factors, and this result can be easily justified by taking into consideration that the details contained in the MS channels are able to provide more specific spatial information for a given HS channel.