IntechOpen
IntechOpen

Open access peer-reviewed article

MRF Models Based on a Neighborhood Adaptive Class Conditional Likelihood For Multimodal Change Detection

Max Mignotte

This Article is part of NUMERICAL ANALYSIS AND SCIENTIFIC COMPUTING Section

Version of Record (VOR)

*This version of record replaces the original advanced online publication published on 28/03/2022

Article metrics overview

447 Article Downloads

View Full Metrics

Article Type: Research Paper

Date of acceptance: March 2022

Date of publication: March 2022

DoI: 10.5772/acrt.02

copyright: ©2022 The Author(s), Licensee IntechOpen, License: CC BY 4.0

Download for free

Table of contents

Introduction
Proposed MRF CD model
Experimental results
Conclusion
Acknowledgments
Data availability
Conflict of interest

Abstract

Statistical methods for automatic change detection, in heterogeneous bitemporal satellite images, remains a challenging research topic in remote sensing mainly because this research field involves the processing of image data with potentially very different statistical behaviors. In this paper, we propose a new Bayesian statistical approach, relying on spatially adaptive class conditional likelihoods which are also adaptive to the considered imaging modality pair and whose parameters are estimated in a first preliminary estimation step. Once that estimation is done, a second stage is dedicated to the change detection segmentation itself based on this likelihood model defined for each pixel and for each imaging modality. In this context, we compare and discuss the performance of different Markovian segmentation strategies obtained in the sense of several non-hierarchical or hierarchical Markovian estimators on real satellite images with different imaging multi-modalities. Based on our original pixel-wise likelihood model, we also compare these Markovian segmentation strategies over the existing state-of-the-art heterogeneous change detection algorithms proposed in the literature.

Keywords

  • change detection

  • heterogeneous satellite captors/sensors

  • Markovian segmentation

  • multimodal or multisource or multisensor or mutilooking satellite images

  • parameter estimation

  • Markovian estimators

Author information

Introduction

Multimodal change detection (MCD) is a recent area of research that has grown and developed considerably over the past decade, mainly due to the rapid development of new sensor systems, new data processing techniques and easier access to remote sensing data. In satellite imagery, multimodal (or heterogeneous) change detection (CD) [1] is the process of detecting or identifying, in a given geographical area, any changes, based on two (or more) images acquired at different instances. The pair (or set) of satellite images (also commonly referred to as bi (or multi)-temporal images to be analyzed) are usually acquired either by different (or heterogeneous) sensors or with the same sensor but with different specifications or conditions.

More precisely, in MCD systems, the heterogeneous bitemporal images may be; (in order of increasing difference from the conventional monomodal CD and difficulty of processing), either provided by different (active or passive) types of sensors, like passive optical and active SAR systems (and leading to bitemporal multisource images), or provided by different passive systems (e.g., different optical sensors) or active remote sensors such as lidars and radar (in the case of bitemporal multisensor or cross-sensor images). Heterogeneous bitemporal images can also be acquired with the same sensing system, but either in different wavelength ranges (hyperspectral or multispectral images) or with different internal settings or different looks (possibly different look angles) or different and complementary speckle noise pre-filtering processes (for mutilooking images) or with unbalanced data or noise distributions possibly due to lighting, weather or phenological or chemical changes which also influences the imaging system (unbalanced image data [2]).

MCD is a difficult image processing problem that can only be solved with a sufficiently flexible, intelligent and robust model for processing image data with different statistical behaviors. This low-level image processing task allows us to solve the same issues usually handled by homogeneous CD techniques [3–8] required for the development, for example, of damage detection and evaluation systems (earthquake, flooding, hurricane, tsunami, forest fire, volcanic activation, etc.) or environmental, agricultural, mineral exploration, or urban growth, monitoring or planning systems etc. MCD has shown an increasing trend and a great interest over the past decade in the geoscience community, mainly because this approach is much less demanding on the origin of the data acquired compared to the conventional homogeneous CD technique. In fact, this low-level processing allows us to fully and intelligently use the multiple heterogeneous (and ever increasing) data sourced from existing archives or from the many and very varied Earth observation satellites existing or planned. In addition, due to advances in this field and the format/specifications evolving through time, it is expected that the heterogeneity of these satellites or aerial image data will be increasing in the years to come. Finally, we include the complementarity of these different imaging methods, which can be used to our advantage. More precisely, a technique of fusion of imaging modalities could potentially be exploited (not only in geo-scientific imagery [9]) to further improve the detection and the analysis of surface changes.

The problem of change detection for homogeneous satellite datasets has been extensively studied in the literature since the advent of digital imaging techniques. The Geoscience research community waited until this problem was recently well enough understood to investigate the MCD problem. This explains why until now, relatively little research has been proposed on MCD.

Despite this, and depending on the modeling strategy used, four main groups of approaches are reported in the recent remote sensing literature and can be easily identified.

The first and most basic techniques are based on similarity metrics [10–12] or rely on empirical (generally hand-crafted) features or local and non parametric operators or detectors [12–14] with supposed good invariance to sensor characteristic.

The second group is made up of non-parametric (possibly statistical) based models whose structure is not a priori fixed in advance (unlike so-called parametric models for which a distribution law with a finite number of parameters is a priori specified as main data structure). It refers to the methods based on a structure consistency [15, 16] or a K-nearest neighbor graph [17] to represent the structure of each image or using a regression strategy based on this structural consistency [18]. This category also contains machine learning (ML), and deep learning (DL) based models which learn representation of (training) data samples (examples) with possibly several levels of abstraction for DL [2, 19–28]. Additionally, image translation with convolutional neural networks (with prior-weighted loss functions that reduce the impact of change pixels on the learning objective) [29] has recently been used as an approach to MCD. It is also made up of unsupervised non-parametric based techniques, that are not based on training examples like the cost function-based model presented in [30].

Thirdly, one can also identify algorithms based mainly on a projection or transformation of the bi-temporal heterogeneous images into a common representation space, in which the pair of multimodal satellite images have the same behavior in statistical terms and on which conventional change detection techniques using homogeneous multitemporal satellite images can then be adopted [31–43]. Belonging to this category but also to the previous one, image mapping from one domain to another domain was also exploited via deep translation based change detection network [44], multiscale deep pyramid feature learning network [45] or cross-resolution difference learning [46].

Finally the last class is given by parametric methods, which we will detail more precisely below since the model described in this work falls into this group. In this strategy, a (possibly finite mixture of) parameterized multivariate distribution law(s) are usually (a priori specified as main data structure and) employed to describe the joint statistic or to account the links between the two different sensor systems or to exploit the statistical dependence between the two measured remote sensing modalities [47]. For example, local statistical analysis of dependence between unchanged areas is derived using the copula representation in [48, 49] and Kullback-Leibler-based comparisons of those statistics are applied to define a similarity map which is then subsequently segmented by thresholding, into change and no-change classes. In [50], the authors base their model on the preliminary estimation of a mixture of flexible multidimensional and multivariate laws using various skewness and kurtosis parameters and supposed to be particularly well adapted for estimating the difference changes in multitemporal Synthetic Aperture Radar (SAR) images given by various multi-looking techniques or with SAR data having different look numbers. In [51–53], a multivariate statistical method, using an estimation step followed by a final segmentation step, has also been proposed; the preliminary stage aims at estimating a statistical model of the sensor response consisting of a mixture of multivariate densities which take into account both the sensor responses to the observed scene and the system noise. Based on this model, a difference map is generated and then subsequently binarized, via a statistical test, to detect the changes. Recently, an original method based on linear algebra, using the Kronecker product between the two image representations, has also been proposed in [54] for the MCD problem.

Finally, the authors in [55] use a pixel pair modeling and find the likelihood laws for these two possible class labels; i.e., “identical” (for the couple of labels; change | change or no-change | no-change) or “different” (for the pair of classes; change | no-change or no-change | change) of these two groups of pixel pairs and, based on this pixel pair modeling, then use the simulated annealing to find the segmentation solution in the MAP sense.

In this work, we present a quite different approach, in a non-parametric statistical framework, whose originality is to rely on a set of class conditional likelihoods also conditioned to the spatial neighborhood of each pixel. This set of spatially adaptive likelihoods allows us to formally define the MCD issue in the Bayesian setting with a reliable and spatially (or neighborhood) adaptive likelihood model whose parameters are estimated in a first preliminary estimation step. Once the estimation stage is achieved, a second stage is dedicated to the binarization (or detection of multimodal change) itself given this set of estimated spatially adaptive likelihoods. In this context, we compare and discuss the accuracy of the CD segmentation map obtained in the sense of several Markovian estimators such as the Maximum Likelihood (ML), MAP (Maximum A Posteriori) [56], MPM (Marginal Posterior Mode) [57], SMAP (or Sequential MAP) estimator using the multiscale and hierarchical Bayesian segmentation framework of Bouman et al. [58] and finally the SCMAP (Scale Causal MAP) estimator proposed in [59] and based on a multi-level prior model involving both scale-causal and spatial interactions combined with a parameterization multigrid technique derived from the finest level.

Proposed MRF CD model

Let yt1 and yt2, a pair of bi-temporal (previously spatially co-registered) satellite images (of N pixels) captured at two separate time periods (usually before and after a natural or man-made event), over the same region of interest, and coming from heterogeneous sensors. Let (yt1, yt2) be a realization of the random variable which is the so-called random field of observations associated to these two images and defined on a rectangular grid S of N sites s. We also consider X = {Xs, s ∈ S} the random label field defined on the same grid related to the bi-temporal satellite images, with each xs, its realization at site s, taking its label in the set of classes 𝛬label = {e0 = no-change,  e1 = change}.

Estimation step

In the unsupervised Bayesian framework proposed in this work, it is necessary, first of all, to estimate the likelihoods (i.e., the so-called marginal or conditional distributions) for both each detection class (namely; e0 = no-change and e1 = change) and for each pixel or more precisely and according to our modeling approach, for each spatial neighborhood 𝜂s around each pixel.

To this end, for each pixel , at site s, we first look for the Nk best (squared) sub images (with size Sw × Sw pixels), existing in the before image yt1, which most looks like, in the Least Square (LS) sense, to the spatial neighborhood 𝜂s (with the same size Sw × Sw) surrounding s. Since the imaging systems used in remote sensing are generally spatially invariant, this similarity search is performed in order to be invariant towards the 8 spatial isometric transformations (rotations and flips) of the neighborhood. In our application, this search is efficiently and quickly achieved by a kd-tree nearest neighbor searching method [60].

In our experiments, we have noticed that the distribution of the grey levels associated to each central pixel of these Nk sub images positioned, at the same coordinate, but in the after image yt2 is well approximated by a Gaussian law whose parameters are estimated via the empirical mean ms and variance . This observation allows us to define the conditional data likelihood PYs∕Xs, 𝜂s associated to the class label e0 = no-change of each pixel s (or equivalently, conditioned to a site s or its spatial neighborhood 𝜂s surrounding it).

Regarding the class label e1 = change, the type of empirical distribution PYs∕Xs, 𝜂s obtained varies a lot and can be very different depending either on the nature of the multi-modalities (considered here; namely multisensor, multisource and mutilooking), or the variety of changed conditions or events (river drying up or flooding, urban construction or inundation, volcano activation, etc.) or with the different possible resolution levels or image sizes. Due to the lack of an appropriate statistical tool to adaptively model this highly variable behavior, the marginal likelihood associated to the class label change should be as less informative as possible. In our application, we have thus adopted a rectangular distribution (also called uniform distribution) over the range of grey-scale levels covered by the after image yt2 thus reflecting the large entropy (or great uncertainty) of this change class-conditional likelihood. Let us note that this type of distribution has often been widely used, as universal non-informative likelihood, for many applications [61]. Finally, the complete data model, different for each pixel s located on the rectangular lattice S, is defined by:

we recall that 𝜇s and , are the empirical mean and variance computed on the set of the grey levels associated to each central pixel of the Nk sub images which first most looks like (in the LS sense) to the spatial neighborhood 𝜂s surrounding s in yt1 and which is then re-located at the same coordinate but in the after image yt2. , and are respectively the Uniform law, and the minimal and maximal grey level existing in the after image yt2.

At this point, it is important to understand that some of the Nk sub images found by the kd-tree search method could be (partially or completely) contaminated by the post-event change (existing in the after image yt2) and this could affect the estimation of the mean (and somewhat the variance) of each pixel-wise likelihood distribution PYs∕Xs, 𝜂s (ys∕e0, 𝜂s) defined for each site s. In fact, this problem is a bit minimized by the fact that firstly, we take, not one, but Nk (closest in the LS sense) sub images and secondly, by the use of a median filter which somewhat correct the unwanted outlier estimations. In addition, in our application, we index each estimation of the mean on the same rectangular grid S (already associated with the two input images) and we use a 3 × 3 spatial median filter to somewhat get rid of contaminated mean estimations. Nevertheless, these two previous strategies are not sufficient to ensure a reliable estimation and that is why we resort to a two-step estimation/segmentation procedure. A first estimation/segmentation step is conducted with the estimation step previously explained (equation (1) followed by median filtering) and a segmentation step (that will be explained in section 2.2) to get the binary CD map . A second estimation/segmentation step is then achieved in which the kd-tree search method of the estimation step looks for the Nk best sub-images, existing in the before image yt1 and which are not contaminated by a pixel whose label has been previously classified e1 = change. It is also interesting to note that this phenomenon of contamination by the post-event change can be considered as noise (both for the estimation step and the segmentation procedure). This being said, the Markovian regularization used in the segmentation step is well adapted to infer a reliable solution in presence of (any) noise.

Segmentation step

Once the estimation of the conditional likelihood distributions (see equations (1)) are defined for each site s, it is straightforward1 to extend the different state-of-the-art Markovian image segmentation algorithms defined as being optimal in the sense of several Markovian estimators such as the ML [56], MAP [56], MPM [57], SMAP [58] and SCMAP criteria [59].

At this point, it is interesting to recall the fundamental difference between these different criteria commonly used in the Bayesian image segmentation framework. To this end, let x be the segmentation or a particular realization (or configuration) of the random label field X (to be estimated). A Bayesian segmentation considers, as an estimation criterion, the (Bayesian) criterion of minimization of the expectation of the loss function C (.) conditioned to the observations:

where Ω designates the space of possible configurations. xGT represents the exact solution and C (xGT, x) represents the price to be paid when x is chosen when the exact solution is xGT. Several Bayesian estimators corresponding to different cost functions can be chosen. MAP is associated with the cost function:
where 𝛿x GT(x) denotes the delta-Dirac mass located in xGT. This cost will penalize, in the same way, all configurations different from xGT. In our case; with a Markovian prior (we have herein considered a classical a priori isotropic, homogeneous Ising/Potts model conditionally dependent on its 8 nearest neighbors. This loss function is also that of the ML criterion for an uniform prior distribution. The SCMAP criterion is also associated to this cost function but relatively to a Markovian prior with a spatial second order neighborhood system including a scale causal link with the (immediately) coarser resolution scale (figure 1). The MPM estimator is associated with a less restrictive cost function penalizing the configuration x in proportion to the number of incorrectly labeled sites:
The MAP estimator is generally poorly adapted to hierarchical Bayesian models. Indeed, it will penalize an error regardless of the scale at which it occurs [58]. Ideally, for these models, errors produced at coarse scales should be penalized more severely (since the misclassification of a label at a coarse scale then causes the misclassification of a large number of labels at the finest resolution). This is what is proposed by the SMAP estimator which is associated with the following cost function:
where the label fields between scales are constructed, with a prior Markov chain in scale, with a quad-tree structure (for the ascending Markovian process) and a pyramid structure (for the descending pass) (see figure 1 at top) in which n is the number of scale levels. In this model, xj is the segmentation at scale level j. Qualitatively, this cost function penalizes a multi-resolution segmentation (according to an hybrid quad-tree-graph structure) in proportion to the size of the largest misclassified region [58]. We will refer the reader to the following studies [56] (ML, MAP), [57] (MPM), [58] (SMAP) and [59] (SCMAP) for the detailed implementation of these Markovian segmentation algorithms (which will be also freely accessible at author’s web address1).

Figure 1.

From left to right; Hierarchical Markovian structure of the SMAP with a quad-tree structure (for the ascending Markovian process) and a pyramid structure (for the descending pass) and SCMAP with a spatial second order neighborhood system including a scale causal link with the (immediately) coarser resolution scale.

ICM segmentation model with adaptive class conditional likelihood

We detail a little more in this section, the simplest Markovian segmentation algorithm called ICM related to the classical a priori Markovian Potts prior with the 8-nearest neighborhood structure (with binary clique potential parameter 𝛽). For more details about ICM, we refer the reader to [56], and for more details on the derivation, implementation and optimization of the other Markovian segmentation strategies, with their prior structure, we refer the reader to [57] (for the MPM), [58] (for the SMAP) and [59] (for the Scale Causal MAP).

In this Bayesian strategy, the segmentation in two classes 𝛬label = {e0 = no-change, e1 = change} of yt2 can be viewed or stated as a statistical labelling problem according to a global Bayesian formulation in which the posterior distribution PX∕Yt2, 𝛷s (x∕yt2, 𝜙s) ∝ exp[−U (x, yt2, 𝜙s)] has to be maximized. In our case, the corresponding posterior energy U (x, yt2, 𝜙s) to be minimized is of the form:

where U1 expresses the adequacy between observations and labels:
and U2 is the energy of the a priori model (𝛽 is the clique potential whatever the type of neighboring pair ⟨s, t⟩), with , a likelihood function parameterized (and preliminary estimated) for each site s, with , a normal law and for , a uniform law (see equation (1) and section2.1).

In order to minimize this energy function, we use a iterative deterministic optimization technique called ICM algorithm for which we recall the different steps of this deterministic iterative minimization algorithm:

  • For the initialization of ICM (iteration [0], in superscript), we choose an initial configuration as close as possible to the optimal segmentation; for example a segmentation in the ML sense:

  • Estimation of from :

  1. For each pixel (or site) in lexicographic order:

    • For each site s, we compute the local posterior energy term for each of the possible classes:

    • We select the class xs which corresponds to the minimum local posterior energy term:

  3. Return to 1.   until a criterion is reached, generally:

Figure 2.

Multimodal datasets (see table 1). (a–c) pre-event image t1, post-event image t2, ground truth segmentation provided by an expert; and (d–e) visualization of the class confusion map (white: true negative, red: true positive, blue: false positive, cyan: false negative) obtained by our Neighborhood and Class Conditional Likelihood estimation combined with the SMAP (d) or SCMAP (e) estimator.

Figure 3.

Multimodal datasets (see table 1). (a–c) pre-event image t1, post-event image t2, ground truth segmentation provided by an expert; and (d–e) visualization of the class confusion map (white: true negative, red: true positive, blue: false positive, cyan: false negative) obtained by our Neighborhood and Class Conditional Likelihood estimation combined with the SMAP (d) or SCMAP (e) estimator.

Figure 4.

CD confusion maps (white: TN, red: TP, blue: FP, cyan: FN) for the dataset #1 (top) and #2 (bottom), obtained in the sense of several Markovian estimators combined with the data likelihood model defined in equation (1). (a) ML (b) MAP (with ICM) (c) MPM (d) SMAP (e) SCMAP.

Experimental results

Heterogeneous dataset description & setup

To show the effectiveness of our approach, we have performed extensive experiments involving different real heterogeneous datasets with different multi-modality types (multi-sensor/source/looking) and exhibiting a wide diversity of changed events and finally provided at different resolution levels and image sizes (see table 1).

For all the tests we have performed, we have only used the luminance component or the greyscale information of the image and thus converted the three color channels (or the multi-spectral bands (possibly with the models introduced in [62, 63])) into one single gray channel when it is necessary. In addition, to reduce the computational load, we have chosen to sub-sample the image so that its maximum length or width is less than 512 pixels (with a decimation technique given by a simple moving average filter) and finally, we have used a double histogram matching method [64] on yt1 and yt2 (more precisely, the before image is histogram matched to the after image to give the pre-processed before image and the after image is then histogram matched to the latter (pre-processed before) image).

Date Location Size (pixels)Event (& Spatial res.)Sensor
109/95–07/96 Sardinia, It 123.6 KOverflow (30 m)Landsat-5 (NIR band)/Optical
207/06–07/07 Gloucester, UK 9.6 MFlooding (0.65 m)TerraSAR-X/QuickBird 2
302/09–07/13 Toulouse, Fr 11.4 MUrbanization (2 m)TerraSAR-X/Pleiades
405/12–07/13 Toulouse, Fr 4 MUrbanization (0.52 m)Pleiades/WorldView-2
501/01–01/02 Congo, Africa 320 KVolcano (10 m)Radarsat (3/5-looks)
601/17–02/17Sutter, CA, USA 439.7 KFlooding (≈15 m)Landsat-8/Sentinel-1A
706/08–06/13 Island town, Ch 167.2 KUrbanization (8 m)Radarsat-2/ Google Earth
806/08–09/12 Shuguang, Ch419 × 342Urbanization (8 m)Radarsat-2/Google Earth
906/08–06/09 Yellow river, Ch 150 KRiver drying up (8 m)Radarsat-2 (1/4-looks)
10 1999–2000 Gloucester, UK 548.5 KFlooding (≈25 m)Spot/NDVI

Table 1

Multimodal datasets with multisensor (#1, #4, #10), multisource (#2, #3, #6, #7, #8) and mutilooking (#5, #9) bi-temporal satellite images.

The internal parameters of our estimation step model (see section 2.1) are; Nk, the number of squared sub-windows that most closely resembles (in the LS sense) the spatial neighborhoods of each pixel and its size Sw. In our application and experiments; Nk = 20 and Sw = 5. For the segmentation step, we have considered, a classical a priori Markovian Potts prior with a 8-nearest neighborhood structure with binary clique potential parameter 𝛽 = 1 for the ICM algorithm (MAP estimator) and for the Gibb’s sampler of the MPM and also for the spatial interactions (or cliques) of the SCMAP model. We have used an inter-scale clique potential setting to 𝛽 = 0.5 for the SCMAP estimator and a hybrid quad-tree and pyramid-graph structure for the SMAP along with a structure depth fixed to d = 9 and an inter-scale regularization parameter for the Markov chains formed by the parent and child nodes in the quadtree (ascending step) and in the pyramid structure (descending step) equals to 𝜃 = 1.0−. In fact, let us note that all these parameters are classic and commonly used in this field. Nevertheless, we have also checked that these parameters also guarantee, on average, the best detection results over the ten heterogeneous data sets. These three internal parameters (Nk, Sw, 𝛽) were fixed after three or four trials and the efficiency of our (low complexity) MRF-MCD algorithm has the property to remain quite insensitive to the variation of its three internal parameters, insofar as these internal parameters vary within a reasonable range (value of the parameter ±50%).

Results & discussion

To discuss the efficiency of our neighborhood-adaptive class conditional data likelihood model defined in equation (1), we have compared the obtained CD segmentation results on our heterogeneous dataset, in the sense of different existing (non-hierarchical or hierarchical) Markovian estimators, a first quantitative study is achieved and evaluated with the same evaluation measures proposed in [55] and [40]; namely the F-measure and the total percentage of good classification (accuracy).

We can notice (see table 2) that hierarchical Markovian estimators, like the SMAP and SCMAP are in fact comparable, in terms of efficiency (while being quite different in terms of hierarchical structures and algorithms) and clearly perform better than the classical non-hierarchical Markovian estimators such as the MAP or MPM. MAP (via the ICM algorithm) and MPM estimators are comparable in terms of performance, while also being algorithmically quite different both of them. Finally, we can compare the score results given by a classification approach without Markovian regularization (ML strategy) and thus the obtained gain for each Markovian regularization strategy over the ML estimator. Comparatively, SMAP is slightly better in terms of mean F-measure than SCMAP, which is slightly better in term of accuracy or overall correct classification percentage average (see table 2).

Estimator #1 #2 #3 #4 #5 #6 #7 #8 #9 #10PCC Mean
ML92.687.481.979.780.485.986.090.386.885.8 85.680
MAP95.792.187.884.981.592.592.192.595.990.8 90.588
MPM95.892.287.884.981.592.592.192.695.990.9 90.629
SMAP96.592.888.385.982.393.794.495.297.094.8 92.095
SCMAP96.194.189.485.882.694.394.895.497.194.8 92.451
Estimator #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 Fm Mean
ML0.400.380.320.210.330.200.280.340.250.55 0.3256
MAP0.660.560.480.290.400.320.420.460.590.72 0.4892
MPM0.660.560.480.290.400.320.420.460.580.72 0.4894
SMAP0.700.570.510.270.360.210.460.630.670.82 0.5193
SCMAP0.630.600.540.250.340.260.440.610.660.82 0.5158

Table 2

Percentage of correctly classified pixels for the different single-scale (ML, MAP, MPM) or hierarchical (SMAP, SCMAP) Markovian estimators. Top: in terms of percentage of correct changed and unchanged pixels (PCC). Bottom: in terms of F measure.

NIR Thermic/Optical [#1] Accuracy Optical/SAR [#2] Accuracy
Proposed method 0.965 0.961 Proposed method 0.928 0.941
Mignotte [41] 0.928 Mignotte [41] 0.971
Touati et al. [14] 0.847 Touati et al. [14] 0.943
Touati et al. [55] 0.964 Touati et al. [28] 0.961
Zhang et al. [21] 0.975 Touati et al. [55] 0.955
PCC [21] 0.882 Touati et al. [30] 0.949
Prendes et al. [52] 0.844
Correlation [52] 0.670
Mutual Inf. [52] 0.580
SAR/Optical [#3] Accuracy Optical/Optical [#4] Accuracy
Proposed method 0.883 0.894 Proposed method 0.859 0.858
Mignotte [41] 0.909 Mignotte [41] 0.859
Touati et al. [14] 0.881 Touati et al. [14] 0.877
Touati et al. [28] 0.892 Touati et al. [28] 0.880
Touati et al. [55] 0.909 Touati et al. [55] 0.862
Touati et al. [30] 0.867 Touati et al. [30] 0.853
Prendes et al. [53, 66] 0.918Prendes et al. [52, 53] 0.844
Prendes et al. [51] 0.854 Correlation [52, 53] 0.679
Copulas [49, 51] 0.760Mutual Inf0. [52, 53] 0.759
Correlation [49, 51] 0.688 Pixel Dif0. [53, 65] 0.708
SAR 3/5-looks [#5] Accuracy Optical/SAR [#6] Accuracy
Proposed method 0.823 0.826 Proposed method 0.937 0.943
Mignotte [41] 0.830 Mignotte [41] 0.952
Touati et al. [14] 0.840
Chatelain et al. [50] 0.749
Correlation [50] 0.713
Ratio edge [50] 0.737
SAR/Optical [#7] Accuracy SAR/Optical [#8] Accuracy
Proposed method 0.944 0.948 Proposed method 0.952 0.954
Mignotte [41] 0.940 Mignotte [41] 0.942
Touati et al. [28] 0.767 Touati et al. [14] 0.847
Touati et al. [28] 0.817
Liu et al. [25] 0.976
SAR/SAR [#9] Accuracy Optical/Optical [#10] Accuracy
Proposed method 0.970 0.971 Proposed method 0.948 0.948
Mignotte [41] 0.979 Mignotte [41] 0.962
Liu et al. [26]0.957–0.964

Table 3

Percentage of good classification (accuracy) on the dataset described in table 1 and obtained comparisons by the two hierarchical (SMAP or SCMAP) segmentation methods based on our pixel-wise neighborhood and class conditional likelihoods) versus the state-of-the-art multimodal change detectors existing in the literature (first upper part of each table) and with the mono-modal change detectors (second lower part of each table) [13, 14, 21, 25, 26, 30, 39, 41, 49–53, 55, 65].

Second, a comparison of the SMAP and SCMAP segmentation results with different state of the art approaches [13, 30, 39, 49, 51] is summarized in table 3. From figures 2 and 3 and from this latter table, we can see that the accuracy rate of these two hierarchical Markovian methods outperforms in average the other state-of-the-art non Markovian approaches and allows us to obtain good CD results across a wide variety of existing satellite imagery heterogeneities. We can also notice that a SCMAP or SMAP hierarchical approach combined with our likelihood model allow us to achieve a relatively constant efficiency whatever the type of multi-modality of satellite imagery encountered. It should be noted, however, that this approach tends to underestimate the change class a little. On the contrary, a non-hierarchical Markovian approach (ICM, MPM) combined with our likelihood model tends to a bit overestimate the change class by detecting some false positives (see figure 4).

Conclusion

In this paper, we have shown that a data likelihood model relying on a set of spatial neighborhood adaptive class-conditional likelihoods, whose parameters have been previously estimated in a first estimation step, combined with a hierarchical Markovian segmentation procedure in a second step, turns out to be both a simple, reliable, computationally efficient and good unsupervised statistical strategy for the change detection issue, whatever the type of multi imaging modality encountered in heterogeneous remote sensing imagery.

Acknowledgments

Author would like to thank the NSERC (Natural Sciences and Engineering Research Council of Canada; RGPIN 2016-04578) for having supported this research work.

Data availability

The code, data, and all that is necessary for reproduction of the results will be freely accessible on the author’s website http://www.iro.umontreal.ca/∼mignotte/ (directory: ResearchMaterial/).

Conflict of interest

The Author declares that there is no conflict of interest associated with this work and this publication.

References

  1. 1.
    Longbotham N., Pacifici F., Glenn T., Zare A., Volpi M., Tuia D., Christophe E., Michel J., Inglada J., Chanussot J., Du Q. Multi-modal change detection, application to the detection of flooded areas: Outcome of the 2009–2010 data fusion contest. IEEE J. Sel. Top. Appl. Earth Obs., 2012; 5(1): 331–342.
  2. 2.
    Su L., Gong M., Zhang P., Zhang M., Liu J., Yang H. Deep learning and mapping based ternary change detection for information unbalanced images. Pattern Recognit., 2017; 66(C): 213–228.
  3. 3.
    Rosin P. L., Ioannidis E. Evaluation of global image thresholding for change detection. Pattern Recognit. Lett., 2003; 24(14): 2345–2356.
  4. 4.
    Castellana L., D’Addabbo A., Pasquariello G. A composed supervised/unsupervised approach to improve change detection from remote sensing. Pattern Recognit. Lett., 2007; 28(4): 405–413.
  5. 5.
    Bovolo F., Camps-Valls G., Bruzzone L. A support vector domain method for change detection in multitemporal images. Pattern Recognit. Lett., 2010; 31(10): 1148–1154. Pattern Recognition in Remote Sensing.
  6. 6.
    Champion N., Boldo D., Pierrot-Deseilligny M., Stamon G. 2D building change detection from high resolution satellite imagery: A two-step hierarchical method based on 3D invariant primitives. Pattern Recognit. Lett., 2010; 31(10): 1138–1147. pattern Recognition in Remote Sensing.
  7. 7.
    Celik T. Bayesian change detection based on spatial sampling and gaussian mixture model. Pattern Recognit. Lett., 2011; 32(12): 1635–1642.
  8. 8.
    Hedjam R., Kalacska M., Mignotte M., Nafchi H. Z., Cheriet M. Iterative classifiers combination model for change detection in remote sensing imagery. IEEE Trans. Geosci. Remote Sens., 2016; 54(12): 6997–7008.
  9. 9.
    Lahat D., Adalı, T., Jutten C. Multimodal data fusion: an overview of methods, challenges and prospects. Proc. IEEE, 2015; 103(9): 1449–1477.
  10. 10.
    Alberga V. Similarity measures of remotely sensed multi-sensor images for change detection applications. Remote Sens., 2009; 1(3): 122–143.
  11. 11.
    Brunner D., Lemoine G., Bruzzone L. Earthquake damage assessment of buildings using vhr optical and sar imagery. IEEE Trans. Geosci. Remote Sens., 2010; 48(5): 2403–2420.
  12. 12.
    Liu G., Delon J., Gousseau Y., Tupin F. Unsupervised change detection between multi-sensor high resolution satellite images. In: 24th European Signal Processing Conf., EUSIPCO 2016, Budapest, Hungary. 2016; pp. 2435–2439.
  13. 13.
    Touati R., Mignotte M., Dahmane M. A new change detector in heterogeneous remote sensing imagery. In: 7th IEEE Int. Conf. on Image Processing Theory, Tools and Applications (IPTA 2017). Montreal, Canada, Qc: 2017; pp. 1–6.
  14. 14.
    Touati R., Mignotte M., Dahmane M. A reliable mixed-norm based multiresolution change detector in heterogeneous remote sensing images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 2019; 12(9): 3588–3601.
  15. 15.
    Sun Y., Lei L., Li X., Tan X., Kuang G. Structure consistency-based graph for unsupervised change detection with homogeneous and heterogeneous remote sensing images. IEEE Trans. Geosci. Remote Sens., 2022; 60: 1–21.
  16. 16.
    Lei L., Sun Y., Kuang G. Adaptive local structure consistency-based heterogeneous remote sensing change detection. IEEE Geosci. Remote Sens. Lett., 2022; 19: 1–5.
  17. 17.
    Sun Y., Lei L., Guan D., Kuang G. Iterative robust graph for unsupervised change detection of heterogeneous remote sensing images. IEEE Trans. Image Process., 2021; 30: 6277–6291.
  18. 18.
    Sun Y., Lei L., Guan D., Li M., Kuang G. Sparse-constrained adaptive structure consistency-based unsupervised image regression for heterogeneous remote-sensing change detection. IEEE Trans. Geosci. Remote Sens., 2022; 60: 1–14.
  19. 19.
    Camps-Valls G., Gomez-Chova L., Munoz-Mari J., Rojo-Alvarez J. L., Ramon M. M. Kernel-based framework for multitemporal and multisource remote sensing data classification and change detection. IEEE Trans. Geosci. Remote Sens., 2008; 46(6): 1822–1835.
  20. 20.
    Du P., Liu S., Xia J., Zhao Y. Information fusion techniques for change detection from multi-temporal remote sensing images. Inf. Fusion, 2013; 14(1): 19–27.
  21. 21.
    Zhang P., Gong M., Su L., Liu J., Li Z. Change detection based on deep feature representation and mapping transformation for multi-spatial-resolution remote sensing images. ISPRS J. Photogramm. Remote Sens., 2016; 116: 24–41.
  22. 22.
    Gong M., Zhang P., Su L., Liu J. Coupled dictionary learning for change detection from multisource data. IEEE Trans. Geosci. Remote Sens., 2016; 54(12): 7077–7091.
  23. 23.
    Zhao W., Wang Z., Gong M., Liu J. Discriminative feature learning for unsupervised change detection in heterogeneous images based on a coupled neural network. IEEE Trans. Geoscience. Remote Sens., 2017; 55(12): 7066–7080.
  24. 24.
    Merkle N., Auer P. F. S., Muller R. On the possibility of conditional adversarial networks for multi-sensor image matching. In: Proc. of IGARSS 2017. Fort Worth, Texas, USA: 2017; pp. 1–4.
  25. 25.
    Liu J., Gong M., Qin K., Zhang P. A deep convolutional coupling network for change detection based on heterogeneous optical and radar images. IEEE Trans. Neural Netw. Learn. Syst., 2018; 29(3): 545–559. https://doi.org/10.1109TNNLS.2016.2636227.
  26. 26.
    Liu Z., Li G., Mercier G., He Y., Pan Q. Change detection in heterogenous remote sensing images via homogeneous pixel transformation. IEEE Trans. Image Process., 2018; 27(4): 1822–1834.
  27. 27.
    Touati R., Mignotte M., Dahmane M. Partly uncoupled siamese model for change detection from heterogeneous remote sensing imagery. J. Remote Sensing GIS, 2020; 9(1): 272–284.
  28. 28.
    Touati R., Mignotte M., Dahmane M. Anomaly feature learning for unsupervised change detection in heterogeneous images: A deep sparse residual model. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 2020; 13(1): 588–600.
  29. 29.
    Luppino L., Kampffmeyer M., Bianchi F., Moser G., Serpico S., Jenssen R., Anfinsen S. N. Deep image translation with an affinity-based change prior for unsupervised multimodal change detection. IEEE Trans. Geosci. Remote Sens., 2022; 60: 1–22.
  30. 30.
    Touati R., Mignotte M. An energy-based model encoding non-local pairwise pixel interactions for multi-sensor change detection. IEEE Trans. Geosci. Remote Sens., 2018; 56(2): 1046–1058.
  31. 31.
    Xu M., Cao C., Zhang H., Xue Y., Li Y., Guo J., Chang C., He Q., Gao M., Li X. Change detection of the tangjiashan barrier lake based on multi-source remote sensing data. In: 2009 IEEE Int. Geoscience and Remote Sensing Symposium. vol. 4, 2009; pp. IV–303–IV–306.
  32. 32.
    Volpi M., de Morsier F., Camps-Valls G., Kanevski M., Tuia D. Multi-sensor change detection based on nonlinear canonical correlations. In: 2013 IEEE Int. Geoscience and Remote Sensing Symposium - IGARSS. 2013; pp. 1944–1947.
  33. 33.
    Wu C., Du B., Zhang L. Slow feature analysis for change detection in multispectral imagery. IEEE Trans. Geosci. Remote Sens., 2014; 52(5): 2858–2874.
  34. 34.
    Liu Z. G., Mercier G., Dezert J., Pan Q. Change detection in heterogeneous remote sensing images based on multidimensional evidential reasoning. IEEE Geosci. Remote Sens. Lett., 2014; 11(1): 168–172.
  35. 35.
    Volpi M., Camps-Valls G., Tuia D. Spectral alignment of multi-temporal cross-sensor images with automated kernel canonical correlation analysis. ISPRS J. Photogramm. Remote Sens., 2015; 107: 50–63.
  36. 36.
    Chen X., Li J., Zhang Y., Tao L. Change detection with multi-source defective remote sensing images based on evidential fusion. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci., 2016; 125–132.
  37. 37.
    Tuia D., Marcos D., Camps-Valls G. Multi-temporal and multi-source remote sensing image classification by nonlinear relative normalization. ISPRS J. Photogramm. Remote Sens., 2016; 120: 1–12. https://doi.org/10.1016j.isprsjprs.2016.07.004.
  38. 38.
    Liu Z., Zhang L., Li G., He Y. Change detection in heterogeneous remote sensing images based on the fusion of pixel transformation. In: 20th Int. Conf. on Information Fusion, FUSION 2017 Xi’an, China. 2017; pp. 1–6.
  39. 39.
    Touati R., Mignotte M., Dahmane M. Change detection in heterogeneous remote sensing images based on an imaging modality-invariant mds representation. In: 25th IEEE Int. Conf. on Image Processing (ICIP’18). Athens, Greece: 2018; pp. 3998–4002.
  40. 40.
    Touati R., Mignotte M., Dahmane M. Multimodal change detection using a convolution model-based mapping. In: Eighth Int. Conf. on Image Processing Theory, Tools and Applications, IPTA 2019 Istanbul, Turkey, November 2019. 2019; pp. 1–6.
  41. 41.
    Mignotte M. A fractal projection and Markovian segmentation-based approach for multimodal change detection. IEEE Trans. Geosci. Remote Sens., 2020; 58(11): 8046–8058.
  42. 42.
    Luppino L. T., Anfinsen S. N., Moser G., Jenssen R., Bianchi F. M., Serpico S. B., Mercier G. A clustering approach to heterogeneous change detection. In: Image Analysis - 20th Scandinavian Conf., SCIA 2017, Tromsø, Norway. 2017; pp. 181–192.
  43. 43.
    Luppino L. T., Bianchi F. M., Moser G., Anfinsen S. N. Remote sensing image regression for heterogeneous change detection. CoRR, abs/1807.11766: 2018.
  44. 44.
    Li X., Du Z., Huang Y., Tan Z. A deep translation (gan) based change detection network for optical and sar remote sensing images. ISPRS J. Photogramm. Remote Sens., 2021; 179: 14–34.
  45. 45.
    Yang M., Jiao L., Liu F., Hou B., Yang S., Jian M. Dpfl-nets: Deep pyramid feature learning networks for multiscale change detection. IEEE Trans. Neural Networks Learn. Systems, 2021; 1–15.
  46. 46.
    Zheng X., Chen X., Lu X., Sun B. Unsupervised change detection by cross-resolution difference learning. IEEE Trans. Geosci. Remote Sens., 2022; 60: 1–16.
  47. 47.
    Storvik B., Storvik G., Fjørtoft R. On the combination of multisensor data using meta-gaussian distributions. IEEE Trans. Geosci. Remote Sens., 2009; 47(7-2): 2372–2379.
  48. 48.
    Mercier G., Moser G., Serpico S. Conditional copula for change detection on heterogeneous sar data. In: 2007 IEEE Int. Geoscience and Remote Sensing Symposium. 2007; pp. 2394–2397.
  49. 49.
    Mercier G., Moser G., Serpico S. Conditional copulas for change detection in heterogeneous remote sensing images. IEEE Trans. Geosci. Remote Sens., 2008; 46(5): 1428–1441.
  50. 50.
    Chatelain F., Tourneret J. Y., Inglada J. Change detection in multisensor sar images using bivariate gamma distributions. IEEE Trans. Image Process., 2008; 17(3): 249–258.
  51. 51.
    Prendes J., Chabert M., Pascal F., Giros A., Tourneret J. A new multivariate statistical model for change detection in images acquired by homogeneous and heterogeneous sensors. IEEE Trans. Image Process., 2015; 24(3): 799–812.
  52. 52.
    Prendes J., Chabert M., Pascal F., Giros A., Tourneret J. Performance assessment of a recent change detection method for homogeneous and heterogeneous images. Rev. Fr. Photogramm. Teledetect., 2015; 209: 23–29.
  53. 53.
    Prendes J. New statistical modeling of multi-sensor images with application to change detection, Ph.D. dissertation, Toulouse, 2015.
  54. 54.
    Chirakkal S., Bovolo F., Misra A., Bruzzone L., Bhattacharya A. A general framework for change detection using multimodal remote sensing data. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 2021; 14: 10665–10680.
  55. 55.
    Touati R., Mignotte M., Dahmane M. Multimodal change detection in remote sensing images using an unsupervised pixel pairwise-based markov random field model. IEEE Trans. Image Process., 2020; 29(1): 757–767.
  56. 56.
    Besag J. On the statistical analysis of dirty pictures. J. R. Stat. Soc., 1986; B-48: 259–302.
  57. 57.
    Marroquin J., Mitter S., Poggio T. Probabilistic solution of ill-posed problems in computation vision. J. Am. Stat. Assoc., 1987; 82(397): 76–89.
  58. 58.
    Bouman C., Shapiro M. A multiscale random field model for Bayesian image segmentation. IEEE Trans. Image Process., 1994; 3(2): 162–177.
  59. 59.
    Mignotte M., Collet C., Pérez P., Bouthemy P. Sonar image segmentation using an unsupervised hierarchical MRF model. IEEE Trans. Image Process., 2000; 9(7): 1216–1231.
  60. 60.
    Arya S., Mount D. M., Netanyahu N. S., Silverman R., Wu A. Y. An optimal algorithm for approximate nearest neighbor searching fixed dimensions. J. ACM, 1998; 45(6): 891–923.
  61. 61.
    Pérez P., Blake A., Gangnet M. JetStream: probabilistic contour extraction with particles. In: Proc. IEEE Int Conf. Computer Vision, ICCV’01. Vancouver, Canada: 2001.
  62. 62.
    Mignotte M. A bi-criteria optimization approach based dimensionality reduction model for the color display of hyperspectral images. IEEE Trans. Geosci. Remote Sens., 2012; 50(2): 501–513.
  63. 63.
    Mignotte M. A multiresolution markovian fusion model for the color visualization of hyperspectral images. IEEE Trans. Geosci. Remote Sens., 2010; 48(12): 4236–4247.
  64. 64.
    Shapira D., Avidan S., Hel-Or Y. Multiple histogram matching. In: IEEE Int. Conf. on Image Processing, ICIP’13. 2013; pp. 2269–2273.
  65. 65.
    Team O. D. The orfeo toolbox software guide, 2014, available at http://orfeo-toolbox.org/.
  66. 66.
    Prendes J., Chabert M., Pascal F., Giros A., Tourneret J.-Y. Change detection for optical and radar images using a Bayesian nonparametric model coupled with a Markov random field. In: Proc. IEEE Int. Conf. on Acoustic, Speech, and Signal Processing, ICASSP’15. Brisbane, Australia: 2015.

Written by

Max Mignotte

Article Type: Research Paper

•

Date of acceptance: March 2022

Date of publication: March 2022

•

DOI: 10.5772/acrt.02

Copyright: The Author(s), Licensee IntechOpen, License: CC BY 4.0

Download for free

© 2022 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution 4.0 License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Impact of this article

447

Downloads

403

Views

4

Crossref Citations

4

Dimensions Citations

Share this article

Join us today!

Submit your Article