Semantic Remote Sensing Scenes Interpretation and Change Interpretation

Abstract

A fundamental objective of remote sensing imagery is to spread out the knowledge about our environment and to facilitate the interpretation of different phenomena affecting the Earth’s surface. The main goal of this chapter is to understand and interpret possible changes in order to define subsequently strategies and adequate decision-making for a better soil management and protection. Consequently, the semantic interpretation of remote sensing data, which consists of extracting useful information from image date for attaching semantics to the observed phenomenon, allows easy understanding and interpretation of such occurring changes. However, performing change interpretation task is not only based on the perceptual information derived from data but also based on additional knowledge sources such as a prior and contextual. This knowledge needs to be encoded in an appropriate way for being used as a guide in the interpretation process. On the other hand, interpretation may take place at several levels of complexity from the simple recognition of objects on the analyzed scene to the inference of site conditions and to change interpretation. For each level, information elements such as data, information and knowledge need to be represented and characterized. This chapter highlights the importance of ontologies exploiting for encoding the domain knowledge and for using it as a guide in the semantic scene interpretation task.

1. Introduction

A fundamental objective of remote sensing is to spread out the knowledge about our environment and to facilitate the interpretation of different phenomena affecting the Earth’s surface. Indeed, satellite images make possible to observe much more phenomena related to the increase of information acquired from multiple sensors. For instance, these phenomena include climatic change, urbanization, deforestation, desertification, and so on. Remote sensing and SIG communities have a great interest on change analysis and interpretation. Therefore, tools and strategies have been maintained for studying and analyzing the Earth’s surface dynamics. The principal objective, here, is to understand and interpret changes that may occur allowing, thus, to define strategies and an adapted decision-making for a better soil management and protection. Change detection, in remote sensing, can be defined as the process of identifying differences in the state of an object or a phenomenon by observing it at different times [1]. Applications associated with change detection include monitoring the evolution of cultures and land use, spatial progression of vegetation, forest and urban monitoring, the analysis of the climate change impacts and other cumulative changes. Several change detection approaches have been proposed in remote sensing. The general objectives of most change detection approaches include identifying the geographic locations and type of changes, quantifying the changes and assessing the accuracy of change detection results [2].

The information levels about changes from the remote sensing imagery can be categorized as (1) change detection level that allows detecting simple binary change (i.e. change vs. non-change). This category includes techniques such as image differencing [3], image rationing [4] and change vector analysis (CVA) [5]. These techniques focus on changes localizing but do not provide any information about the change’s nature. (2) The second category (also called thematic level of change) allows the identification of the detailed change “from-to”. It includes techniques such as post-classification comparison [6] and classified objects change detection (COCD) [7]. For more details about change detection techniques from remotely sensed images, the reader can refer to the work of [2]. The authors have given an overview of different change detection approaches where a comparison between pixel-based change detection and object-based change detection has been presented. Pixel-based change detection methods exploit the spectral characteristics of an image pixel to detect and measure changes. Although these methods have been successfully implemented in many areas for changes detection using remote sensing data, an important limitation of these approaches is that they do not exploit the spatial context of real objects [2]. To overcome this limitation, object-based approaches have been developed. Object-based change detection approaches, as defined by [8], allow to identify differences in geographic objects at different moments by using object-based analysis. This later allows to obtain, from an object image, information such as shape, texture and spatial relationships allowing the exploitation of the spatial context [2]. Consequently, the inclusion of this contextual information allows to understand the semantics of objects [9].

Up to now, both change detection approaches (i.e. pixel-based and object-based methods) have been successful either for detecting simple binary change/non-change (i.e. answering “are there changes”?) or for detailing “from-to” change between different classes (i.e. what change?). However, at the two levels of change detection (change detection and identification), these approaches do not give any information about the cause of changes (i.e. why and how change?), and, therefore, give no hints on how to evaluate their signification for the decision-making task. Consequently, an interpretation change level is needed for generating a description of the character and causality of change. The change interpretation level, here, allows to extract information from data (images) about changes that may occur, that is, to answer the question “why and how a change has been produced?”. As any interpretation level, the change interpretation task is not only based on the perceptual information derived from data, but it requires other knowledge sources such as a prior and contextual knowledge to perform the interpretation task. This knowledge needs to be encoded in an appropriate way for being used as a guide into the interpretation process.

Highlighting the role of the remote sensing imagery for change detection and interpretation, an appropriate semantic interpretation method is needed for change interpretation in satellite images. Such methods should take into account the description and the representation of different information elements at each interpretation level. This chapter focuses on semantic scene interpretation for change interpretation in satellite images. Semantic scene interpretation task is composed of different levels of abstraction. The objectives of this chapter are to describe the semantic scene interpretation strategy including the definition and the representation of different information elements composing that process. It is structured in four sections. Section 2 intends to define required fundamental elements for the interpretation task and presents the role of ontologies for the semantic image interpretation. Afterwards, a description of semantic interpretation methods is discussed. Section 3 reviews and classifies approaches for the semantic remote sensing image interpretation. Section 4 presents a proposed method for semantic change interpretation and describes its different components.

2. Semantic interpretation

Remote sensing utility comes not from the data itself but rather from the information that can be derived from this data [10]. For this reason, the interpretation and data transformation into usable information is an important step for the development of user’s applications. Interpretation plays an important role in the process of data analysis. It helps users to easily understand the extracted information from remote sensing data. Consequently, the interpretation of this data enables user to make policy and management decisions. To be understandable, data must be transformed into information, and then, into knowledge as shown in Figure 1 . In this section, we give a meaning to each information element, such as data, information and knowledge, and then we present the interpretation, in a general sense, and the existing interpretation methods.

Figure 1.

From data to decision-making (relation between information elements).

2.1. Definitions and fundamental elements

There are many definitions and significations of informative elements such as data, information and knowledge. According to [11], Data are facts that are the results of observations or measurements we make on objects (artifacts, sites, seeds and bones). In addition, data are defined as primitive symbolic entities, whose meaning depends on its integration within a context allowing its understanding by an interpreter [12]. Information is a set of facts with a processing capability added, such as context, relationships to other facts about the same or related objects, implying an increased usefulness. Information provides a meaning to data. It is an organized data answering the following basic questions: What? Who? When? Where? Knowledge is information with more context and understanding (answering the following basic questions: why? how? for which purpose?), perhaps with the addition of rules to extend definitions and allowing inference. According to Pr. H. Moukdad of the Dalhousie University [13], knowledge is a reservoir of information that is stored in the human mind. It essentially constitutes the information that can be “retrieved” from the human mind without the need to consult external information sources [13]. Knowledge is internalized or understood information that can be used to make decisions. These three entities can be viewed hierarchically in terms of complexity, data being the simplest and knowledge being the most complex of the three. Knowledge is the product of a synthesis in our mind that can be conveyed by information, as one of many forms of its externalization and socialization [13]. Figure 1 shows the relationship between these three different informative elements.

Information comes from different sources, namely data, prior and contextual knowledge. Therefore, information must be combined in order to extract significant elements for the interpretation and, subsequently, for decision-making. The information combination is a role associated with the Fusion Information and Analytic Technology (FIAT) [14]. Indeed, data fusion is the process of combination of data or information from different sources to estimate or predict states of entities. It consists of developing methods that allow the extraction and conciliation, from different knowledge sources, of (significates) expressive elements for decision-making. However, for any interpretation system, its input and its output can take different meanings according to the considered situation. A statement can be either data or information, and it can be knowledge (a prior and contextual). Such a situation is very frequent in the case of semantic interpretation of remote sensing images where different interpretation levels can be considered depending on the user outcome. On the other hand, high-level positioning considers a contextual attribution role to the input system for data, information and knowledge. Hence, information processing within the FIAT framework imposes a need to characterize and represent information in order to be exploited for the design of intelligent situation analysis and decision support systems. Moreover, information, according to [15], is the data that is relevant to the considered application. Losee [16] added that information is the value currently attached or instantiated to a characteristic returned by a process or function: information is a relational or functional concepts linking data sets. Linking input sets (called definition sets) to outputs set (called content sets) makes information to be informative.

To resume, an information element (data, information or knowledge), is “an entity composed of a definition set and a content set linked by a functional relationship called informative relation, associated with internal and external context”. Figure 2 shows the general structure of an information element. Lillesand et al. [17], suggest that: “interpretation may take place at number of levels of complexity, from the simple recognition of objects on scene to the inference of site conditions”. Therefore, it is important to characterize the information element in each interpretation level. Details of the characterization of the information element are presented in the semantic scene interpretation section.

Figure 2.

Information element structure [14].

2.2. Semantic image interpretation

In remote sensing imagery, image interpretation consists of assigning geographic object types to image objects [18]. A geographic object, according to [19], is an object of a certain minimum size on or near the Earth’s surface (e.g. a forest, lake or mountain), whereas an image object is a discrete region of a digital image that is internally coherent and different from its surroundings [20, 18].

Since long time, images interpretation has been based on pixels’ classification methods [17]. Recently, Castilla and Hay [20] have developed a new approach enabling the image analysis and interpretation based on the image partitioning into objects. This approach, called GEOBIA, relies on geographic objects to image objects based on three different steps: the segmentation, extraction and classification [21, 22, 18]. The segmentation step delineates regions having common characteristics. According to [20, 18], this step is based on the hypothesis that partitioning an image into objects is related to the way humans conceptually organize the landscape to comprehend it. The extraction defines the characteristics of the objects, such as shape, texture or the spectral response (i.e. low-level features such as high values in defined spectral bands) [23]. The classification step assigns a category (i.e. a semantic meaning) to the segmented objects according to the attributes calculated in the extraction phase. This last step aims at enriching the objects of the image in order to assign them a significant semantics (i.e. high-level concepts such as vegetation). This process is performed through the analysis of segment attributes and the interrelationships among segments to identify their geographic labels [23]. Such a concept highlights the importance of contextual information in improving the classification [24]. These techniques have shown efficient results based on expert’s knowledge. However, expert’s knowledge is subjective and cannot be used directly by an automatic process [22]. Consequently, this knowledge limits the automation of the image interpretation procedure. According to [18], the issue of automatic image interpretation consists of developing target recognition algorithms to map geographic objects. At this level, the challenge consists of linking the symbolic semantic information (e.g. vegetation index value) with numerical low-level features (e.g. measured vegetation index value). However, matching the high-level knowledge with the low-level knowledge leads to so-called semantic gap problem. This problem is defined as the lack of coincidence of the information that can be extracted from the visual data and the interpretation that the same data have for a user in a specific situation [25].

Hudelot et al. [26] defined the semantic images interpretation as the process of extraction and inference of high-level knowledge from the observed image. According to [26], “The role of the semantic interpretation is to assign a meaning to the perceived description of the scene, i.e. the data extracted from images. This meaning refers to application domain expertise and terminology”. These authors consider that the semantic image interpretation problem is often limited to a classification problem (i.e. identifying the class of the structured data extracted from the observed images using predefined models). Indeed, an interpreter focuses on identifying the semantic contents of the observed images. However, according to the available knowledge, different answers and interpretations are possible for the interpreter (i.e. interpretation may be done in many ways). Consequently, the semantic is not in the image. It depends on the prior knowledge of the application domain (i.e. high-level knowledge) on the one hand, and the application context on the other hand. Therefore, this knowledge needs to be represented and formulated in an efficient way allowing, then, to improve the semantic images interpretation.

With recent advances in knowledge engineering, ontologies are increasingly used for the formalization of the knowledge of a given domain, in a coherent and consensual manner [27]. Indeed, ontologies are admitted as powerful conceptual tools for describing the knowledge of a domain in a structured and shared way and for the management of unstructured data, in particular in the domain of the semantic web. They provide a relevant methodological framework for the representation of prior knowledge in an image interpretation context [26]. Hence, from the fact that the semantic image interpretation can not only be based on the perceptual information coming from an image, the ontology can be used as a conceptual model encoding the expert’s knowledge and guiding the interpretation task. However, the association of the expert’s knowledge (i.e. qualitative information) with its representation in the image (i.e. numerical and quantitative information) leads to the semantic gap problem as previously described. Therefore, the exploitation of the ontology offers the possibility to overcome this problem. Indeed, during the last few years, ontologies have been mostly used to solve the semantic gap problem by bridging the symbolic information and the information extracted from the images [28]. Gruber [29] defined ontology as an “explicit specification of a conceptualization”. This definition is refined by [30] as a “formal specification of a shared conceptualization”. Conceptualization refers to an abstract model of some phenomenon in the world by identifying relevant concepts of that phenomenon. Explicit means that the identified concepts and the constraints on their use are explicitly defined. Formal refers to the fact that the ontology should be machine readable, and Shared reflects the notion that an ontology captures consensual knowledge not private to some individuals but accepted by a group [31].

An ontology specifies a set of concepts, their characteristics, their instances and their relationships, and axioms that are relevant for modeling a domain of study and permits the inference of implicit knowledge. It separates the expert’s domain knowledge expressed by high-level concepts from the low-level features of image objects [18]. Generally, the association of these two levels can be performed using inference engines (i.e. reasoners) in the ontology. A reasoned is considered as a classification algorithm by remote sensing experts and based on logical rules (expressed in description logics), an automatic reasoner can infer new knowledge from explicit knowledge by ontologies and verify its logical consistency.

3. Semantic interpretation of remote sensing images

In a wide sense, ontologies, presented as explicit knowledge models, are widely used in images interpretation domain. First, particular contributions highlighted the use of ontologies in image processing domain have been presented in the multimedia field [26, 32–35]. Indeed, several multimedia ontologies have been presented and developed either for the description of the image low-level content, or, as standard annotation vocabularies for describing the image high-level content. Approaches proposed in [32–34] are examples of ontologies proposed in this context. Other approaches focusing on images processing or annotation problem have exploited ontologies as an annotation vocabulary to facilitate the mapping between perceptual primitives and high-level concepts. These approaches use a visual or object ontology at the intermediate level (i.e. in between low-level features and domain concepts [26]). According to Maillot [35], the visual concept ontology guides the domain knowledge acquisition process by providing a set of generic visual terms close to natural language and closer to images features. It respectively allows the reduction of the domain knowledge acquisition bottleneck and the semantic gap between domain concepts and low-level features. In Mezaris [36], an object ontology, which is a set of qualitative intermediate-level descriptors, has been proposed. It is used to allow the qualitative definition of the high-level concepts (that user query for) and their relations. Similarly, Hedelot et al. [26] have presented a solution to the symbol grounding problem. The symbol grounding problem refers to the mapping between low-level features (i.e. the numerical image data) and the high-level semantic concepts. The proposed work presents a learning approach for linking low-level features and visual concepts by using an intermediate processing ontology and a prior knowledge-based approach to explicitly build links between the low-level features data and the visual concepts. Ontologies have also been used as a framework allowing the explicit and formal description of the domain application and contextual knowledge. This framework is used as a model to guide the analysis and the interpretation, by exploiting the formal reasoning tools related to ontologies [26]. In this context, the description logics have been used in order to enable the logic formalization of the interpretation or annotation problems [37].

Ontologies have been widely exploited in the remote sensing domain, particularly, for the interpretation or the annotation of remote sensing imagery. Hence, several approaches have been proposed for geographical information analysis and management. The proposed approaches are distinguished according to their objective.

3.1. Ontology-based objects classification

As we have introduced, object-based classification consists of assigning a semantic class label (i.e. high-level concept) to a region (i.e. image object) in the image. In this context, Raskin and Pan [38] used ontology as a knowledge base allowing to classify orthogonal classes such as space, time, Earth realms, physical quantities and integrative science knowledge classes such as phenomena, events, and so on. In Ref. [39], authors have presented a semantic model for the classification of landforms, which are extracted from a digital elevation model using OBIA methods. Andrès et al. [22] have proved that expert’s knowledge explanation via ontologies can improve the automation of satellite images and then they have presented an ontological approach allowing to classify remote sensing images. Jess et al. [40] proposed an ontological framework for ocean satellite images classification, which depicts how a potential building of an ontology model for low and high level of features. Recently, Belgiu et al. [41] have presented a method that consists of coupling an ontology-based knowledge representation for objects classification with the OBIA framework. A very recent semantic object-based classification method (using ontology of high-resolution remote sensing imagery) has been presented in [42]. In this approach, authors started by ontology modeling, then, a classification part is performed based on data-driven machine learning, segmentation, feature selection, sample collection and on an initial classification. Finally, image objects are re-classified based on the ontological model.

3.2. Ontology-based objects recognition

Several studies are focused on the object recognition problem in satellite imagery. For instance, Durand et al. [43] have presented an ontology for the recognition of urban objects in satellite images. This ontology has been enriched later in [44] with other domain concepts and spatial relations and then has been used for the annotation and interpretation of the remote sensing image. In [45], Forestier et al. have developed an ontology for the identification of urban features in satellite images. The proposed method starts by associating a set of low-level characteristics to each image region by using a segmentation algorithm. Then, the knowledge base (i.e. the ontology) is used to assign a semantic to the considered region. This work has been extended and generalized in [46] by adding new knowledge functions (KFs) including spatial relations between objects. Therefore, the proposed approach has been applied for coastal objects recognition. Recently, Luo et al. [47] have presented an ontology-based framework that was used to model the land cover extraction knowledge and interpret high resolution satellite (HRS) images at the regional level. In this work, the land cover ontology structure is explicitly defined, representing the spectral, textural and shape features, and allowing for the automatic interpretation of the extracted results. Similarly, Gui et al. [48] have presented an ontological method for extracting individual buildings with different orientations and different structures from SAR images based on ontological semantic analysis.

3.3. Ontology-based change detection

Modeling different states of objects, or phenomenon, in time allows to detect and identify different changes that can undergo these objects and phenomena. However, few ontology-based approaches have been proposed for change detection in remote sensing domain. For instance, Hashimoto et al. [49] have presented a framework based on ontologies and heuristics for automatic change interpretation. The proposed framework considers remote sensing data analysis as a knowledge information processing, which derives new information about targets with inference from the observed data and a priori knowledge for remote sensing images. In [50], an ontology has been exploited to support spatio-temporal modeling in order to study different land use/cover changes such as splitting, merging, separation and annexation. In this work, the GeoSPARQL ontology including spatial information, the fluent ontology for temporal information and the domain ontology that stored knowledge and contextual information related to the geospatial environment, has been combined allowing to represent and to reason about spatio-temporal dynamics. More recently, Li et al. [51] have implemented an integrated computational framework to support semantic modeling and reasoning about spatio-temporal change of geographical objects in land use/cover (LULCC) data, regarding space, time and topology. In this framework, a spatial ontology has been created to encode essential knowledge about spatio-temporal variation changes such as deforestation and urbanization. Then, based on the knowledge defined in this ontology and on related reasoning rules, the semantic platform allows the semantic query and change reasoning of areas with LULCC.

4. Proposed approach: multi-levels semantic images interpretation

4.1. Semantic scenes interpretation strategy

Semantic image interpretation is defined by the semantics extraction and inference processes of high knowledge from an observed image. Semantics extraction refers to the image interpretation from a human perspective. It consists of obtaining useful spatial and semantic information on the “basic informative granules” (i.e. pixels, objects, zones, global scene) using human knowledge and experience. Generally, existing approaches, for semantics image interpretation, follow a multi-level strategy for describing the image content. According to Marr’s vision [52], this architecture allows to separate the perceptual levels (i.e. syntactic description of the visual content of the image according to descriptors and visual primitives) and the conceptual or semantic levels (i.e. the meaning of the elements present in the image). Hudelot et al. [26] have adapted this architecture for semantic image interpretation in the medical domain. These authors suggested that the semantic level can be divided into three semantic abstraction sub-levels: semantic object level, semantic spatial level and semantic global level. Consequently, we have used this multi-level architecture for semantic scenes interpretation in remote sensing domain and subsequently for changes interpretation.

As shown in Figure 3 , the proposed architecture is composed of different levels of abstraction. Consequently, following the idea that the “interpretation may take place at several levels of complexity, from the simple recognition of objects on scene to the inference of site conditions”, an interpretation task will be accorded to each level. For each level, the interpretation strategy depends on: the input data (i.e. definition set) (e.g. scene), the output goal (i.e. definition content) (e.g. semantic objects classification) and a prior and contextual knowledge (e.g. spatial and temporal relations, contextual criteria, constraints, etc.). However, as we have introduced, there are various meanings associated with the word “information”. Input can take different meanings according to the considered situation. It can be data, information or knowledge. This suggests that an input, in itself, has no unique meaning. Hence, data, information and knowledge are most likely context-dependent representations. Consequently, different situations may be perceived if they are interpreted with different contexts of the observed data. Because there are different meanings of information (i.e. data, information, knowledge), in this chapter, the interpretation process is considered as an information fusion problem. Information fusion focuses on combining different elements for the success of the interpretation step. However, the success of information fusion task is related to the way its basic components are defined and to the quality of their associated knowledge as well as the information or knowledge produced by the fusion process itself [14]. In this section, we focus on defining and characterizing, for each interpretation level, the information element structure and its main components. The main components of information element are as follows [14]:

1. A definition set giving the potential information input element (i.e. what the information refers to);

2. A content set encoding the possible knowledge produced by the information (e.g. measurements or estimations of physical parameters, decisions, hypothesis);

3. An input-output relationship representing the functional link model (e.g. mathematical, physical) that associates the input elements with the produced information contents;

4. An internal context gathering intrinsic characteristics, constraints, or controls about the information relation itself;

5. An external context containing data, information, or knowledge useful to the elaboration of the meaning or the interpretation of the information element.

Figure 3.

General adapted architecture for semantic scenes interpretation.

Formally, an information element J can be represented as follows:

$$\mathsf{J}=\left(\text{information definition}\mathrm{set},\text{informative relation},\text{information content}\mathrm{set},\text{internal context},\text{external context}\right)$$ E1

In the basic information element structure, illustrated in Figure 2 , objects and contents represent entities linked through the informative relation. According to Bosse et al. [14], the nature of these entities may be either hard or soft. Hard means that these entities are quantitatively defined with numbers, individuals, and so on, an example of hard object is raws of pixels and an example of hard contents is features. Soft signifies that entities are qualitatively defined with words, opinions, predictions, and so on. For instance, a rule, defining an image vegetation segment, is a soft object, and the vegetation segment represents the soft content. However, soft entities require a context in which the qualitative descriptors are defined [14]. An informative relation may be impersonal (or hard) when it does not depend on external conditions to link objects to contents. However, it may also behave in a softer way by using cognitive factors such subjective judgements, opinions and perceptions (e.g. (J) representing human experts’ outputs). For example, if we consider the informative relation a sensor making an acquisition, setting parameters value of this sensor belongs to the internal context. However, the conditions why the sensor has been set up with these setting belong to the external context. These conditions include the context of observation. This later represents an important part of the perception process as it is all that an influence on the perception of an event and all that is needed to understand the observation. Therefore, different situations may be perceived, relying on the same set of sensory information items, if they are interpreted within the different contexts of observation. Thus, providing the external context for a specific domain and specific aim makes a system to be intelligent for interpreting a specific situation. Part of this context is the domain knowledge that every human uses to interpret and understand any perception. The exploitation of the ontology offers the best way for representing and reasoning about this knowledge. In the following section, we describe the information element of each semantic image interpretation level and we demonstrate the role of ontologies for representing and reasoning about both internal and external contexts.

4.2. Information element: pixel level

The semantic image interpretation strategy starts with a feature extraction step from the image where raw image data are “converted” into visual features (edges, segments, regions, intersections, etc.), which are supposed to correspond to meaningful parts of semantic objects. What is considered here is not information but a kind of abstract data (i.e. a set of pixels). This level of abstraction corresponds to the information element paradigm when associated with the basic objects that are observed. An informative relation, that is, feature extraction, is used to link objects (input set) to contents (output set). This informative relation embeds knowledge allowing to build this link. However, the informative relation extracting features from the image (i.e. the definition set) probably needs to know the resolution of the sensor producing the image pixels [14] and other knowledge such as segmentation and extraction algorithms and feature properties. This additional information belongs to the internal context of the information element. In addition, contextual information such as sensing conditions including the acquisition date, the sun elevation angle, atmospheric conditions and algorithms characterization (that have to be known “previously” in the feature extraction step). This information belongs to the external context of the information element. Figure 4 shows the information element structure of the pixel level (data).

Figure 4.

Information element in pixel level interpretation.

4.3. Information element: visual primitives level

The visual primitives’ level aims to assign a semantic attribution (i.e. labeling) to the extracted image segments from the shapefile containing the information about image features. An informative relation that links the input set (i.e. shapefile) to the output set (i.e. labeled segments) and uses internal information including features proprieties and segments rules for reasoning about segments labels. The attribution of these labels is based on an external resource (i.e. external context) formulated as a visual primitive ontology. This ontology includes a general description of expert’s knowledge about geographical features representation. The concepts, associated to this structure, are derived from images concepts. Figure 5 illustrates the structure of the information element in the visual primitives’ level. At this level, it is worthwhile to notice that the information definition set (i.e. input definition set) is represented by a shapefile format and the information content set (i.e. output set) by a RDF file format.

Figure 5.

Information element in the primitive-level interpretation.

4.4. Information element: object level

The semantic object level allows to attribute a hard classification to objects in scene. Indeed, the semantic objects interpretation consists of attributing hard classes such as forest, lake, urban and others to the labeled visual primitive extracted in the former step. This later, formulated as knowledge, has been extracted in the visual primitive level, and it is used as an input definition set of the information element in the semantic object level. What is considered, at this level, is not information or data, but a set of knowledge allowing to describe the semantics of objects in the image. The link between the symbolic description (i.e. input definition set) and the semantic content (i.e. output set) is performed through the classification reasoner representing the informative relation of the information element. This informative relation needs other knowledge in order to associate the semantic definition to different image contents. Such knowledge includes prior and contextual knowledge, which are formulated as a domain ontology. Generally, the main concepts of this ontology are geographic objects such as urban, forest, water, and so on, and their hierarchical relationships (i.e. “is-a” relation). The general information element structure of the semantic object level is illustrated in Figure 6 .

Figure 6.

Information element in the semantic object-level interpretation.

4.5. Information element: semantic spatial level

In the semantic spatial interpretation level, the focus is made in order to give a visual description of the whole content of the image scene in a given time. In other words, the objective is to describe different objects, present in the scene, and the existing spatial relations that hold between them. Consequently, this allows to give a conceptual representation of semantic objects and their spatial relation allowing a semantic interpretation at the scene level. A spatial relation extraction reasoner (representing the informative relation of the information element defining the knowledge here) allows to make a link between the semantic objects hierarchy (i.e. input definition set) and the conceptual representation (i.e. output content set). To build this link, the informative relation needs to use some predefined constraints about spatial relationships. These constraints (or spatial rules) are part of the internal context allowing to define spatial relations that existing between different objects in the scene. Spatial relationships include neighborhood relations (such as externally connected (EC), disconnected (DC), and non-tangential proper part (NTPP)); directional relations describing relative orientations of objects (e.g. North and South); and distance relations (such as near and far relationships [53]).

All these spatial relations are formulated in a spatial relation ontology as presented in [26], and then, they are integrated as parts of an external context in the structure of information element. On the other hand, the informative relation requires the integration of the domain knowledge (i.e. domain ontology) with the spatial relation ontology for the global interpretation and understanding of the scene. Notice here, that this domain ontology is used as an external context of the information element in the semantic object level. Therefore, the integration of the spatial relation ontology with the domain ontology (as an external context) in this information element of the semantic spatial level illustrates the growing extent of J context as J level of abstraction increases ( Figure 7 ).

Figure 7.

Information element in spatial level interpretation.

4.6. Information element: global semantic level

The global semantic level (or high semantic level) refers to the semantic scenes interpretation in the time. This allows to describe the semantic content in terms of objects and their relations of different images representing the same scene in the time. To reach this purpose, the global semantic scenes interpretation consists of integrating temporal relations that can hold between images’ objects. The obtained result, output content, is an ontological conceptualization representing different concepts in the images as well as their relationships, namely, semantic, spatial, temporal and filiation relations. Figure 8 shows the structure of the information element in this level. In this structure, the informative relation considers as an input set the semantic spatial representation of different scenes (e.g. two scenes here). In order to allow linking these representations and the output result (i.e. content set), the informative relation, that is, the temporal and filiation relations reasoner, uses temporal and filiations rules as the internal context integrates temporal and filiation relations ontologies. Temporal relations define possible relations that hold between two time intervals that can be used for reasoning about the temporal descriptions of events, actions, beliefs, intentions or causality. Generally, Allen’s Interval Algebra [54] is the most known and widely used model for topological temporal relations between objects in time. The importation of the SWLRTO ontology, as part of external context, offers the possibility to classify different time relations. This allows, for example, to define the rule for the temporal relation before as follows:

$$\mathit{\text{SWRLTO}}\mathit{:}\mathit{\text{hasTime}}\left(?\mathit{\text{i1}},?\mathit{\text{t1}}\right)\text{^}\mathit{\text{SWRLTO}}\mathit{:}\mathit{\text{hasTime}}\left(?\mathit{\text{i2}},?\mathit{\text{t1}}\right)\text{^}\mathit{\text{SWRL}}\mathit{:}\mathit{\text{lessThan}}\left(?\mathit{\text{t1}},?\mathit{\text{t2}}\right)\mathit{\to }\mathit{\text{before}}\left(?\mathit{\text{i1}},?\mathit{\text{i2}}\right)$$ E2

Figure 8.

Information element in the global-level interpretation.

In addition, filiation relations are also of great importance for reasoning about relations between objects in times. Filiation relations have been introduced by Del Mondo [55] and include continuation and derivation relationships. Continuation occurs when an entity (real object) continues to exist from one time to the next with the same identity, and the derivation occurs when an entity creates some others with new identities [55]. Thus, these relations must be integrated to the context allowing, thus, the informative relation to link the input set to the output set.

4.7. Information element: change interpretation

The final objective of the semantic scenes interpretation is the interpretation of changes that may occur. The change interpretation process consists of the detection of the changes that can affect different states of objects and the relations between these changes. Changes can be classified into: (1) domain-independent occurrences (such as growth, shrinkage, disappearance, appearance, etc.) or (2) domain-dependent (or domain-specific) occurrences (such as deforestation, urbanization and desertification). Most researches, analyzing and studying changes, consider the first category as events and the second as processes (i.e. geographic processes). However, relations can exist between the two occurrences. These relations can be either composition or constitution relationships, for example, a deforestation process is specified as the shrinkage and then the disappearance of forest zone, and either can be causal relationships such as cause and initiate relations, for examples, deforestation initiates the shrinkage of a forest, urbanization causes the deforestation, and so on. Consequently, the information element in this level must describe the knowledge about these changes and their relations. Therefore, the role of the informative relation (i.e. change reasoner) is to interpret changes that have occurred from the semantic description of different scenes (input set) allowing subsequently to obtain a semantic representation about changes and their relationships (i.e. the content set). To link the information definition set to the information content set, this informative relation is based on two contextual information: internal and external context. The internal context includes contextual information about objects or different relationships, which are used for the interpretation process. For instance, the spatial reduction of an entity whose coverage is a forest (internal context) is interpreted as phenomena of deforestation. External context includes the representation and descriptions about events and processes. It is about expert (prior) knowledge describing definitions of different occurrences. Descriptions of events and processes can be formulated using upper ontologies such as the basic fundamental ontology (BFO) [56]. This ontology distinguishes between static entities such as forest and dynamic entities such as deforestation. Therefore, combining the BFO ontology with the domain ontology (which represent knowledge of remote sensing images) allows to give a conceptual representation describing events and processes that can be used as an external context to help the interpretation of changes. In Figure 9 , the information element structure and its components in the change interpretation level are presented.

Figure 9.

Information element in the change-level interpretation.

5. Conclusions and discussion

Semantic images interpretation is an important step for any decision-making system. It allows to give a semantic description of the image content. Consequently, this allows an agent (user or machine) to take the best management decision of a given situation. The interpretation may take place at different levels of complexity, from the simple recognition of objects on scene to the inference of site conditions and also to change interpretation. In this chapter, we have mainly focused on the semantic scenes interpretation for change interpretation in remote sensing imagery. Therefore, we have demonstrated that the semantic scenes interpretation can be done in different level, from the low level to the high level. For each level, it is important to characterize the structure of the information element (i.e. data, information or knowledge) and its components (i.e. input set, output set, internal and external context) required for the interpretation process. Consequently, a semantic conceptualization based on ontological concepts for representing the components of the information elements and for the interpretation step has been illustrated in this chapter. Especially, the ontology exploitation has been applied to formulate the expert’s knowledge such as a prior and contextual knowledge. These types of knowledge are important for the semantic scenes interpretation task.

Generally, the structure of the information element is composed of the definition set, content set, informative relation and both internal and external context. In addition, as we have shown in Figure 2 , quality of information (QoI) also must be integrated into the information element structure. Therefore, the characterization of the quality of information is necessary for the interpretation process. QoI about information is through its attributes and their relations. Generally, there are four main aspects of information quality: uncertainty, relevance, reliability and completeness. Future studies include the description of these aspects and their integration in the semantic images interpretation process [18].