SAR image parameters.
The increased availability of civil synthetic aperture radar (SAR) satellite images with different resolution allows us to compare the imaging capabilities of these instruments, to assess the quality of the available data and to investigate different areas (e.g., the Wadden Sea region). In our investigation, we propose to explore the content of TerraSAR-X and Sentinel-1A satellite images via a data mining approach in which the main steps are patch tiling, feature extraction, classification, semantic annotation and visual-statistical analytics. Once all the extracted categories are mapped and quantified, then the next step is to interpret them from an environmental point of view. The objective of our study is the application of semi-automated SAR image interpretation. Its novelty is the automated multiclass categorisation of coastal areas. We found out that the north-west of the Netherlands can be interpreted routinely as land surfaces by our satellite image analyses, while for the Wadden Sea, we can discriminate the different water levels and their impact on the visibility of the tidal flats. This necessitates a selection of time series data spanning a full tidal cycle.
- coastal areas
- SAR imaging
- semantic labelling
- Wadden Sea
The Wadden Sea (Dutch: Waddenzee, German: Wattenmeer, Danish: Vadehavet) is a coastal area with extensive tidal flats in the south-eastern part of the North Sea (see Figure 1). It lies between the coast of N-W continental Europe and the range of Frisian Islands, forming a shallow body of water with tidal flats and wetlands , protected by a 450 km long chain of barrier islands, the Wadden Islands . The Wadden Sea region measures about 22,000 km2 divided between land and sea. About 63% of the region lies in Germany, with about 30% in the Netherlands and 7% in Denmark .
Environment: Most of the landforms in the Wadden Sea region have essentially been created from a marine  or tidal [4, 5] environment. The main exception being the Dutch Wadden island of Texel, the centre of which is a glacial relict. The islands are characterised by dunes and wide, sandy beaches to the North Sea, and a low, tidal coast to the Wadden Sea. It also consists of large intertidal areas, of which two thirds are emergent at low tide. Most of these flats are bare, mainly consisting of sand, but locally along the mainland, and in estuaries, mudflats and salt marshes occur .
Fauna: The Wadden Sea is famous for its rich fauna, such as birds: waders, ducks and geese; that use the Wadden Sea area as a migration stopover or wintering site and also gulls and terns . Typical of the Wadden Sea are large tidal flats, which are characterised by very high benthic biomass and productivity, dominated by molluscs and polychaetes . The Wadden Sea houses a variety of habitats that support diverse fish communities, providing an important nursery and feeding area for many fish of commercial importance [9, 10]. Also, a large population of harbour seals (Phoca vitulina) lives in the Wadden Sea .
Conservation: In 2009, the Dutch-German Wadden Sea was inscribed on the UNESCO World Heritage List, and the Danish part was added later in 2014.
Our interest in this contribution is on the use of remote sensing in the Dutch part of the Wadden Sea, which is a protected area under European legislation . This area is part of the Natura 2000 network  (see Figure 2) and is one of the protected areas being analysed in . Natura 2000 covers 18% of the EU’s land area and 6% of its marine territory. The aim of the network is to ensure the long-term survival of Europe’s most valuable and threatened species and habitats .
In the coastal area research literature, there are several studies treating the Wadden Sea area along the years (for an interdisciplinary overview, see [2, 5]). The Wadden Sea area faces a strong economic impact due to recreation, fisheries and maritime traffic. The last impact is due to, for example, the ports of Bremerhaven, Hamburg and Rotterdam, whereby the traffic runs through or nearby this area, which makes that monitoring of sand banks and any decrease of the water depth in this area is a critical topic for maritime security.
In order to understand the Wadden Sea dynamics, we exploited a number of recent publications already using remote sensing images and addressing the issue of SAR satellite image classification and interpretation [16, 17] in these coastal areas.
For a more detailed description of coastal areas, the work of  is focused on morphological changes (e.g., river estuaries) using X-band radar data. Here, TerraSAR-X Spotlight and StripMap images  are used to monitor river channel and coastline/waterline changes. However, this data set only covers the Island of Helgoland (North Sea, Germany) and the river Elbe estuary.
An extension of  is presented in , where the authors perform automated land-waterline detection with a high level of accuracy using TerraSAR-X StripMap and ScanSAR images. This latter data set covers two areas, the south of Trischen Island and Pellworm Island. The proposed detection technique provides a fast and useful way to identify changes in the coastal topography.
Further,  describes a waterline extraction algorithm, which allows a near real-time determination of changes of coastal outlines of the North Sea using TerraSAR-X and Sentinel-1A images.
In , the authors use a waterline detection method to produce the topography of tidal flats by using 70 SAR images taken between 1996 and 1999 at different water levels. The investigated area is along the North Sea coast of Germany between Schleswig-Holstein and Lower Saxony.
In addition,  investigates the characteristics of five tests areas (Amrum, Pellworm, Wesselburen, Jadebusen and Norderney) using TerraSAR-X and C-band RADARSAT-2  data acquired in 2012 and 2013. The authors demonstrate that the analysis of TerraSAR-X images provides valuable information about the intertidal flats when monitoring the Wadden Sea area. For the detection of small changes (in waterline and morphology), a combination of X-band data with C-band data helps to obtain reliable conclusions. Promising results are also obtained for dual polarisation SAR images, when the images are acquired at incidence angles exceeding 40°.
The organisation of this contribution is as follows. Section 2 presents the characteristics of our data set. Section 3 outlines our image analysis methodology used to generate visual-statistical analytics. Section 4 describes the image classification and annotation results using the methodology presented in Section 3. Finally, Section 5 gives an environmental interpretation and quality assessment of the results from Section 4, while Section 6 contains a conclusion and a comparison of results with Sentinel-2 data.
2. Test and validation dataset
The different resolutions of high-resolution data from TerraSAR-X  and medium-resolution data from Sentinel-1A  allow us to compare the imaging capabilities of these instruments, to assess the quality of the available data and to monitor periodically different areas. Therefore, we analysed both TerraSAR-X and Sentinel-1A amplitude data from the Wadden Sea and its surrounding neighbour areas (outside of the zone marked in purple in Figure 2).
TerraSAR-X is a German radar satellite launched in June 2007. It operates in X-band and is a side-looking SAR instrument that offers a wide selection of operating modes and product-generation options . TerraSAR-X has a repeat period of 11 days. We selected high-resolution mode images (High resolution Spotlight (HS), or StripMap (SM)) with vertical (VV) polarisation, because they provide the best resolution data of the target areas. As for the product-generation options, we took from the available modes the multi-look ground range detected (MGD) level 1b mode, because this is not affected by geometrical interpolation effects over hilly terrain and is well suited for feature extraction. This was also the reason for choosing the radiometrically enhanced (RE) product option that is optimised with respect to radiometry (i.e., reduced speckle).
On the other hand, Sentinel-1 is a C-band radar satellite mission funded by the European Union and operated by the European Space Agency (ESA). It comprises a constellation of two satellites, Sentinel-1A launched in April 2014 and Sentinel-1B launched 2 years later in April 2016. These Sentinels have a repeat period of 6 days for the twin constellation. We selected Sentinel-1A images because they are readily available for our investigated area. Level-1 products are recommended for land cover analysis. For a fair comparison with TerraSAR-X, we chose the interferometric wide (IW) swath mode and its GRD (ground range detected) option containing medium-resolution data. We limited ourselves to single polarisation (VV) data.
When comparing the classification potential of both missions, it turns out that the high-resolution TerraSAR-X data not only provide more detailed object morphologies but also allow more classes to be discriminated with good classification quality. We suppose that this is due to the good signal-to-noise characteristics of the TerraSAR-X radiometrically enhanced products.
Figure 3 shows the geographical locations of all SAR images from our data set. The footprints of three TerraSAR-X image products are marked with red rectangles. Together with these TerraSAR-X images, another two Sentinel-1A image products are analysed in our evaluation. All images were acquired within a short time interval and are projected on a map (see Figure 3) using QGIS .
The scenes were selected based on their availability, their content and their typical diversity of categories. The data set mainly covers the protected area of the Wadden Sea (its Dutch part) and its surrounding areas. The corresponding image parameters are listed in Table 1. All images were tiled into a series of patches with a size of 200 × 200 pixels (in the case of TerraSAR-X) and 128 × 128 pixels (in the case of Sentinel-1A). In total, we obtained 10,834 patches for TerraSAR-X and 34,900 patches for Sentinel-1A and classified them into several categories by using an existing semantic annotation catalogue . Finally, we overlaid the Natura 2000 data  of the Dutch part of the Wadden Sea.
|Image location||Flevoland (Image-1)||Port of Rotterdam (Image-3)||Wadden Sea, Lake IJssel, Marker Lake and the surrounding areas (Image-4)|
|Lake IJssel, Marker Lake, Flevoland,|
and North Holland (Image-5)
|North Holland (Image-2)||Wadden Sea (Image-6)|
|Acquisition time||13-05-2015 (Image-1)||Sept. 21, 2009 (Image-3)||15-05-2015 (Image-4, 5, 6)|
|Instrument mode||StripMap (SM)||High-resolution spotlight (HS)||Interferometric wide swath (IW)|
|Polarisation||VV||VV||VV and VH (Image-4, 5, 6 only VV)|
|Orbit branch||Ascending (Image-1)||Descending||Ascending|
|Product type||Multi-look ground range detected (MGD)||Multi-look ground range detected (MGD)||Ground range detected high resolution (GRDH)|
|Enhancement||Radiometrically enhanced (RE)||Radiometrically enhanced (RE)||—|
|Ground range resolution||6.28 m||2.9 m||20 m|
|Pixel spacing||2.75 m||1.25 m||10 m|
|Equivalent number of looks (range × azimuth)||2.7 × 2.2||3.7 × 2.6||20 × 22|
|Full image size|
(cols × rows)
|20,560 × 11,697 (Image-1)||2119 × 2651 (Image-3)||25,601 × 16,698 (Image-4)|
|10,001 × 8941 (Image-5)|
|14,087 × 11,674 (Image-2)||18,944 × 5013 (Image-6)|
The protected areas are shown in Figure 3 with different colours corresponding to the different tranches in which they became assigned as Natura 2000: tranche 1 (green) per January 2007, tranche 2 (purple) per May 2007, tranche 3 (dark blue) per September 2008, tranche 4 (reddish) per September 2009, and later tranches (light blue) per 2011 to 2016.
3. Image analysis methodology
In the following, we demonstrate that very good results in SAR image classification and semantic annotation can be obtained, if we follow a systematic feature extraction and classification approach described in . Our image classification system is basically composed of the following main modules: data model generation (DMG), database management system (DBMS), knowledge discovery in databases (KDD), and statistical analytics.
The DMG module transforms the original format of original Earth observation products into smaller and more compact product representations that include image descriptors, metadata, image patches, etc.
The DBMS module is used for storing all the generated information and allows querying and retrieval of the available image data.
The KDD module is in charge of finding patterns of interest from the processed data and presenting them to the user. Moreover, the KDD module allows annotating the image content by using machine learning algorithms and human interaction resulting in physical categories.
The statistical analytics module provides classification maps of each dataset, distribution results of the retrieved categories in an image, and the classification accuracy of selected descriptors (primitive features) by computing different metrics.
These four modules are operated automatically and interactively with and without user interaction.
The DMG module represents an automated processing chain that produces data items from an Earth observation product. The main functionality of DMG is metadata extraction, tiling the images with multiple resolutions (resulting in different patch sizes, as described in Section 2), generation of high-resolution quick-look images and primitive feature extraction. The DMG process starts by reading the product data files and extracting the relevant metadata entries. In the next step, the full product image is cut into square-shaped patches, and for each patch, a high-resolution quick-look image is generated. Later, different feature extraction methods can be applied to the generated patches in order to obtain descriptors that characterise the image content. These feature extraction methods are able to describe texture, colour, spectral features, etc. In this study, we use Gabor filters  for feature extraction with five scales and six orientations. The final output of DMG consists of image content descriptors in the form of primitive feature vectors, metadata entries, image patch descriptions and high-resolution quick-look image patches; all this generated information is stored into our database (DBMS).
In the next step, since all the generated information (i.e., primitive features, patches, metadata entries, etc.) is available in the DBMS, the interactive KDD module is enabled and allows us to search for specific images in the DBMS repository. This module implements data mining, machine learning and relevance feedback methods. In particular, it uses an interactive support vector machine (SVM) for active learning support with a χ2 kernel that makes highly accurate classifications even with a small number of examples for each given image annotation category . For more detailed information about the attainable classification quality, see Section 4.
The KDD process is triggered interactively by a user; it accesses the database and selects the image patches and their primitive feature vectors to be used for subsequent classification. In the next step, the quick-look images of the patches are presented to the user via a graphical user interface (GUI). After selection of a target category (from the list of labels available from the semantic catalogue ), the user selects patches with interesting content as positive examples and patches with undesired content as negative examples. This set of positive and negative examples is leveraged as our training data set and passed to the SVM. Then, the SVM is used for training, creates a model for classifying the whole dataset and generates predictions for the remaining patches. The classification results are ordered according to the prediction rankings. In the next step, the ranked results are updated on the screen, and the classified patches are marked in colour, so that the user can perform a new training (with more selected examples) or stop the training (see Figure 4). Finally, when the user is satisfied with the classification results, he/she stops the training and selects a label from the semantic catalogue to annotate this classification, and then, the software stores the classified patches and their semantic label into the database.
Finally, by interactively applying different statistical analytics to the database, we are able to generate analytics about the image content and present these results in different charts, to generate classification maps or change maps (in the case of image time series) and to compute classification accuracy metrics (e.g., precision/recall).
4. SAR image classification and annotation
Once new images are available for further analysis, they are processed as described in Section 3. The annotation methodology is semi-automated, that is, the following functions of the methodology are fully automated: patch tiling, metadata extraction, quick-look generation, feature extraction and insertion of the results into the database. In contrast, the final classification and annotation functions require manual operator interaction. They require interaction because our classification includes an operator to rank the patches via human–machine interaction (i.e., active learning), and the semantic annotation calls for the selection of proper labels for each category.
For high-resolution SAR images of our target areas (e.g., TerraSAR-X data), our annotation scheme has 3 levels with, in total, 150 categories (9 general high-level categories, 73 more specific mid-level categories and 68 categories belonging to a very detailed low level). The very detailed categories describe details of manmade infrastructure, while the categories describing natural environments do not have these refinements.
For medium-resolution and low-resolution SAR images (e.g., Sentinel-1), this annotation scheme can be adapted by concentrating on all categories from high level and some from mid-level.
To each tiled patch, we attach a semantic label. Typically, the total number of categories that can be identified in a high-resolution SAR image varies from 10 to 20 categories depending on the content of the image (e.g., as a function of geographical location and local architecture). In the case of medium-resolution SAR images, the total number of categories varies from 5 to 14 .
Figures 5 and 6 show one representative patch per category of one TerraSAR-X and one Sentinel-1A image, while Figure 7 shows the distribution of categories for TerraSAR-X images, and Figures 8–10 depict the corresponding classification maps. In Figures 11–13, we demonstrate, as in the case of TerraSAR-X, the distribution of categories and the classification maps of the Sentinel-1A data set. Here, one can see the effects of the geometrically coarse classification grid resulting from the unique label patch classification (e.g., Figure 10). A typical classification example of natural scenes is the Wadden Sea image shown in Figure 14.
An interesting phenomenon contained in the figures is the higher number of categories found for TerraSAR-X images (the corresponding Sentinel-1A sub-scenes cover the same area on ground). This phenomenon can be explained by a greater discrepancy between the (visually hidden) feature vectors and by the architectural and geographical locations of these images .
Another phenomenon to be mentioned are the detailed semantic categories that are obtained in the case of TerraSAR-X (e.g., Medium-density residential areas, Mixed forest, etc.) in contrast to the categories in the case of Sentinel-1A (e.g., Inhabited built-up areas, Natural vegetation, etc.). This is due to the higher ground resolution of TerraSAR-X (see Table 1). In total, for TerraSAR-X, there are 23 semantic categories belonging to level 2 and 3 , and, for Sentinel-1A, 16 semantic categories belonging to level 1 and 2 .
Please note that we refer to coastal areas as the areas on both sides of a coastline, while we understand dams as (drivable) barriers separating two water basins and dikes as barriers for holding back flood waters.
Presently, we have reached a stable state of classification accuracy (a precision/recall) rate of 95%/90% for TerraSAR-X and 90%/85% for Sentinel-1A.
In addition, for Sentinel-1A, we plan to include the analysis and combination of its VV and VH polarisation data in order to increase the attainable classification accuracy.
5. Environmental interpretation
The landscapes of the western and northern part of the Netherlands are the result of an ongoing balance between natural processes on the one hand and increasing anthropogenic impact on the other hand at the edge of sea and land (overview in [2, 4, 5]). This resulted in a fine-meshed mosaic of natural habitats and man-made structures. In an alternating series of transgressions and regressions of the sea, especially from 3000 to 1000 years BC, along the Dutch coast, a series of barrier beaches and dunes, interrupted by rivers, were formed [34, 35]. Behind the beaches and dunes, huge packages of silt and clay were deposited and vast packages of peat developed in the (salt, brackish, and fresh) marshes. After that period, the first humans settled in that area, without interfering too much with the natural habitat. Only from 1000 AD, during a transgression of the sea, the interference with the natural landscape became stronger [4, 36]. Dikes were built, at ever increasing heights, and thereby more and more territory was reclaimed, the so-called polders. In those impoldered areas, protected with dikes, cities and agricultural lands developed more and more, as, for example, in the district of North Holland around the city of Amsterdam (Figure 9). In the areas not fully protected by dikes at several locations, the gulleys in between dunes became deeper, and the marshes behind were washed away, in the north of the Netherlands leading to the formation of today’s landscape of the Wadden Sea. That landscape became dominated by a series of islands with sandy beaches and dunes at the seaside and marshes at the landside, with large tidal flats and deep gulleys in between.
With the industrial revolution during the last 200 years, the impact on the landscape became even stronger, especially in those areas nearby ports, as in Rotterdam, one of the world’s largest harbours. Moreover, the land reclamation continued until the last century, with even the closure of the Southern Sea (NL: Zuiderzee) with a 32 km long dike in 1932, giving rise to a huge fresh water lake, Lake IJssel (NL: IJsselmeer), of more than 3300 km2. In this lake during the last decades, several polders were reclaimed for agriculture and new cities and villages now known as the Flevoland district.
We have thus focused on a couple of examples of these different types of habitats with different histories. Firstly, the transect in the Flevoland district (Figure 8) reflects the land reclamation during the last decades. Secondly, the transect of the harbour area of Rotterdam (Figure 10) reflects an area strongly modified since the industrial revolution. Thirdly, the transect in the district of North-Holland (Figure 9) reflects the land reclamation and impacts of the last 500 years and fourthly, the Wadden Sea area (Figure 14) represents the most natural habitat in the north of the Netherlands that arose after a transgression of the sea around 1000 AD.
In Flevoland, the reclaimed polders, in the midst of the fresh water Lake IJsselmeer, were mainly set up as agricultural areas together with some locations for residential areas as is also clear from our satellite image analyses (Figures 7 and 8). Agricultural areas (including forestry) covered 57% of the surface, open water 34% and residential areas 6%. These relative surface-values coincide largely with the inventories of the official Netherlands Statistics office (CBS) , being 51, 41 and 5%, respectively.
The industrialised area of the port Rotterdam is strongly dissected by canals with ships (36% of the surface) (Figures 7 and 10), which corroborates the value issued by the harbour authorities of the port of Rotterdam (38% for water surfaces) . Further, the harbour territory is mainly covered with industrial structures (48% including container stacks, industrial buildings, cranes, minerals, barren sands, storage tanks), which is similar to the 47% according to the harbour authorities of the port of Rotterdam , leaving only little remnants of the coastal estuarine area it was in earlier days.
In the district of North Holland, the surface for residential areas is larger (12%) than in the previous areas, although agriculture (52%) and water (32%) still take a big portion (Figures 7 and 9). These values coincide with those of the Netherlands Statistics office (CBS) (Statistics, 2016), being 12, 46 and 35%, respectively. Nevertheless, care has to be taken especially in this district in the analysis of the water areas, since from the images, it is not feasible to distinguish between fresh water and sea water. Moreover, it is difficult to make a distinction between some detailed habitats in the coastal areas of this district, such as man-made dikes (in the northern and eastern part of this district) and natural dunes (in the western part).
For the Wadden Sea and its nearby surroundings, a completely different division of the major habitats is found than in the previous regions, indicating its natural value and marine character (Figures 11 and 12). The share of deltas or tidal flats (13%) and water (77%) is much higher than in the previous areas, whereas agriculture has only a small share (4%). In earlier inventories of the Dutch Wadden Sea , an almost equal share of 43% for tidal flats and 43% for water was indicated (the other 14% belongs to islands). The lower share of tidal flats in our study is caused by the fact that the Wadden Sea is subject to alternating tidal water levels and therefore dependent on the acquisition time of the individual images.
A more generalised analysis (Figures 11 and 12) of the north-west of the Netherlands shows that there are two different sides of a coin. The heavily anthropogenically impacted regions (Figures 11 mid and 13), (i.e., the west Netherlands encompassing also the Rotterdam harbour area, Flevoland and the North Holland district) differentiate strongly from the more original and natural region in the north of the Netherlands, including the Wadden Sea and its direct surroundings (Figures 11 bottom and 14). The major differences can be found in the surface zones classified as agriculture, inhabited built-up areas and water.
The main habitats and structures for the moderately or strongly impacted areas in the north-west of the Netherlands (Flevoland, Rotterdam port, North Holland) are in general interpreted correctly by means of our satellite image analyses. For the Wadden Sea, one has to take into account the relative distribution of tidal flats versus water surface. A potential solution can be to acquire a series of images covering the full tidal cycle. It is also important to take care about the fact that in the Wadden Sea, although most of the tidal flats are barren, some may be overgrown with sea grasses and some are covered by seaweeds in summer, whereby these may be classified wrongly as agricultural land, assigning them to another category (e.g., agricultural). Again, images from different seasons can be compared to overcome this issue.
Altogether, we conclude that the majority of the habitats, man-made structures as well ecological features, can be classified rather accurately irrespective of a strongly different historic context and different levels of anthropogenic impact of the areas being studied. Nevertheless, it should be taken into account that due to its dynamic history, areas like the north-west Netherlands do call for carefulness regarding the interpretation of some features as, for example, water being fresh or salt and coastal defence structures being anthropogenic dikes or natural dunes and beaches. A further analysis is needed to distinguish between these ecologically rather different habitat types. Therefore, for a proper interpretation and ecological fine-tuning of satellite images, the cooperation with environmental researchers or managers with field experience remains a prerequisite.
After the determination of the classification accuracy of our SAR images, we also determined the pros and cons of including optical sensors such as multispectral Sentinel-2 images . On the one hand, one can automatically download selectable quadrants from large images and obtain image patch relationships that are much easier to understand and classify; on the other hand, we face the problem of potential cloud cover that can persist for a number of days. Figures 15–19 illustrate these relationships for some selected quadrants that cover similar areas as the SAR images. Combining optical sensors with SAR sensors can be a solution to solve the open points arisen in our study, such as the separation between fresh water and sea water (see in Figure 17, the classification results of an optical image that cover an area with these two types of water).
This work was supported by the TerraSAR-X Science Service System (proposals MTH-1118 and LAN-3156) and by the H2020 ECOPOTENTIAL project (grant agreement no. 641762).