Introductory Chapter: Recent Advances in Image Restoration

1. Recent advances

In this chapter, we do not intend to provide a comprehensive survey of existing recent image restoration papers in the literature. Instead, we attempt to provide a glimpse of recent advances from our own perspectives and applications. In particular, we will focus on the following areas. Moreover, we will connect those research areas to some real-world applications.

1.1 Image enhancement

Images can be enhanced from several perspectives: spatial, spatial-spectral, spectral, and spatio-temporal.

1.1.1 Spatial domain

Here, we focus on methods that use only a single image to improve the spatial resolution. The simplest method is the bicubic interpolation, which does not utilize any external information such as point spread function (PSF) [1]. A total of 16 neighbors are used to generate a prediction, and the performance is better than bilinear interpolation, which uses only four neighbors. Recently, there are some new developments. A notable one is the algorithm described in [2], which utilizes the PSF to improve the resolution of a single image. The super-resolution algorithm in [3] is based on edge interpolation. There is also a group of methods based on deep learning [4, 5, 6]. Vast amounts of training images are needed to train the algorithm. Another group is using dictionary-based approach [7, 8]. Both the deep learning and dictionary approaches require many training images, which may be difficult to obtain.

In Sidiya and Li’s chapter [9] in this book, a generative adversarial network (GAN)-based approach was introduced to face image enhancement. The key distinction from conventional GAN is that it is an unsupervised approach, meaning that no ground truth images are required for training. Experimental results showed that the proposed unsupervised approach is only 1–2 dBs inferior to state-of-the-art supervised algorithms. The chapter also pointed out some failure cases, which the authors knew the reasons and will further improve the results in the future.

In Qin et al.’s chapter [10], the authors present a new framework for fusing results from cross view images for 3D mesh reconstruction. Real satellite and ground-view images were used to demonstrate the proposed framework. It was found that the reconstruction accuracy has been improved by close to 1 meter in one of the areas.

1.1.2 Spatial-spectral resolution enhancement: Pansharpening

In many applications, we may have a high-resolution (HR) image with only a few bands and another image having low resolution but many bands. Pansharpening is an image fusion approach that fuses one high spatial resolution image with another low-resolution (LR) multispectral (MS) image. Earlier pansharpening algorithms are limited to images where the panchromatic band overlaps with the MS bands. However, recent advancements have extended the approach to non-overlapping bands [11, 12, 13].

There are two recent survey papers in pansharpening [13, 14, 15, 16]. In recent studies, pansharpened images were observed to improve the performance of some applications [17].

Another chapter in this book by Qu et al. [18] summarizes a Dirichlet-Net for pansharpening. The algorithm was applied to Mastcam image enhancement.

In Restaino et al.’s chapter [19], the authors report an interesting study of using multi-platform data for pansharpening. That is, the low-resolution hyperspectral data and the high-resolution pan or multispectral data come from different satellites.

1.1.3 Spectral enhancement using synthetic bands

In some applications, only MS images are available. It may be useful to synthesize some hyperspectral images using those images so that the performance of some applications can be improved. Recently, there have been some new algorithms such as the Extended Morphological Attribute Profiles (EMAP) algorithm [20] for synthesizing spectral bands.

Here, rather than explaining the details of EMAP, we would like to mention a few recent applications of EMAP. The first one is to use EMAP for soil detection. The original MS images have eight bands. After applying EMAP, 80 synthetic bands were generated. The soil detection performance was improved quite significantly. More details can be found in [21, 22, 23]. Another application is on change detection using heterogeneous images. That is, the images at two different times may not come from the same imager. In [24], we have demonstrated that EMAP has improved the change detection performance in 36 out of 50 cases. In a third application on land cover classification [25], we have observed that, with the help of EMAP, using only 4 bands (RGB and NIR) can achieve reasonably accurate land cover classification performance that is only a few percentage points lower than that of using 144 bands of data.

1.1.4 Spatio-temporal fusion

Here, we consider an interesting application scenario. At time t₁, we have one high-resolution (HR) MS image and a LR MS image. However, at time t₂, we only have a LR MS image. It will be important to use the aforementioned images and synthesize a HR MS image at t₂.

Such scenarios do exist. As shown in Figure 1, one example is the fusion of Landsat (30 m spatial resolution with 16-day revisit period) and MODIS (500 m spatial resolution with almost daily revisit). More details can be found in [26, 27]. Another application scenario is the fusion of Worldview with Planet images [28]. A third temporal fusion study is for Landsat and Worldview images [29]. Once the fused images are available, more frequent change detection can then be performed for a given area.

Figure 1.

Fusion of Landsat and MODIS images to create a high spatial and high temporal resolution image sequence.

Albanwan and Qin’s chapter on spatio-temporal [30] discusses various spatio-temporal fusion methods for remote sensing images. Pixel, feature, and decision level fusion approaches were summarized. A few past results from the authors’ past papers were also included.

1.2 Image denoising

Image noise can be introduced during the image acquisition process. For example, in low lighting conditions, pixel amplitude-dependent noise (Poisson noise) are introduced. In the past, people have investigated sparsity-based methods [31, 32] and deep learning methods [33, 34]. There are also joint denoising and demosaicing algorithms [35].

1.3 Image demosaicing

Many commercial cameras have incorporated the Bayer pattern [36], which is also known as color filter array (CFA) 1.0. An example of CFA 1.0 is shown in Figure 2a. There are many repetitive 2x2 blocks and, in each block, two green, one red, and one blue pixels are present. To save cost, the Mastcam onboard the Mars rover Curiosity [37, 38, 39, 40] also adopted the Bayer pattern. Due to the popularity of CFA 1.0, Kodak researchers invented a red-green-blue-white (RGBW) pattern or CFA 2.0 [41, 42]. An example of the RGBW pattern is shown in Figure 2b. In each 4 × 4 block, eight white pixels, four green pixels, and two red and blue pixels are present. Having more white pixels is believed to help improve the sensitivity of the camera, which is important in low lighting conditions. Numerous other CFA patterns have been invented in the past few decades [43, 44, 45].

Figure 2.

Three CFA patterns. (a) CFA 1.0; (b) CFA 2.0.

It will be good to illustrate the differences between a state-of-the-art method (Demonet [35]) and the NASA’s current demosaicing algorithm known as Malvar-He-Cutler (MHC) [46]. From Figure 3, it can be seen that the demosaiced images by MHC contain some color distortion artifacts whereas the Demonet images do not have noticeable color distortions.

Figure 3.

Comparative of demosaicing images. Left column: NASA’s existing software; right column: State-of-the-art deep learning approach.

There are some new developments in demosaicing CFA 2.0 or RGBW. In [47], a new pansharpening approach was proposed to demosaic RGBW patterns. In [48], a further improved version by combining deep learning and pansharpening was proposed. In [49], a comparative study of the performance of CFA 1.0 and CFA 2.0 for low lighting images was carried out. It was found that CFA 2.0 has advantages over CFA 1.0 in low lighting conditions. It was also observed that denoising can further enhance the demosaicing performance. Finally, a new CFA 3.0 was proposed in [50] in which a comparative study among CFAs 1.0, 2.0, and 3.0 was conducted. It was observed that CFA 2.0 has better performance in terms of peak signal-to-noise ratio (PSNR) in low lighting conditions than CFAs 1.0 and 3.0. CFA 3.0 has better performance than CFA 1.0.

1.4 Image deblurring

Image blurring can be caused by camera motion and built-in factors such as point spread functions in various stages of image formation. In the book [51], a few PSFs are mentioned, including optical, motion, detector, and electronics. For motion-induced blurring, researchers have developed estimation methods [52] to restore the original images.

For NASA’s Mastcam application, an interesting approach was proposed in [53] to improve the left Mastcam images. The idea was to use the left and right Mastcam images to estimate the deblurring kernel, and then a deconvolution is applied to deblur the left images.

1.5 Image inpainting

Image inpainting is a well-known technique for image restoration. One can remove some image contents and then replace those missing contents with fictitious information. Some conventional techniques include Field of Experts (FOE) [54], Laplacian method [55], Local Matrix Completion Sparse (LMCS) [56], and Transformic [57]. More recent methods used GAN for inpainting [58]. We briefly describe these methods below.

FOE: The Field of Experts method (FOE) was developed by Roth et al. [54]. This method uses pre-trained models that are used to filter out noise and obstructions in images.

Laplacian: This method [55] fills in each missing pixel using the Laplacian interpolation formula by finding the mean of the surrounding known values.

Local Matrix Completion Sparse (LMCS) [56]: In LMCS, which was developed by us, a search is performed for each missing pixel to find a pixel with the most similar neighbors. After the search, the missing pixel is replaced with the found pixel. This method performs very well with images containing repeating patterns.

Transformic [57]: The Transformic method was developed by Mansfield et al. [57]. It is similar to the LMCS in that it searches the whole image for a patch that is similar to the neighbors of the missing pixel. However, this method transforms and rotates the searched area to find a better match.

Generative Inpainting (GenIn) [58]: A new inpainting method, Generative Inpainting (GenIn), which is a deep learning-based method [58], was considered in our research. It was developed at the University of Illinois that aims to outperform typical deep learning methods that use convolutional neural network (CNN) models. GenIn builds on CNN and generative adversarial networks in an effort to encourage cohesion between created and existing pixels.

We briefly mention a few recent applications. In [59], the LMCS technique was applied to automatic target recognition. A recent application of LMCS, Transformic, and deep learning can be found in [60] for error concealment in infrared images.

In the chapter by Kwan et al. [61], a number of conventional and deep learning methods were applied to digital terrain model (DTM) extraction.

1.6 Compression artifact reduction

It is surprising that JPEG image compression codec is still being used in some applications nowadays. For instance, the Mars rover Curiosity has a number of imagers, which are all using JPEG for image compression. The current practice at NASA is to use low compression ratios, which can only achieve around three times of compression efficiency. Since the invention of JPEG in early 1990s, there have been quite a few newer and more powerful compression standards in the literature. In the past few years, there are studies on evaluating a number of compression codecs that can achieve perceptually lossless compression [62, 63, 64]. The findings showed that it is feasible to attain 10 to 1 compression with almost no loss of image quality.

2. Future directions

Although there are some encouraging progress in image restoration in recent years, there are still some tough problems ahead. We list a few directions below.

2.1 Image enhancement

Earlier, we have seen that temporal resolution can be enhanced by fusing two sequences of images: one with high revisit times but low resolution and another just the opposite. After fusion, a sequence of high spatial resolution and high temporal resolution images emerges. The new sequence of images can be used for more frequent change detection, land cover classification, etc. However, based on our investigations, the change detection performance is still limited [65]. The reason is that the enhanced images fail to capture the changes sometimes. More research is needed.

Moreover, spectral enhancement sometimes have mixed results. In some applications, we do not see improvement for some reasons [66]. This implies that more research is needed to determine that under what conditions the EMAP-based methods can provide improvement and under what conditions not.

2.2 Image deblurring

For real blurred images collected from cameras, we noticed that some of the open-source codes still could not get good deblurring results. This is perhaps due to the fact that the camera motion may be nonlinear (jerky motion) and existing kernel estimation methods cannot handle such nonlinear motions. Again, more research is needed in this area.

2.3 Image demosaicing

As mentioned earlier, color images using color filter arrays collected in low lighting environments contain Poisson noise that seriously affect the image quality. Image denoising needs to combine with demosaicing in order to yield high-quality images. We believe there is still room for improvement in this area. One possible direction is to investigate deep learning approaches.

2.4 Change detection using heterogeneous images

In many remote sensing applications, we may have high-resolution images at one time but may only have low-resolution images at another time. It will be good to perform change detection across multiple platforms. There are some recent advances [24, 67, 68, 69, 70, 71, 72] in change detection using multimodal or heterogeneous images. For example, Ziemann et al. [68] studied change detection using a mixture of multispectral and synthetic aperture radar (SAR) images. However, more research is still needed to yield consistent results.

Acknowledgments

This work was supported in part by DOE grant DE-SC0019936 and NASA contract number 80NSSC17C0035.