Feature similarities of images restored from various super-resolution methods.
Abstract
For decades, super-resolution has been a widely applied technique to improve the spatial resolution of an image without hardware modification. Despite the advantages, super-resolution suffers from ill-posedness, a problem that makes the technique susceptible to multiple solutions. Therefore, scholars have proposed regularization approaches as attempts to address the challenge. The present work introduces a parameterized diffusion-steered regularization framework that integrates total variation (TV) and Perona-Malik (PM) smoothing functionals into the classical super-resolution model. The goal is to establish an automatic interplay between TV and PM regularizers such that only their critical useful properties are extracted to well pose the super-resolution problem, and hence, to generate reliable and appreciable results. Extensive analysis of the proposed resolution-enhancement model shows that it can respond well on different image regions. Experimental results provide further evidence that the proposed model outperforms.
Keywords
- super-resolution
- resolution
- enhancement
- regularization
- diffusion
1. Introduction
Before deepening into the super-resolution imaging, let us discuss the term resolution. Most people, particularly those not in the imaging field, define resolution broadly as the physical size of an image. For a two-dimensional digital image, this definition implies an area in the image given as the product of the number of pixels in the horizontal and vertical dimensions (pixel or picture element is the smallest unit of information in a digital image). In this context, therefore, a high-resolution image contains a higher pixel count than a low-resolution image. Figure 1(a) includes features with higher perceptual qualities than those in Figure 1(b), but both images have equal sizes. From the figure, therefore, we see that dimension only seems inadequate to define the resolution of an image.
Resolution, more generally, means the quality of a scene (image or video). Five major types of image resolutions are known: pixel resolution, spectral resolution, temporal resolution, radiometric resolution, and spatial resolution. The use of these variations depends on the application. Pixel resolution refers to the total number of pixels a digital image contains. Hence, both images in Figure 1(a) and (b) possess equal pixel resolutions of 2179 × 2011. In other words, each image is approximately 4.4 megapixels (2179 × 2011 = 4,381,969 pixels ≈4.4 megapixels). Unfortunately, pixel count offers fraction of the pieces of information contained in the image. For a colored image with red, green, and blue channels, an individual pixel can only accommodate the details of a single color. Spectral resolution describes the ability of an imaging device to distinguish the frequency (or wavelength) components of an electromagnetic spectrum. Imagine spectral resolution as the degree in which you can uniquely discern two different colors or light sources. Temporal resolution refers to the rate at which an imaging device revisits the same location to acquire data. When dealing with videos, for example, the term implies an average time between consecutive video frames: a standard video camera can record 30 frames per second, implying that every 33 ms, this camera captures an image. In remote sensing, temporal time is usually measured in days to represent time that a satellite sensor revisits a specific location to collect data. Radiometric resolution defines the degree at which an imaging system can represent or distinguish intensity variations on the sensor. Expressed in number of bits (or number of levels), radiometric resolution provides the actual content of information in the image. Spatial resolution explains how an imaging modality can distinguish two objects. In practical situations, spatial resolution describes clarity of an image and defines the resolving power of an image-capturing device. The perceptual quality of an image increases with the spatial resolution. This research presents super-resolution imaging as one of the available techniques to enhance the spatial resolution of an image.
Most people are naturally inclined to high-quality and visually appealing images that contain adequate details. However, this demand is not always achieved because of some imperfections in the imaging process. Therefore, scholars have proposed hardware and software approaches to address the challenge. The former approach requires sensor modification, and it may be achieved by reducing the physical sizes of the pixels—a process that increases pixel density (number of pixels per unit area) on the surface of the sensor [1]. The hardware approach gives perfect resolution enhancement, but the technique endures several drawbacks: (1) it introduces shot noise into the captured images, (2) it makes the imaging device costly and unnecessarily bulkier, and (3) it lowers the charge transfer rate because of the increased chip size [2]. These challenges have prompted scholars to search for software techniques, which are cost-effective and reliable, to improve the spatial resolution of an image without effecting circuitry of the imaging device. In this case, an image can be captured by a low-cost device and processed to generate its corresponding high-quality version.
The classical software approach that has gained a considerable attention of scholars is called super-resolution [3–6], which uses signal processing principles to restore high-resolution images from at least one low-resolution image. Super-resolution techniques can be put into two major categories: single-frame-based, which generates a high-resolution image from the respective single low-resolution image [7, 8], and multi-frame-based, which exploits information from a sequence of degraded images to generate a high-quality image [2, 6]. The current work builds on the multi-frame super-resolution framework, which implicitly encourages noise reduction from the input low-resolution images. The framework bridges total variation (TV) [9] and Perona and Malik [10] smoothing functionals and allows for these functionals to interact in such a way that super-resolution and preservation of critical image features are simultaneously conducted.
2. Image degradation model
The multi-frame super-resolution framework can better be understood through a conceptual degradation model, which shows how an unknown high-resolution image,
Figure 2 can be transformed into
which explains how the degradation model generates frame
where
where
For
which shows that, after shifting and zero filling,
For
which was proved in [12] that it represents pixel-wise mean of measurements. The
The super-resolution problem, whether formulated through
3. Hybrid super-resolution model
3.1. Regularization functionals
Considering the super-resolution ill-posedness property, a hybrid framework combining TV and PM regularization kernels has been formulated. The framework includes additional parameters,
In [9], Rudin et al. established the TV model that explains how noise in the image can be reduced. The model is based on the fact that a noisy image contains a higher total variation, defined by the integral of the absolute gradient of the image or
where
In the similar notion of the TV principle, Perona and Malik proposed an energy functional,
where
The regularization process is often supported by the fidelity potentials
for additive noise,
for multiplicative noise [18],
3.2. Proposed super-resolution model
The hybrid model can be derived from the minimization problem that integrates the corresponding energy functionals from super-resolution, TV, PM, and fidelity. Assuming additive noise and
where
Eq. (12) offers both super-resolution image reconstruction and noise removal capabilities, dictated by TV and PM models. From the equation, as
where
implying protection of edges against smoothing. This automatic interplay between reconstruction and regularization components helps to generate superior super-resolved images.
3.3. Numerical implementation
The solution of the proposed super-resolution model in Eq. (12) was iteratively estimated using the steepest descent method. Therefore, the evolution equation in Eq. (12) can be converted into a numerical system
where
Eq. (15) can be represented in block form by Figure 4. From the Figure, each low-resolution frame,
4. Experimental methodology
Several experiments were executed to determine performance of the proposed super-resolution model relative to the classical approaches. The methodology and procedures under which the experiments were undertaken can be explained as follows: firstly, high-resolution images of bike, butterfly, flower, hat, parrot, Parthenon, plant, and raccoon (Figure 7) were degraded to generate the corresponding low-resolution images (Figure 8, first column). Note that the original images were downloaded from the public domain with standard test images.1 These images were selected because they contain detailed features, and hence it would be easier to test the superiority of various super-resolution methods. As an example, the “Raccoon” image contains small-scale features (fine textures or fur) that most super-resolution approaches may find hard to restore. Degradation of the original images was achieved through warping, blurring, decimation, and noise addition to create sequences of 10 low-quality images with consecutive pairs differing by some rotation and translation motions. To void impacts of registration errors on the reconstruction process, the warping matrix was fixed. Thus, for 10 multiple low-resolution images, the warping matrix for the horizontal and vertical displacements, respectively denoted by ∆
∆ | 0.56 | 1.03 | 0.85 | 0.32 | −0.45 | −0.43 | 0.92 | 1.23 | 0.93 | 0.64 |
∆ | 0.12 | 0.53 | 0.27 | 0.00 | −0.83 | 1.12 | 1.08 | 0.12 | 0.54 | 1.37 |
Next, super-resolution methods based on a variety of regularizers, namely NC00 [20], TV [9], ANDIFF [21], and Hybrid, were applied on the degraded images to restore their original versions. Lastly, the objective metric, namely feature similarity (FSIM) [22], and the subjective metric were used to compare performances of different methods. FSIM incorporates into its formulation some aspects of the human visual system, and hence the metric is considered superior over several other existing image quality metrics. A visually appealing image has a higher value of FSIM, and vice versa.
5. Results and discussions
Visual results show that the classical methods tend to add undesirable artificial features into the reconstructed images (Figure 8). For instance, NC00 introduces bubble-like features around borders, edges, and corners, which are the critical features that emulate the human visual system. The method, on the other hand, does well on homogeneous image regions. The super-resolution method based on TV produces relatively sharper images, but the method also adds artifacts on homogeneous parts of the final images—an effect that degrades the visual quality of the images. The ANDIFF method generates smoother results that contain little artifacts, but the method underperforms for highly-textured images such as the Raccoon. The proposed hybrid model established a proper balance between smoothness and critical feature preservation (Figure 8, last column). Visually, the reconstructed images by our approach are more natural and are free from obvious artifacts. One may argue about a slight blurriness in our results. However, given the higher capability of the proposed method to preserve sensitive image features, this effect may be ignored. Also, the line graphs (taken near the last row across all columns) further confirm that the proposed method is superior because it generates a one-dimensional curve that closely matches the original one (Figure 9).
Numerical results demonstrate that, in all cases of the input images, the proposed super-resolution method achieves higher quality values (Table 1). These convincing objective observations can be explained well from the new formulation in Eq. (12): the hybrid super-resolution model captures the qualities of both PM and TV, an advantage that may promote higher objective quality results. Besides, our formulation incorporates parameters that give an effective interplay between the regularization functionals.
Image | NC00 | TV | ANDIFF | Proposed method |
---|---|---|---|---|
Bike | 0.7139 | 0.7148 | 0.7386 | 0.7642 |
Butterfly | 0.6721 | 0.6733 | 0.7386 | 0.7592 |
Flower | 0.6998 | 0.7084 | 0.7669 | 0.7970 |
Hat | 0.7512 | 0.7624 | 0.8106 | 0.8194 |
Parrot | 0.7672 | 0.7908 | 0.8595 | 0.8738 |
Parthenon | 0.7101 | 0.7287 | 0.7450 | 0.7618 |
Plant | 0.7429 | 0.7416 | 0.8230 | 0.8401 |
Raccoon | 0.7591 | 0.7877 | 0.8046 | 0.8257 |
6. Conclusion
In this work, we have established a hybrid super-resolution framework that combines desirable features of TV and PM models. The framework has been parametrized to mask weaknesses of the models, introduce an automatic interplay between TV and PM regularizations, and promote appealing results. More emphasis was put on super-resolving low-quality images while retaining their naturalness and preserving their sensitive image features. Experimental results demonstrate that the proposed framework generates superior objective and subjective results.
References
- 1.
Park SC, Park MK, Kang MG. Super-resolution image reconstruction: A technical overview. IEEE Signal Processing Magazine. 2003; 20 (3):21-36 - 2.
Maiseli BJ, Elisha OA, Gao H. A multi-frame super-resolution method based on the variable-exponent nonlinear diffusion regularizer. EURASIP Journal on Image and Video Processing. 2015; 2015 (1):22 - 3.
Dong C, et al. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2016; 38 (2):295-307 - 4.
El Mourabit I, et al. A new denoising model for multi-frame super-resolution image reconstruction. Signal Processing. 2017; 132 :51-65 - 5.
Peleg T, Elad M. A statistical prediction model based on sparse representations for single image super-resolution. IEEE Transactions on Image Processing. 2014; 23 (6):2569-2582 - 6.
Zeng X, Yang L. A robust multiframe super-resolution algorithm based on half-quadratic estimation with modified BTV regularization. Digital Signal Processing. 2013; 23 (1):98-109 - 7.
Purkait P, Pal NR, Chanda B. A fuzzy-rule-based approach for single frame super resolution. IEEE Transactions on Image Processing. 2014; 23 (5):2277-2290 - 8.
Yang M-C, Wang Y-CF. A self-learning approach to single image super-resolution. IEEE Transactions on Multimedia. 2013; 15 (3):498-508 - 9.
Rudin LI, Osher S, Fatemi E. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena. 1992; 60 (1–4):259-268 - 10.
Perona P, Malik J. Scale-space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence. 1990; 12 (7):629-639 - 11.
Farsiu S, et al. Fast and robust multiframe super resolution. IEEE Transactions on Image Processing. 2004; 13 (10):1327-1344 - 12.
Elad M, Hel-Or Y. A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur. IEEE Transactions on Image Processing. 2001; 10 (8):1187-1193 - 13.
Ma H, Nie Y. An edge fusion scheme for image denoising based on anisotropic diffusion models. Journal of Visual Communication and Image Representation. 2016; 40 :406-417 - 14.
Tsiotsios C, Petrou M. On the choice of the parameters for anisotropic diffusion in image processing. Pattern Recognition. 2013; 46 (5):1369-1381 - 15.
Xu J, et al. An improved anisotropic diffusion filter with semi-adaptive threshold for edge preservation. Signal Processing. 2016; 119 :80-91 - 16.
Chen Y, Levine S, Rao M. Variable exponent, linear growth functionals in image restoration. SIAM Journal on Applied Mathematics. 2006; 66 (4):1383-1406 - 17.
Chan TF, Esedoglu S. Aspects of total variation regularized L 1 function approximation. SIAM Journal on Applied Mathematics. 2005; 65 (5):1817-1837 - 18.
Lv X-G, et al. A fast high-order total variation minimization method for multiplicative noise removal. Mathematical Problems in Engineering. 2013; 2013 - 19.
Courant R, Friedrichs K, Lewy H. On the partial difference equations of mathematical physics. IBM Journal of Research and Development. 1967; 11 (2):215-234 - 20.
Pham TQ, Van Vliet LJ, Schutte K. Robust fusion of irregularly sampled data using adaptive normalized convolution. EURASIP Journal on Advances in Signal Processing. 2006; 2006 (1):083268 - 21.
Maiseli BJ, Ally N, Gao H. A noise-suppressing and edge-preserving multiframe super-resolution image reconstruction method. Signal Processing: Image Communication. 2015; 34 :1-13 - 22.
Zhang L, et al. FSIM: A feature similarity index for image quality assessment. IEEE Transactions on Image Processing. 2011; 20 (8):2378-2386
Notes
- http://www4.comp.polyu.edu.hk/~cslzhang/NCSR.htm