Color Reconstruction and Resolution Enhancement Using Super-Resolution

Abstract

Image super-resolution (SR) is a process that enhances the resolution of an image or a set of images beyond the resolution of the imaging sensor. Although there are several super-resolution methods, fusion super-resolution techniques are well suited for real-time implementations. In fusion super-resolution, the high-resolution images are reconstructed using different low-resolution-observed images, thereby increasing the high-frequency information and decreasing the degradation caused by the low-resolution sampling process. In terms of color reconstruction, standard reconstruction algorithms usually perform a bilinear interpolation of each color. This reconstruction performs a strong low-pass filtering, removing most of the aliasing present in the luminance signal. In this chapter, a novel way of color reconstruction is presented by using super-resolution in order to reconstruct the missing colors.

1. Introduction

The technical limitations of the imaging devices clearly restrict the spatial resolution of video sequences. The super-resolution (SR) reconstruction technique is usually defined in the state of the art of image processing as a method that combines multiple low-resolution (LR) images with some amount of aliasing to obtain a higher resolution image. Although several methods have been implemented in this field, there are still several future open research challenges [1]. SR can be found in fields like surveillance, remote sensing, astronomy, and an extensive set of consumer electronics applications, among many others [2–5].

This chapter proposes a combination of color reconstruction and resolution enhancement, showing how to apply SR as a novel way to reconstruct missing colors. The quality improvement applying SR mainly depends on two factors: movement in the sequence and presence of aliasing in the image [6, 7]. Therefore, it is important to assure some amount of aliasing in the image without using any artificial system. This aliasing is a direct consequence of the acquisition system.

This chapter is organized as follows: the super-resolution concept and the steps of the considered algorithm are introduced in Section 2, the experimental methodology is presented in Section 3, and the dataset is described in Section 4, while Section 5 shows some color reconstruction algorithms, and Section 6 shows the color reconstruction using super-resolution. Section 7 presents a novel combination between color reconstruction and resolution enhancement. Finally, the conclusions are highlighted in Section 8.

2. Super-resolution

The first frequency domain-based SR algorithm was proposed by Huang and Tsay in 1984 [8], and, after that, several other SR techniques have been developed with different results. A general consideration governing this kind of algorithms is that in order to obtain significant improvements in the resulting image, some amount of aliasing in the input LR images must be guaranteed. This approach is called fusion super-resolution in the literature. The main advantages are that it requires limited processing and memory resources [9] but has the disadvantage of generating artifacts when applied to real-life video sequences.

In order to avoid artifacts, the authors have proposed several improvements based on selective filters and the usage of several cameras [6, 7]. The result consists on a non-iterative dynamic SR algorithm, with reduced computational and memory requirements. It provides static and dynamic SR depending on whether the result is one single frame and a video sequence, respectively. The SR algorithm used in this work has been extensively tested by the authors for general-purpose images and video sequences [9, 10] and also in specific applications, like underwater images [11] or compressed images [12]. As this SR algorithm is based on the direct processing of the information contained in the input images and not in machine learning techniques, it can be used for any kind of situations (including local and global motion, changes in the illumination among frames, etc.), and it does not depend on any training set of images.

2.1. Algorithm description

The SR algorithm considered in this chapter starts after the adjustment of the borders of the images to normalize their dimensions. An LR image sequence is the input for the motion estimation, in which the goal is to find the best motion vectors (MVs) that estimate the real movement (with sub-pixel accuracy) between two consecutive frames. The entire algorithm is composed of a set of successive steps, the first of which is the motion estimation and the second one is the motion compensation, in this case called Shift & Add (S&A), which uses the MVs to build a grid for the frame obtained fusing the LR images. The Holes Filling phase substitutes the zeros with an interpolation of the values coming from adjacent pixels. Since usually big images are considered in the sequence (with the consequent high computational load), the algorithm introduces the concept of window-selective filter (WSF), based on a working window (WW). The aim is to apply SR on the current frame (CF) using only the pictures comprised in the same WW (see Figure 1 ).

Figure 1.

Algorithm scheme that shows the working window (in this case frames between 2 and 6), the current frame (CF), and the other images (always taking into consideration only the Y component). The CF undergoes motion estimation and after that the Shift & Add, followed by the Holes Filling process.

2.2. Motion estimation

In the motion estimation (ME) stage, the image is divided into non-overlapping macro-blocks (MBs), following the block-matching principle (a very popular and efficient ME technique, especially in video compression). In this way, not only global but also local movements are evaluated, referred to the CF. In fact, taking into consideration only the global motion evaluation, the algorithm will fail to create high-resolution (HR) frames when there is a considerable amount of local movement between frames.

2.3. Shift & Add

During this phase, the frames of the WW are used, on a block-per-block basis, to form the final HR image. Firstly, a grid of four times the dimension of the initial LR image is created: it contains the image of the CF after the ME, and the positions (in fact high-resolution pixels) in which no information is present are filled with zeros. MBs of the CF are considered as LR macro-blocks, and every one of them is interpolated to create the base for the corresponding HR block. Only the zero values are interpolated in this process as they represent empty positions.

2.4. Holes Filling

At the MB level, MVs on both horizontal and vertical axes are checked to be at the sub-pixel level, and the sum of absolute difference (SAD) parameter is used as a weight function between MVs and MBs. If new information is detected, it is added to the grid, whereas the points that remain empty are filled with interpolations of adjacent values, a procedure that goes under the name of Holes Filling process.

3. Experimental methodology

The experimental methodology is based on a modification on the capturing system. A common capture system is shown in Figure 2 . In front of the system, there is a lens to focus the scene on the sensor plane. This plane is commonly called image plane. The most common technique to obtain a color image is to sense red, green, and blue colors. In the high-quality products, three sensors are placed, one for each of these colors. Using some mechanical or optical system, the image plane is focused on the three sensors, obtaining three colors at the full sensor resolution.

Figure 2.

Capture system for imaging.

In cheaper applications, only one sensor is used. In that case, in order to obtain three colors, a color filter array (CFA) is placed in front of the sensor. By this way, each sensor cell senses a different color. The most commonly used is the red, green, and blue (RGB) Bayer pattern, which is represented at the beginning of Figure 2 .

After sensing the colors, a color reconstruction algorithm is applied to obtain the colors where they were not sensed. There are several color reconstruction algorithms; most of them are based on different types of interpolation [13–16]. After the color reconstruction, another image processing is performed to obtain the image in YUV format, which is the input of the compression system.

In order to obtain the YUV 4:2:0 image, the following operations are done:

• Calculate the YUV 4:4:4 image as:

  • Y = 0.3 · R + 0.59 · G + 0.11 · B

  • U = Y − R

  • V = Y − B

• Perform a low-pass filtering on U and V images.

• Subsample the U and V images to obtain the YUV 4:2:0 format.

In order to keep aliasing in this work, three aspects are of special relevance:

• Lens

• Sensor configuration

• Color reconstruction algorithm

In this study, some sequences were recorded using high-quality lens. The resolution of this lens was selected to be higher than the sensor resolution. By this way, we avoided the lens optical low-pass filtering (OLPF). In this case, the aliasing appears in the sampling process. This process is guided by two main factors:

• The sensor fill factor

• The color sampling

The sensor fill factor describes the ratio between the light-sensitive area per pixel and the total pixel area. This factor can be increased by using small micro-lens at the top of each pixel. With these lenses, shown in Figure 3 , some of the incident light in the nonsensitive area is concentrated to the light-sensitive area. Those micro-lenses can increase the fill factor up to 60 or 70% but remove some amount of aliasing due to the optical low-pass filter (OLPF) effect produced by the lenses. For this work, we selected a sensor without micro-lenses, thus keeping the aliasing that in other cases would be removed by the OLPF.

Figure 3.

Micro-lenses disposition.

The second factor is the color sampling. In fact, each color image is a reconstructed image. The maximum sampled frequency is represented in Figure 4 . The axes of this figure are the horizontal and vertical space frequencies, with the low frequencies located in the middle of the square. The color squares represent the maximum sampled frequency for each color: the bigger square with a diamond shape represents the frequencies of the green color, and the smaller inner squares represent the frequencies of the red and blue colors. It is important to note that the green signal has higher frequencies than red and blue ones. This limitation in the frequency distribution produces some aliasing in the color spectral domain. In this work a CFA has been used, largely decreasing the resolution of the three color components, but in special the resolution of the red and blue components.

Figure 4.

Color-sampled frequencies in the frequency space.

4. Description of the dataset

Following the methodology described in Section 3, a set of 20 images with random sub-pixel shifts among them was obtained from a Dutch newspaper. The original image exhibits a rich amount of color details and represents a picture of the Big Ben cleaners working on the clock on the left of the image and two columns of text on the right of the image. The sizes of all the sampled images are 624 pixels wide and 464 pixels in height. Each sample image has three color components, with a bit depth of 8 bits per color component (24 bits/pixel). The first LR input image can be seen in Figure 5 .

Figure 5.

Frame 1 of the input image used to reconstruct the color.

5. Color reconstruction algorithms

In this section, a first experiment consisting in the use of the image in YUV 4:2:0 for resolution enhancement is presented. In this case, the image follows all the processing, but different algorithms to reconstruct the color signal are chosen. This processing must keep as much aliasing as possible in the luminance image, to obtain a good performance in SR. Three algorithms were tested: standard reconstruction, SmartGreen1, and SmartGreen3 [17].

5.1. Standard reconstruction algorithms

Standard reconstruction algorithms perform a bilinear interpolation of each color plane. This reconstruction performs a strong low-pass filtering, removing most of the aliasing present in the luminance signal. Super-resolution algorithm was applied using the obtained image in YUV 4:2:0. The results are shown in Figures 6 and 7 . After 20 frames, only a small improvement was achieved due to the limited aliasing present in the luminance image.

Figure 6.

Standard reconstruction with SR of “paper,” frame 1 (left) and frame 20 (right). For the first frame, no SR is possible, and so it is equal to bilinear interpolation.

Figure 7.

Standard reconstruction with SR of “paper,” frame 1 (up) and frame 20 (down) details.

5.2. SmartGreen1

SmartGreen1 [17] performs a bilinear interpolation over the red and blue images. If high frequencies are present, the green image is obtained from the red or blue images in places where there are no green samples. In those places, the green image is equal to the red- or blue-sampled image multiplied by some predefined coefficients.

If this approach works well, the green image can improve the pixel resolution and, therefore, increases the luminance resolution. This algorithm works better in gray areas, where R = G = B. SmartGreen1 seems to keep more aliasing information, and therefore we expected better performance when used in combination with super-resolution. Results are shown in Figures 8 and 9 .

Figure 8.

SmartGreen1 reconstruction with SR of “paper,” frame 1 (left) and frame 20 (right). For the first frame, no SR is possible, and so it is equal to bilinear interpolation.

Figure 9.

SmartGreen1 reconstruction with SR of “paper,” frame 1 (up) and frame 20 (down) details.

5.3. SmartGreen3

SmartGreen3 requires more computational load. In previous versions of SmartGreen, the computed green image from the red and blue values was used directly. This approach worked well in gray areas but made colorful images grayer. In SmartGreen3, this image is only used to enhance the edges and to help in the detection of false color due to color aliasing. The main issue with this algorithm is that it performs a band-pass filtering of the luminance image, removing aliasing. In consequence, we will not increase the quality of the image by using super-resolution. Due to the good performance of this technique, it can be a reference of sharpness for SR as one of the best qualities that the color reconstruction algorithms can achieve. The results obtained using these algorithms are shown in Figures 10 and 11 .

Figure 10.

SmartGreen3 reconstruction with SR of “paper,” frame 1 (left) and frame 20 (right). For the first frame, no SR is possible, and so it is equal to bilinear interpolation.

Figure 11.

SmartGreen3 reconstruction with SR of “paper,” frame 1 (up) and frame 20 (down) details.

The performance obtained using standard reconstruction or SmartGreen1 with SR is not better than using SmartGreen3. Moreover, the improvement of using SR in conjunction with SmartGreen3 is very small, so it is not very wise to apply SR in that case due to the computational load increase.

6. Color reconstruction using super-resolution

As the results obtained were not very satisfactory using the previous approaches, a new approach was tried. Since each color image is subsampled, a large amount of aliasing is present in each one of these images. In fact, when using the Bayer pattern, the luminance image is never fully sensed because it is composed by sampled values of one color and reconstructed values from the other two color images. Because the sensor works in RGB, it is better to work directly in this domain.

Following this idea, it can be seen that the way that the sensor works is similar to our approach to get aliasing in previous steps. Indeed, the red and blue images are subsampled by a factor of two in both directions, and the green image is subsampled, but losing only half of the samples.

Therefore, the proposed idea is to reconstruct each color image as color reconstruction algorithms do, obtaining pixel resolution using super-resolution. This idea is presented in Figure 12 . In this application, each color image is treated as the luminance signal was before, but the motion estimation is done based only on the green values. In this way, we can use SR to replace the color reconstruction algorithm. The first image in the sequence is equivalent to standard reconstruction, but when new images come to the system, the quality increases with the new incoming information.

Figure 12.

Color reconstruction with super-resolution with aliasing (SRA).

To avoid a large amount of calculations, the motion vectors are computed only over the green values, and the same motion vectors are applied to the red and blue matrixes. Another reason to do it in that way is that different motion vectors for different colors can produce color aliasing due to different calculated movements of each color. For this reason, the motion vectors must be coherent over the three color planes.

In this case, SR achieves good results. The images obtained are comparable with SmartGreen3 ones, but there are some issues that remain to be tackled:

• In contrast with traditional color reconstruction algorithms, more than one image is needed to increase the quality.

• Errors in the motion estimation will produce bad results. The motion estimation algorithm must be very accurate, or it has to be used only in certain scenes.

• A large number of calculations are needed to perform the motion estimation.

On the other hand, the most important advantages obtained are:

• It works better in colorful images, while SmartGreen use to shift the false colors to the gray scale.

• In case of good motion estimation, the color aliasing is better suppressed.

Moreover, another approach was tested, where the super-resolution with aliasing (SRA) is applied after the overall processing in the image. This processing increases the differences between the values in the RGB domain. For example, the matrix correcting process and the gamma correction provide more differentiate values. This can be useful for the motion estimator to increase the accuracy of the motion vectors. In this case, the proposed process is to follow the next steps:

• Reconstruct the color using standard reconstruction algorithms, keeping the original values. This is necessary because other processing need R, G, and B data in all the positions.

• Do all the processing: matrix correction, gamma correction, white balance, etc.

• Take the sampled values from the original positions. These values are only modified by the previous processing and not from the color reconstruction algorithms. Using these values, we execute the super-resolution algorithm and reconstruct the lost colors.

With this approach, the obtained results are shown in Figures 13 and 14 . As it is usual in this kind of processing, where no reference image is available, the results are only qualitative, but in Figure 15 , a better color reconstruction is appreciated, with less color aliasing in all the colors, but in special in the green color, which is less shifted to the gray scales.

Figure 13.

Standard color reconstruction.

Figure 14.

SmartGreen3 color reconstruction with false color detection and edge enhancement.

Figure 15.

Super-resolution color reconstruction.

7. Color reconstruction and resolution enhancement

Following the line of the last two applications of Sections 5 and 6, it is possible to join both proposals in a single one. The new idea is to increase the pixel resolution, using each color signal instead of the luminance signal.

In Section 3, it was shown that there are two main sources of aliasing: the color sampling and the fill factor of the sensor. Using these concepts, we can exploit both sources of aliasing in a new application. This new application will deal with color reconstruction and zoom using SR, according to the scheme presented in Figure 16 . Taking into account a single basic block, 16 pixels are reconstructed from two green values and from one blue and red value. Therefore, more images will be necessary to obtain a good quality, eight in the ideal case of having all the missing positions available.

Figure 16.

Color reconstruction and zoom using super-resolution.

The processing followed in this case consisted in:

• Reconstructing the color using a standard reconstruction algorithm, keeping the original values.

• Performing all the other processing: matrix correction, gamma correction, white balance, etc.).

• Taking the sampled values from the original positions. These values are only modified by the previous processing and not from the color reconstruction algorithms.

• Applying SR, using a 4 × 4 matrix instead of a 2 × 2 as in the previous cases.

In this case, the motion vectors are also calculated using the green values and reused for the other color matrixes. The usual way to obtain these images is performing first a color reconstruction algorithm followed by a bilinear interpolation. This approach was used in the images of Figures 17 and 18 . These images are used as a subjective reference to compare against the other images. Following our proposal, some experiments were developed, obtaining the results shown in Figure 19 .

Figure 17.

Standard color reconstruction and bilinear interpolation (left). SmartGreen3 color reconstruction followed by false color detection and bilinear interpolation (right).

Figure 18.

SmartGreen3 color reconstruction followed by false color detection, edge enhancement, and bilinear interpolation (left). Standard color reconstruction followed by edge enhancement and bilinear interpolation (right).

Figure 19.

Super-resolution reconstruction and zoom (left). Super-resolution reconstruction, zoom, and edge enhancement (right).

The obtained results exhibited good quality, although some advantages and disadvantages were found in this new approach. The main disadvantages are:

• More than one image is required to obtain good quality. Nevertheless, this is not a problem for video sequences.

• Bad motion estimation can reduce its quality to bilinear interpolation quality, worse than SmartGreen3 quality.

• It works in RGB data, and it is still necessary to convert to YUV format in some cases, as in video compression.

• It is necessary to suppress the aliasing when super-resolution is not performed.

Also, some advantages are appreciated:

• Better resolution and suppression of color aliasing, achieving a good electronic zooming.

• Reuse of the existing hardware in combination with video compression, for instance, the motion estimation unit.

• Take advantage of undesirable characteristics as aliasing and hand motion.

• Use of signal processing instead of mechanical parts to perform the zoom, contributing in this way to reduce the overall device power dissipation.

8. Conclusions

In this chapter, several applications have been addressed using a real acquisition system. A study of the system has been carried out, analyzing the processing flow in order to identify the possible sources of aliasing. An important problem in this stage is the absence of a reference image to be used in order to obtain quantitative metrics.

Three main applications related with electronic cameras and the image processing and acquisition chain have been addressed:

• Resolution enhancement, where three available color reconstruction algorithms have been studied in order to keep the aliasing at the input. As these algorithms do not allow passing a great amount of aliasing, the SR enhancement is not very noticeable.

• Color reconstruction, where it is exposed how to apply SR as a way to reconstruct the missing colors when a single sensor is used with a CFA.

• Color reconstruction and resolution enhancement, where the two previous applications are combined. In this last case, good results are achieved, making this kind of techniques for such applications recommendable.