Image super-resolution (SR) is a process that enhances the resolution of an image or a set of images beyond the resolution of the imaging sensor. Although there are several super-resolution methods, fusion super-resolution techniques are well suited for real-time implementations. In fusion super-resolution, the high-resolution images are reconstructed using different low-resolution-observed images, thereby increasing the high-frequency information and decreasing the degradation caused by the low-resolution sampling process. In terms of color reconstruction, standard reconstruction algorithms usually perform a bilinear interpolation of each color. This reconstruction performs a strong low-pass filtering, removing most of the aliasing present in the luminance signal. In this chapter, a novel way of color reconstruction is presented by using super-resolution in order to reconstruct the missing colors.
- color reconstruction
- video enhancement
- image fusion
- resolution enhancement
The technical limitations of the imaging devices clearly restrict the spatial resolution of video sequences. The super-resolution (SR) reconstruction technique is usually defined in the state of the art of image processing as a method that combines multiple low-resolution (LR) images with some amount of aliasing to obtain a higher resolution image. Although several methods have been implemented in this field, there are still several future open research challenges . SR can be found in fields like surveillance, remote sensing, astronomy, and an extensive set of consumer electronics applications, among many others [2–5].
This chapter proposes a combination of color reconstruction and resolution enhancement, showing how to apply SR as a novel way to reconstruct missing colors. The quality improvement applying SR mainly depends on two factors: movement in the sequence and presence of aliasing in the image [6, 7]. Therefore, it is important to assure some amount of aliasing in the image without using any artificial system. This aliasing is a direct consequence of the acquisition system.
This chapter is organized as follows: the super-resolution concept and the steps of the considered algorithm are introduced in Section 2, the experimental methodology is presented in Section 3, and the dataset is described in Section 4, while Section 5 shows some color reconstruction algorithms, and Section 6 shows the color reconstruction using super-resolution. Section 7 presents a novel combination between color reconstruction and resolution enhancement. Finally, the conclusions are highlighted in Section 8.
The first frequency domain-based SR algorithm was proposed by Huang and Tsay in 1984 , and, after that, several other SR techniques have been developed with different results. A general consideration governing this kind of algorithms is that in order to obtain significant improvements in the resulting image, some amount of aliasing in the input LR images must be guaranteed. This approach is called fusion super-resolution in the literature. The main advantages are that it requires limited processing and memory resources  but has the disadvantage of generating artifacts when applied to real-life video sequences.
In order to avoid artifacts, the authors have proposed several improvements based on selective filters and the usage of several cameras [6, 7]. The result consists on a non-iterative dynamic SR algorithm, with reduced computational and memory requirements. It provides static and dynamic SR depending on whether the result is one single frame and a video sequence, respectively. The SR algorithm used in this work has been extensively tested by the authors for general-purpose images and video sequences [9, 10] and also in specific applications, like underwater images  or compressed images . As this SR algorithm is based on the direct processing of the information contained in the input images and not in machine learning techniques, it can be used for any kind of situations (including local and global motion, changes in the illumination among frames, etc.), and it does not depend on any training set of images.
2.1. Algorithm description
The SR algorithm considered in this chapter starts after the adjustment of the borders of the images to normalize their dimensions. An LR image sequence is the input for the motion estimation, in which the goal is to find the best motion vectors (MVs) that estimate the real movement (with sub-pixel accuracy) between two consecutive frames. The entire algorithm is composed of a set of successive steps, the first of which is the motion estimation and the second one is the motion compensation, in this case called Shift & Add (S&A), which uses the MVs to build a grid for the frame obtained fusing the LR images. The Holes Filling phase substitutes the zeros with an interpolation of the values coming from adjacent pixels. Since usually big images are considered in the sequence (with the consequent high computational load), the algorithm introduces the concept of window-selective filter (WSF), based on a working window (WW). The aim is to apply SR on the current frame (CF) using only the pictures comprised in the same WW (see Figure 1 ).
2.2. Motion estimation
In the motion estimation (ME) stage, the image is divided into non-overlapping macro-blocks (MBs), following the block-matching principle (a very popular and efficient ME technique, especially in video compression). In this way, not only global but also local movements are evaluated, referred to the CF. In fact, taking into consideration only the global motion evaluation, the algorithm will fail to create high-resolution (HR) frames when there is a considerable amount of local movement between frames.
2.3. Shift & Add
During this phase, the frames of the WW are used, on a block-per-block basis, to form the final HR image. Firstly, a grid of four times the dimension of the initial LR image is created: it contains the image of the CF after the ME, and the positions (in fact high-resolution pixels) in which no information is present are filled with zeros. MBs of the CF are considered as LR macro-blocks, and every one of them is interpolated to create the base for the corresponding HR block. Only the zero values are interpolated in this process as they represent empty positions.
2.4. Holes Filling
At the MB level, MVs on both horizontal and vertical axes are checked to be at the sub-pixel level, and the sum of absolute difference (SAD) parameter is used as a weight function between MVs and MBs. If new information is detected, it is added to the grid, whereas the points that remain empty are filled with interpolations of adjacent values, a procedure that goes under the name of Holes Filling process.
3. Experimental methodology
The experimental methodology is based on a modification on the capturing system. A common capture system is shown in Figure 2 . In front of the system, there is a lens to focus the scene on the sensor plane. This plane is commonly called image plane. The most common technique to obtain a color image is to sense red, green, and blue colors. In the high-quality products, three sensors are placed, one for each of these colors. Using some mechanical or optical system, the image plane is focused on the three sensors, obtaining three colors at the full sensor resolution.
In cheaper applications, only one sensor is used. In that case, in order to obtain three colors, a color filter array (CFA) is placed in front of the sensor. By this way, each sensor cell senses a different color. The most commonly used is the red, green, and blue (RGB) Bayer pattern, which is represented at the beginning of Figure 2 .
After sensing the colors, a color reconstruction algorithm is applied to obtain the colors where they were not sensed. There are several color reconstruction algorithms; most of them are based on different types of interpolation [13–16]. After the color reconstruction, another image processing is performed to obtain the image in YUV format, which is the input of the compression system.
In order to obtain the YUV 4:2:0 image, the following operations are done:
Calculate the YUV 4:4:4 image as:
Y = 0.3 · R + 0.59 · G + 0.11 · B
U = Y − R
V = Y − B
Perform a low-pass filtering on U and V images.
Subsample the U and V images to obtain the YUV 4:2:0 format.
In order to keep aliasing in this work, three aspects are of special relevance:
Color reconstruction algorithm
In this study, some sequences were recorded using high-quality lens. The resolution of this lens was selected to be higher than the sensor resolution. By this way, we avoided the lens optical low-pass filtering (OLPF). In this case, the aliasing appears in the sampling process. This process is guided by two main factors:
The sensor fill factor
The color sampling
The sensor fill factor describes the ratio between the light-sensitive area per pixel and the total pixel area. This factor can be increased by using small micro-lens at the top of each pixel. With these lenses, shown in Figure 3 , some of the incident light in the nonsensitive area is concentrated to the light-sensitive area. Those micro-lenses can increase the fill factor up to 60 or 70% but remove some amount of aliasing due to the optical low-pass filter (OLPF) effect produced by the lenses. For this work, we selected a sensor without micro-lenses, thus keeping the aliasing that in other cases would be removed by the OLPF.
The second factor is the color sampling. In fact, each color image is a reconstructed image. The maximum sampled frequency is represented in Figure 4 . The axes of this figure are the horizontal and vertical space frequencies, with the low frequencies located in the middle of the square. The color squares represent the maximum sampled frequency for each color: the bigger square with a diamond shape represents the frequencies of the green color, and the smaller inner squares represent the frequencies of the red and blue colors. It is important to note that the green signal has higher frequencies than red and blue ones. This limitation in the frequency distribution produces some aliasing in the color spectral domain. In this work a CFA has been used, largely decreasing the resolution of the three color components, but in special the resolution of the red and blue components.
4. Description of the dataset
Following the methodology described in Section 3, a set of 20 images with random sub-pixel shifts among them was obtained from a Dutch newspaper. The original image exhibits a rich amount of color details and represents a picture of the Big Ben cleaners working on the clock on the left of the image and two columns of text on the right of the image. The sizes of all the sampled images are 624 pixels wide and 464 pixels in height. Each sample image has three color components, with a bit depth of 8 bits per color component (24 bits/pixel). The first LR input image can be seen in Figure 5 .
5. Color reconstruction algorithms
In this section, a first experiment consisting in the use of the image in YUV 4:2:0 for resolution enhancement is presented. In this case, the image follows all the processing, but different algorithms to reconstruct the color signal are chosen. This processing must keep as much aliasing as possible in the luminance image, to obtain a good performance in SR. Three algorithms were tested: standard reconstruction, SmartGreen1, and SmartGreen3 .
5.1. Standard reconstruction algorithms
Standard reconstruction algorithms perform a bilinear interpolation of each color plane. This reconstruction performs a strong low-pass filtering, removing most of the aliasing present in the luminance signal. Super-resolution algorithm was applied using the obtained image in YUV 4:2:0. The results are shown in Figures 6 and 7 . After 20 frames, only a small improvement was achieved due to the limited aliasing present in the luminance image.
SmartGreen1  performs a bilinear interpolation over the red and blue images. If high frequencies are present, the green image is obtained from the red or blue images in places where there are no green samples. In those places, the green image is equal to the red- or blue-sampled image multiplied by some predefined coefficients.
If this approach works well, the green image can improve the pixel resolution and, therefore, increases the luminance resolution. This algorithm works better in gray areas, where R = G = B. SmartGreen1 seems to keep more aliasing information, and therefore we expected better performance when used in combination with super-resolution. Results are shown in Figures 8 and 9 .
SmartGreen3 requires more computational load. In previous versions of SmartGreen, the computed green image from the red and blue values was used directly. This approach worked well in gray areas but made colorful images grayer. In SmartGreen3, this image is only used to enhance the edges and to help in the detection of false color due to color aliasing. The main issue with this algorithm is that it performs a band-pass filtering of the luminance image, removing aliasing. In consequence, we will not increase the quality of the image by using super-resolution. Due to the good performance of this technique, it can be a reference of sharpness for SR as one of the best qualities that the color reconstruction algorithms can achieve. The results obtained using these algorithms are shown in Figures 10 and 11 .
The performance obtained using standard reconstruction or SmartGreen1 with SR is not better than using SmartGreen3. Moreover, the improvement of using SR in conjunction with SmartGreen3 is very small, so it is not very wise to apply SR in that case due to the computational load increase.
6. Color reconstruction using super-resolution
As the results obtained were not very satisfactory using the previous approaches, a new approach was tried. Since each color image is subsampled, a large amount of aliasing is present in each one of these images. In fact, when using the Bayer pattern, the luminance image is never fully sensed because it is composed by sampled values of one color and reconstructed values from the other two color images. Because the sensor works in RGB, it is better to work directly in this domain.
Following this idea, it can be seen that the way that the sensor works is similar to our approach to get aliasing in previous steps. Indeed, the red and blue images are subsampled by a factor of two in both directions, and the green image is subsampled, but losing only half of the samples.
Therefore, the proposed idea is to reconstruct each color image as color reconstruction algorithms do, obtaining pixel resolution using super-resolution. This idea is presented in Figure 12 . In this application, each color image is treated as the luminance signal was before, but the motion estimation is done based only on the green values. In this way, we can use SR to replace the color reconstruction algorithm. The first image in the sequence is equivalent to standard reconstruction, but when new images come to the system, the quality increases with the new incoming information.
To avoid a large amount of calculations, the motion vectors are computed only over the green values, and the same motion vectors are applied to the red and blue matrixes. Another reason to do it in that way is that different motion vectors for different colors can produce color aliasing due to different calculated movements of each color. For this reason, the motion vectors must be coherent over the three color planes.
In this case, SR achieves good results. The images obtained are comparable with SmartGreen3 ones, but there are some issues that remain to be tackled:
In contrast with traditional color reconstruction algorithms, more than one image is needed to increase the quality.
Errors in the motion estimation will produce bad results. The motion estimation algorithm must be very accurate, or it has to be used only in certain scenes.
A large number of calculations are needed to perform the motion estimation.
On the other hand, the most important advantages obtained are:
It works better in colorful images, while SmartGreen use to shift the false colors to the gray scale.
In case of good motion estimation, the color aliasing is better suppressed.
Moreover, another approach was tested, where the super-resolution with aliasing (SRA) is applied after the overall processing in the image. This processing increases the differences between the values in the RGB domain. For example, the matrix correcting process and the gamma correction provide more differentiate values. This can be useful for the motion estimator to increase the accuracy of the motion vectors. In this case, the proposed process is to follow the next steps:
Reconstruct the color using standard reconstruction algorithms, keeping the original values. This is necessary because other processing need R, G, and B data in all the positions.
Do all the processing: matrix correction, gamma correction, white balance, etc.
Take the sampled values from the original positions. These values are only modified by the previous processing and not from the color reconstruction algorithms. Using these values, we execute the super-resolution algorithm and reconstruct the lost colors.
With this approach, the obtained results are shown in Figures 13 and 14 . As it is usual in this kind of processing, where no reference image is available, the results are only qualitative, but in Figure 15 , a better color reconstruction is appreciated, with less color aliasing in all the colors, but in special in the green color, which is less shifted to the gray scales.
7. Color reconstruction and resolution enhancement
Following the line of the last two applications of Sections 5 and 6, it is possible to join both proposals in a single one. The new idea is to increase the pixel resolution, using each color signal instead of the luminance signal.
In Section 3, it was shown that there are two main sources of aliasing: the color sampling and the fill factor of the sensor. Using these concepts, we can exploit both sources of aliasing in a new application. This new application will deal with color reconstruction and zoom using SR, according to the scheme presented in Figure 16 . Taking into account a single basic block, 16 pixels are reconstructed from two green values and from one blue and red value. Therefore, more images will be necessary to obtain a good quality, eight in the ideal case of having all the missing positions available.
The processing followed in this case consisted in:
Reconstructing the color using a standard reconstruction algorithm, keeping the original values.
Performing all the other processing: matrix correction, gamma correction, white balance, etc.).
Taking the sampled values from the original positions. These values are only modified by the previous processing and not from the color reconstruction algorithms.
Applying SR, using a 4 × 4 matrix instead of a 2 × 2 as in the previous cases.
In this case, the motion vectors are also calculated using the green values and reused for the other color matrixes. The usual way to obtain these images is performing first a color reconstruction algorithm followed by a bilinear interpolation. This approach was used in the images of Figures 17 and 18 . These images are used as a subjective reference to compare against the other images. Following our proposal, some experiments were developed, obtaining the results shown in Figure 19 .
The obtained results exhibited good quality, although some advantages and disadvantages were found in this new approach. The main disadvantages are:
More than one image is required to obtain good quality. Nevertheless, this is not a problem for video sequences.
Bad motion estimation can reduce its quality to bilinear interpolation quality, worse than SmartGreen3 quality.
It works in RGB data, and it is still necessary to convert to YUV format in some cases, as in video compression.
It is necessary to suppress the aliasing when super-resolution is not performed.
Also, some advantages are appreciated:
Better resolution and suppression of color aliasing, achieving a good electronic zooming.
Reuse of the existing hardware in combination with video compression, for instance, the motion estimation unit.
Take advantage of undesirable characteristics as aliasing and hand motion.
Use of signal processing instead of mechanical parts to perform the zoom, contributing in this way to reduce the overall device power dissipation.
In this chapter, several applications have been addressed using a real acquisition system. A study of the system has been carried out, analyzing the processing flow in order to identify the possible sources of aliasing. An important problem in this stage is the absence of a reference image to be used in order to obtain quantitative metrics.
Three main applications related with electronic cameras and the image processing and acquisition chain have been addressed:
Resolution enhancement, where three available color reconstruction algorithms have been studied in order to keep the aliasing at the input. As these algorithms do not allow passing a great amount of aliasing, the SR enhancement is not very noticeable.
Color reconstruction, where it is exposed how to apply SR as a way to reconstruct the missing colors when a single sensor is used with a CFA.
Color reconstruction and resolution enhancement, where the two previous applications are combined. In this last case, good results are achieved, making this kind of techniques for such applications recommendable.