1. Introduction
Multispectral images usually present complimentary information such as visual-band imagery and infrared imagery (near infrared or long wave infrared). There are strong evidences that the fused multispectral imagery (in gray scales) increases the reliability of interpretation (Rogers & Wood, 1990; Essock et al., 2001) and thus good for machine analysis (computer vision); whereas the colorized multispectral imagery improves observer performance and reaction times (Toet et al. 1997; Varga, 1999; Waxman et al., 1996) and thus good for visual analysis (human vision).
Imagine a nighttime navigation task that may be executed by an aircraft equipped with a multispectral imaging system. Analyzing the synthesized (fused or colorized) multisensory image will be more informative and more efficient than simultaneously monitoring multispectral images such as visual-band imagery (e.g., image intensified, II), near infrared (NIR) imagery, and infrared (IR) imagery, which may be displayed either on several split panels on a big screen or on several small screens. The focus of this chapter is how to synthesize a color presentation of multispectral images in order to enhance night vision. It is anticipated that the successful applications of night vision colorization techniques will lead to improved performance of remote sensing, nighttime navigation, target detection, and situational awareness. This colorization approaches mentioned here involve two main techniques, image fusion and colorization, which are briefly reviewed as follows, respectively.
Two commonly used fusion methods are the discrete wavelet transform (DWT) (Pu & Ni, 2000; Nunez et al., 1999) and various pyramids (such as Laplacian, contrast, gradient, and morphological pyramids) (Jahard et al., 1997; Ajazzi et al., 1998), which both are
On the other hand, a
Toet (2003) proposed a night vision (NV) colorization method that transfers the natural color characteristics of daylight imagery into multispectral NV images. Essentially, Toet’s natural color-mapping method matches the statistical properties (i.e., mean and standard deviation) of the NV imagery to that of a natural daylight color image (manually selected as the “target” color distribution). However, this color-mapping method colorizes the image regardless of scene content, and thus the accuracy of the coloring is very much dependent on how well the target and source images are matched. Specifically, Toet’s method weights the local regions of the source image by the “global” color statistics of the target image, and thus will yield less naturalistic results (e.g., biased colors) for images containing regions that differ significantly in their colored content. Another concern of Toet’s “
In this chapter, we will discuss and explore how to enhance human night vision by presenting a color image with a set of multispectral images. Certainly, a color presentation of multispectral night vision images can provide a better visual result for human users. We would prefer the color images resembling natural daylight pictures that we are used to; meanwhile the coloring process shall be efficient enough ideally for real time applications. A segmentation-based colorization procedure is first reviewed, and a channel-based color fusion is then introduced. The remainder of this chapter is organized as follows. The multispectral image preprocessing, registration and fusion are described in Section 2. Next, the
2. Multispectral image preprocessing
The multispectral images that we acquired include visible (RGB color) images, image intensified (II, enhanced visible) images, near infrared (NIR; spectral range: 0.9~1.7 μm) images, and long-wave infrared (LWIR; spectral range: 7.5~13 μm) images. Before performing multispectral colorization, image preprocessing, image registration, and image fusion are required.
2.1. Standard preprocessing
Standard image preprocessing such as
Night-vision images (NIR and LWIR) were acquired under different background and conditions, which may cause images to have different background (brightness) and contrast (dynamic range). We employed a general
where IN is the normalized image,
The image contrasts of near infrared (NIR) images are significantly affected by illumination conditions. Nonlinear enhancement like histogram equalization or histogram matching usually increases noises while enhancing a NIR image. A linear enhancement such as
2.2. Image registration
We used the FMT method only accounting for translation alignment although it can be alternated for scaling and rotation (but not reliable). The image alignment by scaling and rotation is accomplished with affine transforms using NMI metric. The image transforming parameters can be estimated by maximizing the NMI value. Calculation of NMI and interpolation of transforming (e.g., fractional scaling) are quite time consuming. However, the searching spaces of parameters (for scaling and rotation) are small because two cameras are sitting on the same fixture by turns and aiming at the same target. This expedites the registration process on the other hand.
Different FOV of multispectral images is another challenge for image registration. For example, FLIR SC620 camera (used in our experiments) is a two-band imaging device with a LWIR camera (640×480 pixels; FOV: 24˚) and a built-in visible camera (2048×1536 pixels; FOV: 32˚). Before registration with LWIR image cropping the visible image is desired. To find the
2.3. Image fusion
Image fusion is a necessary step for the color fusion discussed in this chapter. Image fusion serves to combine multiple-source imagery using advanced image processing techniques. Laplacian pyramid and DWT-based fusion methods are briefly reviewed, while the details of image fusion were documented elsewhere (Zheng et al., 2005).
The Laplacian pyramid was first introduced as a model for binocular fusion in human stereo vision (Burt & Adelson, 1985), where the implementation used a Laplacian pyramid and a maximum selection rule at each point of the pyramid transform. Essentially, the procedure involves a set of band-pass copies of an image is referred to as the Laplacian pyramid due to its similarity to a Laplacian operator. Each level of the Laplacian pyramid is recursively constructed from its lower level by applying the following four basic steps: blurring (low-pass filtering); sub-sampling (reduce size); interpolation (expand); and differencing (to subtract two images pixel by pixel) (Burt & Adelson, 1983). In the Laplacian pyramid, the lowest level of the pyramid is constructed from the original image.
The regular DWT method is a multi-scale analysis method. In a regular DWT fusion process, DWT coefficients from two input images are fused pixel-by-pixel by choosing the average of the
where
For the detail coefficients (the other three quarters of the coefficients) at each transform scale, the larger absolute values are selected, followed by neighborhood morphological processing, which serves to verify the selected pixels using a “filling” and “cleaning” operation (i.e., the operation fills or removes isolated pixels locally). Such an operation (similar to smoothing) can increase the consistency of coefficient selection thereby reducing the distortion in the fused image.
3. Segmentation-based colorization
In segmentation-based colorization (i.e., local coloring) method, multispectral night vision imagery is rendered segment-by-segment with the statistical color properties of natural scenes by using the color mapping technique. Eventually, the colorized images resemble daylight pictures. The main steps of segmentation-based colorization are given below: (1) A false-color image (source image) is first formed by assigning multispectral (two or three band) images to three RGB channels. The false-colored images usually have an unnatural color appearance. (2) Then, the false-colored image is segmented using the features of color properties, the techniques of nonlinear diffusion, clustering, and region merging. A set of “clusters” are formed by analyzing the histograms of the three components of the diffused image in
3.1. Color space transform
In this subsection, the RGB to
The actual conversion (matrix) from RGB tristimulus to device-independent XYZ tristimulus values depends on the characteristics of the display being used. Fairchild (1998) suggested a “general” device-independent conversion (while non-priori knowledge about the display device) that maps white in the chromaticity diagram to white in the RGB space and vice versa.
The XYZ values can be converted to the
A logarithmic transform is employed here to reduce the data skew that existed in the above color space:
Ruderman et al. (1998) presented a color space, named
The three axes can be considered as an achromatic direction (
3.2. Image segmentation
The nonlinear diffusion procedure has proven to be equivalent to an adaptive smoothing process (Barash & Comaniciu, 2004). The diffusion is applied to the false-colored NV image here to obtain a smooth image, which significantly facilitates the subsequent segmentation process. The clustering process is performed separately on each color component in the
3.2.1. Adaptive smoothing with nonlinear diffusion
3.2.2. Image segmentation with clustering and region merging
The diffused false-colored image is transformed into the
The local extremes (peaks or valleys) are easily located by examining the crossover points of the first derivatives of histograms. Furthermore, “peaks” and “valleys” are expected to be interleaved (e.g., valley-peak-valley-…-peak-valley); otherwise, a new valley value can be calculated with the midpoint of two neighboring peaks. In addition, two-end boundaries are considered two special valleys. In summary, all intensities between two valleys in a histogram are squeezed in their peak intensity and the two end points in the histogram are treated as valleys (rather than peaks). If there are
Clustering is done by separately analyzing three components (
where
where
3.3. Automatic segment recognition
A
Similar to a training process, a look up table (LUT) has to be built under supervision to classify a given segment (sj) into a known color group (
3.4. Color mapping
3.4.1. Statistic matching
A “
where
After this transformation, the pixels comprising the multispectral source image have means and standard deviations that conform to the target daylight color image in
3.4.2. Histogram matching
4. Channel-based color fusion
The segmentation-based colorization described in Section 3 can usually produce colorized night-vision images closely resembling natural daylight pictures. However, this segmentation-based coloring procedure involves many processes and heavy computations such as histogram analysis, color space transform, image segmentation, and pattern classification. It will be a grand challenge for real time applications. Therefore, we propose a fast color fusion method, termed as
The general framework of channel-based color fusion is as follows, (i) prepare for color fusion, preprocessing (denoising, normalization and enhancement) and image registration; (ii) form a color fusion image by properly assigning multispectral images to red, green, and blue channels; (iii) then fuse multispectral images (gray fusion) using
In night vision imaging, there may be two or several bands of images available, for example, visible (RGB), image intensified (II), near infrared (NIR), medium wave infrared (MWIR), long wave infrared (LWIR, also called thermal). The discussions of following subsections focus on how to form a channel-wise color fusion with the available multispectral images.
4.1. Color fusion with two-band images
Upon the available images and common applications, we will discuss two-band color fusion of (II
4.1.1. Color fusion of (II ⊕ LWIR)
Suppose a color fusion image (FC) consists of three color planes, FR, FG, FB, the color fusion of II and LWIR images are formed by using the following expressions,,
where
4.1.2. Color fusion of (NIR ⊕ LWIR)
A color fusion of NIR and LWIR is formulated by,
where I_Gmax =
4.1.3. Color fusion of (RGB ⊕ LWIR)
Two-band color fusion of RGB and LWIR is described as follows,
where IRed, IGreen and IBlue are the three channel images of a RGB image; I_Rmax =
4.1.4. Color fusion of (RGB ⊕ NIR)
The color fusion of RGB and NIR is defined as,
where I_Gmax =
4.2. Color fusion with three-band images
Due to the available image databases, we only discuss one application of three-band color fusions, (RGB
where I_Gmax =
5. Experimental results and discussions
Two sets of multispectral images were used in our experiments, which were taken at night time and referred as to “NV-set 1” and “NV-set 2”. In NV-set 1, three pairs of multispectral images (as shown Figs. 1-3), image intensified (II) and long wave infrared (LWIR), were analyzed by using the
The two input images and the fused images used in the coloring process are shown in Figs. 1-3a, Figs. 1-3b and Figs. 1-3c, respectively. The image resolutions are given in figure captions. Two input images in NV-set 1 were preregistered. The false colored images (not shown in Figs. 1-3) were obtained by assigning image intensified (II) images to blue channels, infrared (IR) images to red channels, and providing averaged II and IR images to green channels. The rationale of forming a false-color image is to assign a long-wavelength NV image to the red channel and to assign a short-wavelength NV image to the blue channel. The number of false colors were reduced with the nonlinear diffusion algorithm with AOS (additive operator splitting for fast computation) implementation that facilitated the subsequent segmentation. The segmentation was done in
merging operations (the clustered images are not shown in Figs. 1-3). The parameter values used in clustering and merging are
Two-band channel-based color fusion (described in Eqs. 9) was applied to the II and LWIR images (shown in Figs. 1-3a, b), and the results are illustrated in Figs. 1-3f. The color fusion results are very good especially in representing vegetation. Compared to the segmentation-based colorization results, the channel-based color fusion seems less realistic such as the sky and roads shown in Figs. 1-2f. However, the processes of channel-based color fusion eliminate the needs of segmentation and classification, and also reduced the color transforms. The processing speed of is much faster than that of segmentation-based colorization.
In NV-set 2, four pairs of multispectral images (as shown Figs. 4-7), color RGB, near infrared (NIR) and long wave infrared (LWIR), were analyzed by using the channel-based color fusion algorithm as described in Section 4. The results of channel-based color fusion are presented in Figs. 4-8.
The three-band input images used in the color fusion process are shown in Figs. 4-7a, b and c, respectively. The image resolutions are given in figure captions. The RGB images and LWIR images were taken by a FLIR SC620 two-in-one camera, which has LWIR camera (of 640×480 pixel original resolution and 7.5~13 μm spectral range) and an integrated visible-band digital camera (2048×1536 pixel original resolution). The NIR images were taken by a FLIR SC6000 camera (640×512 pixel original resolution and 0.9~1.7 μm spectral range). Two cameras (SC620 and SC6000) were sat on the same fixture by turns and aimed at the same direction. The images were captured during sunset time and dusk time in fall season. Of course, image registration as described in Section 2.2 was applied to the three band images shown in Figs. 4-7, where manual alignments were employed to the RGB images shown in Figs. 6-7a since those visible images are so dark and noisy. To better present the RGB images, contrast and brightness adjustments (as described in figure captions) were applied. Notice that piecewise contrast stretching (Eq. 1) was used for NIR enhancements. The fused images using
The two-band channel-based color fusion of (RGB
The segmentation-based colorization demonstrated here took two-band multispectral images (II and LWIR) as inputs. Actually, this segmentation-based colorization procedure can accept two or three input images (e.g., II, NIR, LWIR). If there are more than three bands of images available (e.g., II, NIR, MWIR, LWIR), we may choose the low-light intensified (visual band) image and two bands of IR images. As far how to choose two bands of IR images, we may use the image fusion algorithm as a screening process. The two selected IR images for colorization should be the two images that can produce the most (maximum) informative fused image among all possible fusions. For example, given three IR images, IR1, IR2, IR3, the two chosen images for colorization, IC1, IC2, should satisfy the following equation:
We exhibited the channel-based color fusion with possible combinations of two-band and three-band multispectral images. The processing speed of channel-based fusion is much faster than segmentation-based colorization, while the colors in channel-based fusion are less natural than the colors in segmentation-based colorization. Parameter settings in channel-based color fusion (Eqs. (9-13)) may be varied with different bands of images and with image capturing time and season, which can be conducted and stored before a field application.
6. Conclusions
In this chapter, a set of color fusion and colorization approaches are presented to enhance night vision for human users, which can be performed automatically and adaptively regardless of the image contents. Experimental results with multispectral imagery showed that the colored images contain clear information, and realistic colors. Specifically, the segmentation-based colorization (local-coloring) procedure is based on image segmentation, pattern recognition, and color mapping, which produces more colorful and more realistic colorized night-vision images. On the other hand, the channel-based color fusion procedure generates very impressive color-fusion images using linear transforms and channel assignments, which can be implemented very efficiently for real-time applications. The synthesized multispectral imagery with proposed colorizing approaches will eventually lead to improved performance of remote sensing, nighttime navigation, and situational awareness.
Acknowledgments
This research is supported by the U. S. Army Research Office under grant number W911NF-08-1-0404.
References
- 1.
Ajazzi B. Alparone L. Baronti S. Carla R. 1998 Assessment of pyramid-based multisensor image data fusion , in Proc. SPIE3500 237 248 . - 2.
Barash D. Comaniciu D. 2004 A common framework for nonlinear diffusion, adaptive smoothing, bilateral filtering and mean shift , 22(1),73 EOF 81 EOF . - 3.
Brown L. 1992 A survey of image registration techniques , ACM Comput. Surv.,24 325 376 . - 4.
Burt P. J. Adelson E. H. 1983 The Laplacian pyramid as a compact image code , Com-31 (4),532 EOF 540 EOF . - 5.
Burt P. J. Adelson E. H. 1985 Merging images through pattern decomposition ,575 173 182 . - 6.
Chen Q. Defrise M. Deconinck F. 1994 Symmetric Phase-Only Matched Filtering of Fourier-Mellin Transforms for Image Registration and Recognition , ,16 12 1156 1168 . - 7.
Essock E. A. Mc Carley J. S. Sinai M. J. De Ford J. K. 2001 Human perception of sensor-fused imagery , in , R. R. Hoffman and A. B. Markman, Eds., Lewis Publishers, Boca Raton, Florida. - 8.
Essock E. A. Sinai M. J. et al. 1999 Perceptual ability with real-world nighttime scenes: imageintensified, infrared, and fused-color imagery , 41(3), 438-452. - 9.
Fairchild M. D. 1998 Color Appearance Models , Addison Wesley Longman Inc.,0-20163-464-3 MA. - 10.
Gonzalez R. C. Woods R. E. 2002 (Second Edition), Prentice Hall,0-20118-075-8 Saddle River, NJ. - 11.
Hill D. L. G. Batchelor P. 2001 Registration methodology: concepts and algorithms, in Medical Image Registration , Hajnal, J. V.; Hill, D. L. G.; & Hawkes, D. J. Eds, Boca Raton, FL. - 12.
Jahard F. Fish D. A. Rio A. A. Thompson C. P. 1997 Far/near infrared adapted pyramid-based fusion for automotive night vision , in (IPA97),886 890 . - 13.
Keys L. D. Schmidt N. J. Phillips B. E. 1990 A prototype example of sensor fusion used for a siting analysis , in 1990, ACSM-ASPRS Annual Conf. Image Processing and Remote Sensing 4,238 249 . - 14.
Keysers D. Paredes R. Ney H. Vidal E. 2002 Combination of tangent vectors and local representations for handwritten digit recognition , , ,2396 538 547 , Windsor, Ontario, Canada. - 15.
Nunez J. Otazu X. et al. 1999 Image fusion with additive multiresolution wavelet decomposition; applications to spot1landsat images , J. Opt. Soc. Am. A16 467 474 . - 16.
Perona P. Malik J. 1990 Scale space and edge detection using anisotropic diffusion ,12 629 639 . - 17.
Pohl C. Genderen J. L. V. 1998 Review article: multisensor image fusion in remote sensing: concepts, methods and applications , 19(5),823 EOF 854 EOF . - 18.
Pu T. Ni G. 2000 Contrast-based image fusion using the discrete wavelet transform , 39(8),2075 EOF -2082. - 19.
Rogers R. H. Wood L. 1990 The history and status of merging multiple sensor data: an overview , in Technical Papers 1990, ACSMASPRS Annual Conf. 4,352 360 . - 20.
Toet A. 2003 Natural colour mapping for multiband nightvision imagery ,4 155 166 . - 21.
Toet A. Jspeert I. J. K. 2001 Perceptual evaluation of different image fusion schemes , in: I. Kadar (Ed.), , The International Society for Optical Engineering, Bellingham, WA,436 441 . - 22.
Toet A. Jspeert I. J. K. Waxman A. M. Aguilar M. 1997 Fusion of visible and thermal imagery improves situational awareness in: J.G. Verly (Ed.), , International Society for Optical Engineering, Bellingham, WA,177 188 . - 23.
Varga J. T. 1999 Evaluation of operator performance using true color and artificial color in natural scene perception (Report ADA363036), Naval Postgraduate School, Monterey, CA. - 24.
Waxman A. M. Gove A. N. et al. 1996 Progress on color night vision: visible/IR fusion, perception and search, and low-light CCD imaging , Proc. SPIE2736 96 107 , Enhanced and Synthetic Vision 1996, Jacques G. Verly; Ed. - 25.
Zheng Y. Essock E. A. 2008 A local-coloring method for night-vision colorization utilizing image analysis and image fusion ,9 186 199 . - 26.
Zheng Y. Essock E. A. Hansen B. C. 2005 An advanced DWT fusion algorithm and its optimization by using the metric of image quality index , 44 (3), 037003-1-12. - 27.
Zheng Y. Essock E. A. Hansen B. C. 2004 An advanced image fusion algorithm based on wavelet transform-incorporation with PCA and morphological processing , Proc. SPIE5298 177 187 . - 28.
Zheng Y. Hansen B. C. Haun A. M. Essock E. A. 2005 Coloring Night-vision Imagery with Statistical Properties of Natural Colors by Using Image Segmentation and Histogram Matching , ,5667 107 117 .