Comparison of multi-focus image fusion
Image decomposition is important to image fusion and affects the information extraction quality, even the whole fusion quality. Wavelet theory has been developed since the beginning of the last century. It was first applied to signal processing in the 1980’s, and over the past decade it has been recognized as having great potential in image processing applications, as well as in image fusion. Wavelet transforms are more useful than Fourier transforms, and it is efficient in dealing with one-dimensional point-wise smooth signal [3-5]. However the limitations of the direction make it not perform well for multidimensional data. Images contain sharp transition such as edges, and wavelet transforms are not optimally efficient in representing them.
Recently, a theory for multidimensional data called multi-scale geometric analysis (MGA) has been developed. Many MGA tools were proposed, such as ridgelet, curvelet, bandelet, contourlet, etc [6-9]. The new MGA tools provide higher directional sensitivity than wavelets. Shearlets, a new approach provided in 2005, possess not only all above properties, but equipped with a rich mathematical structure similar to wavelets, which are associated to a multiresolution analysis. The shearlets form a tight frame at various scales and directions, and are optimally sparse in representing images with edges. Only the curvelets has the similar properties with shearlets [10-14]. But the construction of curvelets is not built directly in the discrete domain and it does not provide a multiresolution representation of the geometry. The decomposition of shearlets is similar to contourlets, while the contourlet transform consists of an application of the Laplacian pyramid followed by directional filtering, for shearlets, the directional filtering is replaced by a shear matrix. An important advantage of the shearlet transform over the contourlet transform is that there are no restrictions on the direction numbers. [15-19]
In recent years, the theory of the shearlets, which is used in image processing, has been studied gradually. Now the applications of shearlets are mainly in image denoising, sparse image representation  and edge detection [21, 22]. Its applications in image fusion are still under exploring.
2. Shearlets [12, 20]
2.1. The theory of Shearlets
In dimension , the affine systems with composite dilations are defined as follows.
Where , , are both invertible matrices, and , the elements of this system are called composite wavelet if forms a tight frame for .
Let A denote the parabolic scaling matrix and S denote the shear matrix. For each and ,
The matrices described above have the special roles in shearlet transform. The first matrix controls the ‘scale’ of the shearlets, by applying a fine dilation faction along the two axes, which ensures that the frequency support of the shearlets becomes increasingly elongated at finer scales. The second matrix , on the other hand, is not expansive, and only controls the orientation of the shearlets. The size of frequency support of the shearlets is illustrated in Fig. 1 for some particular values of a and s.
for different values of a and s.
In references , assume , where is the anisotropic dilation matrix and is the shear matrix, which are given by
For , , let be given by
Where is a wavelet, and ; , and . This implies , and .
In addition, we assume that
There are several examples of functions satisfying the properties described above. Eq. (3) and (4) imply that
for any , where , the functions form a tiling of . This is illustrated in Fig.2 (a). This property described above implies that the collection
is a Parseval frame for . And from the conditions on the support of and one can easily observe that the function have frequency support,
That is, each element is support on a pair of trapezoids, of approximate size , oriented along lines of slope .(see Fig.2 (b)).
Similarly we can construct a Parseval frame for , where is the vertical cone,
and , where and are defined as (2) and (3), then the Parseval frame for is as follows,
To make this discussion more rigorous, it will be useful to examine this problem from the point of view of approximation theory. If is a bas is or, more generally, a tight frame for , then an image f can be approximated by the partial sums
Where is the index set of the largest inner products . The resulting approximation error is
and this quantity approaches asymptotically zero as increases.
The approximation error of Fourier approximations is , of the Wavelet is , and the approximation error of Shearlets is , which is better than Fourier and Wavelet approximations.
2.2. Discrete Shearlets
It will be convenient to describe the collection of shearlets presented above in a way which is more suitable to derive numerical implementation. For and , Let
Where are defined in section 2. For , each term is a window function localized on a pair of trapezoids, as illustrated in fig.1a. When or , at the junction of the horizontal cone and the vertical cone, is the superposition of two such function.
Using this notation, for , we can write the Fourier transform of the Shearlets in the compact form
The Shearlet transform of can be computed by
Indeed, one can easily verify that
And form this it follows that
3. Multi-focus image fusion based on Shearlets
3.1. Algorithm framework of multi-focus image fusion using Shearlets
3.1.1. Image decomposition
Image decomposition based on shearlet transform is composed by two parts, decomposition of multi-direction and multi-scale.
Multi-direction decomposition of image using shear matrix or .
Multi-scale decompose of each direction using wavelet packets decomposition.
In step (1), if the image is decomposed only by , or by , the number of the directions is . If the image is decomposed both by and , the number of the directions is . The framework of Image decomposition with shearlets is shown in Fig. 3.
3.1.2. Image fusion
Image fusion framework based on shearlets is shown in Fig. 4. The following steps of image fusion are adopted.
The two images taking part in the fusion are geometrically reg is tered to each other.
Transform the original images using shearlets. Both horizontal and vertical cones are adopted in this method. The number of the directions is 6. Then the wavelet packets are used in multi-scale decomposition with .
Fusion rule based on regional absolute value is adopted in this algorithem.
The choice of low frequency coefficients.
Low frequency coefficients of the fused image are replaced by the average of low frequency coefficients of the two source images.
The choice of high frequency coefficients.
Calculate the absolute value of high frequency coefficients in the neighborhood by Eq.(18) Where is the size of the neighborhood, denotes the two source images, is the regional absolute value of image within 3 neighborhood with the center at , means the pixel value at from .
Select the high frequency coefficients from the two source images.
Where is the high frequency coefficients of the fused image.
Finally the region consistency check is done based on the fuse-decision map, which is shown in Eq.(20).
According to Eq.(20), if the certain coefficient in the fused image is to come from source image , but with the majority of its surrounding neighbors from , this coefficient will be switched to come from .
The fused image is gotten using the inverse shearlet transform.
3.2. Simulation experiments
Multi-focus image of Bottle
The following group images are selected to prove the validity proposed in this section.
The two source images, Fig.5.(a) and (b), are the multi-focus images, which focus on the different parts. The fusion methods of these experiments are shearlets, contourlets, Haar, Daubechies, PCA and Laplacian Pyramid (LP). Fusion rule mentioned above is used in this experiment. The following image quality metrics are used in this experiment: Standard deviation (STD), Difference of entropy (DEN), Overall cross entropy (OCE), Entropy (EN), Sharpness (SP), Peak signal to noise ratio (PSNR), Mean square error (MSE) and Q.
Fig.5. (c) is the ideal image, Fig.5.(d) ~Fig.5.(i) are the fused images with different methods. From the subjective evaluation of Fig.6 and objective metrics from Table 1, we can see that shearlet transform have more detail information, disperse the gray level and higher sharpness of the fused image than other methods do.
Multi-focus Images of CT and MRI
The source images are the CT (Computer Tomography) and MRI (Magnetic Resonance Imaging) images. And Entropy (EN), Sharpness (SP), Standard deviation (STD) and Q is used to evaluate the effect of the fused images.
Fig.6 (a) is a CT image, whose brightness has relation with tissue density and the bone is shown clearly, but soft tissue is invisible. Fig.6 (b) is a MRI image, whose brightness has relation with the number of hydrogen atoms in t issue, so the soft t issue is shown clearly, but the bone is invisible. The CT image and the MRI image are complementary, the advantages could be fused into one image. The desired standard image cannot be acquired, thus only entropy and sharpness are adopted to evaluate the fusion result. Fusion rule mentioned above is used in this experiment.
4. Remote sensing image fusion based on Shearlets and PCNN
4.1. Theory of PCNN
PCNN, called the third generation artificial neural network, is feedback network formed by the connection of lots of neurons, according to the inspiration of biologic v is ual cortex pattern. Every neuron is made up of three sections: receptive section, modulation and pulse generator section, which can be described by discrete equation [23-25].
The receptive field receives the input from the other neurons or external environment, and transmits them in two channels: -channel and -channel. In the modulation on field, add a positive offset on signal from -channel; use the result to multiply modulation with signal from -channel. When the neuron threshold , the pulse generator is turned off; otherwise, the pulse generator is turned on, and output a pulse. The mathematic model of PCNN is described below [26-30].
Where is the constant time of decay, is the threshold constant time of decay, is the threshold amplitude coefficient, are the link amplitude coefficients, is the value of link strength, and are the link weight matrix.
4.2. Algorithm framework of remote sensing image fusion using Shearlts and PCNN
When PCNN is used for image processing, it is a single two-dimensional network. The number of the neurons is equal to the number of pixels. There is a one-to-one correspondence between the image pixels and the network neurons.
In this paper, Shearlets and PCNN are used to fuse images. The steps are described below:
Decompose the original images A and B respectively into many different directions via Shear matrixs (In this chapter, ).
Calculate the gradient features in every direction to form feature maps, .
Decompose feature map of all directions using DWT, are high frequency coefficients after the decomposition.
Take into PCNN, and fire maps in all directions are obtained.
Take the Shearlets on original images A and B, the high frequency coefficients in all directions are and , and the low are and . The fused high frequency coefficients in all directions can be selected as follow:
The fusion rule of the low frequency coefficients in any direction is described below:
Where is the variance of .
The fused image is obtained using the inverse Shearlet transform.
4.3. Simulation experiments
In this section, three different examples, Optical and SAR images, remote sensing image and hyperspectral image, are provided to demonstrate the effectiveness of the proposed method. Many different methods, including Average, Laplacian Pyramid (LP), Gradient Pyramid (GP), Contrast Pyramid (CP), Contourlet-PCNN (C-P), and Wavelet-PCNN (W-P), are used to compare with our proposed approach. The subjective v is ual perception gives us direct Comparisons, and some objective image quality assessments are also used to evaluate the performance of the proposed approach. The following image quality metrics are used in this paper: Entropy (EN), Overall cross entropy (OCE), Standard deviation (STD), Average gradient (Ave-grad), , and .
In these three different experiments, the parameters of values of PCNN are showing as follows:
Experiment 1: , and the iterative number is .
Experiment 2: , and the iterative number is .
Experiment 3: , and the iterative number is .
As optical and SAR images, remote sensing image and hyperspectral image are widely used in military, so the study of these images in image fusion are of very important.
Fig.9-11 gives the fused images with Shearlet-PCNN and some other different methods. From Fig.9-11 and Table3, we can see that image fusion based on Shearlets and PCNN can get more information and less distortion than other methods. In experiment 1, the edge feature from Fig. 9(a) and spectral information from Fig. 9(b) are kept in the fused image by using the proposed method, which is showing in Fig.9(c). In Fig.9 (d), the spectral character in the fused image, fused by Contourlet and PCNN, is distorted and the from visual point of view, the color of image is too prominent. From Fig.9 (e)-(f), spectral information of the fused images is lost and the edge features are vague. Fig. 10 are the fused Remote sensing image, which is able to provide more new information since it can penetrate clouds, rain, and even vegetation. With different imaging modalities and different bands, its features are different in each image. In Fig.10(c) and (d), band 8 has more river characteristics but less city information, while band 4 has opposite imaging features. Fig.10 (c) is the fused image using Shearlets and PCNN. The numerical results in Fig.5 and Table 1 show that the fused image based on Shearlets and PCNN keep better river information, and even involve excellent city features. In Fig 10.(d), in the middle of the fused image using Contourlet and PCNN, has obvious splicing effect. Fig.11(c) is the fused Hyperspectral image. Fig.11(a) and (b) are the two original images, The track of the airport is clear in Fig.11(a), however, some planes information are lost. Fig. 11(b) shows the different information. In the fused image, the track information is more clearly, and aircrafts characters are more obvious. But lines on the runways are not clear enough in the fused images using other methods. From Table 3 we can see that most metric values using the proposed method are better than other methods do.
The theory of Shearlets is introduced in this chapter. As a novel MGA tool, shearlets offer more advantages over other MGA tools. The main advangtage of shearlets is that it can be studied within the framework of a generalized Multi-Resolution Analysis and with directional subdivision schemes generalizing those of traditional wavelets. This is very relevant for the development of fast algorithmic implementations of the many directional representation systems proved in the last decade.
In this chapter, we have succeed in demonstrations that shearlets are very competitive formulti-focus image and remote sensing image fusion. As a new MGA tool, Shearlet is equipped with a rich mathematical structure similar to wavelet and can capture the information in any direction. And the edge and orientation information are more sensitive than gray according to human visibility. We take full advantage of multidirection of Shearlets and gradient information to fuse image. Moreover, PCNN is selected as a fusion rule to select the fusion coefficients. Because the character is tics of directional and gradient facilitate motivating PCNN neurons, the more precise image fusion results are gotten. Several different kinds of images, shown in the experiments, prove that the new algorithm we proposed in this chapter is effective.
After development in recent years, the theory of Shearlets is gradually improving. But the time complexity of Shearlets decomposition has been the focus of the study. Which need further study, especially in its theory and applications. We will focus on other image processing methods using shearlets in our future work.
S. G. Mallat, Theory for Multiresolution Signal Decomposition: The Wavelet Representation, IEEE Transaction on Pattern Analysis and Machine Intelligence, 11(1989),pp: 674-693.
A. Krista, Z.Yun, D.Peter, Wavelet Based Image Fusion Techniques — An introduction, review and comparison, International Society for Photogrammetry and Sensing, 62(2007), pp: 249-263.
J. P. Antoine, P. Carrette, R. Murenzi, B. Piette, Image Analysis with Two Dimensional Continuous Wavelet Transform. Signal Processing, 31(1993), pp: 241-272.
F. G. Meyer, R. R. Coifman, Brushlets: A Tool for Directional Image Analysis and Image Compression. Applied and Computational Hamonic Analysis, 4(1997), pp: 147-187.
N. Kingsbury, Complex Wavelets for Shift Invarian Analysis and Filtering of Signals. Applied and Computational Hamonic Analysis, 10(2001), pp: 234-253.
P. Brémaud , Mathematical principles of signal processing: Fourier and wavelet Analysis, New York, 2002.
Y. Xiao-Hui, J. Licheng, Fusion Algorithm for Remote Sensing Images Based on Nonsubsampled Contourlet Transform, Acta Automatica Sinica,34(2008), pp: 274-281.
E. J. Candes, and D. L. Donoho, Continuous curvelet transform. I. Resolution of the wavefront set, Applied Computational Harmonic Analysis, 19 (2005), pp:162-197.
M.N.Do, M.Vetterli, The Contourlet Transform: An Efficient Directional Multiresolution Image Representation, IEEE Transaction on Image Processing, 14(2005), pp: 2091-2106.
M. QiGuang, W. BaoShu, Multi-Focus Image Fusion Based on Wavelet Transform and Local Energy, Computer Science, 35(2008), pp: 231-235.
Wang-Q Lin, The Discrete Shearlet Transform: A New Directional Transform and Compactly supported Shearlet Frame, 5(2010), pp:1166-1180.
G. Easley, Demetrio Labate and Wang-Q Lim, Sparse Directional Image Representations using the Discrete Shearlet Transform, Applied Computational Harmonic Analysis, 25(2008), pp: 25-46.
G.Kutyniok. and D. Labate, Construction of Regular and Irregular Shearlet Frames, Journal of Wavelet Theory and Applications, 1(2007), pp: 1-10.
G. Kutyniok and Wang-Q Lin, Image Separation Using Wavelets and Shearlets, Lecture Notes in Computer Science, 6920(2012), pp:416-430.
K.Guo and D.Labate, Optimally Sparse Multidimensional Representation using Shearlets, SIAM Journal on Mathematical Analysis, 39(2007), pp: 298-318.
G. Kutyniok and D. Labate, Resolution of the Wavefront Set using Continuous Shearlets, Trans. Amer. Math. Soc, 361(2009), pp:2719-2754. Shearlet webpage, http://www.shearlet.org.
K. Guo, W. Lim, D. Labate, G. Weiss, E. Wilson. The theory of wavelets with composite dilations[J]. Harmonic Analysis and Applications. 4(2006), pp:231–249.
K. Guo, W. Lim, D. Labate, G. Weiss, E. Wilson. Wavelets with composite dilations and their MRA properties[J]. Appl. Comput. Harmon. Anal.,20(2006), pp:231–249.
K. Guo, D. Labate and W. Lim, Edge Analysis and Identification using the Continuous Shearlet Transform, Applied Computational Harmonic Analysis, 27(2009), pp:24-46.
G. Kutyniok and Wang-Q Lin, Image Separation Using Wavelets and Shearlets, Lecture Notes in Computer Science, 6920(2012), pp:416-430.
K. Guo, W. Lim, D. Labate, G. Weiss, E. Wilson. Wavelets with composite dilations[J]. Electron. Res. Announc. Amer. Math. Soc., 10(2004), pp:78–87.
R. Eckhorn, H. J. Reitboeck, M Arndt et al. Feature linking via synchronization among distributed assemblies: Simulation of results from cat cortex, Neural Computation, 2(1990), pp: 293-307.
R. Eckhorn, H. J. Reitboeck, M Arndt et al. Feature linking via Stimulus-Evoked Oscillstions: Experimental Results form Cat Visual Cortex and Functional Implications form Network Model. In:Proc Int JCNN, Washington D C. 1(1989), pp:723-730.
W. Jin, Z. J. Li, L. S. Wei, H. Zhen, The improvements of BP neural network learning algorithm, Signal Processing Proceedings ,2000, WCCC-ICSP 2000. 5th International Conference on, 3(2000),pp: 1647-1649.
R. P. Broussard, S. K. Rogers, M. E. Oxley et al, Physiologically motivated image fusion for object detection using a pulse coupled neural network , IEEE Trans. Neural Network,10(1999), pp:554-563.
W. Chen, L.C. Jiao, Adaptive tracking for periodically time-varying and nonlinearly parameterized systems using multilayer neural networks, IEEE Trans. on Neural Networks, 21(2010), pp:345-351.
W. Chen, Z.Q. Zhang, Globally stable adaptive backstepping fuzzy control for output-feedback systems with unknown high-frequency gain sign, Fuzzy Sets and Systems, 161(2010), pp: 821-836.
X.B.Qu, J.W.Yan, Image Fusion Algorithm Based on Features Motivated Multi-channel Pulse Coupled Neural Networks, Bioinformatics and Biomedical Engineering, 1(2008),pp: 2103-2106.
X.B.Qu, C.W. Hu, J.W.Yan, Image Fusion Algorithm Based On Orientation Information Motivated Pulse Coupled Neural Networks, Intelligent Control and Automation, 1(2008),pp: 2437-2441.
G.R. Easley, D. Labate, and F. Colonna, Shearlet Based Total Variation Diffusion for Denoising, IEEE Trans. Image Process. 18(2009),pp: 260-268.