Open access

Multitechnique Fusion of Imaging Data for Heterogeneous Materials

Written By

Kateryna Artyushkova, Jeffrey Fenton, Jabari Farrar and Julia Fulghum

Submitted: 15 October 2010 Published: 24 June 2011

DOI: 10.5772/16903

From the Edited Volume

Image Fusion and Its Applications

Edited by Yufeng Zheng

Chapter metrics overview

2,924 Chapter Downloads

View Full Metrics

1. Introduction

Complete characterization of a complex multicomponent heterogeneous material requires information not only on the surface or bulk chemical components, but also on stereometric features such as size, distance, and heterogeneity in three-dimensional space. Probing vertical structures is equally important for nanocomposite materials with surface segregation, overlayers, concentration gradients or multiple layers. Such complexity of heterogeneous materials makes it difficult to uniquely distinguish between alternative morphologies using a single analytical method and routine data acquisition and analysis.

The combination of sputtering capabilities and high lateral resolution in images led to the wide spread use of three-dimensional imaging studies using Time-of-flight Secondary Ion Mass Spectrometry (TOF-SIMS). (Wucher et al., 2007; Delcorte, 2008; Jones et al., 2008; Rafati et al., 2008) Main drawback of this approach is limited quantitative information. X-ray Photoelectron Spectroscopy (XPS) has benefits of being quantitative and offers very similar capabilities in combining ion sputtering and imaging, and there have been a handful number of studies using this approach. (Gao et al., 2003; Artyushkova, 2010) Disadvantage of combining sputtering and imaging is its destructive nature and possibility of induced modification that may introduce artifacts within images.

XPS has the advantage of being one of the only surface analysis techniques that provides readily interpretable, surface-specific, chemical information, which is a core analytical method of choice in obtaining surface chemical composition. (Briggs & Grant, 2003) The development of commercial imaging XPS instrumentation has occurred in parallel with imaging developments in other spectroscopic techniques. Improved spatial resolution and decreased analysis time make it possible to correlate XPS analysis with a host of other imaging techniques which have comparable fields of view, but different information content from different depth levels. There is now some field of view (FOV) overlap between XPS and a variety of techniques, including Atomic Force Microscopy (AFM), imaging FTIR, confocal microscopy (CM), SIMS and Secondary Electron Microscopy (SEM). The overlapping FOVs for the techniques listed in Table 1 make correlative data analyses and a fusion of multiple analytical perspectives achievable and valuable for obtaining quantitative structural information in three dimensions.

Imaging using confocal microscopy provides three-dimensional, high-resolution, non-destructive imaging of sample features. (Fellers & Davidso; Pawley, 2006) Scanning permits many slices to be imaged and high resolution, three-dimensional volumes may be created for a variety of specimens in a non-destructive fashion. Topographic and phase imaging AFM may be used for obtaining topographical and chemical information from surface/air interface of materials. (Behrend et al., 1999; Garcia et al., 1999; Garcia & Perez, 2002) By using relatively large scanned areas, the AFM images can be correlated with XPS and CM.

Table 1. Properties of analytical techniques of choice for image fusion

In recent years, a lot of research has been directed towards visualization of scientific data from various sophisticated, but unique visualization perspectives. (Adriaensens et al., 2000; Varshney, 2000; Peri, 2001; Viergever et al., 2001; Mahler, 2004; Artyushkova & Fulghum, 2005; Matsushita et al., 2007; Baum et al., 2008; Macii et al., 2008; Rieder et al., 2008; Ropinski et al., 2009; Artyushkova, 2010) This includes problem of visualizing data from multiple modalities (multimodal) and visualizing data by combining perspectives within the same modality (unimodal). In multimodal visualization image data from different sensors that use different physical principles are combined to form a new image that contains more interpretable information that could be gained using the original information.

Table 2. Benefits of technique fusion

Data fusion is concerned will the problem of how to combine data from multiple sensors to perform inferences that may not be possible from a single method alone. Benefits of technique fusion are shown in Table 2. Combining data acquired from the same area on a sample by different techniques should reveal more information than would be obtained if each data type was processed separately. Fusing imaging data from XPS, CM and AFM will be discussed in the current chapter for the purpose of obtaining fused volume representing quantitative structural morphological information from multicomponent heterogeneous systems, such as polymer blends. For such direct correlative studies, a variety of issues must be addressed. The techniques sampling properties, including sampling depth, field of view, spatial resolution and image information content must be considered in designing experiments that result in the acquisition of data from the same area. Correlating the data requires area marking, image matching, feature selection, image alignment, image registration and, finally, image fusion.

Advertisement

2. Experimental

2.1. Material preparation

Poly (vinyl chloride) (PVC), poly (methyl methacrylate) (PMMA) were used as received from Scientific Polymer Products, Inc. Fluorescein-labeled poly(styrene) (PS*) (excitation λ= 494 nm, emission λ = 518 nm) and poly(butadiene) (PB, MW = 233.0 kDa) were obtained from Polysciences, Inc. MEH-PPV (Poly(2-methoxy-5-(2’-ethyl-hexyloxyl)-p-phenylene vinylene of 1000kDa was used as received from from H.W. Sands Corporation.

For PVC/PMMA blends, films were prepared by combining the PMMA and PVC in a 2% w/v solution in HPLC-grade tetrahydrofuran. 0.2 wt% Rhodamine dye is added. The solutions of 50/50 mixtures were allowed to sit for at least 24 h and deposited on Teflon® watch glasses using pipettes. Films were allowed to air dry for 24-48 h before peeling and analysis of both air- and substrate- side.

For PS*/PB and PS*/PMMA blends, as received materials were used in a 2% (w/v) solution in HPLC grade toluene. Solutions containing a 50/50 mixture of the two polymers were allowed to sit for 24h before being solvent cast onto silicon wafers.

10/90 PMMA/PPV blend was made from 2% (w/v) solutions of PMMA and MEH-PPV dissolved in toluene and deposited onto a cleaned glass slides.

2.2. XPS analysis

The XPS spectra and images were acquired on a Kratos AXIS Ultra photoelectron spectrometer using a monochromatic Al Kα source operating at 300W. The base pressure was 2x10-10 torr, and operating pressure was 2x10-9 torr. Charge compensation was accomplished using low energy electrons. Standard operating conditions for good charge compensation are -3.1 V bias voltage, -1.0 V filament voltage and filament current of 2.1 A. For all samples medium magnification images of 350x 350 microns in size were acquired.

For PVC/PMMA sample, O 1s, Cl 2p and C 1s images were acquired for 3 minutes each at pass energy (PE) of 80 eV. High resolution C 1s 55 micron spectra were acquired from bright and dark areas selected within Cl 2p images for 6 minutes at PE 20 eV.

For PS*/PMMA sample, 7 images were acquired at following binding energies (BEs): 5 type of C (288.5, 286.1, 283.7,282.3, 281.7) and 2 types of (531, 529.3) for 3 minutes and 80 eV PE. Small area O 1s and C 1s 55 micron spectra were acquired from 4 areas within images for 6 minutes each at 20 eV PE.

For PPV/PMMA sample, 6 images were acquired at following BEs: 4 types of C (286.5, 285.8, 282,8, 281.5) and 2 types of O (530.4, 528.8). Small area O 1s and C 1s 55 micron spectra were acquired from 4 areas within images for 6 minutes each at 20 eV PE.

For PS*/PB blend, an images-to-spectra dataset was acquired. In this experiment, images are acquired as a function of binding energy (BE) across the range of interest, in this case the C 1s peak, from 289 eV to 279 eV with a 0.2 eV step. This corresponds to a binding energy range from 291 eV to 281 eV after charge correction. Pass energy of 80 eV and acquisition time of 2 minutes per image was employed.

2.3. Confocal analysis

Confocal images were obtained with Zeiss LSM 510 confocal laser-scanning microscope. For PVC/PMMA blend with rhodamine dye Helium Neon laser (543nm excitation wavelength) was used. For PS* containing blends and PPV/PMMA blends argon-ion laser with an excitation wavelength of 488nm in the single channel mode. 20x dry and 60x oil objectives were used at iris diameter of 1.4. Depth series at 0.5-2 micron step were acquired depending on the objective used.

2.4. AFM analysis

AFM images were obtained on a Digital Instruments Multimode NanoScope IIIa scanning probe microscope. Height and phase images were recorded simultaneously under ambient conditions in tapping mode. Commercial Si3N4 cantilevers with force constants of 2.5-8.5 N/m, and resonance frequencies between 120-190 kHz were used.

2.5. Software

GUI (Graphical User Interface) for Image processing and fusion was written in house in Matlab. (The MathWorks) The GUI included all the necessary steps for image fusion, such as multivariate image analysis, image preprocessing using variety of filters, resizing, cropping, rotating, image registration using MI, concentration mapping, image morphing and visualization using rendered volumes. The image PCA routine in the PLS_Toolbox 5.2(Eigenvector Research) in MATLAB was used to extract principal component images and loadings. Rigid registration (translation and rotation) using MI was utilized using in-house-written routine in Matlab using MI function by Dennis Lucero. Morphing algorithm of Beier and Neely was implemented using Dean Krusienski’s routine.

Advertisement

3. Results and discussion

In attempt to completely understand structure and chemistry of heterogeneous multicomponent materials using direct correlative multitechnique analysis, several steps have to be undertaken: the same area on the sample has to be analyzed by multiple analytical methods of choice, spectra and images acquired have to be processed and analyzed, and resulted information has to be fused to provide a final image representative of the 3D sample structure.

The heterogeneity in chemistry of the sample must be reflected in all of the analytical methods chosen for the analysis with understanding that each analytical imaging method has different mechanism of generating contrast. Chemical sensitivity of XPS offers great advantage for this purpose, however in some cases more elaborate experimental schemes might be needed for distinguishing between similar polymers in multicomponent polymeric samples or for increasing signal to noise ratio. Several ways to generate a fluorescent signal in CM can be thought of, the first being addition of the dye into the mixture with assumption that it will partition preferentially in one of the mixture constituents, the second is the use of fluorescently tagged polymers and the third, use of natively fluorescent polymers.

The next important consideration is different sampling depth of the technique. This being a main advantage of using multitechnique characterization, serves as somewhat limiting factor in terms of sample preparation requirements imposed by each method and types of data that can be obtained. Vertical heterogeneity may result in concentration gradients and/or surface segregation, while lateral heterogeneity may result in topographical heterogeneity. The AFM method is the most sensitive to the topography of the surface, and there is a limit of topographical height difference that instrument can handle. Large features of phase-separation, which can be easily detected in XPS and other methods and can be beneficial for registration purposes, can increase topographical heterogeneity to the point that AFM images of reasonably large size will be impossible to obtain. The next method of surface sensitivity is XPS, which is sampling top 8-10 nm of the surface. For some of the polymeric systems, big differences in macroscopic parameters of the constituents, can cause significant surface segregation, where within the bulk of the sample, there will be phase separation of the mixture and as the consequence, it will be reflected in CM images, while the top surface may be a homogeneous layer of one of the polymer resulting in featureless XPS images.

AFM is a limiting technique in the physical size of the images that can be acquired. The larger the heterogeneity of the sample, the smaller image size can be acquired. It is clear, that all other methods that can handle larger FOV’s will be reduced to the AFM area of analysis.

Three types of polymer blends were designed for the purpose of demonstrating different approaches of image fusion. The first is 50/50 PVC/PMMA blend where the Rhodamine dye has been added to provide the fluorescent contrast for confocal microscopy. This blend represents the well-characterized system, as our and other groups extensively characterized it previously. Enough knowledge about the morphology of this blend allows us to consider it as test system, as it is well known how the composition changes laterally and vertically. In this blend, however, there is a problem of not 100% exclusive preferential dissolution of dye in one of the blend constituents. Another type of blend are mixture of fluorescin-labled PS (PS*) with PMMA, PB and PVC, i.e. PS*/PMMA, PS*/PB and PS*/PVC. The problem of partial dissolution of dye in both polymers is solved by this approach, as fluorescence is coming exclusively from the labeled polymer making confocal data interpretation less ambiguous. And the third blend used in this report is a mixture of natively fluorescent polymer, MEH-PPV with PMMA. This represents another way of solving ambiguities of confocal data interpretation. Throughout the discussion we will use examples from all three types of polymer blends.

Following steps required to be done prior to image fusion will be discussed now:

  1. Area marking

  2. Image acquisition

  3. Pixel-to-feature conversion

  4. Image registration of feature-level images: -resizing, preprocessing, AIR using MI

  5. Registration of original data on pixel level

3.1. Area marking

The task of marking an area on the sample so that it can be easily located within each analytical method of choice and, at the same time, don’t introduce damage or artifacts to the sample have been approached through a variety of means. The simplest approach is to mark sample areas with a felt pen, but this becomes difficult when the analysis area approaches the dimensions of the pen tip. Optical features on the sample provide internal marks, but may not always be present. TEM grids have also been used to locate areas, but the height differential between the grid and the sample is frequently greater than the vertical range of the AFM scanner, not allowing imaging the grid within FOV of the image.

An alternative method for marking an analysis area is AFM lithography, which allows the outline of the analysis area to be drawn using contact mode AFM. (Sugimura & Nakagiri, 1997; Cleveland et al., 1998; Ngunjiri & Garno, 2008; Lu et al., 2009) The tip is scanned under load forces that are experimentally determined to remove sample from the surface. We’ve successfully used AFM tip to introduce marking lines for several samples of varying morphologies (patterned and polymer blends). (Artyushkova et al., 2009)

Figure 1 shows optical image of the sample with a grid and two sets of vertical lines drawn by AFM tip. Grid helps to locate an area in confocal and XPS analysis, especially when lines are not observable in chemical images, which sometimes might be the case.

Figure 1.

Optical image of the sample with grid and sets of lines

3.2. Image acquisition

After the lines have been drawn, the AFM images between lines are acquired. The exact distance from left top line is measured to the location of a center of AFM images. This knowledge is very useful in checking the output results from image registration as will be discussed further. Figure 2 shows height and phase images acquired from PS*/PMMA sample.

Marks introduced by AFM tip can be easily located in all other instruments using optical cameras. Confocal image from the same sample is shown in Figure 2. The high intensity corresponds to fluorescent PS-enriched phase of the blend. The lines marked by AFM are readily visible. From the distance between the top of the line to the center of AFM images, one can register AFM image within confocal image. As discussed below, results of automatic image registration can be evaluated based on this knowledge.

Figure 2.

Confocal, AFM and XPS images from marked area on PS*/PB sample. Lines are visible in CM but not in XPS chemical images

When the sample is transferred for the analysis into XPS spectrometer, the optical camera is used to locate the analysis area. The lines marked are readily visible in optical images. Chemical images are then acquired from the matched area. Figure 2 shows an XPS C 1s image acquired at BE of the main peak – 285 eV after charge correction. Lines are not visible, but having an optical image as reference, one can be certain that this XPS image is acquired from exactly the same area as AFM and confocal images.

3.3. Feature selection

Multisensor fusion can take place at the pixel, feature or symbol level of representation. (Varshney, 2000; Peri, 2001; Mahler, 2004; Macii et al., 2008) In pixel-level fusion, a new image is formed through the combination of multiple original images to increase information content associated with each pixel. Pixel-level fusion is recommended for images with similar exterior orientation, similar spatial, temporal and spectral resolution, and capturing similar physical phenomena. When images record information from very different chemical or physical phenomena, if they are collected from different platforms, or have significantly different sensor geometry, preference should be given to the feature-level fusion, where original images are converted to the images representative of the features, i.e. segmented images, for example.

In case of multitechnique fusion using XPS, AFM and confocal microscopy, images are acquired from different platforms sampling different physical and chemical properties. Moreover, multiple images per each technique are available, so they have to be converted to the feature images representing particular chemical component or phase. Depth array is acquired in CM. Several elemental and chemical XPS images are usually acquired. Often images-2-spectra data are acquired XPS, which consist of 50-70 images within the binding energy range of element of interest with small binding energy step. Two AFM images are usually acquired, one being representative of the height and another of the phase image. These multidimensional multivariate images may be converted to a single image representing features of interest for feature-based data fusion.

Image segmentation, classification or Multivariate Image Analysis (MIA) are among the methods capable to extract feature information from multivariate images. The goal of MIA methods is to extract the most significant information from an image data set, while reducing the dimensionality of the data. Principal Component Analysis (PCA) transforms multivariate images into a number of score images. (Artyushkova & Fulghum, 2001; Artyushkova & Fulghum, 2004) The first principal component accounts for as much of the variability in the data as possible, and each succeeding component accounts for a decreasing amount of the remaining variability. The objective of PCA is to identify images which are globally correlated or anti-correlated. This information is displayed as loadings of the different images, and the pixels responsible for the correlations can then be displayed in component score images. PCA is a method of choice for feature selection due its simplicity, uniqueness of solution and speed.

Data from PPV/PMMA blend are used as an example of using PCA as feature-selection method from both XPS and CM domains. This is an example of the sample, where large clearly-observed features in optical and chemical XPS and CM images allow for easy analysis area identification and matching. 6 chemical photoelectron images are acquired at BE’s corresponding to two types of oxygen and four types of carbon. Principal component analysis of these 6 images results in 3 principal components. Figure 3 shows score images and loadings for these components. The 1st PC has high contribution from all photoelectron

Figure 3.

PCA analysis applied to series of 6 XPS chemical images from PPV/PMMA blend

Figure 4.

PCA analysis applied to series of confocal depth images

images and results in intense and most homogeneous distribution. It has been shown that the 1st component is a representative of mixture of topographical and inelastic background. (Artyushkova and Fulghum 2004) The 2nd component image is more heterogeneous with loading having high positive contribution due to images #2, 5 and 6 and negative contribution due to image #4. Images 2, 5 and 6 are correspond to the BE where the highest contribution is from PMMA, while image 4 is a representative of PPV. So, PC score image 2 can be considered as being representative of PMMA-enriched phase of the blend and it will be used as a feature-representation of XPS domain.

For feature selection from confocal microscopy imaging data, PCA was applied to 9 confocal images acquired from the same area with 2 micron vertical step (Figure 4). Application of PCA to XPS imaging data allows identification of spatial distribution of chemical phases present in the sample and as the consequence one of the PC scores can be used as feature image representing one of the phases. PCA analysis of confocal depth series serves different purpose than that of XPS data. Each original confocal image within depth series represents PPV-enriched phase at different depth. PCA remove noise and provides confocal image representative of the PPV enriched phases with improved signal-to-noise. Alternatively, one can select most intense and contrast image from the depth array as a feature-representative of confocal data. The 5th and 6th images have a highest contribution into the 1st PC, so either PC score image 1 or original image 5 or 6 can be used as feature-representation of confocal data.

AFM topography and phase images contain very different type of information about the samples morphology; therefore either one or both of them can be used directly for image fusion.

3.4. Image registration

After the images representative of a particular chemical phase have been selected for each technique, they must be registered, or brought to the same spatial grounds.

The first step is bringing them to the same scale in pixel-to-micron ratio. The original scale of images is very different. XPS images are always acquired at 256x256 pixels. The smallest size is 200x200 micron. The pixel-to-micron ratio for XPS data is 1.28. AFM data are always acquired at 512x512 pixels size. The size of the AFM image acquired for all the samples is usually less than 100 microns. For 60x60 microns size image, the pixel-to-micron ratio is 8.53 for AFM data. The size of confocal image can be varied and it is usually less than 200 microns. For an analysis area of 123x185 micron image acquired at 512x768 pixels has the pixel-to-micron ratio of 4.15. As the loss of data is not desirable, it is not practical shrinking AFM and confocal data to match the size of XPS images. At the same time, if one would increase the size of XPS image to match the size of, let’s say, AFM image, it will result in unreasonably large (1700x1700 pixels) size leading to excessively time- and computer- consuming image registration. Therefore, in each individual case of matching sizes of images in terms of their pixel-to-micron ratio, compromise should be found. For example, if confocal image is decreased in two times to a size of 256x384 pixels, the size of AFM image that matches the pixel-to-micron ratio of confocal is 124x124 pixels, while that for XPS becomes 416x416 pixels. These are reasonable sizes of images to process further.

The next step is spatial transformation of images to bring them to the same spatial coordinates. From the way the sample is positioned with respect to the detectors in all three instruments, only translation and rotation must be involved in spatial transformation, which is an example of rigid linear transformation. Manual registration may involve setting up ground control points (GCPs), i.e. points whose coordinates are known for both reference and target images. Manually selected GCPs are used for calculating linear rigid spatial transformation. This approach requires distinct features observable in all images, which is not always the case (see XPS image in Figure 2). It is also subjective to human errors. Alternative approach is automatic image registration (AIR) which iteratively adjusts spatial transformation parameters (rotation and translation) so as to maximize some similarity measure computed between the transformed target image and the corresponding reference image. (Black et al., 1996; Viergever et al., 2001; Zitova & Flusser, 2003; Bentoutou et al., 2005; DelMarco et al., 2007; Liu et al., 2008) There is a variety of automatic intensity based measures, which do not require the definition of GCPs or features.

The criterion of maximization of mutual information (MI) has proven to be a breakthrough in the field of image registration. It is a leading method in multimodal registration. (Maes et al., 1997; Chen et al., 2003; Maes et al., 2003; Bentoutou et al., 2005; Luan et al., 2008)The MI, originating from information theory, is a measure of a degree of grey level dependency between two images. The mutual information I of two images 1 and 2 is defined in terms of the entropies H(1) and H(2) of the images and their joint entropy H(1,2), as follows:

I(1,2)=H(1)+H(2)H(1,2)E1

Entropy is a measure of uncertainty; of how well it is possible to predict the grey value of an arbitrary pixel in an image given the probability density distribution function of the grey values. An image containing a large number of different grey values has a high entropy value. If two images are misregistered, the sharpness of the peaks in the joint histogram will decrease. From the definition of mutual information the registration of images depends on maximization of their mutual information.

The strength of mutual information is that generally it does not require preprocessing of the images. It is more suitable for multimodality image matching than other statistical measures, such as cross-correlation measures, which rely on an equivalence relation between the intensities of two images, whereas mutual information depends on the existence of a statistical distribution relation, hence posing less demands on the relation between the images’ grey values.

Figure 5.

Left – MI surface plot calculated for all values of shift in x and y for image rotated by angle Θ. i and j - coordinates of maximum of MI

MI is computed for window pairs from target and reference images and its maximum is searched. The maximum MI provides three parameters, i.e. angle of rotation Θ, and coordinates of i and j for maximum MI. i and j are the coordinates of the left corner of the rectangular area matching the size of target image that has to be cropped out from the rotated reference image by the angle Θ (see figure 5).

To facilitate AIR using MI of very different types of image pairs, various preprocessing routines may be utilized. Different lateral resolution may provide different levels of details in the same features captured by both techniques. Some smoothing filtering may be required to bring the images to be registered to similar details appearance. Sometimes image smoothing is not sufficient. In confocal image, for example, small bright features exist within the big dark feature. One way to simplify the images is to convert them to objects. In the analysis of the objects in images it is essential that we can distinguish between the objects of interest and the background. The techniques that are used to find the objects of interest are usually referred to as segmentation techniques - segmenting the foreground from background. Thresholding produces a segmentation that yields all the pixels that, in principle, belong to the object or objects of interest in an image. Figure 6b) shows thresholded images of original XPS and CM images representative of PVC-enriched phase in PVC/PS* blend in Figure 6a). The intensity distribution is simplified now so that there much fewer possible matches of best registration results. An alternative to this is to find those pixels that belong to the borders of the objects. Techniques that are directed to this goal are termed edge finding techniques. Example of edges extracted from XPS and confocal images is shown in Figure 6c).

Another parameter that should be paid attention to is phase inversion. Bright feature in one image can correspond to dark feature in another image. Image inversion might be necessary as preprocessing step.

Figure 7 shows registered XPS and confocal images for PS*/PVC blend in Figure 6. Large features in both images allow visual estimation of MI registration results as satisfactory.

Figure 8 shows registered images for PS*/PMMA sample. Original images for this blend are shown in Figure 2. AFM and confocal images were registered first. Features in both images are very similar, so the results can be visually estimated as satisfactory. The knowledge of position of the center of AFM images with respect to the line visible within the confocal image allows checking the accuracy of registration. XPS image does not have any distinct features in images to judge the registration results and there are no lines visible. XPS image was registered with AFM and with confocal separately and the fact that both pairs gave exactly the same translation and rotation parameters confirms that the image shown is indeed from exactly the same area as confocal and AFM.

Figure 6.

Image preprocessing. XPS (left) and Confocal (right) images representative of PVC-enriched phase in PVC/PS* blend a) original images, b) thresholded and c) edges extracted.

Figure 7.

Registered XPS (a) and Confocal (b) images for PVC/PS* blend

Figure 8.

Registered XPS, Confocal and AFM images for PS*/PMMA blend

3.5. Image fusion

Now that all the feature-level images are registered and therefore represent different types of chemical and physical information from exactly the same area, they can be fused. We will now discuss different examples of multitechnqiue image fusion.

3.5.1. Example 1. Confocal microscopy and XPS

Fusing confocal and XPS data provides a benefit of extended spatial coverage. CM provides information on how features change with depth, but the data are not quantitative, as intensity in confocal images represents fluorescence which is directly proportional to concentration of the fluorescent phase but exact or even approximate concentrations are unknown. XPS, on the other hand, is a quantitative method. Small area spectra from areas of interest within the images provide concentration of chemical phases. The 1st example of CM-XPS fusion involves use of quantitative information available from XPS top and, if possible, bottom, images to quantify confocal images and to build quantitative confocal volume, where each pixel represents a concentration of a particular chemical phase. The idea behind this fusion is presented in figure 9.

Figure 9.

Quantifying confocal depth series by using quantitative information from small area XPS spectra acquired from both sides of sample

Confocal array from PVC/PMMA blend has been quantified from quantitative chemical information provided by small area XPS spectroscopy. From XPS analysis of this blend it is discerned that the air side of the sample is heterogeneous and enriched in PMMA, while the substrate side is more homogeneous and enriched in PVC. The minimum and maximum concentration of PVC obtained from XPS small area spectra was used to map intensity in the top and bottom confocal images to 22/40 and 55/75, respectively. The intensities in intermediate confocal slices are then mapped by taking the weighted average between those numbers. Figure 10 plots max and min intensities in original and mapped confocal images as a function of depth. The resulted confocal volume representative of PVC enriched phase now can be displayed and analyzed as shown in Figure 11. Rendered volume allows us to see the exterior of the material from top and bottom side of the sample. Isosurfaces at three values of concentration, 30, 50 and 70 % PVC in the blend are displayed. It can be seen that air side is very heterogeneous with small islands with 30% of PVC in them. The 50/50 blend is located in the middle of the sample, while the bottom side is somewhat heterogeneous with higher PVC concentration.

Figure 10.

Quantifying intensities in confocal images. Minimum and maximum values of intensity in a) original and b) quantified images.

Figure 11.

Visualization of quantified confocal volume: Rendered volume – top (a) and bottom (b), three cross-sections, c), Isosurfaces of d)70% PVC, e) 50% PVC and f) 30% PVC

Another way to explore the quantitative confocal volume is to plot intensities within the regions of interest (ROIs) as a function of depth. These are plotted on Figure 12 for 4 ROI’s selected of confocal image representing PVC concentration profiles. The concentration profiles within dark areas are more linear than in bright areas.

Figure 12.

Concentration depth profiles of PVC% in the blend from areas selected on confocal image

3.5.2. Example 2. Confocal microscopy and XPS

The very top surface determines most of the material’s properties. Imaging and small area XPS represents an integral signal from total top 10 nm of the material. The top confocal image can be considered as an integrated image of top 1 micron thickness of the material. 3-D visualization of morphology from imaging data acquired from different depths from the same areas on the sample is a key factor for understanding 3-D structure. The idea behind the 2nd example of fusing CM and XPS images is visualization of changes in morphology and chemistry in the top 1 micron of the sample.

XPS and confocal images representative of PPV-enriched part of PPV/PMMA blend have been registered; images are shown in figure 13. The bright areas are enriched in PPV. The XPS image is then quantified using numbers obtained from small area spectroscopy, being 30% and 65% as lowest and highest concentration of PPV. Confocal image is quantified to approximately the same concentrations of PPV as in XPS image, i.e. 35% as lowest and 70% as largest PPV percentage. Approximately the same sample composition is expected for 1 micron depth. Small degree of concentration gradient is introduced by slight increase of concentration in both dark and bright parts of the image. The main focus of merging XPS and confocal images in this example is on morphological (shape) changes with depth.

The next step is merging these two quantitative images representing spatial distribution of PPV-enriched phase at two different depths into one display. This represents the problem of reconstructing a solid object from a series of parallel planar cross-sections. (Lin et al., 1989) To recreate a volume between quantitative images of polymer blends at different depths, additional slices have to be created in between pairs of images. Interpolation, which produces one or more intermediate images which smoothly and locally turn the 1st image into the 2nd one, is required. A linear interpolation, a simplest type, averages the intensities in two images. However in real heterogeneous samples, where surface segregation may exist, pixels belonging to the same feature in one slice do not necessarily connect to pixels exactly beneath them in the next slice. There are several ways to address this correspondence problem. An example of such approach is image morphing. The morphing is realized by coupling image warping (interpolation of shape) with color interpolation. Image warping applies 2D geometric transformations to the images to retain geometric alignment between their features, while color interpolation blends their color to produce in-between images. (Wolberg, 1998; Tal & Elber, 1999; Artyushkova & Fulghum, 2005; Penska et al., 2007) It begins with establishing correspondence between images with pairs of feature primitives, e.g., mesh nodes, line segments, curves, or points. A pairwise correspondence between two successive images is pre-computed and stored as a pair of morph maps. The feature correspondence is then used to compute mapping functions that define the spatial relationship between all points in both images. The warp function is used to interpolate the positions of the features across the morph sequence. Once both images have been warped into alignment for intermediate feature positions, ordinary color (intensity) interpolation generates in-between images.

Figure 13.

pairs of lines outlining features in both confocal (reference) and XPS (target) images are drawn for calculating warping function for image morphing

Figure 14.

Visualization of volume obtained through image fusion of XPS and confocal image. Rendered volumes from top (XPS) and bottom (confocal) are shown. Three cross-section and isosurface representing 50/50 PPV/PMMA blend are also shown

Lines are used as feature primitives in this image morphing algorithm. First the lines are drawn on the reference image so that they describe the shape of features the best. Then the corresponding lines are drawn on the target image to describe the same features. The more lines you draw the better the morphing results. The features for this blend change shape quite significantly so many lines have to be drawn for an accurate morphology description. The confocal image has more features to describe, so it is used as reference image. Then lines corresponding to the same exact feature in XPS images are drawn. The figure 13 shown both images with 26 pairs of lines defined. 20 images are reproduced between XPS and confocal image using the warping function calculated from pairs of lines.

Figure 15.

Concentration depth profiles showing PPV% within 1 micron of depth from selected areas on XPS and Confocal image.

The visualized volume by rendering, isosurface (50/50 PPV/PMMA composition) and cross-sections are shown in Figure 14. Smooth transition in quite significant changes of morphological features is observed. The most important way to visualize the data is to plot profiles for regions of interest. As the volume represents the real chemical concentration of PPV, one can analyze the morphology this way. Regions of interest shown on XPS and confocal images together with concentration profiles (Figure 15), indicate several important observations. For some dark areas on XPS images (red and blue), the concentration increases a little bit towards dark areas in confocal images. For one of the areas (green), the concentration does not change. For few bright areas on XPS images (magenta, violet), the concentration also increases towards depth. For two other bright areas in XPS images, the concentration decreases with depth, so phase inversion is observed. One can really appreciate the quantitative level of morphological details fusion of XPS and confocal images provides.

3.5.3. Example 3. AFM and XPS

One can approach fusion of XPS and AFM images as the problem of combining images of drastically different resolutions, captured with different instruments and with different spectral characteristics into single image with the goal of facilitating interpretation and analysis. (Nunez et al., 1999; Pohl, 1999; Svab & Ostir, 2006; Ling et al., 2008; Rao et al., 2008; Artyushkova et al., 2009) For example, a typical tradeoff which often occurs in remote sensing is between spatial resolution and resolution in wavelength. It may be possible and desirable to combine a low resolution color image and a high resolution monochromatic image to produce a color image with high spatial resolution. Similarly, X-ray photoelectric spectroscopy (XPS) has very high resolution in spectral dimension (binding energy) but relatively low spatial resolution. In contrast, atomic force microscopy (AFM) images have significantly higher spatial resolution, but can be considered to be monochromatic, since they represent surface topography. The goal of this image fusion type is to improve the lateral resolution of XPS images using high lateral resolution AFM images.

The simplest approach to increasing the spatial resolution of a color image using a monochromatic image of higher spatial resolution is to convert from a red, green, blue (RGB) basis to intensity, hue, saturation (IHS) basis. (Pohl, 1999) While the RGB is a rectangular coordinate system, IHS is a cylindrical coordinate system. After converting the low resolution color image to IHS, the grey values of the monochromatic image are transformed using a simple linear greyscale transformation so that the smallest and largest values correspond to those in the low resolution intensity image. The low resolution intensity image is then replaced with the monochromatic image of higher resolution and the coordinate transformation is inverted, yielding a color image of higher resolution.

There are no RGB channels per se in photoelectron imaging. R, G and B channels expected to be spectrally independent. If one acquires a set of elemental and chemical XPS images, then three independent photoelectron images can be considered as RGB channels. In order for photoelectrons images to be spectrally independent, they have to represent different chemical compounds. For systems with more than two chemical species such as polymer blends which are used throughout this chapter, one can envision selecting three photoelectron chemically independent images as RGB channels, and applying RGB-to-IHS conversion algorithm for resolution merge with AFM image. In two-component systems, however, there are no three spectrally independent images. One image as representative of one of the component of the mixture is an inverse of a representative of another component. So, just a single XPS image (either an original photoelectron image or PC score) representing particular chemical phase present in the mixture is what is usually available in case of polymer blend samples. In this case single photoelectron image can be converted into RGB image by applying false-color mapping to a gray scale image. Concept behind RGB-2-IHS resolution merge for XPS and AFM images is shown in Figure 16. Low-spatial resolution XPS image representing particular chemical phase is color-mapped, this RGB image is converted to HIS components. Intensity component is then being replaced by high spatial-resolution AFM image and inverse transformation of IHS-2-RGB is done to obtain resolution merged image.

This procedure has been tested on XPS and AFM images acquired from PS*/PB blend. In this case images-to-spectra data set has been acquired within C 1s region as described in Experimental section. In order to extract one image representative of either PS* or PB-enriched phase of the blend, PCA was applied to images-to-spectra data set. The 1st PC is

Figure 16.

Concept behind resolution-merging XPS and AFM images.

topographic and phase AFM images from the same area are shown in Figure 17 together with XPS false color-mapped PC score image. According to chemical composition of the blend obtained through small area XPS valence band (VB) spectra, this polymer blend is very homogeneous, i.e. PS*% is ranging from 75 to 85% in darkest and brightest areas within XPS images. The XPS image representative of PS*-enriched phase obtained through XPS shows large features that have slightly different composition but still it is detectable by XPS. AFM image accesses really different level of details due to much higher level of lateral resolution, showing small circular particle like features, while it is not able to detect such small chemical differences between large phase-separated areas. This shows how two techniques can be complimentary due to different sampling properties and different resolution scales. In order to represent this complementary information in one image, AFM and XPS images have been fused. For that, XPS image has been then color-mapped using hot as false color map in Matlab (figure 17). By performing RGB-2-IHS conversion and substituting value of I for AFM topography or phase image and performing IHS-2-RGB conversion, one obtains the resolution-merged images (figure 17). As one can see, in those images, small chemical heterogeneity from XPS images is preserved as color, while a great level of details is now available from AFM high spatial resolution images. One can consider fused images as either AFM images with added chemistry as color or better-quality XPS images. As discussed above, color does not mean a lot in case of XPS images, so one can convert fused images back to original grey color scheme and compare original XPS images to fused grey scale images. The same conclusions can be made from these images. Fusion with phase AFM images gives especially nice correlation that is might due to the fact that phase imaging reflects chemical information.

Another example of resolution-merging XPS and AFM images represent quite a different perspective of this approach. The original registered XPS (O 1s) and AFM images from PVC/PMMA blend are shown in Figure 18. As one can see, the bright features in O 1s image, i.e. PMMA-enriched phase, correspond to higher features in AFM images. Small area spectroscopy confirms that the surface is enriched in PMMA, as composition variation from dark to bright areas on the sample is between 60 to 90% PMMA concentration. The level of

Figure 17.

Resolution-merge of XPS (PS* enriched phase) and AFM images for PS*/PB blend

details provided by both techniques is quite similar, which shows that there is no micro-separation of phases within large phase-separated features, but AFM images shows much better edge information and better resolution. The same procedure of RGB-2-IHS of false-colored PMMA-enriched phase XPS image with AFM image was performed. The merged image shown in Figure 18 does not show exact feature-to-feature correspondence, indicating that the chemical heterogeneity is not in direct correspondence to topography of the surface.

Advertisement

4. Conclusion

Fusing imaging data from XPS, CM and AFM is discussed in the current chapter for the purpose of obtaining fused data representing quantitative structural morphological information from multicomponent heterogeneous systems, such as polymer blends. Steps leading to image fusion discussed in details are area marking, area identification, image acquisition, feature selection, image alignment and image registration. Three examples of image fusion were discussed. In the first, confocal microscope depth series were quantified and visualized based on small area photoelectron spectroscopy acquired from exactly the same areas of polymer blends. In the second, image morphing between image acquired at 10 nm sampling depth by XPS and image acquired at ~1 micron depth by CM provided very detailed information on how morphology of polymer blend changes from the very top surface down into the bulk. And in the third, resolution merge between low spatial resolution XPS image and high spatial AFM image was tested for two types of polymer blend samples.

Combining data acquired from the same area on a sample by different analytical techniques reveals more information than would be obtained if each data type was processed individually.

Figure 18.

Resolution-merge of XPS (PMMA enriched phase) and AFM images for PVC/PMMA blend

References

  1. 1. AdriaensensP.StormeL.CarleerR.VanderzandeD.GelanJ.LitvinovV. M.MarissenR. 2000 Visualization of tensile stress induced material response at a crack tip in polymers under critical load by NMR imaging. Macromolecules, 33 13 (June 2000), 48364841 , 0024-9297
  2. 2. ArtyushkovaK. 2010 Structure determination of nanocomposites through 3D imaging using laboratory XPS and multivariate analysis. Journal of Electron Spectroscopy and Related Phenomena, 178 292302 , 0368-2048
  3. 3. ArtyushkovaK.FarrarJ. O.FulghumJ. E. 2009 Data fusion of XPS and AFM images for chemical phase identification in polymer blends. Surface and Interface Analysis, 41 2 (February 2009), 119126 , 0142-2421
  4. 4. ArtyushkovaK.FulghumJ. E. 2001 Identification of chemical components in XPS spectra and images using multivariate statistical analysis methods. Journal of Electron Spectroscopy and Related Phenomena, 121 1-3 , (December 2001), 3355 , 0368-2048
  5. 5. ArtyushkovaK.FulghumJ. E. 2004 Mathematical topographical correction of XPS images using multivariate statistical methods. Surface and Interface Analysis, 36 9 (September 2004), 13041313 , 0142-2421
  6. 6. ArtyushkovaK.FulghumJ. E. 2005 Angle resolved imaging of polymer blend systems: From images to a 3D volume of material morphology. Journal of Electron Spectroscopy and Related Phenomena, 149 1-3 , (November 2005), 5160 , 0368-2048
  7. 7. BaumK. G.HelgueraM.KrolA. 2008 Fusion Viewer: A New Tool for Fusion and Visualization of Multimodal Medical Data Sets. Journal of Digital Imaging, 21 (October 2008), S59S68 , 0897-1189
  8. 8. BehrendO. P.OdoniL.LoubetJ. L.BurnhamN. A. 1999 Phase imaging: Deep or superficial? Applied Physics Letters, 75 17 (August 1999), 25512553 , 0003-6951
  9. 9. BentoutouY.TalebN.KpalmaK.RonsinJ. 2005 An automatic image registration for applications in remote sensing. Ieee Transactions on Geoscience and Remote Sensing, 43 9 (September 2005), 21272137 , 0196-2892
  10. 10. BlackK. J.VideenT. O.PerlmutterJ. S. 1996 A metric for testing the accuracy of cross-modality image registration: Validation and application. Journal of Computer Assisted Tomography, 20 5 (September 1996), 855861 , 0368-8715
  11. 11. BriggsD.GrantJ. T.editors 2003 Surface Analysis by Auger and X-Ray Photoelectron Spectroscopy: IM Publications, Chichester, UK.
  12. 12. ChenH. M.AroraM. K.VarshneyP. K. 2003 Mutual information-based image registration for remote sensing data. International Journal of Remote Sensing, 24 18 (September 2003), 37013706 , 0143-1161
  13. 13. ClevelandJ. P.AnczykowskiB.SchmidA. E.ElingsV. B. 1999 Energy dissipation in tapping-mode atomic force microscopy. Applied Physics Letters, 75 20 (October 1999), 26132615 , 0003-6951
  14. 14. DelcorteA. 2008 On the road to high-resolution 3D molecular imaging. Applied Surface Science, 255 4 (December 2008), 954958 , 0169-4332
  15. 15. Del MarcoS. P.TomV.WebbH. F. 2007 A theory of automatic parameter selection for feature extraction with application to feature-based multisensor image registration. Ieee Transactions on Image Processing, 16 11 (November 2007), 27332742 , 1057-7149
  16. 16. Eigenvector Research, I. PLS_Toolbox 5.2
  17. 17. FellersT. J.M. W. Davidso "Introduction to Confocal Microscopy." Available from http://www.olympusconfocal.com/theory/confocalintro.html.
  18. 18. GaoJ. X.LiuE.ButlerD. L.ZengA. P. 2003 Compositional depth profile analysis of coatings on hard disks by X-ray photoelectron spectroscopy and imaging. Surface & Coatings Technology, 176 1 (November 2003), 93102 , 0257-8972
  19. 19. GarciaR.PerezR. 2002 Dynamic atomic force microscopy methods. Surface Science Reports, 47 6-8 , (September 2002), 197301 , 0167-5729
  20. 20. GarciaR.TamayoJ.SanPaulo. A. 1999 Phase contrast and surface energy hysteresis in tapping mode scanning force microsopy. Surface and Interface Analysis, 27 5-6 , (May 1999), 312316 , 0142-2421
  21. 21. JonesE. A.LockyerN. P.VickermanJ. C. 2008 Depth profiling brain tissue sections with a 40 keV C-60(+) primary ion beam. Analytical Chemistry, 80 6 (March 2008), 21252132 , 0003-2700
  22. 22. LinW. C.ChenS. Y.ChenC. T. 1989 A New Surface Interpolation Technique for Reconstructing 3d Objects from Serial Cross-Sections. Computer Vision Graphics and Image Processing, 48 1 (October 1989), 124143 , 0073-4189X.
  23. 23. LingY.EhlersM.UseryE. L.MaddenM. 2008 Effects of spatial resolution ratio in image fusion. International Journal of Remote Sensing, 29 7 (April 2008), 21572167 , 0143-1161
  24. 24. LiuX. J.YangJ.ShenH. B. 2008 Automatic image registration by local descriptors in remote sensing. Optical Engineering, 47 8 (August 2008), 087206 0091-3286
  25. 25. LuH. H.LinC. W.HsiaoT. C.LeeC. K.HsuS. M. 2009 Nanopatterning on Silicon Wafers Using AFM-Based Lithography-for Solar Cells. Journal of Nanoscience and Nanotechnology, 9 3 (March 2009), 16961700 , 1533-4880
  26. 26. LuanH. X.QiF. H.XueZ.ChenL. Y.ShenD. G. 2008 Multimodality image registration by maximization of quantitative-qualitative measure of mutual information. Pattern Recognition, 41 1 (January 2008), 285298 , 0031-3203
  27. 27. MaciiD.BoniA.De CeccoM.PetriD. 2008 Tutorial 14: Multisensor data fusion. Ieee Instrumentation & Measurement Magazine, 11 3 (June 2008), 2433 , 1094-6969
  28. 28. MaesF.CollignonA.VandermeulenD.MarchalG.SuetensP. 1997 Multimodality image registration by maximization of mutual information. Ieee Transactions on Medical Imaging, 16 2 (April 1997), 187198 , 0278-0062
  29. 29. MaesF.VandermeulenD.SuetensP. 2003 Medical image registration using mutual information. Proceedings of the IEEE, 91 10 (October 2003), 16991722 , 0018-9219
  30. 30. MahlerR. P. S. 2004 "Statistics 101" for multisensor, multitarget data fusion. IEEE Aerospace and Electronic Systems Magazine, 19 1 (January 2004), 5364 , 0885-8985
  31. 31. MatsushitaY.SekiguchiT.SaitoK.KatoT.ImaiT.FukushimaK. 2007 The characteristic fragment ions and visualization of cationic starches on pulp fiber using ToF-SIMS. Surface and Interface Analysis, 39 6 (June 2007), 501505 , 0142-2421
  32. 32. NgunjiriJ.GarnoJ. C. 2008 AFM-based lithography for nanoscale protein assays. Analytical Chemistry, 80 5 (March 2008), 13611369 , 0003-2700
  33. 33. NunezJ.OtazuX.ForsO.PradesA.PalaV.ArbiolR. 1999 Multiresolution-based image fusion with additive wavelet decomposition. IEEE Transactions On Geoscience and Remote Sensing, 37 3 (May 1999), 12041211 , 0196-2982
  34. 34. PawleyJ.editor 2006 Handbook of Biological Confocal Microscopy. Berlin: Springer.
  35. 35. PenskaK.FolioL.BungerR. 2007 Medical applications of digital image morphing. Journal of Digital Imaging, 20 3 (September 2007), 279283 , 0897-1889
  36. 36. PeriJ. S. J. 2001 Approaches to multisensor data fusion. Johns Hopkins Apl Technical Digest, 22 4 (October-December 2001), 624633 , 0270-5214
  37. 37. PohlC. 1999 Tools and methods for fusion of images of different spatial resolution in International Archives of Photogrammetry and Remote Sensing, 32 W6, Spain: Valladolid.
  38. 38. RafatiA.ShardA. G.AlexanderM. R.DaviesM. C. 2008 Characterisation of drug loaded polymer films using ToF-SIMS and XPS depth profiling. NMAET III & SSBII 10 meeting. Teddington, UK.
  39. 39. RaoC. V.RaoK. M. M.ReddyP. S.PujarG. 2008 A novel method for enhancement of radiometric resolution using image fusion. International Journal of Applied Earth Observation and Geoinformation, 10 2 (June 2008), 165174 , 0303-2434
  40. 40. RiederC.RitterF.RaspeM.PeitgenH. O. 2008 Interactive visualization of multimodal volume data for neurosurgical tumor treatment. Computer Graphics Forum, 27 3 (May 2008), 10551062 , 0167-7055
  41. 41. RopinskiT.HermannS.ReichR.SchafersM.HinrichsK. 2009 Multimodal Vessel Visualization of Mouse Aorta PET/CT Scans. IEEE Transactions on Visualization and Computer Graphics, 15 6 (November-December 2009), 15151522 , 1077-2626
  42. 42. SugimuraH.NakagiriN. 1997 AFM lithography in constant current mode. Nanotechnology, 8 3A (September 1997), A15A18 , 0957-4484
  43. 43. SvabA.OstirK. 2006 High-resolution image fusion: Methods to preserve spectral and spatial resolution. Photogrammetric Engineering and Remote Sensing, 72 5 (May 2006), 565572 , 0099-1112
  44. 44. TalA.ElberG. 1999 Image morphing with feature preserving texture. Computer Graphics Forum, 18 3 (December 1999), C339C348 , 0167-7055
  45. 45. TheMath.WorksI.MatlabR2008a. Natick, Massachusetts.
  46. 46. VarshneyP. K. 2000 Multisensor data fusion. Intelligent Problem Solving: Methodologies and Approaches, Proceedings, 1821 13 , 0302-9743
  47. 47. Viergever MA, Maintz JBA, Niessen WJ, Noordmans HJ, Pluim JPW, et al. 2001 Registration, segmentation, and visualization of multimodal brain images. Computerized Medical Imaging and Graphics 25 147151 .
  48. 48. WolbergG. 1998 Image morphing: a survey. Visual Computer, 14 8-9 , 360372 , 0178-2789
  49. 49. WucherA.ChengJ.WinogradN. 2007 Protocols for three-dimensional molecular imaging using mass spectrometry. Analytical Chemistry, 79 15 (August 2007), 55295539 , 0003-2700
  50. 50. ZitovaB.FlusserJ. 2003 Image registration methods: a survey. Image and Vision Computing, 21 11 (October 2003), 9771000 , 0262-8856

Written By

Kateryna Artyushkova, Jeffrey Fenton, Jabari Farrar and Julia Fulghum

Submitted: 15 October 2010 Published: 24 June 2011