Chemical formula, colour on RLM, and average atomic number of epoxy resin and some minerals.
The recent developments in electronics and computing have brought a radical change to the microstructural characterization of materials. The integration of digital image acquisition and digital image analysis with microscope automation methods is giving rise to a rich set of techniques in the new field of Digital Microscopy (Paciornik & Maurício, 2004).
Modern microscopes of all kinds (optical, electron, scanning probe) are controlled by software and have digital image acquisition. This setup allows many integrated tasks to be run under the control of automation routines like, for instance, specimen scanning and automatic focusing. Additionally, some microscopes can be fully automated. Thus, it is possible to integrate specimen scanning, image acquisition and storage, processing, analysis and report generation in a single routine.
Besides the automation of routine tasks in the microscopes, Digital Microscopy really opens new possibilities for microstructural characterization. In this context, multimodal microscopy emerges as a promising trend. Multimodal microscopy aims at combining complementary types of information from a given sample in order to build a multidimensional data set. It generates multi-component images combining layers obtained from different microscopy modalities, or from the same microscope in diverse conditions. For instance, multimodal microscopy may consider different signals in scanning electron microscopy (SEM) and different contrast modes in optical microscopy. Sometimes, multimodal microscopy is also referred as co-site, correlative or collaborative microscopy.
The key step of a multimodal microscopy methodology is the registration between images from a given field and/or set of fields. Image registration is the process of overlaying two or more images of the same scene taken at different conditions or by different sensors. Actually, registration is a crucial procedure in all image analysis tasks in which the final information is obtained from the combination of various data sources. Typically, registration is required to combine or compare images in remote sensing and medical imaging applications.
Once the multimodal set of images is acquired and registered, image segmentation can be employed to discriminate phases, regions or objects of interest. Due to the nature of this problem, multidimensional pattern recognition techniques arise as potential methods for image segmentation. Then, after segmentation, one is able to measure size, shape, intensity, and position parameters, leading to the possibility of automatic characterization of microstructural features.
The present chapter presents a multimodal methodology that combines images obtained by reflected light microscopy (RLM) and SEM. The so-called RLM-SEM co-site microscopy (Gomes & Paciornik, 2008a, 2008b) was developed to solve some ore microscopy problems that cannot be solved by either RLM or SEM.
2. Ore microscopy
Ore microscopy is an essential tool for ore characterization. It was generally employed in its various modalities (stereoscopic, transmitted and reflected light, SEM, etc.) for mineral identification and quantification, and in the determination of mineral texture and liberation analysis. In certain conditions, ore microscopy is the single approach to access this kind of information. In the mining industry, it is extensively used to provide parameters to the Geometallurgy procedures for exploration, production planning, and processing plant design and optimization purposes.
Transmitted and reflected light microscopy, respectively for transparent and opaque minerals, are probably the most traditional techniques of mineralogical identification. During the last two centuries, diverse analytical methods based on various properties of minerals were developed and refined. Referring to reflected light microscopy, it is worth to mention properties such as reflectivity, colour, reflection pleochroism, internal reflections, hardness, preferential polishing, chemical reactivity, crystalline habit, and crystalline texture, among others. There are some classical text-books that cover both theoretical and practical aspects of ore microscopy such as Galopin & Henry (1972), Gribble & Hall (1992), Criddle & Stanley (1993), and Craig & Vaughan (1994).
However, these traditional methods generally require an expert mineralogist and only few of them can be applied in automated systems. Thus, optical microscopy was being left aside in favour of SEM in ore characterization methodologies. In fact, in the last decades, research and development of microscopy in Applied Mineralogy field were focused on SEM.
The SEM is a very versatile analytical instrument. It builds images through synchronization of the electron beam scanning and one of the many signals that come from the interaction between the electron beam and the specimen. Thus, the pixels present intensities proportional to the signal measured by one of the SEM detectors such as, for instance, back-scattered electrons (BSE) or secondary electrons (SE) detectors. If the SEM has a coupled energy dispersive X-ray spectrometer (EDS), it becomes even more versatile, and can also perform elemental chemical analysis with a resolution down to approximately 1 µm on the surface. This is the great advantage of SEM – a large variety of electron-specimen interactions can be used to form images and to provide information with different physical meanings (Reimer, 1998; Goldstein et al., 2003).
The most used signal for ore characterization is BSE that can furnish topography information and atomic number contrast. Nevertheless, if the specimen is plane, each pixel is proportional to the average atomic number of its corresponding region on the specimen surface (Jones, 1987). Therefore, BSE images of polished samples are indirectly compositional images, in which mineral phases can be correlated to characteristic intensities or grey levels that are proportional to their average atomic numbers. Figure 1 shows a BSE image of iron ore in which four phases can be recognized: the embedding resin (the black background), quartz (dark grey), goethite (the grey particle at centre), and hematite (white). Table 1 presents chemical formula, colour on RLM, and average atomic number of epoxy resin and some minerals.
Based on BSE and EDS techniques, some automated systems for ore characterization were developed and commercially launched (Petruk, 1988; Sutherland & Gottlieb, 1991; Gu, 2003). These systems are SEM’s especially dedicated to quantitative mineral analysis. They can identify minerals using BSE and EDS signals, and perform quantification routines through integrated image analysis software. Their capabilities may include particle-by-particle analysis, mineral phase classification and quantification, and mineral liberation analysis. Therefore, they became dominant for ore characterization, both in academy and industry, due to their enormous analytical capacity and relative simplicity of use.
Nevertheless, in recent years, there was a growing use of optical microscopy applied to ore characterization. Basically, three facts contributed to this trend: better optics, better digital image acquisition devices (Pirard et al., 1999), and the advent of Digital Microscopy. The progress in microscope optics, mainly due to infinity correct tubes and new advanced objective lenses, provided images with reduced spherical aberration and free of colour distortions (Davidson & Abramowitz, 1999), which are more suitable to image analysis and consequently to quantitative microscopy.
The colour has always been one of the most important properties used for mineral identification under a microscope (Piller, 1966). Moreover, there are some relevant minerals that are not distinguishable in the SEM, but can be discriminated through their colours in the reflected light microscope, such as, for instance, hematite and magnetite, which are the major iron ore minerals. Hematite and magnetite have similar average atomic numbers, respectively 20.59 and 21.02, and consequently show similar grey levels in BSE images, preventing their discrimination. The segmentation of hematite and magnetite in such kind of images requires a strong image contrast. However, this contrast condition avoids the segmentation of other present phases. In practice, not even SEM-based systems for automated mineralogy can discriminate hematite and magnetite, because the discrimination of these minerals is not possible through EDS due to their similar chemical composition (Gomes & Paciornik, 2008b).
On the other hand, transparent minerals and the embedding resin generally cannot be distinguished by their specular reflectances. For instance, quartz and epoxy resin have practically the same reflectance through the visible light spectrum (Neumann & Stanley, 2008). Actually, this is a classical problem in ore microscopy that renders unfeasible this kind of analysis through reflected light microscopy.
|Phase||Chemical formula||Colour on RLM||Average atomic number|
|Epoxy resin||C21H25ClO5||Dark grey||7.90|
|Goethite||Fe2O3.OH||Grey / Brown||19.23|
|Native copper||Cu||Bright yellow||29.00|
Figure 2 shows a pair of images of an iron ore sample acquired by reflected light microscopy and SEM. Comparing them, one can observe that the segmentation between quartz and epoxy resin in the BSE image is easy, but it is not viable in the optical image. On the other hand, hematite and magnetite present distinct colours, respectively light grey and pinkish grey, in the optical image, but have practically the same grey level in the BSE image.
Another example of minerals of difficult discrimination can be observed in Figure 3. It shows images of the same field of a copper ore sample acquired by reflected light microscopy and SEM. In the optical image, chalcopyrite can be easily identified by its characteristic brass yellow colour, but pyrite and pentlandite present a very similar colour (pale yellow). On the other hand, in the BSE image, chalcopyrite and pentlandite are practically indistinguishable, due to their similar average atomic numbers (23.54 and 23.36, respectively). Nevertheless, pyrite is slightly darker than pentlandite, because pyrite has a lower average atomic number (20.66).
The RLM-SEM co-site microscopy was developed to overcome these challenges. This methodology can improve the SEM analytical capacity adding specular reflectance (colour) information from RLM. The methodology was employed with some mineral samples, aiming at the discrimination of phases that are not distinguishable by either RLM or SEM, but can be discriminated through the combined use of both techniques.
3. Image registration
Image registration comprises the operation to determine the correspondence point to point between two or more images of the same area (or volume) obtained by different sensors or in different conditions, and the subsequent process of overlaying them.
Image registration is a fundamental procedure in all image analysis tasks in which the final information is gained from the combination of various data sources. Only after the registration, a multi-component image that represents a multimodal database can be properly composed and analyzed.
Typically, image registration is employed for composition and comparison of multi-spectral images in Remote Sensing (Schowengerdt, 1983). It also has several applications in Medicine, such as diagnosis, preparation of surgeries, treatment evaluation, etc. It is used, for instance, for fusion of anatomical and functional information, which are usually obtained through different medical imaging techniques (van den Elsen et al., 1993; Maintz & Viergever, 1998).
There is in the literature a wide variety of image registration methods based on different principles and employed for diverse applications (Zitova & Flusser, 2003; Goshtasby, 2005). Anyway, registration consists in the determination and implementation of a geometric operation (spatial transformation) between images in order to correlate the spatial coordinates of both images. Therefore, the fundamental aspect of any registration method is the spatial transformation used to correctly overlay images. Although many types of variations may be present in images, a suitable transformation must remove only spatial distortions between them (Brown, 1992). Other differences, due to the diversity of information that each image represents, must be maintained, since these are the interesting characteristics that one aims to expose.
In fact, image registration is more complex than a simple image alignment. It is not limited to translation and rotation of images. It may be composed of a combination of six distinct basic transformations: translation, rotation, scale, shear, projection, and other non-linear and local distortions. Figure 4 presents the six basic transformations, showing their effects in a sample base image.
Spatial transformations can convert the coordinates of the sensed image to the coordinates of the base or reference image. Thus, they correlate these digital images pixel by pixel, allowing the assemblage of a multi-component image.
The transformations that involve only translation and rotation are generally called rigid body or Euclidean transformations, since the Euclidean distances within images are preserved (Szeliski, 2004). In contrast, the other ones are classified as non-rigid or elastic. Although this nomenclature is the most commonly found in the literature, including the present text, it is not a consensus. Some authors consider scale as a rigid body transformation too, and there are still others that also include shear and projection in the class of rigid transformations (Crum, et al., 2004).
In this context, multimodal microscopy procedures that are performed intrinsically in a unique microscope constitute probably the simplest cases, generally involving only rigid body transformations. Sometimes it is even possible to acquire images that are directly registered.
Multimodal microscopy methodologies on reflected light microscope can be carried out with or without specimen removal from the stage. In the first case, specimen removal, for instance, for chemical etchings like in a classical metallographic approach, generally imply some displacement between images. Therefore, translation and occasionally rotation corrections are required. Soto et al. (2004) and Paciornik & Gomes (2009) present case studies of multimodal methods that involve specimen removal for chemical etchings.
On the other hand, optical methodologies without specimen removal can be sometimes performed without translation corrections (De-Deus et al., 2007). Nevertheless, there are some exceptions. For instance, Pirard (2004) proposed a multispectral imaging technique applied to ore characterization in which shifts of the order of several pixels occur between images obtained from different wavelengths; and Iglesias et al. (2011) developed a multimodal microscopy methodology based on the combination of cross-polarized and bright field images in which there were small misalignments between them.
The SEM forms an image through scanning its electron beam in a raster across the specimen and then it synchronizes the scanning with a signal from one of its detectors. Thus, in a given field, it can acquire several different images, which are ready to compose a multi-component image without the need for a registration procedure. However, in practice, older equipments usually exhibit some translation between images from different detectors. In this case, a translation correction is not enough to properly register the images, because SEM’s generally present complex and non-linear distortions (Goldstein et al., 2003) that must be considered.
In the RLM-SEM co-site microscopy, the registration procedure involves rigid and non-rigid transformations. The specimen handling between the microscopes and the different stages imply that translation and rotation adjustments are necessary. Besides, non-rigid transformations are required due to the complex distortions that occur in images from SEM. Even a fine calibration of the equipment is not capable of preventing them.
A registration procedure consists of a sequence of mathematical operations that determine the suitable spatial transformation and then defines and applies the geometric operation that properly performs the registration. The base of a registration procedure is the kind of information used by its algorithm. Therefore, as stated in the already classical review paper of Zitova & Flusser (2003), there are two main classes of algorithms according to their nature: area-based and feature-based.
Area-based algorithms, also called template matching, estimate the correspondence between images (or parts of them) in order to determine which transformations provide the best correspondence. The correlation between two signals (cross-correlation) is a standard approach to template matching algorithms that can be particularly efficient if it is computed in the frequency domain using the fast Fourier transform (Lewis, 1995). Area-based algorithms are in general simpler than feature-based ones. They are applied mostly in cases involving only rigid and scale transformations. Besides, they are more sensitive to noise in images. For instance, the multimodal methodologies presented by Soto et al. (2004), Paciornik & Gomes (2009), and Iglesias et al. (2011) employed cross-correlation in the frequency domain for the registration of their optical images.
Feature-based algorithms consist of four steps: feature detection, feature matching, mapping function design, and image transformation and resampling (Zitova & Flusser, 2003). Two sets of features, which are salient and distinctive objects such as corners, line intersections, edges, etc., are manually or automatically detected in both base and sensed images. These features are represented by the so-called control points (points themselves, centers of gravity, line endings, etc.). The aim is to find the pairwise correspondence between control points and then to map a suitable transformation from them. Therefore, the sensed image is transformed through the determined mapping function and an appropriate interpolation technique is employed in order to treat non-integer coordinates.
The detection of control points and the determination of their correspondence in base and sensed images are crucial and difficult tasks. The method named Scale Invariant Feature Transform (SIFT), proposed by Lowe (2004), has been shown computationally efficient and robust upon diverse distortions and multisensor cases.
In contrast to the area-based methods, the feature-based ones do not work directly with image intensity values. Control points constitute higher level information. This fact makes feature-based methods suitable for applications in which diverse sensors with different data structures and physical meanings are involved. Besides, it allows registering images with any nature of distortions, including non-linear and local ones (Zitova & Flusser, 2003).
Furthermore, in multimodal microscopy methodologies, an alternative approach can facilitate the determination of control points. By introducing indentation marks in the sample through a microdurimeter, the control points can be properly defined as their centers of gravity. In fact, this method can be useful for the registration of one or few fields in which specific microstructural features are of interest. However, it becomes impractical when the number of fields is large, as usually occurs in ore characterization procedures.
The RLM-SEM co-site microscopy methodology employs a feature-based method for registration that is described in the section 5.3.
4. Image processing and analysis
A typical image processing and analysis sequence comprises the steps of image acquisition, digitization, pre-processing, segmentation, post-processing, feature extraction and classification (Gonzalez & Woods, 2007).
Pre-processing, or image enhancement, is the first step after image digitization and is used to correct basic image defects, normally created during the image acquisition step. Typical operations, at this step, are background correction, edge enhancement and noise reduction. Pre-processing is useful for qualitative reasons, as it improves the visibility of relevant features in the image, but it is even more important to prepare the image for the following step of segmentation.
Segmentation is the technical term used for the discrimination of objects in an image. Segmentation is probably the most complex step in the sequence because it tries to represent computationally a cognitive process that is inherent to the human vision. When we look at an image we use many different inputs to distinguish the objects: brightness, boundaries, specific shapes or textures. Our brains process this information in parallel at high speed, using previous experience. Computers, on the other hand, do not have the same associative power. The recognition of objects in an image is made through the classification of each pixel of the image as pertaining or not to an object.
There are many segmentation methods based on different principles such as thresholding, edge detection, texture analysis, mathematical morphology, etc. Each one is generally more suitable for a specific application. Categorically, there is not an ideal generic method that is always the best one. Some classical references in this area are: Haralick et al. (1973), Otsu (1979), Haralick (1979), Beucher & Lantuéjoul (1979), Marr & Hildreth (1980), Pun (1981), Canny (1986), and Adams & Bischof (1994).
The most common segmentation method is thresholding. It is based on the assumption that pixels pertaining to a given class of objects (e.g. a specific mineral phase) have similar colour or greyscale intensity, and this colour is different from the background and from other classes of objects. In other words, there must be sufficient contrast between different phases in the material. If that is the case, then the segmentation is based on selecting colour/intensity thresholds that represent the various phases.
Noise, uneven illumination, edge effects contribute to degrade the discrimination between phases, and that is why a pre-processing step may be so relevant. Evidently, phase contrast maybe too low, depending on the microscope used, as described before, and that is where combining information from different types of signals becomes critical.
In many situations the results of segmentation contain artefacts, such as spurious objects, touching or partially overlapping objects, etc. A very common artefact in mineralogical images is segmenting a phase together with the edges of a different phase that share the same intensity range. Some of these defects can be minimized with an appropriate pre-processing step, such as delineation (edge enhancement), but many must be corrected after segmentation, in the so-called post-processing step.
Post-processing makes intense use of morphological operators such as erosion, dilation, opening, closing, and more sophisticated functions like the watershed separation method for touching objects (Serra, 1982, 1988). Ideally, the final result is an image in which just the relevant objects are present and separated in groups that correspond to each phase present in the sample. However, as described below, this is rarely the case, and further analysis of the segmented objects must be done to complete the discrimination.
Given a segmented post-processed image containing a set of objects, several measurements are available. Field features such as number of objects and area fraction are some of the simplest ones. Object specific features are more sophisticated and include measurements of size, shape, position, intensity and texture of each object in the image (Friel, 2000). These features are critical for the classification step.
4.1. Multi-component image analysis
Common colour images, generated by either scientific or general-purpose digital cameras, are generally 24-bit RGB images. Actually, they consist of multi-component images composed by three images that represent the primary colours (red, green and blue) with 8-bit quantization each (Orchard & Bouman, 1991). The RGB system is the most common colour representation system employed by cameras and displays. It was developed to be similar with the tri-stimulus response of human vision. Figure 5 shows a sample RGB image above its three components, respectively, R, G and B. In this figure, one can observe some samples of how the primary colours are mixed to compose other colours and grey levels.
The components of a multi-component image can represent information of any source and physical meaning. It is not necessary that the data within components are correlated. They just must have spatial correspondence pixel-to-pixel, i.e., they must be registered.
Multi-component images with up to three components have the advantage that they can be viewed as RGB colour images in standard image viewer software. In this case, the so-called pseudo-colours denote the properties which are represented in the components. Pseudo-colour images constitute a useful approach for data visualization.
Each component image can be singly processed and analyzed as a common intensity or grey level image. However, this processing should be carefully performed so that the spatial relations within images are not undesirably modified. Geometric operations must be especially avoided.
A multi-component image can also be conceived as a matrix in which each element is a vector, not a scalar value. Each vector represents a pixel, and each element of vector is the value of this pixel in one of the component images. In a RGB image, each pixel consists of a three-element vector that represents the intensity of the three primary colours. Therefore, in a multi-component image, the probability density function becomes multivariate and consequently its histogram of intensities becomes multidimensional. Figure 6 shows a RGB image of a copper ore sample in the reflected light microscope and its bi-dimensional RG histogram.
Image segmentation by thresholding can be generalized to an n-dimension problem. In this case, a threshold becomes a decision boundary, whose form depends on the number of components. One component leads to decision boundaries that are scalar values (thresholds); two components imply that decision boundaries are straight lines; three components involve decision boundaries that are planes; and so on. Nevertheless, there are more sophisticated segmentation methods that offer more complex decision boundaries, such as curves, surfaces, etc. Besides, they can also be discontinuous. In this context, multidimensional pattern recognition techniques arise as potential methods for segmentation of multi-component images.
Pattern recognition is the scientific discipline whose goal is the classification of objects (patterns) into a number of classes from the observation of their characteristics (Theodoridis & Koutroumbas, 2003). It aims to build a simpler representation of a data set through its more relevant features in order to perform its partition into classes (Duda et al., 2001).
Pattern recognition techniques can be used to classify objects (pixels, regions, etc.) within images. Any part of image that has at least one measurable property can be considered as an object and consequently it can be classified. In multi-component images, a pixel consists of a vector in which the elements represent its values in the components.
The classification of pixels is actually an image segmentation procedure. Each class of pixels can properly represent a phase or mineral in an ore microscopy case. If the phases of interest are known, this problem becomes supervised (Duda et al., 2001). In a supervised classification procedure, this known information is exploited so that the classification system learns how the different classes can be recognized.
A supervised classification procedure involves three stages: training, validation and classification. The training stage comprises sampling of known pixels of each class in order to compose the so-called training set that is used as knowledge base. Therefore, the classifier is trained, i.e., it is designed based on the known information. Following, in the validation stage, another known set of pixels, the validation set, is classified aiming to estimate the performance of the classifier, considering its generalization capacity (Toussaint, 1974). If the validation stage indicates that the training was successful, the classification is then possible and consequently the segmentation procedure is implemented.
It is worth to mention that although the RGB system is vastly predominant in image acquisition devices, it is not generally the most appropriate colour representation system for classification purposes because its three components (R, G and B) are very correlated, due to their strong dependence from the light intensity (Littmann & Ritter, 1997). Besides, it doesn’t represent colours in a uniform scale, preventing measurements of similarity between colours through their distance in RGB space (Cheng et al., 2001). There are many colour systems described in the literature (Sharma & Trussell, 1997). Actually, one can define any colour system from linear or non-linear transformations of RGB (Vandenbroucke et al., 2003). Systems like rgb, HSI, L*a*b* and L*u*v* present less correlated features and consequently tend to be more suitable for classification (Gomes & Paciornik, 2008a).
5. Combining reflected light microscopy and SEM
This section describes the RLM-SEM co-site microscopy methodology by reviewing two case studies, in which it was applied for the characterization of a copper ore and an iron ore. These case studies were originally presented by Gomes & Paciornik (2008a and 2008c, respectively).
The RLM-SEM co-site microscopy comprises four sequential steps: image acquisition in RLM; image acquisition in SEM; registration; and image processing and analysis. The three first steps consist of generic routines of the methodology, but the image processing and analysis procedure depends on the case study. The segmentation of minerals in both case studies was performed through supervised classification of pixels, exploiting their multidimensional nature. However, used features and classification methods differ.
Ore microscopy procedures commonly involve acquisition and analysis of tens to hundreds of images per cross-section in order to provide a representative sampling. Therefore, in the development of the RLM-SEM co-site microscopy, automatic routines for field scanning and image acquisition were implemented for both used microscopes.
5.1. Image acquisition in reflected light microscopy
A motorized and computer controlled microscope with a digital camera (1300 x 1030 pixels) was used. A routine was implemented for microscope and camera control, and for image acquisition. It integrates and automates many common procedures such as specimen x-y scanning, automatic focusing, background correction and imaging.
The following image acquisition procedures and conditions were employed:
Before image acquisition, a SiC reflectivity standard was used to generate background images for each objective lens, which were subsequently employed for automatic background correction (Pirard et al., 1999) of every acquired image.
Illumination was kept constant by direct digital control of the lamp voltage.
Camera sensitivity, exposure and white balance were optimized initially for a representative field of the sample and kept constant there on.
Objective Lenses: 5X (NA 0.13); 10X (NA 0.20); 20X (NA 0.40), leading to resolutions of 2.11, 1.05, and 0.53 µm/pixel, respectively.
Single fields regularly spaced on the sample were imaged through specimen scanning with a motorised stage and automatic focusing.
Each field position was recorded in a database for subsequent image acquisition on in SEM.
All images were acquired at 24 bit RGB colour quantisation.
5.2. Image acquisition in SEM
A digital SEM was used to acquire a BSE image (1024 x 768 pixels) of each field imaged on RLM. In this procedure, the sample must be placed in the SEM stage at a similar arrangement as positioned in the RLM stage. It is unnecessary and impractical to place the sample in the exact same way, but a similar arrangement can make image registration easier and faster.
The magnification of the SEM was set to keep similar resolutions to optical images. Other SEM operational parameters were manually tuned for a representative field of the sample and then kept constant. After that, the field positions database was loaded with the acquisition routine developed in the SEM control software. It converts RLM stage coordinates to SEM stage coordinates and subsequently performs automatic specimen scanning and image acquisition. Thus, respectively for copper ore and iron ore samples, 121 (11 x 11) and 81 (9 x 9) fields per cross-section were imaged with the RLM and the SEM. Figure 7 presents a field of the copper ore sample as imaged on RLM and SEM.
5.3. Image registration
An automatic method for the registration of RLM and SEM images was developed. The distortions were considered according to their sources, and the registration procedure was accomplished through sequential steps. The first step comprises distortions from the SEM, such as astigmatism and local distortions. The second step adjusts rotation and the third one corrects translation. At the end, the registered images are cropped to represent exactly the same field.
The first step is carried out through a feature-based registration algorithm. It maps the SEM characteristic distortions based on several control points that are automatically detected. These distortions do not depend on samples. They are a function of SEM operational parameters, such as magnification and working distance. Therefore, this step was employed only one time for each SEM set-up, i.e., once for each magnification whose pixel size corresponds to the pixel size obtained through one of the objective lenses of the RLM.
A standard specimen with regular distributed and easily extractable control points must be imaged on RLM and SEM. In the present case, two copper grids (200 mesh for 5X and 10X lenses, and 400 mesh for 20X lens) were used. These images were analyzed by an automatic routine in order to determine the centroid (center of gravity) of each grid hole, the control points. Then, a suitable spatial transformation was computed from the pairs of control points using the local weighted mean method proposed by Goshtasby (1988). Therefore, this spatial transformation was applied to just remove distortions in every SEM image of the ore sample.
Figure 8 shows the pairs of control points superimposed to the RLM image of the grid (10X). The white circles represent the control points extracted from the RLM image, and the white dots are the points obtained from the corresponding SEM image. As can be observed, circles and points are not aligned. Besides, the misalignment varies, evidencing the complexity of these distortions.
The second step of the registration method aims at finding the rotation angle between images and subsequently to adjust rotation. This angle is due to sample manipulation and its different arrangement in the sample holders of the microscopes. Thus, this rotation is constant in all fields of a sample in a given experiment.
An iterative algorithm is used to determine the angle that maximizes the normalized cross correlation between a pair of images. The algorithm uses coarse-to-fine approach. It evolves in order to adjust the angle down to 0.01 of precision. This procedure is applied for one pair of images and the obtained angle is used to correct rotation in all SEM images.
After the second step, the SEM images are free of distortions and they are put in the same coordinate system of the RLM images. Therefore, only translation problems remain. Thus, in the third step, the RLM and the SEM images are finally registered through the maximization of normalized cross correlation. At the end, they are cropped to represent exactly the same field. Figure 9 shows the images present in Figure 7, after registration.
5.4. Image processing and analysis
The registered SEM and RLM images went through a pre-processing step of delineation to reduce the well-known halo effect, making them more suitable for the subsequent segmentation procedures. Figure 10 shows a detail of an image obtained by RLM, before and after delineation operation. Comparing them, one can observe that delineation improves phase transitions and consequently allows better segmentation results.
5.4.1. Copper ore
The copper ore sample had a complex mineralogy mainly composed of thirteen minerals, which were then taken as individual classes (quartz, three different silicates, apatite, magnetite, pentlandite, chalcopyrite, covelline, bornite, sphalerite, chalcocite, and native copper). Besides mineral phases, epoxy resin was taken as a class too. Thus, the training stage involved sampling of pixels from the fourteen classes. In practice, 6000 pixels of each phase were interactively selected from several images.
The delineated RLM images were converted from RGB to the rgb, HSI, L*a*b* and L*u*v* colour systems with the goal of revealing colour information hidden by the correlated RGB system. These conversion operations generate ten new components (r, g, b, H, S, I, a*, b*, u*, v*), increasing the system dimensionality from four to fourteen.
A Bayes classifier (Duda et al., 2001) was employed and the fourteen components (BSE intensity and the thirteen colour components) were used as features. The validation was carried out through holdout estimate (Toussaint, 1974), reaching a success rate larger than 99.5%.
The classification result was a grey level image per field in which each phase was represented by a distinct grey level. Thus, pixels classified as phase one have intensity one, pixels recognised as phase two have intensity two, and so on. Therefore, a suitable look-up table can be applied in order to attribute a different pseudo-colour to each phase and consequently to make their visualisation easier. Figure 11 presents the classification result for the images shown in Figure 9 and its look-up table.
5.4.2. Iron ore
The mineralogical assemblage of the iron ore sample was simple. It was mainly composed by hematite, magnetite, goethite and quartz. Therefore, in this case study, five classes were considered (epoxy resin, quartz, goethite, haematite, and magnetite).
The segmentation process was split in two supervised classification procedures. The first one recognised epoxy resin, quartz, goethite, and a hematite-magnetite composed phase through the classification of pixels in SEM images, using the BSE intensity as feature in a Bayes classifier. Then, the second classification procedure was able to discriminate hematite and magnetite. It was carried out through the classification of pixels in RLM-SEM composed images. In this case, their four components (R, G, B, and BSE intensity) were used as features and a Bayes classifier was employed. The training stage for both classification procedures comprised interactive sampling of 1000 pixels of each one of the five classes from a RLM-SEM composed image. Figure 12 presents the segmentation result for the images shown in Figure 2 and its applied look-up table.
The information from RLM and SEM presents different structures and physical meanings. RLM data consists of a vector of three 8-bit values, which denote specular reflectance in the RGB colour system. On the other hand, SEM data is represented as 8-bit values of BSE intensity that provides average atomic number contrast. Thus, it is very difficult to compute suitable measurements of similarity between patterns (pixels) that can be used to recognise the classes. In this kind of problem, Valev and Asaithambi (2001) pointed out that different classifiers can be used to complement one another. This is the approach employed for the segmentation in the iron ore case study.
The increase of dimensionality carried out in the copper ore case study can make the classification task easier, since hidden information is discovered and consequently patterns are better described. However, the training data must grow exponentially with the dimensionality in order to prevent the so-called curse of dimensionality (Marques de Sá, 2001). In practice, this is not an issue for pixels classification as a typical image of 1024 x 768 pixels, for instance, has about 0.8 million of pixels, and it is easy to obtain several thousands of pixels of each class.
The resulting segmented images in both case studies reveal small amounts of misclassified pixels in borders between phases. It occurs mainly due to little cracks and relief, in spite of the good sample preparation and the delineation pre-processing. This misclassification is quite small and it can be negligible in mineralogical identification and quantification procedures. However, it becomes more significant for microstructural characterisation purposes, such as mineral liberation analysis. Therefore, post-processing routines should be developed.
Multimodal microscopy extends the capabilities of traditional microscopy techniques, improving the discrimination of mineral phases in ores. By combining Optical Microscopy and Scanning Electron Microscopy it takes advantage of the complementary contrasts provided by these techniques.
This method relies on microscope automation, digital image acquisition, processing and analysis. Over the last years many of these techniques have become readily available in both commercial and free software environments.
The use of supervised classification methods relies on operator experience, during the training stage, but once the classifier is optimized and validated, the effective classification of unknown samples is fully automatic and fast.
The developed method is applicable to other materials for which individual microscopy techniques do not provide enough discrimination between the relevant phases.
One of the authors (S. Paciornik) acknowledges the support of CNPq, the Brazilian Research Council.