Features of the proposed models.
Breast cancer is one of the global leading causes of death among women, and an early detection is of uttermost importance to reduce mortality rates. Screening mammograms, in which radiologists rely only on their eyesight, are one of the most used early detection methods. However, characteristics, such as the asymmetry between breasts, a feature that could be very difficult to visually quantize, is key to breast cancer detection. Due to the highly heterogeneous and deformable structure of the breast itself, incorporating asymmetry measurements into an automated detection system is still a challenge. In this study, we proposed the use of a bilateral registration algorithm as an effective way to automatically measure mirror asymmetry. Furthermore, this information was fed to a machine learning algorithm to improve the accuracy of the model. In this study, 449 subjects (197 with calcifications, 207 with masses, and 45 healthy subjects) from a public database were used to train and evaluate the proposed methodology. Using this procedure, we were able to independently identify subjects with calcifications (accuracy = 0.825, AUC = 0.882) and masses (accuracy = 0.698, AUC = 0.807) from healthy subjects.
- breast cancer
- bilateral registration
Cancer is one of the leading causes of death worldwide. In 2008, nearly 13% (7.6 million) of all deaths were cancer related. Among all types of cancer, lung, liver, colon, breast, and cervical are the most frequent ones. Recent studies predict 13.1 million cancer deaths for 2030 . Among women, breast cancer is the deadliest type of cancer. Nearly 1.8% of all worldwide deaths are breast cancer related .
Till today, there is no cure for breast cancer, and since the trigger to develop any type of cancer is still a mystery, there is not an effective way to prevent the occurrence. Early detection of breast cancer plays a key role in a positive prognosis. There are several imaging technologies that might be used by specialists for the early detection of breast cancer, such as magnetic resonance imaging, ultrasound, and X-ray mammogram. The last technique is the primary tool used to diagnose and detect breast cancer worldwide, and it has been proved to be the best cost-effective tool to diagnose the disease .
In clinical practice, mammography allows for the detection of early signs of tumors before they become apparent . Common signs of early cancer inside the breast tissue are micro-calcifications, architectural distortions, and masses . During the screening procedure, radiologists use those signs to generate a standardized evaluation of the risk of cancer in a given patient, called Breast Imaging-Reporting and Data System (BI-RADS). This report helps oncologists to decide a course of action among women at risk of developing breast cancer .
The broad use of mammogram has driven the development of computer-aided detection (CADe) and computer-aided diagnosis (CADx) systems. While both approaches aim to assist radiologists to detect and diagnose breast cancer as early as possible, CADx systems are used as a second opinion  and CADe ones aim to improve visualization of the lesions (with up to 35% improvement in detection rate ). However, although it has been shown that CADe systems have helped radiologists to better interpret findings , it has also been demonstrated that in some cases they may make interpreting the images more difficult, reducing the accuracy of early cancer detection . Furthermore, these systems also may increase the workload of the radiologists .
A typical CADe system, whose workflow is shown in Figure 1, consists of two algorithms applied sequentially, one to detect suspicious regions or regions of interest (ROI), and one to refine such regions. The former includes the preprocessing of the images, segmentation of the breast tissue, and the detection of the ROI itself. The latter process is performed to reduce the number of false positives , and usually relies on machine learning techniques [9–11]. Lastly, the results are presented to the radiologist, highlighting in the original mammography the regions that the analysis deemed highly suspicious. As seen in the same image, CADx systems follow the same workflow as CADe ones. However, besides highlighting areas of higher risk to the radiologist, additional algorithms are used to analyze each ROI and generate a computer-based diagnosis. It is important to mention that, currently, few CADe and CADx commercial systems have been approved by the Food and Drug Administration of the United States of America .
Many methodologies used by CADx systems analyze only one breast, or even just a subregion of the breast, at a time. That is, they evaluate the left and right breasts as independent objects, unlike radiologists, who analyze images of both breasts simultaneously to evaluate their asymmetry. Radiologists do so because asymmetry is related to early signs of breast cancer (i.e. parenchymal distortion, bright spots, masses, etc.) [14, 15] and it may be used to reduce the rate of false positive detection of masses [16, 17]. Asymmetry can refer to either a longitudinal study, where current and prior mammograms are compared, or a bilateral study, where differences between the left and right breast are analyzed.
A few CADx systems have already tried to incorporate asymmetry studies to enhance diagnosis [14, 18–20]. Some researchers have studied the use of a feature-based asymmetry analysis, where the mammograms are processed individually and the differences between the individual analyses are used as a mean to quantify asymmetry . This approach has also been used to characterize risk factors, such as breast density, and predict near-term breast cancer .
Another method that evaluates asymmetry, this one trying to mimic the approach used by radiologists, is the mammogram subtraction. In this approach, differences between mammograms are enhanced by performing a rigid registration (alignment) of the images. However, this methodology was originally employed only in longitudinal studies , comparing the same breast at two different times, since the highly heterogeneous and deformable tissue of the breast has hindered the inclusion of subtraction approaches in bilateral asymmetry studies .
Miller et al.  proposed a technique for the detection of bilateral symmetry using a semi-automated texture-based procedure that segments the glandular tissue, measuring the shape between views, and thus detecting the occurrence of asymmetries. The algorithm obtained an accuracy of 0.867 on a validation dataset of 30 screening mammogram pairs. Later, Miller et al.  presented a method for the detection of bilateral asymmetry based on measures of shape, topology, and distribution of brightness. This method was tested on 104 mammogram pairs, yielding a classification accuracy of 0.74.
Lau et al.  proposed a method for the detection of breast tumors that extracted measures of brightness, roughness, and directionality, and was based on localized asymmetry. This method was evaluated using 10 pairs of mammograms where asymmetry was a significant factor in the radiologist’s diagnosis. A sensitivity of 0.92 was obtained, with 4.9 false positives per mammogram. However, the alignment was tuned manually using control points.
Ferrari et al.  characterized asymmetry as variations in oriented textural patterns, obtained using directional filtering with Gabor wavelets at different orientations and scales. Using a database with 80 images resulted in a classification accuracy of up to 0.744.
Rodriguez-Rojas et al.  presented a CADx system targeted to detect high-risk cancer patients. To do so, automated breast tissue segmentations were performed on 200 Mexican subjects labeled as either low- or high-risk according to their BI-RADS score. Then, 50 features were extracted, and bilateral differences between mammograms were defined by subtracting corresponding features in both mammograms. Finally, a genetic algorithm selected a predictive combination of features. Using this methodology, they were able to classify low-risk and high-risk cases with an area under the receiver operating characteristic (ROC) curve (AUC) of 0.88 on a 150-fold cross-validation set. The features included in the model were associated with the differences in signal distribution and tissue shape.
In summary, and as presented, most asymmetry detection methods are either feature-based, rely on simple bilateral subtraction techniques [14, 27], or depend on an ROI provided by a radiologist [24, 25]. Thus, in order to efficiently measure asymmetry, a better and automatic registration must be performed . To do so, alignment has been improved by using the nipple as a reference point  and by co-registering both breasts using a robust point matching approach . Nevertheless, none of those works include a fully automated bilateral registration. In this chapter, a methodology that incorporates an automatic asymmetry analysis with both a feature-based and a pixel-wise bilateral subtraction into a CADx system is presented.
The proposed methodology follows the CADx workflow presented in the previous section. However, asymmetry measurements are used to aid in the diagnosis. To obtain such measurements, two additional stages are incorporated into the workflow: registration and pixel-wise subtraction. Additionally, a series of image transformations are incorporated to enhance different characteristics of the breast in the mammograms. This work is based on and follows previous efforts [30–32].
Figure 2 shows how the bilateral asymmetry information was incorporated into the CADx system. Briefly, soft tissue is first segmented, the image of the left breast is then registered to its right counterpart and a bilateral subtraction of the co-registered images is performed; images are then filtered and features are extracted; a multivariate model is selected using a train set; and finally, the model is evaluated on a validation set. A detailed explanation of each stage is presented in the following sections.
A total of 1796 digitalized film mammograms from 449 different subjects were used. From those, 45 were classified as healthy subjects (HS) (mean age of 59.3 and standard deviation (SD) of 9.8 years), 197 as subjects with malignant calcifications (CS) (mean age of 58 and SD of 10.9 years), and 207 as subjects with malignant masses (MS) (mean age of 64.1 and SD of 10.1 years). Each subject had the four standard mammograms taken, namely, left and right craniocaudal (CC), and left and right mediolateral oblique (MLO) projections.
In order to avoid problems associated with intra-scanner variability [17, 22, 33], all mammograms in this study were obtained from the Howtek dataset of the Digital Database for Screening Mammography public database , in which all mammograms were digitalized using a Howtek 960 scanner using a sampling rate of 43.5 micrometers per pixel and a 12-bit depth.
Segmentation, also called categorization by computer vision definitions, allows delimiting one or several parts of a given image assigning one class label (e.g. bone, muscle, fat, skin, calcification, and mass). This process is defined by the division or segmentation of the image into several homogeneous regions disjointed from their surroundings. A commonly used automatic segmentation of the breast tissue is based on the estimation of the background noise. For this study, an initial segmentation mask was created by estimating the background noise in the image and discarding all pixels below five standard deviations of the noise level. Then, holes were removed by applying closing morphological operations with a 3 × 3 supporting region, as described by Eq. (1):
where ⊕ and ⊖ are the grayscale dilation and erosion morphological operations, respectively. B (x, y) is a 3 × 3 structural element. A (x, y) is the image being segmented and S (A) is the resulting segmentation of the A (x, y) image. The largest connected region is used as the segmentation mask while all other high-intensity regions are removed from the images. Figure 3 shows an example of the results of the segmentation procedure.
Image registration can be defined as the intensity and spatial mapping between two images . Given two input images F and M, image registration can be expressed as R′ = g[T(F)], where T is a spatial transformation function, g an intensity transformation function, and R′ the registered image. The transformation function is not always necessary; a lookup table can be used to pinpoint intensities. A visual example of image registration is presented in Figure 4, where an image M is being registered to match image F.
Image registration has been widely used in medical applications [28, 36, 37]. However, the soft nature of the breast tissue makes them highly deformable, and rigid registration procedures, in which only rotation, translation, and scaling functions are used, are not sufficient. Therefore, nonrigid registration methods are necessary [38, 39]. There are many approaches to deal with medical imaging registration, the most recent comparison of algorithms based on a retrospective evaluation was published by West et al. , but it was constrained to do intra-patient rigid registration. Also recently, Diez et al.  and Celaya-Padilla et al.  compared registration algorithms with breast images as a source, and both concluded that the B-Splines approach was the most consistent.
Breast image registration based on a B-Splines transformation is defined as follows: given two input images (F = target image, M = image being registered), M is deformed by modifying a mesh of control points following a maximization of a similarity measure based on steepest descent gradient [6, 15]. The deformed image is compared to F using a similarity metric. If the images are similar enough, the process stops. Otherwise, the process reiterates.
Figure 5 shows a multi-resolution pyramid approach  for the B-Spline implementation. There, the images are first registered using low-resolution images, the B-Spline transformation parameters are moved into the next higher resolution and parameter optimization is run again, and so on. This often avoids issues with local minima in the parameter search space and reduces computational time .
For this study, the image to be registered was first horizontally flipped. Then, both the moving image and the target image were resampled into a lower resolution image. Next, the pyramids for the multi-resolution were generated. Afterwards, the registration process detailed in Figure 5 was carried out. And finally, the original moving image was deformed using the final parameters of the registration. For this implementation, mutual information  was used as the similarity metric. In Figure 6, the checkerboard of an example result from the B-Spline registration procedure is presented. There, it can be seen that the registered image was successfully aligned with its counterpart.
2.4. Image subtraction
Once the images were co-registered, a pixel-wise absolute difference was computed between the left and right images, as defined by Eq. (2) as follows:
where Ir(x, y) represents the right image, Il(T(x, y)) represents the left image registered to the right image space, and I∆(x, y) represents the map of absolute differences. Figure 7 shows an example of the differential image for two given input images.
2.5. Image enhancement
To study the appearance of the architectural distortions, two enhancing filters were applied to the images: a morphological high-frequency enhancement filter (H) designed to enhance fiber-like tissues, and a Laplacian of Gaussian filter (L) that enhances high-frequency patterns inside the breast tissue. Additionally, since the texture between normal and abnormal tissues is different , two texture maps were created. The first map computed the local standard deviation (σ) of the mammograms, and the second map computed the local fractal dimension (F). All image processing was implemented in C++ using Insight Segmentation and Registration Toolkit (ITK) libraries for image manipulation following previous efforts [32, 43].
2.6. Feature extraction
There are several features that may be quantified when aiming to detect early signs of cancer. For this analysis, 43 features were extracted from each image. These features can be grouped in three main categories: shape (i.e. area, perimeter, compactness, elongation, region centroid, region scatter), signal (i.e. mean, median, energy, variance, standard deviation, dynamic range, z mean, entropy, skewness, kurtosis, z range, fraction greater than z deviations, fraction lower than z deviations, value at fraction, 5% trimmed mean, 5% trimmed standard deviation, 5% trimmed z Mean), and morphology (i.e. total signal, signal centroid, signal scatter, and signal surface). Details of the full feature extraction procedure can be found in Ref. .
The enhancement filters and texture maps presented in Section 2.5 were applied to the four screening mammograms (i.e. left and right CC, and left and right MLO) and to the two bilateral subtraction images (CC and MLO), yielding a set of 15 images for both the CC and the MLO views: Ir, Il, IΔ, Hr, Hl, HΔ, Lr, Ll, LΔ, σr, σl, σΔ, Fr, Fl, and FΔ, where I is the raw image, H, L, σ, and F are the enhanced images described in Section 2.5, and r, l, and Δ, stand for the right, left, and bilateral subtraction images, respectively. Features were then extracted from this set of images. Additionally, to study the feature-based asymmetry analysis, the average and absolute difference of each left-right pair of measurements was also analyzed, resulting in 860 additional features, resulting in a total of 2150 features per subject.
2.7. Feature selection
The first step of the feature selection process consisted discarding highly correlated to avoid redundancy. For any pair of features with a Spearman correlation coefficient larger than 0.96, one feature was randomly selected to be kept, and the other removed from the selection. The dataset was normalized using the empirical distribution of the healthy subjects and a z-normalization was performed using the rank-based inverse normal transformation .
In order to select the most accurate and compact set of features from each dataset, the least absolute shrinkage and selection operator (LASSO) method was used . The shrinkage and selection method minimizes the sum of squared errors and penalizes the regression coefficients, as described by Eq. (3) as follows:
Given a set of input measurements x1…xn and an outcome y, the lasso method fits a linear model where xiis the covariate vector for the ith case and yiis the outcome, t is a tuning parameter that determines the amount of regularization, and N is the number of cases.
The multivariate search was performed using a class balanced data sample of 100 subjects for training and the remaining subjects as a blind test set. The models were calibrated using a leave-one-out cross-validation strategy, training the models at every split using N – 1 subjects and evaluating the model using the remaining subjects . The final reported performance was obtained by applying the final model gathered on the training stage and evaluating it in the blind test set.
A total of 1796 mammograms were successfully segmented. The image sets of nine subjects had to be removed from the experiment due to problems with the registration process, six were from MS, two from CS, and one from HS. All the remaining subjects were included in the subsequent stages of the analysis. The 2150 extracted features were filtered by the correlation process, removing 826 features.
Table 1 shows the features that were selected for each model: the CS versus HS (n = 12), and the MS versus HS (n = 16). The former achieved an accuracy of 0.825 with an AUC of 0.882 and the latter an accuracy of 0.698 with an AUC of 0.807. Figure 8 shows the ROC curves for both the models.
|CS versus HS||MS versus HS|
The proposed methodology is fully automated and does not require manual intervention as previous proposals [16, 17]. Although the approach is similar to others , we did not attempt to remove the pectoral muscle from the segmentation mask, since the presence of abnormal axillary lymph in this area is an indicator of occult breast carcinoma . However, from the computational point of view, the feature extraction process may be affected if the region processed is not well focused .
The proposed registration process achieved a good performance having only 2.0% of the subjects that had to be discarded due to registration issues. This performance is remarkable when considering the amount of deformation undergoing in a mammography procedure. The B-spline deformation is an improvement over rigid or affine co-registration methods . The advantage of the deformable registration has been recognized as a key element in breast analysis and has been successfully used in longitudinal studies . Regarding digital subtraction, the differences in the X-ray projection, and image acquisition and digitizing artifacts may affect the detection of asymmetric patterns. Our results indicate that even in the presence of registration artifacts, the digital subtraction added information that was successfully incorporated during the feature selection process.
The B-Spline transformation algorithm, proposed for the bilateral mammogram registration presented, shows a clear improvement after the registration. Due to lack of temporal mammograms, temporal registration was not tested. Nevertheless, the methodology could be implemented in such task. However, the temporal registration should be re-optimized using a new set of parameters.
The enhanced images and texture maps enriched the feature set providing a four-fold increase in extracting features per patient, which were also incorporated in the final classification models. Regarding symmetry, the strategy of exploring bilateral symmetry has been explored by other researchers where a series of features (signal, texture, breast density, etc.) were computed from each mammogram and the absolute difference between both breasts was obtained to measure breast tissue asymmetry, and used it to predict the likelihood of developing cancer . We extended this idea by registering the left and right images using a deformable transformation, which increased the number of features per patient by 25%.
This study shows that healthy subjects, subjects with calcifications, and subjects with masses can accurately be classified through models generated via mammography registration and a feature selection methodology. The analysis of the feature selection strategy demonstrated that even when using a different approach for the feature selection strategy, the proposed methodology achieved similar results as the previously presented ones. Therefore, we can say that the methodology is robust to the feature selection strategy.
The methodology demonstrated that the image subtraction of registered images generates information that aids in the identification of subjects with lesions, such as malignant masses and calcifications. The methodology also incorporated the use of feature-based asymmetry into the CADx system. The combination performance achieved has the potential to be used to queue cases with a high chance of malignant findings, or may have the practical use of triaging mammograms in developing countries where there is a deficiency of expert readers.