Deep Learning-Based Segmentation of Cellular Membranes in Colorectal Immunohistochemical Images

Abstract

The segmentation of cellular membranes is essential for getting crucial information in diagnosing several cancers, including lung, breast, colon, gastric cancer, etc. Manual segmentation of cellular membranes is a tedious, time-consuming routine and prone to error and inter-observer variation. So, it is one of the challenges that pathologists face in immunohistochemical (IHC) tissue images. Although automated segmentation of cellular membranes has recently gained considerable attention in digital pathology applications, little research is based on machine learning approaches. Therefore, this study proposes a deep framework for semantic segmenting cellular membranes using an end-to-end trainable Convolutional Neural Network (CNN) based on encoder and decoder architecture with Atreus Spatial Pyramid Pooling (ASPP). The backbone of the encoder depends on the residual architecture. The performance of the proposed framework was evalu ated and compared to other benchmark methods. As a result, we show that the proposed framework exhibits significant potential for cellular membranes segmentation in IHC images.

1. Introduction

Immunohistochemistry (IHC) is an efficient staining technique used in pathology to localise a certain antigen in a tissue specimen. Hence, It is now employed in state-of-the-art research to identify specific antigens within a tissue sample from formalin-fixed paraffin-embedded (FFPE) tissue, e.g. in tissue microarrays (TMAs) and 3D dimensional spheroids grown from cells. The cellular membranes segmentation of IHC images is usually required in histopathology to provide more relevant information for diagnosing particular cancers because specific tumour antigens are expressed in certain cancers. Glucose transporter-1 (GLUT-1) is one of the well-known biomarkers bound with the cellular membrane that induces and increases hypoxic conditions in different solid tumours, including breast, prostate, and colorectal cancer [1]. The production of Hypoxia-Inducible Factors (HIFs) proteins in tumour hypoxia regions activates GLUT-1 genes that promote hypoxia [2]. Oxygen gradient, supply and distribution in hypoxic areas lead to the difference in size and extension in all solid tumour regions [3]. Tumour hypoxia is a unique hallmark of cancer due to the difference in oxygen demand and supply, which produces cancer stem cell niche, resistance to therapy (chemotherapy and radiotherapy), immune damping, poor clinical prognosis and genomic instability [3, 4]. Currently, tumour hypoxia is receiving significant attention as the centre for the hallmarks of cancer; this is because of its many characteristics of chemotherapy and radiotherapy resistance and a primary prognostic factor [5]. In clinical practice, assessing the development and spread of hypoxia across solid tumours is an essential routine performed by pathologists to describe the appropriate therapy. In the deep hypoxic environment, It is getting more difficult for chemotherapy and radiotherapy to reach tumour sites. Thus, the need for using hypoxia-activated pro-drugs (HAPs) as targeted therapy is needed.

In recent years, Computer-Aided Diagnosis (CAD) technologies have emerged as one of the potential solutions for histopathological image analysis. The CAD technologies have been used to quantitatively and objectively evaluate IHC biomarkers in a whole tissue slide or a specific region of interest delineated by a pathologist. So, they have been employed to assist histopathologists in some laborious routines, such as visual examination of IHC images for scoring and segmenting the cellular membrane. Thus, the advantages of CAD technologies are avoiding inconsistency in the diagnosis among pathologists, improving the diagnosis quality and reducing the diagnostic time. On the other hand, machine learning-based CAD technologies rely heavily on hand-crafted features that can be significantly prone to feature extractor bias. In addition, relevant domain knowledge is necessary to select the valuable features. Thus, hand-crafted techniques can only deal with some low-level information of images. In contrast, deep learning-based CAD techniques are characterised by; 1) Their ability to extract high-level abstract features from images automatically in a standardised way [6, 7], 2) Their ability to analyse entire slides in detail rather than focusing on a region of interest (ROI) [8], 3) Their ability to learn complex mapping functions directly from the input data and 4) Their ability to avoid personal user bias, as it does not require manual extraction of specific visual features [9]. Hence, they deliver unbiased outcomes for dataset images [9]. So, this work proposes deep learning based-segmentation of the cellular membranes in colorectal IHC images.

The remainder of this chapter is organised as follows. Section 2 presents the related works. Section 3 provides the materials, proposed framework and evaluation indices that are used in this work. Section 4 reports the experimental results from the proposed model. Finally, in Section 5, we discuss analyses of the results and conclude the chapter.

2. Related work

Automated segmentation of cellular membranes has received much attention lately. Various machine learning-based approaches for segmenting the cell membranes in IHC images have been proposed [10, 11, 12, 13, 14]. Chang et al. [10] employed the colour channel to extract the morphology, texture and intensity features that then have been utilised to train the support vector machine (SVM) classifier. Tuominen et al. [12] employed conventional machine learning techniques and ImageJ in preparing their ImmunoMembrane web application. Kuo et al. [14] employed a watershed algorithm for nucleus segmentation. But all these algorithms are not up to the performance of deep learning approaches. Therefore, there is a demanding need to employ deep learning approaches, which are advanced machine learning approaches for solving the current issues related to membrane segmentation. Several researchers [15, 16, 17] investigated deep learning approaches for cell membranes segmentation. Khameneh et al. [15] employed the SVM classifier to specify ROI and then the deep U-net model for segmenting membrane regions. Saha et al. [16] proposed a long short term memory (LSTM) architecture to detect cell membrane and nucleus. Gaur et al. [17] proposed deep CNN based on the active learning technique for membrane segmentation.

An atrous convolution layer is introduced in deepLab-v1 [18] network to widen the receptive field of view over the input feature maps without a decrease in spatial dimensions and an increase in the number of network parameters. And then, multiple parallel atrous convolutional layers with different dilution rates are proposed in the DeepLab v2 [19] network to segment objects at multiple scales. These layers are known as Atrous Spatial Pyramid Pooling (ASPP) model. After that, the ASPP model is improved in DeepLab v3 [20] to concatenate the image level features, a 1x1 convolution and three 3x3 atrous convolutions with different dilution rates. Encoder-Decoder structure and ASPP model are integrated into Deeplab v3+ [21] for applying the depth-wise separable convolution in both ASPP and Decoder modules. The encoder module reduces the spatial dimensions of the feature maps through the repeated application for the convolution and pooling layers, whilst the decoder module gradually recovers the spatial dimensions by using de-convolution and upsampling layers. Then, skip connections are introduced between the encoder and decoder modules to have sharper segmentation results.

In this work, we propose a trainable CNN based-detector to incorporate encoder, ASPP and decoder. We leveraged the identity mappings proposed by He et al. (2016a,b) in their Residual architectures. So, the encoder part employs the pre-trained ResNet-50 network [22] trained on the ImageNet [23] dataset as the feature extractor. Hence, through the proposed detector, we can overcome the challenges of 1) training the entire network from scratch, 2) the data scarcity problem and its consequences, and 3) over-fitting and poor generation of features. The main contributions of our proposed framework are as follows:

• The proposed detector introduces a new way of deep learning-based semantic segmentation of cell membranes.

• The implementation of the ASPP structure improves the performance of the proposed detector.

• The proposed framework incorporated an encoder-decoder network and dilated/atrous convolutions units to tackle the presence of pooling layers issue that reduces the feature maps dimensions ignore the positional information of objects.

• It is value addition in terms of the main quantification in the existing techniques for segmenting the cellular membrane.

3. Material and evaluation methods

3.1 Image dataset acquisition

We evaluated the proposed detector on real IHC dataset images. Dataset images were acquired from the institute of cancer therapeutics at University of Bradford, United Kingdom. The authors have obtained ethical approval for publication. The images were obtained by immunohistochemistry staining on colorectal cancer adenocarcinoma of human clinical specimens. GLUT-1 and ALDH7A1 prepared the immunohistochemistry staining on HT-29 Tissue Microarray (TMA). The TMA clinical sample slide holds 150 cores and has the number G063 (Biomax.us). These cores represent the whole side of the clinical sample and give a total of 50 cases of colorectal cancer in each whole TMA slide. Whereas 100 cores are colorectal tissues, and 50 cores were either malignant, adjacent tissue to the cancer tissue or normal tissues. The clinical samples were collected from colorectal cancer patients (male and female) in July and August 2019. The IHC images were scanned using an Aperio Digital Pathology Slide Scanners (Aperio AT2) and then captured with ×20 magnification and 200 μm diameter. The whole cores and examples of GLUT-1 expression of IHC colon adenocarcinoma images are shown in Figures 1 and 2, respectively.

Figure 1.

The G063 TMA slide with 150 cores.

Figure 2.

Examples of GLUT-1 expression at different stages of IHC colon adenocarcinoma images; magnification of upper panel is ×5 and bottom panel is ×20.

3.2 Pre-processing of dataset images

In a pre-processing step, there are procedures will be applied to the dataset images as follows;

3.2.1 Derivation of dataset images

We derived dataset images by hand-picking the ROI from TMA cores that comprised the most representative cells membrane stained. At first, we ran Aperio AT2 program and then selected the ROI regions at a (5× magnification) low resolution. We captured the images from the ROI regions with 20× magnification. A total of 400 IHC images were extracted with size 512 × 512 pixel and then stored in jpeg compression format.

3.2.2 Annotation of dataset images

Two trained pathologists manually annotated the cellular membranes of IHC images according to the proportion of Glut-1 and ALDH7A1 staining. The ground truth images are generated in MATLAB R2020a environment.

3.2.3 Stain normalisation of dataset images

In order to highlight the diaminobenzidine (DAB) stain regions of reactive membranes in the IHC images, we utilised a colour normalisation method described in [24].

3.2.4 Partition of dataset images

We split the dataset images randomly into 80% training set (320 images) and 20% testing set (80 images). The testing set does not utilise for training our proposed detector.

3.2.5 Augmentation of dataset images

Data augmentation is an essential step to generate additional artificial training images by using some transformations for increasing the deep network performance [25, 26]. In this work, we augmented the training images and their ground truth images of our IHC dataset by rotating them with angles of 90, 180 and 270 degrees and then flipping in the horizontal and vertical direction. We chose the rotate and flip transformations to enlarge the training images without affecting the quality of input images [27] and thus avoid the features poorly generalisation and over-fitting problems [28].

3.3 The evaluation index

The segmentation performance of the proposed detector was assessed by using the popular four evaluation criteria. These criteria use the following metrics; TP, FP, FN and TN denote respectively the number of true positive, false positive, false negative and true negative from all images in the dataset. True positive (TP) is counted as the intersection of a segmented cell membrane with its ground truth; otherwise, it is counted as false positive (FP). False negative (FN) is calculated as the missed parts of the ground truth, and true negative (TN), parts of the image beyond the union segmentation plus ground truth.

3.3.1 Network accuracy metric

This criterion is used to measure a network’s ability to segment. It indicates correctly predicted observations against total observations and it is calculated as follows:

$$\mathit{\text{Accuracy}}=\frac{\mathit{TP}+\mathit{TN}}{\mathit{TP}+\mathit{FP}+\mathit{FN}+\mathit{TN}}$$E1

3.3.2 Detection accuracy metric

This criterion is used F1-score metric to measure the detection accuracy of individual cellular membranes. The F1-score is defined by both Precision and Recall metrics. Precision metric indicates the correctly predicted positive observations against total predicted positive observations, whilst Recall metric indicates correctly predicted positive observations against total actual positive observations. F1-score is calculated as follows:

$$F_1\mathit{\text{Score}}=\frac{2.\mathit{\text{Precision}}.\mathit{\text{Recall}}}{\mathit{\text{Precision}}+\mathit{\text{Recall}}}=\frac{2\mathit{TP}}{2\mathit{TP}+\mathit{FN}+\mathit{FP}}$$E2

where

$$\mathit{\text{Precision}}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FP}}$$

and

$$\mathit{\text{Recall}}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FN}}$$

3.3.3 Shape similarity metric

This criterion is used Intersection over Union (IoU) also known as Jaccard Similarity Coefficient to compare similarities between segmented cell membranes and their ground truth. The Jaccard index is calculated as follows:

$$\mathit{\text{Jaccard}}\left(\mathit{IoU}\right)=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FP}+\mathit{FN}}$$E3

3.3.4 Dice coefficient score

This criterion is used to measure the agreement among segmented cell membranes and their ground truth at object level. The Dice score metric is used the ground set (G) and segment set (S). G; a group of pixels belonging to a ground truth object, and S; a group of pixels belonging to a segmented object. It is utilised to measure similarity between G and S and produces scores between 0 and 1, where 1 indicates perfect segmentation. It is calculated as follows:

$$\mathit{\text{Dice}}\left(G,S\right)=\frac{2\left|G\cap S\right|}{2\left|G\cap S\right|+\left|G\right|+\left|S\right|}$$E4

3.4 Proposed methodology

We propose a semantic level segmentation of celluar membranes using an end-to-end trainable CNN based on integrates three modules; an encoder, an ASPP, and a decoder. We adapt the ResNet-50 [22] pre-trained on ImageNet [23] as the backbone for encoder module. The inputs are first passed through an extended ResNet50 network, followed by an ASPP module for multi-scale image processing and a decoder module to resize the images to the original input dimensions and produce sharp segmentation results. Figure 3 shows our architecture and its three main modules. In the following section, there is a brief description of each module:

Figure 3.

Show the proposed network architecture, “CONV” represents the convolution blocks that followed by rectified linear unit activation layer (ReLU) and batch normalisation layer (BN); “ASC” represents the atrous separable convolution blocks; “#F” represents the output number of filters for block; “S” represents the stride of all convolutions; P is padding.

3.4.1 Encoder module

It acts as a feature extractor that uses several residual units to reduce the size of an input image. It contains the following blocks;

• An initial block consists of a 7 × 7 convolutional layer with stride 2 and a 3 × 3 max-pooling layer with stride 2, so the output size after this block is reduced by four times (Output stride = 4), and its output is 64 channels.

• Res2 Unit is composed of three blocks (residual error units), all of which are small convolutions of 1 × 1 or 3 × 3, and stride is 1, so the output size after Res2 block is still lower than the original image by four times, the Res2 output is 256 channels.

• Res3 Unit is composed of four blocks, among which the 3 × 3 convolution kernel stride of the first block is 2. Therefore, the output size after Res3 is reduced by eight times relative to the original image (output-stride = 8), and the output of Res3 is 512 channels.

• Res4 Unit is composed of six blocks. The first block is a regular residual block, and the 3 × 3 convolution of the following five blocks use a hole convolution with rate = 2. Therefore, the output size after Res4 is reduced by eight times relative to the original image (output-stride = 8), but the receptive field is reduced by four times close to the original image. The Res4 output is 1024 channels.

• Res5 Unit is composed of three blocks. The first block is the same as Res4, which is a residual with rate = 2. The rate of the 3 × 3 convolution in the last two blocks is 4. Therefore, the output size after Res5 is reduced by eight times relative to the original image (output-stride = 8), but the receptive field is as large as the original image. The Res5 output is 2048 channels.

3.4.2 ASPP module

It applies four parallel Atrous Separable Convolutions (ASC) with different dilation rates; this allows analysing the extracted features at different scales. Whereas, The ASC is a depthwise convolution with atrous convolutions followed by a pointwise convolution. The ASPP output is 1024 channels.

3.4.3 Decoder module

The outputs of the ASPP module are concatenated and passed through a 1 × 1 convolution to reduce the number of channels to 256. This result is upsampled by a factor of four and concatenated with low-level features of the same dimension.

Since the input structure should be aligned with the output structure, thus it is appropriate to share the low-level information, such as edges or shapes, with the higher ones. Then, we apply two 3 × 3 ASCs and, finally, a 1 × 1 convolution with two-channel, so that a binary mask is obtained. This result is upsampled by a factor of four to recover the original size of the image.

4. Experiment and results

In this section, we describe our experiment, analyse the experimental results obtained and then compare the obtained results with the other networks results produced for the same purpose.

4.1 Setup the experiment

The experiment was implemented in MATLAB R2020a on a PC with Intel⊗R, Dual-Core i7–7700 at 3.60 GHz CPU, 64 GB RAM, and NVIDIA GeForce GTX.

1070 GPU. The experiment was carried out using our IHC dataset images as follows;

• The augmented training dataset images were divided randomly into training set 80% (x images) and test set 20% (y images).

• The network is constructed by initialising the weights from a pre-trained ResNet50 model.

• The proposed network was trained by tuning hyperparameters as shown in Table 1.

Hyperparameter	Value
Optimizer	SGDM
LearnRateSchedule	‘piecewise’
LearnRateDropPeriod	10
LearnRateDropFactor	0.3
Momentum	0.9
InitialLearnRate	0.001
L2Regularization	0.005
MaxEpochs	30
MiniBatchSize	8
Shuffle	‘every-epoch’
ValidationPatience	10
ValidationFrequency	1 per epoch

Table 1.

Hyperparameter.

4.2 Results analysis

4.2.1 Traing Progress analysis

The evaluation of proposed detector accuracy and loss function over training time on our IHC dataset images is shown in Figure 4. After each training epoch, the accuracy and the loss function were calculated on the training and validation sets to observe its ability to generalise and avoid overfitting. Throughout the training period, there were raised in the performance in the training set, which corresponded to a decrease in loss value. This behaviour is because the model is still learned to be able to generalise well; however, when the model is able to generalise, the validation loss fluctuates with close to the training loss. As shown in Figure 4, the proposed detector converges in the training process through the first 40 epochs, and there are wide fluctuations in performance in the training set, which correspond to fluctuations in loss value. This is expected behaviour during the first epochs of training since the model is still unstable; however, when the model stabilises, the training loss becomes steady with a slight variation close to the validation loss.

Figure 4.

The training, loss and validation values over the training time.

4.2.2 Performance analysis

A 2 × 2 confusion matrix was used to represent the prediction results of the cellular membranes. The matrix was built on two rows and two columns: membranes and non-membranes representing the classes. The 2 × 2 normalised confusion matrix is shown in Figure 5. To statistically analyse the behaviour of our detector at the pixel level, we calculated metrics from the test set using equations; 1, 2 and 3, as reported in Table 2. Whereas Dice-Coefficient was calculated at object level using eq. 4.

Figure 5.

Normalised confusion matrix for proposed detector.

Metrics	Proposed Detector
Accuracy	0.9227
Recall	0.9335
Precision	0.8874
F1-score	0.9099
IoU	0.8347
Dice Index	0.829

Table 2.

The statistical metrics of our detector.

4.2.3 Comparative analysis

To our knowledge, there are few works that employed a deep learning models to segment cell membranes in IHC images. Hence, to validate our detector and to compare its performance with the state of the art segmentation methods, we first implemented some the public pre-trained models including FCN-8 [29], U-Net [30] and SegNet [31]. The comparative analysis of our detector against the other networks is reported in Table 3. When comparing the results produced by our detector with others, we derive that the proposed detector achieved good performance metrics. We get high performance than the popular networks; SegNet, U-Net and FCN-8. At pixel-level, it achieved an F-score value of 0.910. At the object level, it achieved a Dice score value of 0.829.

Deep model	F1 Score
Eycke et al. [32]	0.844
SegNet	0.858
U-Net	0.862
FCN-8	0.783
Proposed Detector	0.910

Table 3.

Comparative analysis of different models on the IHC dataset.

5. Conclusions

Although the number of training images in our IHC dataset was small, it is observed from the obtained results in these experiments that the proposed detector has significantly achieved promising results in semantic segmentation. This is due to its architecture, which employs the following; firstly, it uses atrous/dilated convolution layers as a way to widen the field of view over the input feature maps without increasing the number of parameters. It also uses the ASPP module to deal with the different scales problem of objects in the image. Furthermore, It uses an encoder-decoder architecture. Hence it reduces the resulting output dimensions through passing multiple convolution layers with strides of 1 or more to avoid pooling layers in the network. Finally, it passes the output through a decoder with learnable parameters to regain the original dimensions. For this reason, we have chosen this architecture for the proposed detector to segment the cellular membranes of colorectal IHC images semantically.

In conclusion, we have presented an end-to-end trainable deep neural network to tackle the problem of cellular membranes in colorectal IHC images. The proposed architecture has achieved a good performance compared with other methods. Hence, the proposed detector is able to objectively and automatically detect glands, thus easing the burden of pathologists.

Conflict of interest

The authors declare that they have no conflict of interest.