Image analysis and quantification of Haematopoietic stem cells (HSCs) position within their surrounding microenvironment in the bone marrow is a fast growing area of research, as it holds the key to understanding the dynamics of HSC-niche interactions and their multiple implications in normal tissue development and in response to various stress events. However, this area of research is very challenging due to the complex cellular structure of such images. Therefore, automated image analysis tools are required to simplify the biological interpretation of 3D HSC microenvironment images. In this chapter, we describe how 3D intravital microscopy data can be visualised and analysed using a computational method that allows the automated quantification of HSC position relative to surrounding niche components.
- intravital microscopy
- 3D image analysis
- bone marrow visualisation
- haematopoietic stem cell niche
- object segmentation and classification
Somatic stem cells have the extraordinary ability to maintain their own pool, while replenishing dead cells and regenerating tissues after injuries throughout our lifetime. Dividing stem cells have the potential to differentiate into other cell types such as blood cells, skin cells or brain cells, or maintain themselves through a process called “self-renewal”. The stem cells that form blood and immune cells are called haematopoietic stem cells (HSCs). These stem cells are responsible for turnover and maintenance of red blood cells, platelets and immune cells. During differentiation, HSCs generate multi-potent and lineage-committed progenitor cells prior to attaining maturity, which results in the generation of billions of new blood cells every day .
The bone marrow (BM) is the main site, where HSCs and their immediate progeny reside, precisely contained in a regulated and a very complex environment called the niche. The niche microenvironment has a direct impact on the function of HSCs, determined by the interaction of the HSCs themselves with particular cellular and molecular components in their surroundings [1, 2].
To observe and understand such interactions, single-cell resolution intravital microscopy of fluorescent HSCs and niche components is a vital tool, allowing the study of stem cells, their behaviour and interactions during steady state, aging and disease [3, 4]. This method has been successfully utilised to directly observe the HSCs with their niche in mouse bone marrow. Fluorescent dyes were used to label HSCs prior to transplantation into transgenic reporter mice that expressed GFP under the control of an osteoblast-specific promoter. Osteoblast cells reside on the bone marrow endosteal surface and contribute to the formation of bone . This kind of fluorescent niche component served as a fundamental tool for the visualisation of the HSC endosteal niche and resulted in multiple observations indicating that functional HSCs localise near osteoblastic cells [5–7]. Intravital microscopy of HSCs injected into wild type and other bone marrow stroma/haematopoietic reporters indicated the importance of GalphaS receptor subunit on homing HSCs , of interactions with nestin-positive mesenchymal stem and progenitor cells  and with regulatory T cells . Time-lapse imaging has revealed that infection-exposed HSCs interact with larger endosteal niches [7, 11] and that aged HSCs interact with the bone marrow differently than young ones . However, a clear understanding of how the coordinated action of multiple niche components regulates HSC fate is not clear, and considering the localization of HSCs relative to multiple surrounding cellular and structural constituents of the bone marrow microenvironment is the first step towards unravelling HSCs dynamics and signal exchanges.
Even though manual distance measurement of HSCs to other niche components is possible [1, 6, 13], it suffers from various limitations including the extensive time spent on conducting such manual analysis as well as intra- and inter-researcher inconsistencies and human error. Therefore, a specialised image analysis tool capable of completing such tasks will add the great benefits of producing consistent unbiased results, and will simplify the interpretation of the biological microenvironment events of the bone marrow niche.
During the last decade, several attempts have been made to develop image analysis approaches to automate the quantification of medical and biological images, from cell segmentation and detection to tumour diagnosis and classification [14–16]. However, the direct application of such methods to three-dimensional (3D) intravital microscopy images of the bone marrow is not possible. This is due to the complex structure of the tissue, decreased signal with increased depths, unpredictable scattering of both the excitation and emission photons and the irregular encasing bone structure, which restricts the resolution of intravital microscopy images when compared to
Consequently, we developed an image analysis pipeline, to specifically tackle all the main challenges associated with intravital microscopy 3D images of bone marrow. The approach starts by detecting and segmenting HSCs and surrounding niche components, for instance, the bone and osteoblast. Then it classifies each segmented HSCs and assigns them into one of three classes, and finally it quantifies them based on their location to surrounding niche components. This analysis will allow the extraction of variable quantitative features, which could potentially reveal novel aspects of HSC biology, and especially their relation to the bone marrow microenvironment .
This chapter is organised as follows: first an introduction of the bone marrow and haematopoietic stem cells niche highlighting the importance of intravital microscopy and automated image analysis is provided (Section 2), followed by a detailed description of intravital microscopy of the bone marrow and the main challenges faced during the image analysis stage (Section 3). The automated image analysis tool is then described (Section 4), detailing how the three main segmentation parameters can be optimised for a particular cell or niche component (Section 5). Finally, a description of how a supervised machine learning classifier can be used to classify both HSCs and vasculature is presented (Section 6), followed by a brief description of how to obtain 3D positional measurements of HSCs to other niche components (Section 7).
2. Intravital microscopy of mouse bone marrow
Nevertheless, how multiple niche components affect and regulate HSCs fate is still a significant question that scientists are attempting to answer. Recording the localization of HSCs relative to multiple surrounding cellular and structural constituents of the bone marrow microenvironment is the first step towards understanding such phenomena.
Confocal fluorescence and second harmonic generation microscopy of HSCs and their niche components in the mouse calvarium (top of the skull) bone marrow provides a powerful tool to observe cellular interactions and has been successfully used to detect and study fluorescently labelled HSCs and GFP expressing osteoblasts and other niche components in transgenic reporter mice [7, 11].
Confocal microscopy provides a better 3D optical resolution than epifluorescence microscopy because it restricts the light that reaches the photomultiplier through a pinhole. While in epifluorescence microscopy in-focus image objects are mixed with out-of-focus image information arising from regions outside the focal plane, in confocal microscopy the pinhole blocks out-of-focus signal. This technology has allowed major advances in the field of biological imaging, due to its cost-effective solution and ease of use. In addition, confocal microscopes have improved sensitivity, resolution and intensity compared to epifluorescent microscopes. Confocal microscopy is also less phototoxic and can therefore be used to generate not only static but also dynamic data of living cells within their tissues, through time-lapse acquisition. All these qualities make confocal microscopes an ideal instrument for
Two-photon excitation of a fluorophore provides 3D optical sectioning similar to confocal imaging. However, it uses a wavelength roughly twice the length of the absorption peak of the specimen being imaged. In this chapter, we describe a particular application of two-photon excitation, named second harmonic generation (SHG). This takes advantage of a characteristic specific to certain molecules, such as collagen, which is the emission of photons at half of the wavelength of the exciting photons. Because collagen is one of the main components of bone, SHG signal allows detection of calvarium bone without the need to use specific transgenic fluorescent reporter mouse strains. An example of raw bone marrow
3. Application and challenges of bone marrow intravital microscopy image analysis
The combination of confocal and SHG microscopy has greatly simplified high-resolution fluorescence imaging of animal tissues and organs. In Section 2, we have briefly described the advantages of the combination of confocal/SHG microscopy. Here, we describe the experimental set-up used and the challenges posed by the intravital images when they have to be further analysed. In the following sections, we describe in details how HSCs and bone marrow components can be automatically segmented, classified and measured.
To visualise HSCs, they were first labelled
Another challenge imposed on image analysis from
To address these issues, we developed a local heterogeneity-based image segmentation (LH-SEG) approach  that employs multi-resolution segmentation  and mean intensity difference to neighbour thresholding. The approach works by comparing local morphological and intensity characteristics of objects, which most often are smaller than the cells and structures recorded. These detected objects are then grouped based on their homogeneity with other neighbouring objects within a defined distance. Because the method is applied to each 2D slide, it ensures reliable edge detection and segmentation across cells and structures with high intensity heterogeneity, despite the decreased signal (intensity) at increasing depths.
The dye often diffuses from the labelled cells into the surroundings, and is diluted upon cell division, resulting in a loss of brightness and intensity of signal.
Labelling HSCs does not always provide a homogenous staining, which can result in some HSCs being brighter than others.
The dye often leads to background noise as result of cell debris and aggregate signal.
Figure 2 shows examples of the points listed above.
To overcome these issues, LH-SEG detects cells with variable intensities and signal brightness. Moreover, a machine learning classification protocol based on morphological and textural features recognises and classifies all segmented DiD signal, to differentiate real HSCs from cell debris and aggregate.
4. Image analysis
Image analysis is the process of extracting meaningful information from images, using manual or automated methods (the latter known as computer vision techniques) [20, 21]. The selection of the appropriate image analysis method for a certain type of images determines the success rate of the analysis. Therefore, understanding the challenges associated with the images acquired is the first step towards developing an effective image analysis solution.
Automated image analysis has a number of advantages over the human manual analysis. Human vision can easily be biased by pre-conceived concepts, affecting the output results and the rigorous testing of hypothesis. Manual analysis can also be time-consuming compared to automated analysis tasks, in which large datasets can be batch processed without the need of human monitoring, allowing users to perform other tasks while the analysis is being carried out.
Microscopy images are often complex, with a range of artefacts and background noise, which require variable image processing steps before any meaningful quantification can be extracted from objects and region of interests. An overall image analysis protocol, including image processing, analysis and data output, needs to be designed and tailored according to the targeted image datasets. A general outline of the image analysis pipeline described in this chapter is provided in Figure 3.
4.1. Image acquisition and processing
Image acquisition is the process of obtaining the raw image data using a microscope. Such data usually contain a number of imperfections, for example, due to oversaturation, out of focus signal and uneven excitation due to the irregular structure of the tissue itself.
To overcome these issues, filtering techniques are usually introduced to the image analysis as a pre-processing step. Smoothing and de-nosing filters such as the median filter and Gaussian filter are applied to enhance image quality and reduces the noise introduced by the image acquisition process.
In the case of our
4.2. Image segmentation
Object detection is a crucial step, as it determines the objects of interest in an image. This step ultimately governs the quality of quantifications extracted from each object. A very effective method of object detection is image segmentation. Segmentation is the process of dividing an image into smaller meaningful segments, by selecting a group of pixels to represent a region, or an object, contained within a border. Objects are segmented such that the pixels enclosed within one border share certain characteristics that define them as an object.
Successful segmentation is dependent on the method used, which critically needs to be tailored for each particular set of images. Images with a high level of contrast between objects of interests and background and with uniform intensities can be segmented using simple thresholding and edge detection approaches. More complex images with high intensity heterogeneity and intricate structures such as
The main challenges imposed upon segmentation of bone marrow
This segmentation approach was mainly developed to minimise artefacts due to loss of signal with increased depths. The first step of LH-SEG, multi-resolution segmentation, starts by dividing each 2D slice of the 3D stack into smaller segments grouped by their homogeneity in shape and texture. Homogeneity for each pixel is then calculated by selecting a scale parameter α. The parameter α is optimised for each object type (HSC, osteoblast and bone), taking into consideration their morphological and textural characteristics. Optimisation of this parameter for each object type is described in detail in Sections 4.3 and 5.
Adjacent homogeneous image segments are then merged based on their mean intensity difference to neighbourhood (MDN) threshold. MDN threshold calculates the difference between an image segment and its neighbouring image segments using the mean intensity values described in  as follows:
Figure 4 shows how, at the end of this process, segmented images on each
4.3. Parameter optimisation
Image segmentation is a fundamental step of image analysis, which ultimately determines the level of success of the overall quantification. Automated image analysis approaches can only perform sufficiently well when their parameters are optimally set for a certain type of images or dataset. In this section, we describe how the optimal settings can be selected for datasets containing intravital microscopy bone marrow 3D stacks. Three fundamental parameters are targeted: (1) the multi-resolution segmentation scale parameter
The selection of the scale parameter
MDN threshold has two parameters: MDN threshold value and the distance to neighbouring image segments
5. Parameter optimisation
In this section, we present examples of the results obtained when different parameter values for
5.1. Multi-resolution parameter
Selecting a small value such as
Selecting a small value such as
Selecting a small value such as
5.2. MDN threshold value
5.2.1. for HSCs
The optimisation of the MDN threshold value depends on the correct detection of HSC edges. To observe the effects of changing the MDN threshold on HSC segmentation, we set
5.2.2. for osteoblasts
To observe the effects of changing the MDN threshold on osteoblastic cell segmentation, we set
5.2.3. for bone
To observe the effects of changing the MDN threshold on bone segmentation, we set
5.3. Distance to neighbour
To illustrate the process of selecting the optimal distance to neighbour
We set the optimal multi-resolution scale parameter
To show the effects of changing the neighbourhood size
To demonstrate the effect of the selection of the value of parameter
To show the effect of changing the neighbourhood size
Note: Parameter selection can be further observed on the
6. Machine learning
Machine learning (ML) has become a valuable artificial intelligence tool, increasingly used for analysis of complex image data [23, 24]. ML serves two main objectives: classification and regression. Classification approaches are performed when a computer is given a set of options and is expected to divide them into a subset of categorise. Regression is the process of selecting the appropriate response to a particular situation from a set of possible responses. Furthermore, ML classifiers can be categorised into two main models: supervised and unsupervised classifiers. Unsupervised approaches utilise a clustering technique, where ML looks for resemblances across data, and then splits the same data into clusters. Clusters are then used to define the classes. Conversely, supervised approaches require a training set, where the ML learns information by extracting a set of features from that particular set, and the ML classifier is then expected to classify a new set of data and categorise it into a
6.1. Decision tree classifier
The decision tree (DT) classifier is a structured approach that builds classification models from an input dataset to predict the output of an unknown dataset. The DT classifier starts the classification procedure structurally from the top node and uses the feature vectors to split it into further nodes [25, 26]. The selection of the feature vectors is measured by the purity of a particular subset, such that if a pure subset is produced, the splitting stops, otherwise splitting continues until a pure subset is allocated. Impurity of a specific subset is calculated by the entropy and is defined in  as follows:
In this chapter, we describe the application of the DT classifier for classifying DiD signal (potential HSCs) and vasculature structures. The DT classifier was selected in this study due to its computational simplicity and illustrative attributes represented in the DT output. The classifier also has the advantage of being able to select discriminative features without a prior step of feature optimisation, allowing the incorporation of a wide range of features for testing at the training stage.
6.2. Training and testing the classifier
LH-SEG provides a sufficient level of DiD objects detection. However, not all the DiD objects are HSCs. This is due to cell debris and the diffusion of the dye, which could lead to aggregates producing signal similar in shape and intensity to that of the HSCs.
In order to identify HSCs, we propose the use of DT classifier. We first trained our classifier to distinguish three classes of DiD objects: Class-1 represents HSCs distinguished by their smoother, round surface and high intensity as observed for quiescent HSCs; Class-2 comprises HSCs that have less rounded shapes and present small uropod-like protrusions , as previously observed in time-lapse images of migratory HSCs ; and Class-3 contains DiD objects that are not HSCs and characterised by their exceedingly uneven morphology. A selection of objects was manually selected for each category and fed into the classifier as training set. The exact number of objects needed for training each class depends on the variability of the structures contained, such that objects with regular, predictable characteristics such as Class-1 HSCs would require fewer objects for training, while irregular objects containing high variability such as class-3 would require a higher number of objects for training to ensure the classifier collects sufficient numbers of characteristics to handle the complexity and inconsistency of objects. After training, the classifier is tested on different datasets from those used for training with a 3-fold cross-validation approach. Classification results provided a high accuracy for all classes and successfully classified the DiD object to the three proposed classes as described in  and Figure 9A and B.
Blood vessels inside the bone marrow are another component that plays a vital role in the maintenance of HSCs. Interactions between vasculature and HSCs have become one of the main headlines for many recent studies [29, 30]. As demonstrated in our pervious study, segmentation of vasculature of
7. 3D positional measurements
Following segmentation and classification of DiD-labelled HSCs, HSCs position relative to their nearest osteoblast and bone is measured as described in , and could be measured to their nearest vessel or other segmented bone marrow components. The overall automated distance measurements from HSCs to osteoblast and bone were equivalent to previously published data obtained using manual distance measurements and described in [5, 6]. Further quantitative measures such as the morphological and textural characteristics of each segmented and classified object can also be obtained using this tool.
In this chapter, we describe an approach for the visualisation, analysis and quantification of HSCs within their surrounding microenvironment in the bone marrow using intravital imaging and image analysis approaches.
Confocal fluorescence and SHG microscopy combined facilitate the collection of large amounts of image data of HSCs and their niche components. They specifically allowed the first observations of the cellular interactions of
An image analysis approach combining image segmentation (LH-SEG) and machine learning classification detects and defines the edges of HSCs, osteoblast, bone and blood vessels. Subsequently, detected HSC and vasculature objects are fed into a decision tree classifier for categorising the detected signal into further classes. HSC signal was categorised into three classes representative of their morphological and intensity characteristics, proving high levels of accuracy for detection of HSCs and elimination of DiD debris and background signal. Furthermore, vasculature in the bone marrow was categorised into four classes based on their intrinsic morphological and position features. This will simplify the analysis of this component and allow simultaneous analysis of other components, increasing our understanding of the complexity of the HSC niche. After all the biologically important objects have been segmented and classified, it is then possible to derive quantitative information from these features and further translate it into a deeper understanding of the biological events observed.
This automated image analysis pipeline was particularly designed to address the major challenges imposed on
In conclusion, once the correct parameters are selected as we describe, this approach is applicable to the analysis of a broad range of 3D and intravital microscopy images from other tissues and organs, leading to a faster pace of discovery not only in the haematopoiesis and bone marrow field, but also in other somatic stem cell studies, and developmental and cancer biology studies. Further development of the described tool can be expanded to more programming languages and software platforms. This will accelerate the productivity of the method and allow an even wider range of applications.
C. Lo Celso, C. P. Lin, and D. T. Scadden, “In vivo imaging of transplanted haematopoietic stem and progenitor cells in mouse calvarium bone marrow,” Nat Protoc, vol. 6, pp. 1–14, Jan 2011.
S. J. Morrison and D. T. Scadden, “The bone marrow niche for haematopoietic stem cells,” Nature, vol. 505, pp. 327–34, Jan 16 2014.
L. Ritsma, S. I. Ellenbroek, A. Zomer, H. J. Snippert, F. J. de Sauvage, B. D. Simons, et al., “Intestinal crypt homeostasis revealed at single-stem-cell level by in vivo live imaging,” Nature, vol. 507, pp. 362–5, Mar 20 2014.
A. Kohler, V. Schmithorst, M. D. Filippi, M. A. Ryan, D. Daria, M. Gunzer, et al., “Altered cellular dynamics and endosteal location of aged early haematopoietic progenitor cells revealed by time-lapse intravital imaging in long bones,” Blood, vol. 114, pp. 290–8, Jul 9 2009.
C. Lo Celso, H. E. Fleming, J. W. Wu, C. X. Zhao, S. Miake-Lye, J. Fujisaki, et al., “Live-animal tracking of individual haematopoietic stem/progenitor cells in their niche,” Nature, vol. 457, pp. 92–6, Jan 1 2009.
S. W. Lane, Y. J. Wang, C. Lo Celso, C. Ragu, L. Bullinger, S. M. Sykes, et al., “Differential niche and Wnt requirements during acute myeloid leukemia progression,” Blood, vol. 118, pp. 2849–56, Sep 8 2011.
N. M. Rashidi, M. K. Scott, N. Scherf, A. Krinner, J. S. Kalchschmidt, K. Gounaris, et al., “In vivo time-lapse imaging of mouse bone marrow reveals differential niche engagement by quiescent and naturally activated haematopoietic stem cells,” Blood, May 21 2014.
G. B. Adams, I. R. Alley, U. I. Chung, K. T. Chabner, N. T. Jeanson, C. Lo Celso, et al., “Haematopoietic stem cells depend on Galpha(s)-mediated signalling to engraft bone marrow,” Nature, vol. 459, pp. 103–7, May 7 2009.
S. Méndez-Ferrer, T. V. Michurina, F. Ferraro, A. R. Mazloom, B. D. MacArthur, S. A. Lira, et al., “Mesenchymal and haematopoietic stem cells form a unique bone marrow niche,” Nature, vol. 466, pp. 829–834, 2010.
J. Fujisaki, J. Wu, A. L. Carlson, L. Silberstein, P. Putheti, R. Larocca, et al., “In vivo imaging of Treg cells providing immune privilege to the haematopoietic stem-cell niche,” Nature, vol. 474, pp. 216–219, 06/09/print 2011.
R. A. Khorshed, E. D. Hawkins, D. Duarte, M. K. Scott, O. A. Akinduro, N. M. Rashidi, et al., “Automated Identification and Localization of haematopoietic Stem Cells in 3D Intravital Microscopy Data,” Stem Cell Rep, vol. 5, pp. 139–53, Jul 14 2015.
H. Geiger and K. L. Rudolph, “Aging in the lympho-haematopoietic stem cell compartment,” Trends Immunol, vol. 30, pp. 360–5, Jul 2009.
A. Sanchez-Aguilera, Y. J. Lee, C. Lo Celso, F. Ferraro, K. Brumme, S. Mondal, et al., “Guanine nucleotide exchange factor Vav1 regulates perivascular homing and bone marrow retention of haematopoietic stem and progenitor cells,” Proc Natl Acad Sci USA, vol. 108, pp. 9607–12, Jun 7 2011.
G. Chung and L. Vese, “Image segmentation using a multilayer level-set approach,” Computing and visualisation in Science, vol. 12, pp. 267–285, 2009/08/01 2009.
S. Y. Yeo, X. Xie, I. Sazonov, and P. Nithiarasu, “Level set segmentation with robust image gradient energy and statistical shape prior,” in IEEE International Conference on Image Processing, pp. 3397–3400, 2011.
K. Nandy, J. Kim, D. P. McCullough, M. McAuliffe, K. J. Meaburn, T. P. Yamaguchi, et al., “Segmentation and quantitative analysis of individual cells in developmental tissues,” Methods Mol Biol, vol. 1092, pp. 235–53, 2014.
P. Rompolas, K. R. Mesa, and V. Greco, “Spatial organization within a niche as a determinant of stem-cell fate,” Nature, vol. 502, pp. 513–518, 10/24/print 2013.
L. Ritsma, S. I. J. Ellenbroek, A. Zomer, H. J. Snippert, F. J. de Sauvage, B. D. Simons, et al., “Intestinal crypt homeostasis revealed at single-stem-cell level by in vivo live imaging,” Nature, vol. 507, pp. 362–365, 03/20/print 2014.
G. Mallinis, N. Koutsias, M. Tsakiri-Strati, and M. Karteris, “Object-based classification using Quickbird imagery for delineating forest vegetation polygons in a Mediterranean test site,” ISPRS J Photogramm Remote Sens, vol. 63, pp. 237–250, 3//2008.
F. Long, J. Zhou, and H. Peng, “Visualisation and analysis of 3D microscopic Images,” PLoS Comput Biol, vol. 8, p. e1002519, 2012.
S. Uchida, “Image processing and recognition for biological images,” Dev Growth Differ, vol. 55, pp. 523–549, 2013.
B. Moon, “A Gaussian smoothing algorithm to generate trend curves,” Korean J Comput Appl Math, vol. 8, pp. 507–518, 01 2001.
E. Mjolsness and D. DeCoste, “Machine learning for science: state of the art and future prospects,” Science, vol. 293, pp. 2051–5, Sep 14 2001.
C. Sommer and D. W. Gerlich, “Machine learning in cell biology—teaching computers to recognize phenotypes,” J Cell Sci, vol. 126, pp. 5529–39, Dec 15 2013.
C. Agarwal and A. Sharma, “Image understanding using decision tree based machine learning,” in International Conference on Information Technology and Multimedia, pp. 1–8, 2011.
O. Aydemir and T. Kayikcioglu, “Decision tree structure based classification of EEG signals recorded during two dimensional cursor movement imagery,” J Neurosci Methods, vol. 229, pp. 68–75, May 30 2014.
M. F. Krummel and I. Macara, “Maintenance and modulation of T cell polarity,” Nat Immunol, vol. 7, pp. 1143–9, Nov 2006.
R. A. Khorshed and C. Lo Celso, “Machine learning classification of complex vasculature structures from in- vivobone marrow 3d data,” (ISBI), IEEE International Symposium on Biomedical Imaging, Prague, 2016.
A. Mendelson and P. S. Frenette, “haematopoietic stem cell niche maintenance during homeostasis and regeneration,” Nat Med, vol. 20, pp. 833–846, 08//print 2014.
Daniel J. Nolan, M. Ginsberg, E. Israely, B. Palikuqi, M. G. Poulos, D. James, et al., “Molecular signatures of tissue-specific microvascular endothelial cell heterogeneity in organ maintenance and regeneration,” Dev Cell, vol. 26, pp. 204–219, 29 July 2013.