The New Landscape of Diagnostic Imaging with the Incorporation of Computer Vision

Abstract

Diagnostic medical imaging is a key tool in medical care. In recent years, thanks to advances in computer vision research, a subfield of artificial intelligence, it has become possible to use medical imaging to train and test machine learning models. Among the algorithms investigated, there has been a boom in the use of neural networks since they allow a higher level of automation in the learning process. The areas of medical imaging that have developed the most applications are X-rays, computed tomography, positron emission tomography, magnetic resonance imaging, ultrasonography and pathology. In fact, the COVID-19 pandemic has reshaped the research landscape, especially for radiological and resonance imaging. Notwithstanding the great progress that has been observed in the field, obstacles have also arisen that had to be overcome to continue to improve applications. These obstacles include data protection and the expansion of available datasets, which involves a large investment of resources, time and academically trained manpower.

1. Introduction

A large part of diagnosis in various specialties of medical care relies heavily on image analysis. Depending on the type of technique used, more or less detail of the structures of interest can be obtained. It will also depend on the type of technique, whether the image is in two dimensions or if there are several slices that then can form a three-dimensional reconstruction [1]. Some specific techniques can also produce video output [2]. All of these different formats can be adapted for use as training and testing material for computer vision models. Computer vision, a subfield of artificial intelligence (AI), comprises all those techniques that allow a computer system to understand an image or a set of images and produce as a result a numerical or symbolic output. This output can be used to make a decision about the image [3]. When these models are applied to healthcare images, the output can be used to make a clinical decision [4]. Within computer vision algorithms, we have those that are handcrafted, where a person analyzes the set of images to be classified and chooses the features to be extracted from those images. For example, if we want to classify cardboard boxes in a scene, the person will probably choose to detect the edges of the box and the texture of the cardboard, as a first step [3]. Now, thanks to advances in research in this area, neural networks can also be used. A neural network is a computational algorithm composed of a set of interconnected nodes called artificial neurons, which are similar in function to neurons in the human nervous system. A neuron in this network receives information from the preceding neuron, processes it, and transmits it to other neurons. Networks can be simple, with very few layers of neurons, or complex, with many layers and many interconnections [5]. These models have the advantage that the determination of the features is automatic and does not need to be handcrafted. However, neural networks require a large amount of data to be able to perform accurate feature extraction with minimal error. In addition, as the number of connections between its different neurons is very high, it becomes complex to elucidate which features have been selected to produce an output from an initial image [3, 5, 6]. In the following chapter, we will discuss the most common AI applications of neural networks as computer vision models in the clinical medical field. In addition, we will analyze the different obstacles that the field of AI has encountered in its development along with the advancement that these vision applications have brought to the medical field.

2. Methods

A targeted review of the literature was carried out using the criteria “AI,” “Computer Vision,” and “Medical Imaging.” The databases consulted were PUBMED and Google Scholar until January 2022, selecting only articles in English. Our initial search revealed 860 articles of which a subgroup of 130 was selected. The inclusion criteria focused on the quality of the research, the robustness of the models, the transfer to the clinical setting, and the optimization of the parameters for the rational use of resources.

3. Computer vision with neural networks

Computer vision (CV) and AI research have several decades of steady progress. Specifically, the part of this discipline that uses convolutional neural networks (CNN) for image processing had its first boom with handwritten digit identification in 1989. This application was developed by Yann LeCun using some of the insights previously proposed by Kunihiko Fukushima [5]. Since computational capabilities at that time were scarce, there was little research in this area between 1990 and 2000. Thanks to the progressive increase in processing and storage capacities, in 2012 the AlexNet model was tested in competition with great success and from then on, the field of computer vision began to be populated with numerous applications [7]. The applications varied according to the type of task required and the type of dataset used. Also at this time, several authors began to investigate further the structure of the different published models and started to work on their taxonomy. Thus, we have articles that examined the components of various CNNs and their interconnection [8] and others that analyzed the different architectures, their engineering challenges, and their possible future applications [4, 9].

4. Healthcare computer vision

The advancement of computer vision in the field of medical imaging awakened in the late 2000s [4]. As partially mentioned earlier, the advances were made possible by advances in deep learning (DL) research, increased local processing capabilities with graphic processing units (GPUs), and the creation of medical image datasets [10]. The creation of larger and more complete datasets was mainly due to the increasing digitization of medical records in several countries. These electronic health records (EHR) are able to store, in addition to the images that will constitute the raw material, the labels that will be used to guide the training of the models [11]. These EHRs started out as a tool to generate billing codes for different medical practices. Then they changed their use, becoming digital support for clinical practice [12]. This change allowed its adoption not only in institutions or networks of institutions but also in entire regions and countries [12, 13]. The extension of the coverage territory allowed to expand even more the image datasets and included more patient variability, which is key to obtain models with wide generalization power.

5. Operation of computer vision algorithms

When applying AI models, specifically computer vision models to different types of medical images, we can perform different tasks. According to Huo et al., these tasks can be classified into four categories [14]. The first one is classification, in which the input is an image and the output is a label. The label can be numerical (e.g., 1, 2) or it can be text (e.g., cancerous, noncancerous) [14, 15]. The second is detection, which consists of the identification of an object in the image by means of a bounding box. This task offers an extra degree of information since in addition to locating the object it can inform about its position by means of coordinates in the input image [14, 16, 17]. The third is segmentation, which provides the highest degree of information about an image. In this task, each pixel receives a label, and the final result is a mask that groups several pixels. This enables the segmentation of precise structures within medical images, such as glomeruli or metastatic zones in pathology slides, or entire organs, such as the bladder in CT images [14, 18, 19, 20]. The last task is synthesis. It consists of generating images from noise or other images. For this, two different models work antagonistically, one generates the images de novo from available data and the other model tries to discriminate this artificially generated image from a real image. With each iteration of the process, both the generator and the discriminator become more efficient, which produces images with high similarity to the real ones [14, 21]. This task allows for example to generate of more training samples to populate datasets and thus, to achieve models with more generalization power [22, 23].

6. Transfer learning and data augmentation

As neural networks increase their number of layers and the connections between them, their complexity increases. Neural networks with many layers have demonstrated more than satisfactory performance in several tasks, many of them superior to human performance [24]. However, when working with these complex networks it is necessary to have a large amount of data for training, to avoid overfitting, and to expand the power of generalization. The use of networks with few layers trained on small datasets has also been researched, which has shown that there is a tendency to overfitting or underfitting [25]. In the AI medical field, it is very difficult to have very large datasets, as this demands a lot of specialized manpower. Specialized manpower (doctors, biologists, geneticists, etc.) is the one that analyzes the data and aggregates the labels to train the algorithms [25, 26]. Therefore, two solutions have been found to deal with this problem of small datasets. The first one consists of data augmentation. This group of techniques creates images in virtual form from the original images of the dataset. For example, you can alter the position of the images, rotate them on their axis, and change the contrast and brightness, just to mention a few [25, 27]. The second solution is transfer learning. This technique consists of training a complex network on a massive dataset (ImageNet) usually of common images (dogs, cats, etc.), and then performing a finetuning. The finetunning is the training with the specific medical dataset, which only alters the weights of the last layers of the neural network. This helps to obtain better results than training the network from scratch on the specific dataset [25, 28].

7. Model performance evaluation

Being able to measure the performance of our models is crucial to be able to evaluate their suitability for different tasks. Also, when performing finetunning, it is important to be able to have performance measures to know which parameters promote the best results. First of all, every time we test a model, we will have part of a dataset (in this particular case, images) that already has the labels assigned to it. The assignment of labels is done by the medical professionals specialized in the pathology being worked on. When the model processes the samples and predicts the new labels, these are compared with the original ones (called ground truth). With the result of the comparison, what is called a confusion matrix is constructed [29, 30]. This structure contains the true positives (TP) and true negatives (TN) and false positives (FP) and false negatives (FN). A TP or TN is established when the prediction and the ground truth are the same for a given sample (e.g., is a TN when the model predicted negative and the image was negative). On the contrary, a FP or FN is established when there is no coincidence between the model and the ground truth (e.g., the model predicted negative and the ground truth indicated a positive sample, therefore, the sample is a FN) [29]. Almost all the other global metrics that are usually reported in the different publications are derived from the four previous metrics. For example, the accuracy of a model corresponds to the number of samples correctly predicted by the model over the total number of samples. Then, considering the previous metrics, the correctly predicted samples would be included in the sum of the TP and TN. Additionally, the total number of samples would not be more than the sum of the TP, TN, FP, and FN [29].

8. Healthcare applications of AI and computer vision

8.1 X-ray imaging

Medical X-ray imaging consists of the emission of these rays by a transmitter that passes through the area of the patient. According to the radiographic density (depending on the density of the tissue and the atomic number of its components), the structures in the area will absorb the rays differentially, which will result in lights and shadows [31]. In the AI field of computer vision applied to X-ray, there is a preponderance of work in the area of the thoracic cavity [32]. Thus, we found work focused on the detection of pulmonary nodules with models trained on images from one pool of patients and tested in a different pool. We also found longitudinal work, where the model was trained and tested on images from the same patients, with images separated by a time window [32, 33, 34]. Another large part of the work focused on the detection of pneumonia. Several models were trained on datasets from different hospitals, which showed variations in various image features between hospitals. As expected, the models showed better metrics when trained and tested on data from the same hospital [32, 35, 36, 37]. With the advent of COVID-19, there was an explosion of research in the detection of this pathology in X-ray images. Thus, numerous models were created that attempted to distinguish COVID-19 pneumonia from viral or bacterial pneumonia. These developments were key since they allowed screening and managing patients automatically and to avoid spreading the contagion of COVID-19 patients [32, 38, 39, 40, 41]. Work was also carried out to contribute to the detection of tuberculosis in chest images. These models demonstrated satisfactory performance in screening tuberculosis images with respect to normal lungs or other pulmonary pathologies. However, the models did not show the ability to distinguish between active and quiescent disease [32, 42, 43]. Additionally, part of the research was also directed to the detection of pneumothorax. This part of the development was of important value in patient triaging, especially in determining the size and position of the pneumothorax and its changes over time in the same patient. Several of these models have already received FDA clearance as assistive devices in the emergency unit [32, 44, 45, 46]. As a final part of this section, to a lesser extent than the previous ones, models were also built for the detection of other types of pulmonary involvement, such as consolidation, edema, emphysema, fibrosis, and pleural effusion [32, 47].

8.2 Computed tomography

Computed tomography (CT) integrates many X-ray images taken from different angles thanks to the high-speed rotating platform that rotates on the same axis where the patient lies. The type of images it produces is cross-sectional [1]. Using AI computer vision techniques, it is possible to operate directly on a fixed plane (one section) or to use complete volumes (several consecutive sections). Most of the research in this area is classification (about 36%), followed by segmentation (27%), detection (22%), and others (15%) [30]. Broadly speaking we can list the works in this area in the identification of organs (kidney [48], liver [49, 50], lungs [51, 52], and heart [53, 54]) and in the identification of substructures or lesions (artery calcification [55], nodules [56], polyps [57, 58], and lymph nodes [59, 60, 61]). Among the most commonly used measures to report the performance of the different models are accuracy, sensitivity, specificity, AUC-ROC, and F1 score [1]. The processing of the images as input is also diverse. It is possible to use 3-dimensional inputs, that is, several consecutive slices that form a volume. Projection methods, such as maximum intensity projection, can also be used to transform a 3-dimensional input into a 2-dimensional one [1, 62].

8.3 Positron emission tomography

Positron emission tomography (PET) is a technique that allows the observation of metabolic processes in different tissues of the patient’s body. Radiolabeled compounds that follow a specific metabolic pathway are injected, the radiation is detected by sensors and then the complete image is reconstructed with the areas of highest activity [1]. 18F-fluorodeoxyglucose (FDG) is one of the most widely used radioactive substrates as a marker in PET [1, 63]. Among the applications of AI computer vision to this medical imaging modality, we have the segmentation of tumor areas in the brain [64], heart [65], head and neck [66], and nasopharynx [67] to adjust the dose and position of the radiotherapy intervention. With respect to classification tasks, work has been published on esophageal cancer [68], Alzheimer’s disease typing [69], and Hodgkin’s lymphoma [1, 70].

8.4 Magnetic resonance imaging

Magnetic resonance imaging (MRI) is a technique that uses high-intensity electromagnetic fields and radiofrequency waves to detect changes in the rotational axis of protons, mostly in water molecules. Water makes up almost all the tissues in the body and the difference in the percentage of water influences the axis changes. Deep learning applications in the field of MRI can be grouped into two broad categories. The first is related to the physical aspects and the generation of images on the device. In this category, you can find works that focus on image restoration, image reconstruction, and multimodal image registration [71]. The second category emphasizes applications for medical purposes, in which the determination of pathology or its progress is the main purpose [71, 72, 73, 74]. Focusing on the second category, we find works on brain aging [75], brain vascular lesions [76], Alzheimer’s disease [77], multiple sclerosis [78], glioma [79], and meningioma [80]. In the abdominal cavity, we find works of identification and segmentation of organs [81], polycystic kidneys [82], and renal transplantation [83]. Finally, isolating the spine as the focus of the study, we found works on labeling and separation of vertebrae [84], spinal stenosis grading [85], and identification and segmentation of spinal metastasis [86]. It is important to mention that organ segmentation is a very important focus in deep learning applications for MRI images. With the definition of organ contours in each plane (slice), the determination of the organ coordinates and the addition of consecutive areas, volumes can be calculated. The calculation of volumes is of crucial importance since they can be used to determine the dilation of organs (e.g., splenomegaly). The measurement of dilation is not only an important initial measurement. Thanks to the volumetric determination, it is possible to follow up on patients to observe the efficiency of treatments [81].

8.5 Ultrasonography

Ultrasonography (US) consists of the use of ultrasound (usually at a frequency greater than 20,000 hz) to form images of the inner regions of the organism. To do this, a probe emits waves and they bounce back at different speeds according to the type of tissue [87]. From this technique, we can count on two different outputs. One is an image (frame) where the structure of medical interest is located. The other is a complete video where we can visualize, for example, blood flow or muscle contraction. Within the research in AI computer vision applied to US, most of the works include the analysis of individual frames. In this way, frames can be produced directly from the device or they can be extracted from ultrasound videos. When extracting frames from videos, the regions of interest for the specific task is usually timely located and the rest are discarded [88]. Other less common and more integrative methodologies can use videos directly as input. They produce the division into frames, use a model (CNN) to extract features from each frame, and then integrate all the extracted features with a recurrent model (e.g., long short-term memory network) following a timeline [2]. Focusing on applications, in those works that performed classification we found the study of breast lesions [88, 89, 90], thyroid nodules [88, 91], liver fibrosis [88, 92], and focal liver disease [88, 93]. Regarding the detection of lesions, some works focused on papillary thyroid carcinoma [94] and breast cancer [95]. Continuing in the detection task, but moving from lesions to the detection of the fetal standard plan, several papers proposed different methodologies [88, 96, 97]. These works constituted important pillars for the improvement of automatic guidance tools in the fetal US that could be embedded in image production software. Finally, in the segmentation task, several works have been registered with approaches in areas similar to those mentioned above, such as breast lesions [88, 98] and lymph node contouring [88, 99, 100]. However, in this part, there is also an application that has several works and that has an important diagnostic value in the clinical setting. This application is the detection of atheroma plaques in the carotid artery and the automation of this process would allow screening and prevention in a faster and more cost-effective way [101, 102]. In fact, a multicenter clinical study has already been published to evaluate the feasibility of the technique [102].

8.6 Computational pathology

Classical pathology consists, very briefly, of the preservation, treatment, and staining of very small portions of tissue in slides. Stains can be standard ones, which highlight general structures, such as nuclei or cytoplasm, or immunohistochemical stains, in which specific cellular markers are targeted [103]. Thanks to advances in storage capabilities and the availability of cloud computing, the last few years have seen a migration from direct microscopic observation of stained tissues to the digitization of slides. Digital slides are stored in a specific file type called whole slide image (WSI), where it is possible to store the different magnification planes with very high compression. The scanning of the slides and the production of WSI for different uses, such as telepathology, constitute a branch of pathology called digital pathology [30, 104]. In addition, the increasing production and cataloging of WSIs for the diagnosis of different diseases made it possible to use them as training and testing materials for computer vision algorithms. This application of algorithms in WSIs has been called computational pathology and most of the published works use deep learning as a basis for different tasks. In a very general manner, one could describe the process of creating a computational pathology pipeline for any disease. Once the WSIs of the pathology to be studied are available, the final magnification to work with must be selected (20×, 40×) and consecutive patches of the different zones (disease and healthy tissue) must be generated [30]. The patches are generated due to the large size of the WSIs (the highest magnification can exceed 3e10 pixels). Consequently, the patches are used as input to the model and the model will learn, according to the task, to identify tumor and non-tumor zones [30]. In test WSIs, the same technique can be used to generate patches, process them with the model and then reconstruct the final image with a heat map. The heat map will identify the regions with the highest probability of belonging to a class (healthy or tumor). Jiang et al. categorize the implementation of computational pathology in oncology into five purposes, which are tumor diagnosis, subtyping, grading, staging, and prognosis [30]. Thus, we can find applications of these five purposes for breast cancer [30, 105, 106, 107, 108], lung cancer [30, 109, 110, 111], colorectal cancer [30, 112, 113, 114, 115], gastric cancer [30, 116, 117], prostate cancer [30, 118, 119], and thyroid cancer [30, 120, 121]. Another set of applications of computational pathology lies in the automatic analysis for the identification of rejection in organ transplantation. Several papers have been published for kidney [122, 123] and heart [124] transplantation.

9. COVID-19 research landscape remodeling

The COVID-19 pandemic created a compelling need for innovation in testing to generate solutions that were cheap, easy to use, fast, and ubiquitous. Since lung imaging is a useful diagnostic tool, during the pandemic many research groups began to look for solutions using AI and computer vision [125]. As lung imaging is an important resource in emergency medicine for optimal triage of patients with suspected COVID-19 infection, computer vision solutions aimed to be a rapid analysis element that could speed up patient management times. From 2019 to 2020, a nearly two-fold increase in the number of publications on the artificial intelligence applied to medical imaging was observed. Moreover, starting from zero publications in 2019, by 2020 about 15% of all deep learning research associated with medical imaging was on COVID-19. With respect to the focus on the type of medical imaging, it was observed that of all the proposed computer vision solutions, almost half (49.7%) were focused only on X-rays. The remaining modalities were CT (38.7%), multimodality (10.2%), and ultrasonography (1.5%) [125]. As the research progressed, the usefulness of ultrasound as a tool for the diagnosis and management of COVID-19 was also observed. The ease of maintaining sterility, the possibility of performing bedside operations, the reduced time to obtain the image, and the possibility of using only one operator for the procedure have made this imaging modality highly suitable for this pandemic. The group of Born et al. opened the door to the use of deep learning with ultrasound for COVID-19 screening [126, 127]. Several groups followed with different proposals and today, the field has grown considerably by extending applications to other pathologies [128, 129].

10. Challenges for the field

As we briefly mentioned in one of the previous sections, one of the biggest challenges facing the field of AI and computer vision applied to medicine is the availability of datasets. Generating general datasets, although it is a task that requires time, can be done in a more laborsaving way. For instance, it does not require a high degree of training to classify common images. In fact, some search engines ask their users when they access specific content to first select from a group of images those that have a traffic light in it. That generates labels and in this way very large datasets are built. As we also mentioned before, in order to generate medical image datasets, trained doctors are needed to perform the same activity. That requirement makes the process complex, time-consuming, and expensive [25, 26]. Another problem facing the field is the variability between different hospital centers’ samples. As we have already explained before, the greater the amount of data that the algorithm trains with, the higher its generalization power. However, when the data comes from different hospitals, even if they are in the same city, samples of the same medical condition may suffer variations in color, brightness, contrast, and position, to mention just a few. These variations respond to the different equipment used by hospitals and the different sample preparation techniques that different laboratories may have. This variability is manifested in its maximum expression in computational pathology. Moreover, the most current works usually include studies with different scanners and from different hospitals to analyze the robustness of the model [124]. Another challenge that specifically affects computational pathology is the weight of each sample. As we mentioned before, the WSI of the pathology samples contains a considerable amount of pixels, especially at their highest magnification level. This makes it challenging to be able to share the images and store complete datasets. It is worth mentioning that also operating digitally with these images raises the hardware requirements to high levels. For this reason, parallelization tasks or image batch processing can become complex, which also increases processing times [130]. Finally, a crucial aspect must be addressed. Operating with medical images requires a high degree of data protection and the use of anonymization techniques. In order to use hospital data, an ethics committee must first review the scope of the project. The ethics committee will determine the degree of consent that patients must provide in order to use their data. In many retrospective studies, depending on the amount of private data being used, committees may approve the waiving of informed consent (IC). For example, if patients have already consented to the original study and no further identifying data will be added to the project, this may be a favorable setting for not requiring additional IC. However, that decision rests solely with the committee and this entity will decide the constraints of the project. Ethics committees may be slow to grant project approval, especially if the scope of the project is extensive. Also, should new ICs be required, this can also add cost and time to the project [131].

11. Innovating through challenges

The challenges that have crossed the field of AI and computer vision in healthcare have also promoted the search for solutions. This search has sparked ideas and achieved some interesting proposals that are slowly being incorporated into daily practice. To begin with, the problem of generating labels in WSIs gave rise to a new technique called multiple instance learning (MIL). This technique uses as labels only the diagnosis of the patient (which is usually available in EHRs). Thanks to this new approach, a group managed to analyze 44,732 WSIs without any kind of data curation, incredibly speeding up project times [132]. As we also mentioned, the variability between samples from different hospitals is a problem that threatens the creation of large datasets. One of the solutions to this problem was the creation of stain normalization. This is a method that in one of its variants uses autoencoders and allows to standardize of the color distribution in the images, using another image as a template [133]. Thanks to this method, it is possible to have more homogeneous images, even if they come from different laboratories. Regarding the weight of the WSIs, generally, only a small part of the image is used by the deep learning models for the task they perform. For example, as the image passes through the successive layers of a CNN, the information is reduced. In the last layers, only the essential information remains that will complete the task with the least possible error. Using this principle, one group created the concept of neural compression. Basically what this group proposed is to create abstract representations of the WSI images after passing through successive steps in a convolutional network. In this way, noise is removed at each step and only a small, compressed representation remains [134]. This concept would help store WSIs more efficiently with only the information needed for the task. Finally, to provide the greatest privacy protection to patients and also speed up data exchange processes, blockchain networks and interplanetary file system (IPFS) can be used. In this way, the information is decentralized, which reduces the risk of data leakage. In addition, the different hospitals participating in the study can provide the files, which can be fragmented and hashed according to IFPS. The entire process would be governed by one or several smart contracts, which would ensure that only authorized nodes contribute data or extract data. Smart contracts may also contain portions of sensitive information, which would eliminate the need for human interaction and the possible breach of confidentiality [135, 136, 137].

12. Conclusions

The use of AI and computer vision algorithms, especially neural networks, has advanced greatly in recent years. The various applications with different types of medical images have made numerous diagnostic and prognostic applications available to the medical field. The field of oncology has seen the greatest number of developments. Particularly, computational pathology applied to oncology has developed a high degree of diversification in vision tasks, achieving models that could perform diagnosis, subtyping, grading, staging, and prognosis. However, just as innovative applications have emerged, the field has also had to overcome obstacles, which are still complex to analyze for some conditions today. The difficulty of constructing medical datasets, the variability of samples between different institutions, and the mandatory data protection are some of them. However, these obstacles have promoted the creation of ideas to overcome them and that is how we have neural compression and stain normalization that can be great allies to exponentially expand the datasets. Finally, the COVID-19 pandemic was a major trigger for research in AI and computer vision applied to the field of medical imaging, specifically lung imaging. It could be seen that a modeler of the research landscape was the feasibility in the clinical field. In fact, the ease of use, the short operating time, and the possibility of maintaining sterility were part of the parameters that promoted the use of ultrasonography expanding the research with deep learning in this imaging modality. Despite these great advances, more studies must be done to further refine computer vision models to ensure that patients receive the best quality of medical care.