Effective Screening and Face Mask Detection for COVID Spread Mitigation Using Deep Learning and Edge Devices

Abstract

The emergence of COVID-19, stemming from the SARS-CoV-2 virus, has led to a widespread outbreak affecting countless individuals and inducing dire circumstances globally. Mitigating the transmission of COVID-19 has necessitated the implementation of effective measures such as rigorous COVID screening and physical safeguards, including practices like social distancing and the utilization of face masks. Notably, the application of advanced technologies such as deep learning, a subset of artificial intelligence (AI), has played a pivotal role in devising novel strategies for both detecting COVID-19 and curbing its propagation. This chapter presents a comprehensive overview of COVID screening methodologies based on deep learning, with a specific focus on biomedical image processing and the detection of face masks. Furthermore, it delves into initial endeavors concerning COVID image analysis and the creation of a mobile face mask detection system, designed to operate on edge devices. The ensuing discussions encompass detailed case studies, showcasing the practical implications and efficacy of these initiatives.

1. Introduction

The global COVID-19 pandemic, originating from the Coronavirus disease 2019 (COVID-19), continues to persist across more than 200 countries and territories, evoking significant apprehension within the international community [1, 2]. This crisis has engendered profound human losses and economic ramifications, reshaping lives in various countries through the implementation of lockdown measures. In order to curb the dissemination of COVID-19 and curtail its associated mortality, early detection of the disease assumes critical importance. Effective and prompt screening and testing are pivotal for the proficient management of individuals afflicted by COVID-19 [3, 4]. The evolution of advanced techniques has led to the proposition of increasingly efficient screening technologies, aimed at attenuating the transmission of COVID-19.

1.1 COVID-19 screening

The goal of a COVID-19 screening test is to identify potential cases of the disease in individuals who are asymptomatic. This proactive approach aims to mitigate the spread of COVID-19 by detecting infections early, allowing for timely and effective treatment. During the initial stages of the COVID-19 pandemic, the primary method for detecting viral infections was the Reverse Transcription-Polymerase Chain Reaction (RT-PCR) test [5]. However, the effectiveness of the RT-PCR assay has been questioned due to its limited sensitivity [6], which can be attributed to various factors like sample preparation and quality control issues [7]. Furthermore, the current nucleic acid tests’ sensitivity necessitates repeated testing for a significant portion of suspected patients to achieve a reliable diagnosis. This underscores the need for the development of a complementary tool capable of providing lung-imaging information. Such a tool would serve as an invaluable resource for medical professionals, aiding them in enhancing the accuracy of COVID-19 diagnoses.

Chest X-rays and thoracic computed tomography (CT) scans are readily available imaging tools that offer significant support to medical practitioners in diagnosing lung-related ailments [8, 9, 10]. The application of artificial intelligence (AI) in enhancing image analysis of chest X-rays and thoracic CT data has garnered substantial interest, particularly in the development of effective COVID-19 screening techniques. AI, a burgeoning technology in the realm of medical imaging, has played a dynamic role in the battle against COVID-19 [11]. This is in contrast to traditional imaging processes that heavily rely on human interpretation, as AI offers imaging solutions that are safer, more accurate, and more efficient. Notably, the utilization of deep learning for data representation has exhibited remarkable success in image processing [12]. Convolutional neural networks (CNNs) [13, 14, 15, 16, 17, 18] have effectively tackled the challenge of representing digital images, particularly on extensive datasets such as the ImageNet dataset [19]. These advances demonstrate the potential of deep learning in transforming biomedical image analysis.

The potency of deep learning methods, such as Convolutional Neural Networks (CNNs), has been prominently demonstrated in the classification of COVID-19 cases. Ghoshal et al. [20] introduce a Bayesian Convolutional Neural Network designed to estimate diagnostic uncertainty in COVID-19 predictions. This approach incorporates 70 lung X-ray images from COVID-19 patients sourced from an online COVID-19 dataset [21], as well as non-COVID-19 images from Kaggle’s Chest X-Ray Images (Pneumonia), where Bayesian inference is employed to enhance detection accuracy. Narin et al. [10] focus on COVID-19 infection detection using X-ray images, employing a comparative analysis of three deep learning models: ResNet50, InceptionV3, and Inception-ResNetV2. The evaluation results indicate that the ResNet50 model surpasses the performance of the other two models. Zhang et al. [22] similarly harness the ResNet architecture for COVID-19 classification using X-ray images. An anomaly score is estimated to optimize the COVID-19 score, which in turn is used for classification. Wang et al. [23] introduce COVID-Net, a framework tailored for the detection of COVID-19 cases through X-ray images. Primarily, most ongoing studies employ X-ray images for discriminating between COVID-19 cases and other instances of pneumonia and healthy subjects. However, the limited quantity of available COVID-19 images raises concerns regarding the methods’ robustness and generalizability, urging further investigation.

Furthermore, it is of utmost importance to delineate the areas affected by COVID-19 infection, as this yields comprehensive insights crucial for accurate diagnosis. Semantic segmentation plays a pivotal role in identifying and quantifying COVID-19 by recognizing regions and associated patterns. This technique enables the assessment of regions of interest (ROIs) encompassing lung structures, lobes, bronchopulmonary segments, and infected regions or lesions within chest X-ray or CT images. Extracting handcrafted or learned features for diagnosis and other applications becomes feasible through the use of segmented regions. The advancement of deep learning has significantly propelled the evolution of semantic image segmentation. In the context of CT scans, the networks employed for COVID-19 include established models like U-Net [24, 25, 26], UNet++ [27], and VB-Net [28] to segment ROIs. Furthermore, segmentation approaches for COVID-19 can be categorized into two main groups: those oriented toward lung regions and those aimed at lung lesions. The former group focuses on distinguishing lung regions, encompassing entire lungs and lung lobes, from surrounding (background) regions in CT or X-ray images [29, 30]. For instance, Jin et al. [29] utilize UNet++ to detect the entire lung region. The latter group seeks to isolate lung lesions (or artifacts such as metal and motion) from lung regions [31, 32]. In addition to screening techniques, physical solutions for mitigating the spread of COVID-19 also hold efficacy.

1.2 Face mask detection

An effective physical measure to counter the spread of COVID-19 is the utilization of face masks in public settings [33]. Figure 1 illustrates the varying transmission risks between an infected individual and an uninfected person. When an infected person does not wear a mask, the risk of transmitting the virus to an uninfected person is substantially high, as depicted in the first row. This risk diminishes to a moderate level if either of them wears a face mask (depicted in the second row). The lowest risk of infection occurs when both individuals are wearing masks [34], as depicted in the third row. Thus, wearing face masks effectively reduces the spread of COVID-19. However, ensuring universal adherence to mask-wearing mandates poses challenges. AI-driven face mask detection [35] has emerged as a technique to identify compliance with face mask requirements and can serve as a reminder for those not wearing masks. Developing a face mask detection system from scratch presents challenges due to the scarcity of labeled images. As a result, deep transfer learning [36] offers a promising solution to this predicament. This technique involves adapting pre-trained models to the task of face mask detection. The complete process of constructing face mask detection models through deep transfer learning is delineated in Figure 2.

Figure 1.

Different risk of transmission between infected person (left column) and uninfected person (right column).

Figure 2.

The flow of building face mask detector.

During the model training phase, a limited set of annotated images is loaded as training data. These images are then used to fine-tune pre-trained deep learning models, ultimately creating a robust fake mask detector. In the testing phase, datasets with labeled ground truth are loaded. The face mask detector is applied to this data, and its performance is evaluated using predefined metrics. Subsequent sections delve into comprehensive details regarding COVID-19 screening methodologies and fake mask detection on edge devices, all supported by illustrative case studies.

The structure of this chapter is as follows: In Section 2, we provide an overview of previous research on AI-driven COVID-19 screening techniques and face mask detection. Moving to Section 3, we delve into specific case studies, presenting associated outcomes that highlight effective screening and face mask detection strategies. These endeavors leverage deep learning and edge devices to mitigate the spread of COVID-19. Finally, Section 4 encapsulates the conclusions and outlines avenues for future research.

2. Related work

2.1 COVID-19 screening via AI-enhanced image processing

The advancement of AI-driven image processing has played a pivotal role in significantly advancing COVID-19 screening techniques, encompassing both COVID-19 classification and COVID-19 segmentation. Considerable attention has been directed toward the classification of COVID-19 cases versus non-COVID-19 cases, with a focus on employing deep learning models. These models aim to effectively differentiate between COVID-19 patients and non-COVID-19 subjects, wherein the latter group includes individuals with common pneumonia and those without pneumonia. The diagram depicted in Figure 3 illustrates the process of COVID-19 classification on a chest X-ray image. The COVID-19 classifier receives a chest X-ray image as input and provides an output that classifies the image as either indicating COVID-19 or non-COVID-19 status.

Figure 3.

A diagram of COVID-19 classification on chest X-ray images.

Considerable research efforts have been directed toward COVID-19 classification. Chen et al. [27] pursued COVID-19 classification using segmented lesion patterns extracted via UNet++. Their dataset included diverse patient cases, such as COVID-19 patients, viral pneumonia patients, and non-pneumonia patients. Given the visual similarity between common pneumonias, particularly viral pneumonia, and COVID-19, distinguishing these conditions becomes crucial for effective clinical screening. To address this, a 2D CNN model was proposed, employing manually delineated region patches for classification between COVID-19 and typical viral pneumonia. Additionally, Wang et al. [37] combined segmentation information with a proposed 2D CNN model to classify COVID-19 cases by considering handcrafted features of relative infection distance from the lung’s edge. Xu et al. [38] utilized candidate infection regions segmented by V-Net. They combined these region patches with handcrafted features representing the distance from the edge of the region to perform COVID-19 classification using a ResNet-18 model. Zheng et al. [24] employed U-Net for lung segmentation and utilized 3D CNNs to predict COVID-19 probabilities based on the segmented features. Their dataset consisted solely of chest CT images of COVID-19 and non-COVID-19 cases. Similarly, Jin et al. [29] introduced a UNet++ − based segmentation model to identify lesions and a ResNet50-based classification model for diagnosis. Their larger dataset encompassed chest CT images of 1136 cases, including 723 COVID-19 positives and 413 COVID-19 negatives. In another work, Jin et al. [39] employed a 2D Deeplab v1 model for lung segmentation and a 2D ResNet152 model for slice-based identification of positive COVID-19 cases. In summary, ongoing efforts in COVID-19 classification primarily focus on learning from significant volumes of medical images. However, the application of these techniques is hindered by the considerable data requirements for building effective classifiers.

The segmentation of COVID-19 cases is achieved through image semantic segmentation techniques bolstered by deep learning models such as U-Net and V-Net [40, 41]. An illustrative instance of image semantic segmentation on a chest X-ray image is presented in Figure 4. This approach enables the precise identification and isolation of COVID-19-affected regions within medical images.

Figure 4.

An example of image semantic segmentation on a chest X-ray image.

The U-Net architecture is a powerful tool for segmenting both lung regions and lung lesions, facilitating the construction of effective image segmentation models [31]. U-Net, developed using a fully convolutional network [42], features a distinctive U-shaped design encompassing two symmetric paths: an encoding path and a decoding path. The layers at the corresponding levels in these two paths are interconnected through shortcut connections, fostering the acquisition of improved visual semantics and intricate contextual details. Zhou et al. [43] introduced UNet++, which inserts a nested convolutional structure between the encoding and decoding paths, further enhancing segmentation performance. In a similar vein, Milletari et al. [44] developed the V-Net, employing residual blocks as fundamental convolutional units and optimizing the network using a Dice loss. Furthermore, Shan et al. [28] devised VB-Net, enhancing segmentation efficiency by incorporating convolutional blocks with bottleneck blocks. Numerous variations of the U-Net architecture and its derivatives have been explored, yielding promising segmentation outcomes in COVID-19 diagnosis [27]. Advanced attention mechanisms are incorporated to identify the most discriminant features within deep learning models. Oktay et al. [45] introduced the Attention U-Net, capable of capturing intricate structures in medical images, rendering it suitable for segmenting lesions and lung nodules in COVID-19 applications. The integration of COVID-19 classification and segmentation empowers the implementation of multifaceted screening techniques across different levels.

2.2 Face mask detection

Face mask detection has been a subject of extensive research, and these efforts can be broadly categorized into two classes. The first approach treats it as an object detection task, where the goal is to localize the face mask area using bounding boxes. For instance, Mingjie Jiang et al. [46] proposed a one-stage face mask detector that employs a pre-trained ResNet for transfer learning and a feature pyramid network to extract semantic information. They introduced a novel context attention mechanism to enhance the detection of mask features. Similarly, Loey et al. [47] presented an object detection process using a combination of ResNet and YOLO V2. Another approach views face mask detection as an image classification problem [35]. Researchers in this category employed various convolutional neural networks (CNNs) such as MobileNet [48], Inception V3 [49], VGG-16 [50], and ResNet [51]. MobileNet, designed for edge devices, operates at high speed due to its smaller model size and complexity. Inception V3 utilizes factorizing convolutions to maintain robustness while reducing connections. VGG-16 explores the impact of depth on accuracy in large-scale image classification tasks.

In addition to training models from scratch, some researchers use pre-trained face detectors to extract faces, and then apply mask detection classification models on the detected faces [52, 53]. For example, Lippert et al. utilize OpenCV’s pre-trained face detector and a VGG-16-based classifier for face mask detection. The deployment of machine learning models on resource-constrained devices has gained popularity, as executing models locally is often preferable to sending data to the cloud due to issues such as limited bandwidth and privacy concerns.

In summary, face mask detection has been tackled through two main avenues: object detection using bounding boxes and image classification using various CNN architectures. These diverse approaches aim to enhance the accuracy and efficiency of detecting face mask presence and adherence.

2.3 Edge device

Machine learning models are executed on small IoT or edge devices, which highlights the relevance of edge computing However, executing such models on these devices is not a straightforward task due to their computational intensity. Models must be lightweight or compatible with these devices to be feasible. NVIDIA Jetson TX2 and Nano are popular test devices. These NVIDIA devices serve as embedded AI computing solutions. The Jetson TX2 features an NVIDIA Pascal GPU with 8 GB of memory, while the Nano is equipped with a Maxwell GPU and 4 GB of memory. Both devices are well-suited for computer vision and machine learning tasks. Figure 5 depicts the devices used in this chapter, and Table 1 provides a detailed comparison between the two devices.

Figure 5.

NVIDIA Jetson Nano (left) and NVIDIA Jetson TX2 (right).

	Jetson TX2	Jetson Nano
CPU	Dual-Core NVIDIA Denver 2 64-Bit CPU	Quad-core ARM Cortex-A57 MPCore processor
GPU	256-core Pascal @1300 MHz	NVIDIA Maxwell architecture with 128 NVIDIA CUDA® cores
Memory	8GB 128-bit LPDDR4	4 GB 64-bit LPDDR4
Storage	32GB eMMC 5.1	16 GB eMMC 5.1

Table 1.

Comparison between NVIDIA Jetson TX2 and NVIDIA Jetson Nano.

3. Case study

3.1 Case study on COVID-19 classification

In the context of this chapter, chest X-ray images were selected for both model development and validation. It was driven by the cost-effectiveness and greater accessibility of chest X-rays compared to CT scans, particularly in communities with limited medical resources. Additionally, chest X-rays offer a swift imaging solution, making them particularly attractive for large-scale patient screening during the COVID-19 pandemic. Therefore, utilizing chest radiography for screening COVID-19 patients is considered a practical, efficient, and rapid approach [54, 55]. For validation purposes, this chapter utilized a comprehensive chest X-ray dataset “COVIDx” [23]. This dataset comprises a vast collection of 18,543 chest radiography images from 13,725 unique cases. It is important to note that when evaluating the distribution of classes between the training and testing datasets, a significant dissimilarity in class distribution becomes apparent.

This chapter employs two types of deep learning models: Convolutional Neural Networks (CNNs) and ResNet. CNNs have played a significant role in advancing various visual processing tasks like image classification [56], object detection and tracking [57, 58], and semantic segmentation [59]. The progress of CNNs has been facilitated by large datasets like ImageNet [56] and YouTube-BoundingBoxes [60], which provide ample training data for building large-scale models. The general architecture of CNN for image classification is depicted in Figure 6. SOTA CNN architectures such as AlexNet [56], VGG [50], and GoogleNet [61] have propelled advancements in image classification. These architectures leverage millions of annotated samples from large datasets to successfully estimate appropriate parameters. Furthermore, CNNs have been enhanced by combining them with other deep learning models. For example, Wang et al. [62] combined CNN with Recurrent Neural Networks (RNNs) for multi-label image classification. Additionally, CNNs combined with autoencoders [63, 64] have demonstrated effectiveness in tasks like face detection. In this chapter, three small CNNs were trained and tested on a subset of the COVIDx dataset to demonstrate a proof-of-concept experiment.

Figure 6.

Convolutional neural network architecture for image classification.

ResNet [51] is an artificial neural network architecture inspired by the structure of pyramidal cells in the cerebral cortex. It introduces skip connections, or shortcuts, which allow the network to bypass certain layers. The concept behind ResNet is that training a network to learn a residual mapping is simpler than training it to directly learn the underlying mapping. This is achieved using residual blocks, as depicted in Figure 7. A crucial modification in ResNet compared to a standard CNN is the “skip connection” for identity mapping. This identity mapping has no parameters; it simply adds the output from the previous layer to the next layer. However, the dimensions of $x$ and $F\left(x\right)$ might differ. Since convolutions usually reduce spatial resolution, e.g., a $3\times 3$ convolution on a $32\times 32$ image results in a $30\times 30$ image, the identity mapping is expanded using a linear projection $W$ to match the channels of the residual. This allows the input $x$ and $F\left(x\right)$ to be combined as input to the subsequent layer. Given the effectiveness of ResNet, various ResNet architectures will be employed in this chapter to screen COVID-19 using the complete COVIDx dataset.

Figure 7.

General architecture of the residual block of ResNet [51].

This chapter examines the proposed models from two distinct viewpoints. The first perspective involves assessing their capability to effectively identify COVID-19 cases from a limited dataset, using compact CNNs. The second perspective entails investigating whether these models can utilize the ResNet architecture to identify COVID-19 cases within an extensive dataset, all without relying on transfer learning techniques. For the small dataset scenario, a subset of 350 images was extracted from the original COVIDx dataset [23] for training and testing. Among these, 300 images were allocated for training, while the remaining 50 were reserved for testing. Three small CNNs were employed for this evaluation. The obtained training and testing accuracies for these three models are depicted in Figure 8. The results indicate that the initial shallow CNN (referred to as CNN1) encountered issues with under-fitting. However, as additional layers were incorporated to extract more intricate features, CNN3 exhibited superior accuracy performance.

Figure 8.

Performance on training and testing accuracy for three small CNNs.

In the context of a large dataset, all images encompassed within the COVIDx dataset were harnessed to establish a classifier for COVID-19 identification. The initial step encompassed data preprocessing, involving the compression of images. The original X-ray images within the dataset measured $1024\times 1024\times 3$ in dimensions. To expedite the training process, the image size was compressed to $64\times 64\times 3$. Subsequently, training was initiated using this preprocessed dataset. The outcomes of the training and testing phases are depicted in Figure 9.

Figure 9.

Performance comparison on training and testing.

The findings unveil the existence of an optimal configuration, specifically ResNet-34, which outperforms alternative models. Beyond ResNet-34, a consistent better performance becomes apparent as the number of ResNet layers increases to 152. This can be attributed to the phenomenon where, with the progressive augmentation of layers, the models tend to overfit while the available data remains inadequate to effectively train the model. It is noteworthy that although the training time slightly extended with the inclusion of additional layers in the ResNet models, the performance gains were limited. Furthermore, the results underscore the accomplishment of this chapter in achieving commendable performance by training ResNet models from scratch, devoid of reliance on transfer learning techniques.

While supervised deep learning demonstrates impressive performance in classifying COVID-19 images, its practical application is hindered by the need for a substantial volume of annotated medical images for training. Given the limitations in available COVID-19-related data resources and the significant costs associated with labeling medical images, this approach becomes less feasible, further exacerbated by labeling inaccuracies that may arise [65]. To address this challenge, the focus has shifted toward semi-supervised deep learning, which has garnered considerable attention due to its capacity to enhance model generalization by leveraging both labeled and unlabeled data [66, 67, 68, 69]. This paradigm involves training deep neural networks through the simultaneous optimization of a standard supervised classification loss on labeled samples and an unsupervised loss on unlabeled data [67, 69]. Semi-supervised learning models aim to amplify the information derived from unlabeled data [70] or impose regularization on the network to enforce smoother and more consistent classification boundaries [68].

In the realm of COVID-19 research, particularly in tasks like COVID-19 image classification and image segmentation, semi-supervised learning has emerged as a solution to mitigate the scarcity of labeled data [71, 72, 73, 74, 75, 76]. However, within the domain of COVID-19 image classification, the studies conducted by Zhou et al. [76], Calderon et al. [72], and Paticchio et al. [74] have not thoroughly examined model performance using large-scale X-ray image datasets such as COVIDx [23]. Furthermore, they have not conducted comprehensive comparisons against state-of-the-art methods, particularly in scenarios where labeled data is severely limited, constituting less than 10% of the dataset. In response, this chapter introduces a semi-supervised deep learning model for COVID-19 image classification, systematically evaluating its performance on the COVIDx dataset [23].

Using the ResNet architecture, this chapter devised a two-path semi-supervised learning ResNet (referred to as SSResNet), which comprises three key components: a shared ResNet, a supervised ResNet, and an unsupervised ResNet. These two paths are formed by coupling the shared ResNet with either a supervised ResNet or an unsupervised ResNet. Both labeled and unlabeled data are leveraged in the computation of the unsupervised loss, utilizing the mean squared error loss (MSEL). Conversely, only labeled data contributes to the calculation of the supervised loss, employing the cross-entropy loss (CEL). To counterbalance data imbalance, a weighted cross-entropy loss (WCEL) was ingeniously crafted, assigning greater weight to the COVID-19 class. The primary objective of minimizing MSEL lies in enhancing image representation, while the reduction of WCEL aims to boost classification performance. For a comprehensive outline of the methodology, refer to Algorithm 1. The efficacy of the proposed model was thoroughly assessed using the extensive X-ray image dataset COVIDx. The experimental findings distinctly establish that the proposed model excels in the realm of COVID-19 image classification. Notably, even when trained with a notably limited quantity of labeled X-ray images, the model showcases remarkable performance.

Algorithm 1. Learning of Semi-supervised ResNet (SSResNet).

Require: training sample $x_i$, the set of training samples $S$, label $y_i$ for $x_i$ ($i\in S$)

1. for $t$ in [1, num epochs] do

2.  for each minibatch $B$ do

3.    $z_{i\in B}\leftarrow f_{{\theta }_{\mathit{\text{shared}}}}\left(x_{i\in B}\right)$ $⊳$ shared representation

4.    $z_{i\in B}^{\mathit{sup}}\leftarrow f_{{\theta }_{\mathit{sup}}}\left(z_{i\in B}\right)$ $⊳$ supervised representation

5.    $z_{i\in B}^{\mathit{\text{unsup}}}\leftarrow f_{{\theta }_{\mathit{\text{unsup}}}}\left(z_{i\in B}\right)$ $⊳$ unsupervised representation

6.    $l_{i\in B}^{\mathit{\text{WCEL}}}\leftarrow -\frac{1}{\mid B\mid }{\sum }_{i\in B\cap S}\mathit{\text{logϕ}}\left(z_i^{\mathit{sup}}\right)\left[y_i{w}_i\right]$ $⊳$ supervised loss component

7.    $l_{i\in B}^{\mathit{\text{MSEL}}}\leftarrow \frac{1}{C\mid B\mid }{\sum }_{i\in B}{‖z_i^{\mathit{sup}}-z_i^{\mathit{\text{unsup}}}‖}^2$ $⊳$ unsupervised loss component

8.    $\mathit{\text{Loss}}\leftarrow l_{i\in B}^{\mathit{\text{WCEL}}}+\lambda \times l_{i\in B}^{\mathit{\text{MSEL}}}$ $⊳$ total loss

9.    update ${\theta }_{\mathit{\text{shared}}}$, ${\theta }_{\mathit{sup}}$, ${\theta }_{\mathit{\text{unsup}}}$ using optimizer, e.g., ADAM

 return ${\theta }_{\mathit{\text{shared}}}$, ${\theta }_{\mathit{sup}}$, ${\theta }_{\mathit{\text{unsup}}}$

Upon analyzing the class distribution disparities between our training and testing datasets, a noteworthy distinction came to light. Consequently, we undertook the task of reconstructing the data structure. This involved partitioning the dataset into distinct training and testing subsets, ensuring that their class distributions closely aligned. Specifically, 70% of the data was allocated for the training dataset, while the remaining 30% constituted the testing dataset. For a comprehensive breakdown of the reconstructed dataset, including sample distribution details, please refer to Table 2.

Dataset	Normal	Pneumonia	COVID-19	Total
Training	6195	6708	75	12,978
Testing	2656	2876	33	5565
Total	8851	9584	108	18,543

Table 2.

Sample distribution in different classes for training and testing datasets.

It is evident that the distribution of samples in our dataset is highly skewed, particularly in relation to the COVID-19 class. This imbalance presents a significant hurdle in achieving a high-performing classifier. To surmount this issue, our proposed model employs a weighted cross-entropy loss. This innovative approach involves according greater weight to the minority class (COVID-19) throughout the training process. For a comprehensive understanding of this technique, please refer to section two.

In the experiment, the key hyper parameters for training the proposed model are: Minibatch size: 256, Number of epoch: 50, Optimizer: Adam optimizer, and Initial Learning rate: 0.1. They are determined by trial and error. Moreover, the details of the model architecture are illustrated in Table 3. We employ COVID-Net¹ as a baseline supervised model to present the state-of-the-art performance of COVID-19 image classification for comparison. Furthermore, we compared the proposed model with SRC-MT that is the state-of-the-art semi-supervised learning since it outperformed $\mathrm{\Pi }$ model and mean teacher model in the area of medical image classification.

Name	Description
Input	Medical Images
Shared ResNet	one convolutional layer, 2 residual block
	batch normalization, one pooling layer
Supervised ResNet	one convolutional layer, 2 residual block
	batch normalization, one pooling layer
Unsupervised ResNet	one convolutional layer, 2 residual block
	batch normalization, one pooling layer
Output	image class $\varphi \left(z^{\mathit{sup}}\right)$ and
	a new representation $z^{\mathit{\text{unsup}}}$

Table 3.

The proposed network architecture.

Table 4 showcases a comprehensive comparison of the classification performance between SRC-MT and the newly introduced model (SSResNet). On the whole, the overall accuracies attained by SRC-MT surpass those achieved by the proposed SSResNet. Nonetheless, an interesting observation emerges when considering scenarios where merely 5% of labeled samples were utilized for training. In this specific scenario, the MacroF metric of SSResNet surpasses that of SRC-MT. This outcome indicates that the proposed model exhibits greater efficacy in identifying COVID-19 samples. Essentially, the utilization of the unsupervised path in SSResNet appears to remarkably enhance data representation, thereby leading to a more pronounced improvement in COVID-19 classification performance compared to SRC-MT.

Semi-supervised model	Accuracy (%)	MacroP (%)	MacroR (%)	MacroF (%)
SRC-MT (5%)	90.67	61.08	60.75	60.59
SRC-MT (7%)	89.82	89.92	74.13	78.95
SRC-MT (9%)	92.79	93.61	79.15	84.15
Our model	Accuracy (%)	MacroP (%)	MacroR (%)	MacroF (%)
SSResNet (5%)	84.95	61.18	66.76	62.41
SSResNet (7%)	84.21	63.67	67.85	62.83
SSResNet (9%)	81.79	59.34	70.99	59.19

Table 4.

Comparing performance between SRC-MT and proposed model (semi-supervised ResNet (SSResNet)).

Furthermore, this study delved into a granular examination of the performance for each class, elucidating the outcomes through the employment of confusion matrices, as depicted in Figure 10. A noteworthy observation emerges from this analysis: The proposed model outperforms SMC-TC in terms of recognizing COVID-19 cases. This observation underscores the capability of SSResNets to glean more effective features from unlabeled data, resulting in a heightened ability to discern COVID-19 samples. Notably, as the ratios of labeled data are incrementally augmented, there is a marked improvement in the accuracy of COVID-19 recognition. This underscores the proposition that the inclusion of an unsupervised path serves to elevate image representations, consequently bolstering classification performance. In essence, the integration of unlabeled data distinctly contributes to a substantial enhancement in COVID-19 classification performance, primarily attributable to the amplification of image representations facilitated by the unsupervised path within the SSResNet.

Figure 10.

Comparison of confusion matrix generated with SRC-MT and SSResNets trained on different ratios of labeled data.

3.2 Face mask detection

This chapter also introduces the process of constructing a mobile model designed for face mask detection on edge devices. The primary goal of this endeavor is to develop an intelligent Internet of Things (IoT) device capable of performing real-time video processing to ascertain whether an individual is wearing a face mask in public settings. This technology finds practical application in scenarios such as enforcing mask-wearing within facilities or buildings. In this context, a smart IoT camera functions as a vigilant observer, detecting instances where individuals are not complying with the mask mandate and triggering alarms as necessary. To achieve this, mobile devices like FPGAs, integrated into the camera system, execute the face mask detection models. For a visual representation of this concept, refer to Figure 11, which presents an illustrative diagram depicting the potential of employing face mask detection to control access to a door.

Figure 11.

A diagram of a face mask alarm system. A face mask alarm mounted on the door consists of a camera, a face mask detector, and a convertor. The camera will send the personal image to the face mask detector consisting of mobile GPUs and artificial neural networks. Then, the detector will detect the face mask on the image by running the neural networks on the mobile GPUs and sent the detection result. Finally, the convertor will show a speech reminder such as “Face Mask Required” back to the person if the detection result indicates there is no face mask on the face.

Instead of embarking on the construction of models from scratch, the approach taken involves the utilization of pre-trained models that have undergone training using more extensive datasets across broader classes. This pre-trained ensemble comprises four distinct convolutional neural networks (CNNs): MobileNet V2, Inception V3, VGG 16, and ResNet 50. Leveraging transfer learning, as opposed to constructing models from scratch, often leads to superior performance. These pre-trained models have been honed on the comprehensive ImageNet dataset [56]. The salient attributes extracted from these pre-trained models serve as the foundation, conveyed to a novel classifier positioned at the terminal end of the network. Subsequently, a mask detection classifier is trained on top of the pre-trained model. A pivotal distinction discernible among these models is their input size. Inception V3 adopts an image size of $299\times 299\times 3$, while the other three models employ a size of $224\times 224\times 3$. The efficacy of these models is rigorously verified through inference runs on mobile devices, encompassing the NVIDIA Jetson TX2 and NVIDIA Jetson Nano platforms.

Dataset and experiment setup are presented below.

• Dataset: We utilized publicly available datasets², which consist of 3890 images categorized into two classes: with face and without face. Among these, 1916 images featured individuals wearing masks, while 1930 images depicted individuals without masks, after removing redundant entries. This dataset was deliberately structured to maintain a balanced distribution for our classifier. However, it does present some exceptional cases, such as images containing multiple faces or instances where faces are partially occluded by other body parts.

• Experiment setup: A learning rate of 0.0001 was employed, while the experiment consistently utilized a batch size of 10. This modest batch size was deliberately selected to facilitate extended training while operating within memory limitations. Furthermore, the experimentation spanned 100 epochs. The loss function of choice was binary cross-entropy.

The experiment involves the utilization of publicly available datasets, sourced from https://github.com/chandrikadeb7/Face-Mask-Detection. This dataset comprises a total of 3890 images that include both images with faces and images without faces. Within this collection, there are 1916 images depicting individuals wearing masks and 1930 images without masks. This dataset design maintains a balanced distribution, which is optimal for classification. It is important to note that certain exceptions are present, such as images containing multiple faces or faces partially covered by other body parts. In configuring the models, a learning rate of 0.0001 is employed. Throughout the experiment, a batch size of 10 is consistently used. This choice is rooted in the goal of facilitating more extensive training while optimizing memory usage. The training process spans 100 epochs, and for the loss function, the binary cross-entropy is selected.

In the model implementation, a diverse set of tools come into play. TensorFlow [77], a widely adopted open-source software library, forms the core framework for machine learning applications, supporting operations across various processing units including CPUs, GPUs, and TPUs. The flexibility it offers, in terms of both architectural design and working with pre-trained models, contributes to its popularity. Keras [78], which functions atop TensorFlow, is another critical component. Designed for expedited execution of deep learning models, Keras enhances the speed of development. For this research, Keras is employed to leverage pre-trained deep learning models, which are then fine-tuned to create the face mask detection classifier through transfer learning.

This study delved into the robustness of the models by subjecting them to training on limited sample sizes. Traditionally, working with small training datasets results in diminished training and testing performance. However, given the unique context of face mask detection in the context of the COVID-19 outbreak, acquiring extensive data for training and testing becomes a challenge. Thus, developing models for face mask detection that excel with small training data becomes pivotal for creating effective applications. Additionally, these models must be locally deployable and capable of achieving tasks at an optimal speed.

To address these requirements, various ratios of training data were employed to assess model performance. The outcomes, presented in Table 5, were obtained by running these models on mobile NVIDIA GPUs. The experimental results yield noteworthy insights. MobileNet V2 emerges as the swiftest model, processing nearly 40 frames per second (FPS) on the Jetson TX2 platform. On the other hand, VGG 16 attains the highest accuracy among the models. Notably, when trained using just 1% of the available training data, Inception demonstrates superior performance.

Models	Training				Testing on TX2			Testing on Nano
	$R_{\mathit{tr}}$	$N_{\mathit{tr}}$	Loss	Training accuracy	Testing accuracy	Inference time (ms)	FPS	Testing accuracy	Inference time (ms)	FPS
MobileNet V2	1%	30	0.107	0.9776	0.6957	26.97	37.27	0.6926	42.40	23.58
	5%	150	0.095	0.9733	0.7685	25.10	39.86	0.7659	42.00	23.80
	10%	300	0.114	0.9622	0.7842	25.10	39.87	0.7801	42.90	23.31
	20%	601	0.136	0.9490	0.7735	25.53	39.14	0.7699	43.03	23.24
ResNet 50	1%	30	0.071	0.9666	0.5000	70.90	14.11	0.7433	195.05	5.13
	5%	150	0.050	0.9845	0.7976	71.00	14.09	0.8023	176.45	5.67
	10%	300	0.035	0.9900	0.7976	70.13	14.25	0.7962	173.05	5.78
	20%	601	0.027	0.9927	0.8651	71.00	14.08	0.8599	173.65	5.76
Inception V3	1%	30	0.167	0.9556	0.8275	89.27	11.21	0.8275	187.00	5.35
	5%	150	0.136	0.9578	0.8704	87.80	11.41	0.8713	188.53	5.31
	10%	300	0.136	0.9445	0.9066	92.33	10.85	0.9057	905.72	5.22
	20%	601	0.167	0.9379	0.8972	89.57	11.17	0.8977	192.77	5.19
VGG 16	1%	30	0.089	0.9667	0.6857	2138.77	0.47	0.6761	239.37	4.18
	5%	150	0.006	0.9978	0.9231	2139.60	0.47	0.9173	245.23	4.08
	10%	300	0.010	0.9956	0.9370	2136.00	0.47	0.9325	236.17	4.23
	20%	601	0.006	0.9994	0.9607	2139.57	0.47	0.9580	237.70	4.21

Table 5.

Performance comparison on face mask detection on Jetson TX2 and Jetson Nano. FPS refers to the number of images processed per second. $R_{\mathit{tr}}$ and $N_{\mathit{tr}}$ denote the ratio of training data and the number of training images.

Observations indicate that training accuracy surpasses 90% for each model, indicative of overfitting given the markedly lower testing scores. As the proportion of training data is gradually increased to 5, 10, and 20%, the performance of all models exhibits enhancement. This suggests that augmenting the dataset samples contributes to alleviating overfitting issues. VGG secures the highest accuracy, and while training on 20% of the data, MobileNet attains the lowest accuracy. ResNet and Inception showcase comparable performance levels throughout this experimentation.

Additionally, it is observed in Table 6 that the pretrained models performed better on recognizing images containing masks regarding the precision, recall, and Fscore. Additionally, these pretrained models can achieve promising performance in terms of Fscore.

Models	Class	Precision	Recall	Fscore
MobileNet V2	With Mask	0.99	0.83	0.90
	Without Mask	0.86	0.99	0.92
ResNet 50	With Mask	1.00	0.96	0.98
	Without Mask	0.96	1.00	0.98
Inception V3	With Mask	1.00	0.98	0.99
	Without Mask	0.98	1.00	0.99
VGG 16	With Mask	1.00	1.00	1.00
	Without Mask	1.00	1.00	1.00

Table 6.

Comparison of precision, recall, and Fscore.

In summary, we leveraged pre-trained models such as MobileNet V2, ResNet 50, Inception V3, and VGG 16. These models were chosen due to their established performance in various applications. When considering model complexity, MobileNet V2 stands out for its efficiency, boasting the lowest complexity among the aforementioned models. On the other hand, VGG-16 adheres to a classical architecture characterized by a substantial number of parameters, leading to an overall higher complexity. InceptionV3, with its emphasis on capturing multi-scale features, adopts a more intricate architecture than MobileNet V2. In the context of speed and accuracy, as showcased in the performance comparison table labeled “Performance comparison on face mask detection on Jetson TX2 and Jetson Nano,” MobileNet V2 outperforms the others in terms of speed. In contrast, VGG-16 exhibits the slowest processing speed, aligning with the principle that more complex models tend to operate at a slower pace. Furthermore, it is noteworthy that VGG-16 achieves optimal performance relative to the other models, driven by its expansive model capacity. This insight underscores the trade-off between complexity and performance, where VGG-16’s higher capacity contributes to its superior performance despite the trade-off in processing speed.

3.3 Challenges to real-world applications

Deploying COVID screening for real-world applications still presents several challenges:

• COVID screening: To be effective in real-world scenarios, COVID screening requires a streamlined and efficient process. However, existing techniques, particularly deep learning-based X-ray COVID detection, involve intricate steps, including chest X-ray tests and subsequent COVID classification. Moreover, the quality of data obtained from chest X-ray tests might not be optimal for accurate COVID classification. Additionally, the reliance on high-performance computing hardware for deep learning-based classification hinders its applicability in underdeveloped regions.

• Face Mask Detection: It encounters several obstacles that hinder its real-world applications. Firstly, the high accuracy achieved in controlled experiments might not translate directly to real-world scenarios due to significant differences in image backgrounds, object sizes and positions, and object overlaps. Secondly, face mask detection must work across various environments, including both indoor and outdoor settings. Adapting to outdoor environments requires techniques capable of handling diverse weather conditions and backgrounds, which poses difficulties for current methodologies. Lastly, users prefer seamless face mask detection that does not disrupt their daily activities. This demands the development of techniques that can perform detection without requiring users to actively interact with a camera.

In both cases, addressing these challenges requires innovative solutions that can simplify processes, adapt to diverse conditions, and accommodate user preferences without compromising performance or accuracy.

4. Conclusion

The deployment of technologies such as COVID-19 screening and counterfeit mask detection plays a crucial role in curbing the spread of the COVID-19 virus. This chapter has introduced advanced AI-driven methods designed to enhance both COVID-19 screening and the identification of face masks. Specifically, the utilization of deep learning for COVID-19 classification has demonstrated significant potential in the creation of effective screening tools. Moreover, the application of deep learning for face mask detection on mobile devices has exhibited promising results. In the pursuit of developing these innovative techniques, valuable insights are gained that can be applied in future pandemic situations. In the future, potential work includes: (1) Overfitting is characterized by machine learning models achieving impressive performance during the training phase, but faltering when tested on new data. To ensure the efficacy of large models for COVID screening and face mask detection, it is imperative to address this concern. In our forthcoming work, we aim to tackle this issue through the implementation of data augmentation techniques, leveraging expansive pre-trained image models. More specifically, these large pre-trained image models hold the capacity to generate synthetic data for both COVID screening and face mask detection. This synthetic data will encompass varying backgrounds, enriching the diversity of the training dataset and reducing overfitting tendencies. Additionally, we intend to incorporate human feedback as an integral part of the loop. This iterative process will involve human assessment to gauge the quality of the generated data, fostering continuous improvements in the data augmentation strategy. This combined approach of data augmentation and human feedback holds significant promise in enhancing the generalization capabilities of the models, thereby enabling robust and high-performing COVID screening and face mask detection systems in real-world scenarios; (2) AI ethics continue to be a significant concern across various applications, encompassing issues such as privacy breaches and biased data. In our future efforts, we are dedicated to addressing these ethical challenges head-on. Our strategy involves the application of privacy-preserving techniques to safeguard user privacy, coupled with measures to mitigate biased data. To protect user privacy, we are poised to employ our pioneering privacy-preserving edge intelligent computing framework. This entails training autoencoders in an unsupervised manner on individual edge devices. Subsequently, the latent vectors derived from these autoencoders are transmitted to the edge server for classifier training. This approach effectively reduces communication overhead while safeguarding end-users’ sensitive data from exposure. In tackling biased data concerns, our plan is to integrate fair pre-processing techniques from AIF360, an AI fairness toolkit.³ These techniques will be strategically applied during the data collection phase for both COVID screening and face mask detection. By doing so, we aspire to counteract biases that may emerge in the data, ensuring equitable and unbiased outcomes. By proactively addressing privacy and bias concerns through cutting-edge privacy-preserving frameworks and fairness techniques, we aim to develop AI solutions that not only excel in performance but also uphold the highest standards of ethical conduct.

Acknowledgments

This research work is supported by the US NSF award 2018945, 2205891, and 2302469.

Conflict of interest

The authors declare no conflict of interest.