RGFSAMNet: An interpretable COVID-19 detection and classification by using the deep residual network with global feature fusion and attention mechanism

3.1 Data Collection

SARS-COV-2 [43] and the Chest radiography [44] [45] datasets were publicly available datasets that we employed to evaluate our proposed model CONet to detect the presence of COVID-19. The SARS-COV-2 dataset contains CT scan pictures of COVID-19 patients and healthy persons. Hospitalized patients in Sao Paulo provided these statistics. The images depict persons who have been hospitalized due to SARS-COV-2 infection (COVID) and those who have other lungs damaged by the virus, including 60 women and 60 men with abnormalities. It was gathered from a total of 120 patients who participated. This dataset comprises 2482 CT scans, including 1252 from COVID-19-positive patients and 1230 from COVID-19-negative patients.

The suggested model CONet detects COVID-19 using chest radiography and SARS-COV-2. A chest radiography dataset includes COVID-19, pneumonia, and healthy patient X-rays. Pneumonia, COVID-19, and Normal each have 4192 images, totaling 12,576. We need these images to train and evaluate our system for identifying COVID-19 from other lung illnesses. Image samples from multiple situations allow a robust evaluation of the model’s generalization and performance on unseen data.

Both datasets aim to support scientific research on artificial intelligence systems capable of analyzing CT scans and chest radiography pictures to detect SARS-CoV-2 infection in individuals.

3.2 Methodology

The proposed methodology consists of some main steps, including data preprocessing, model development, hyperparameters setting, and training and testing the proposed models on the dataset. Firstly, we load the lung CT scan and Chest X-ray image datasets into a working environment. Once the dataset is loaded, the data pre-processing operations, such as image resizing, data normalization, and data augmentations, are performed. Then, we utilized the deep residual learning context with global feature fusion concepts and attention mechanism to build the proposed deep learning model named “RGFSAMNet”. Furthermore, we will be dividing the dataset into training and testing. In this model, we divide the dataset into 80% for the training set, 20% from the training for validation, and the remaining 20% for testing. We will develop a tailored approach to train our new model in batches and allow it to run for a few epochs. Ultimately, it is necessary to assess the model’s performance by evaluating it using the testing dataset. The semantic representation of the proposed methodology is shown in Figure 2:

• Download figure
• Open in new tab

Fig 1.

Sample images for both datasets

• Download figure
• Open in new tab

Fig 2.

Schematic representation of the proposed methodological framework.

3.2.1 Data Pre-processing

Data preprocessing involves several processes, including extracting labels from images, converting images to RGB format, conducting feature scaling, and resizing images.

Feature scaling

Feature scaling usually applies to the independent variables of input data to standardize them within a fixed range such as [0, 1]. In this method, the CT scan and X-ray pictures in the dataset are initially converted to RGB format and then enlarged to 224x224 pixels using OpenCV procedures. The conventional range of image pixel values is [0, 255]. Here 255 is the maximum value. At the beginning, two distinct datasets are loaded with images and records. Subsequently, the data is transformed into NumPy arrays to facilitate feature scaling. Max normalization changes the pictures’ data to the range [0, 1] by dividing each pixel value by 255. On the other hand, the label data is encoded one-hot to change the category labels into binary format [46]. As a result, the normalized number is shown by the letter x^′. where x is the original intensity of the image.

Data splitting

We used a local directory to access the dataset and directly accessed them using the “operating system” library of Python and subsequently processed with the OpenCV library. The images were then converted into appropriate data types to facilitate model training. It’s necessary to partition the dataset into training and testing subsets. For this model, the dataset is divided into 80% for the training set, 20% from the training set for validation, and 20% testing set [47].

Data Augmentation

In data augmentation, we tried to expand our dataset to enhance its diversity artificially. The primary goal is to introduce variations through minor modifications of the original data. An ImageDataGenerator class is provided by the Keras deep learning framework to fit the model through image data augmentation. In this study, we applied data augmentation to the training data using the ImageDataGenerator class, generating multiple versions of the same images. This approach is particularly advantageous when working with a limited dataset [48]. The augmentation was only applied to the training data as detailed in Table 1, as well as the visualization of the augmented samples, is shown in Figure 3 and here we have chosen to:

View this table:

• View inline
• View popup
• Download powerpoint

Table 1. Data augmentation parameters that we utilized in this research.

• Download figure
• Open in new tab

Fig 3.

Samples of augmented images of Chest CT-scan & X-ray.

3.2.3 Proposed Model Descriptions

The latest developments in deep learning systems have impacted artificial intelligence by making the models more sophisticated and accurate in a wide range of tasks. In a neural network, Deep states are a term that measures the number and size of the layers in the network, indicating its depth. The models may learn the features at a high level by using this amount of depth, which can improve their performance on complex tasks.

The entire structure is represented by a deep residual learning mechanism-based block (ResNet-50) as the base model. Deep CNN model mainly comprises three main layers: Convolution, Max-pooling, and Dense. For extracting features, the convolution layer uses filters from the input data. The max-pooling layer usually increases computational performance by minimizing the size of these features. The dense layer connects all previous layers. The dense layer is also known as a fully connected layer.

The deep CNN models can be optimized for classification and detection by modifying hyperparameters.

The proposed RGFSAMNet model architecture is shown in Figure 4 and described below in detail.

• Download figure
• Open in new tab

Fig 4.

Schematic architecture of the proposed model.

Input Layer

The input layer of the CNN contains the numerical values representing the pixels of the input image. The input is a 224x224x3 pixel array for a 224x224 dimension image with three color channels, specifically the RGB color channel [56].

Convolutional Layer

The Convolutional Layer is the key component of CNN. The Convolutional Layer produces a new image that displays the features of the input image. The convolutional layer applies a filter to each input image during the process. The filter on this layer is a 2-dimensional array with dimensions of 5x5, 3x3, or 1x1.

Then the filter generates a feature map, later used by the Action Layer [57]. The 2D convolutional operator for the received image (X) and kernel (k) is:

Where Y [i, j] is the output at position (i, j). X[i· s + m− p, j· s + n− p] is the input at position (i· s + m −p, j ·s + n −p), K[m, n] is the kernel at (m, n), H and W are the height and width of the kernel, i and j are the output coordinates, s is the stride (defining the step size of the kernel as it moves across the input image) and p is the padding (the number of pixels added to the border of the input image).

Base Model

For our proposed model, we used the ResNet-50 pre-trained model as the base model. ResNet-50, one of the major advancements in deep learning, is used in this project. Training the model before ResNet came into existence seemed to be one of the hardest tasks due to fading gradients. Skip connection is one of the major strengths of the ResNet model. ResNet-50 has 50 neural network layers. The network takes the input image in multiples of 32 as width and height. ResNet-50 is a modified version of the Residual Network (ResNet) structure, specifically developed to facilitate the training of extremely deep neural networks. The network is composed of 50 layers, which include convolutional, batch normalization, and ReLU layers. Additionally, shortcut connections enable the network to bypass one or more levels. The fundamental mathematical principle behind ResNet is the incorporation of residual blocks, in which the input is combined with the output. This approach effectively addresses the issue of the vanishing gradient problem and facilitates the training of more complex networks [12].

ResNet-50, like other ResNet architectures, is built on the concept of residual learning. The left upper part of Figure 4 shows the block diagram of the residual learning block. The core idea is expressed mathematically as:

Here, H(x) is the desired underlying mapping that the network aims to learn, x is the input to a residual block, and F (x) is the residual mapping that the network needs to learn. The term F (x) + x represents the output of the residual block, where F (x) is the output of the stacked non-linear layers (e.g., convolutional, batch normalization, and ReLU layers) and x is the shortcut connection that bypasses these layers. In a network with multiple layers like ResNet-50, this approach is repeatedly used in many residual blocks, allowing the network to learn increasingly complex mappings while mitigating the problem of vanishing gradient. Shortcut connections aid in the propagation of gradients during backpropagation, hence streamlining the training of deeper neural networks.

Global Feature Fusion

Global feature fusion is generally accepted to combine multiple global pooling strategies, such as Global Average Pooling (GAP) and Global Max Pooling (GMP), for extracting more comprehensive and robust features from a deep CNN model. These techniques will be able to retain both the global spatial information and most significant feature activations [58].

Global Average Pooling (GAP) [59] is a pooling operation that calculates the average of each feature map across the spatial dimensions, reducing each feature map to a single scalar value. This operation effectively compresses the spatial dimensions, maintaining the global structure of the feature maps. For a given feature map F∈ℝ^H×W×C with height H, width W , and C channels, the GAP for each channel c can be computed as: where F_ijc is the value of the feature map at position (i, j) in channel c. The result GAP_c is a scalar value representing the average of the entire feature map for the c-th channel. This operation reduces the feature map F∈ℝ^H×W×C to a vector GAP ∈ℝ^C .

Global Max Pooling (GMP) [60] selects the maximum value from each feature map across the spatial dimensions. This helps in capturing the most prominent or activated feature in each channel, making it useful for identifying strong activations.

For a feature map F ∈ℝ^H×W×C , the Global Max Pooling (GMP) for each channel c is computed as: where GMP_c is the maximum value from the entire feature map for the c-th channel.

This operation reduces the feature map F∈ ℝ^H×W×C to a vector GMP∈ ℝ^C .

The idea of global feature fusion is to combine the features extracted by both GAP and GMP, leveraging the benefits of both approaches. While GAP captures the overall distribution of the feature map, GMP focuses on the most salient features. One simple fusion method is concatenation, where the outputs of GAP and GMP are concatenated along the channel dimension:

Here GAP ∈ℝ^C is the vector resulting from global average pooling. GMP∈ℝ^C is the vector resulting from global max pooling. F_fusion ∈ℝ^2C is the concatenated feature vector, containing both average and max-pooled features.

Attention Mechanism

The attention mechanisms in image classification give the model a way to learn which parts of the input are more important for the task at hand; this is therefore helpful for performance by effectively underlining features that will be crucial. A short description of attention is that it calculates the weighted sum of features in a manner such that more important features have higher weights. In an image, the features are the pixel values in feature maps that are created by a deep learning model [61].

The attention function can be summarized as:

Where is the query matrix. is the key matrix. is the value matrix. d_k is the dimensionality of the key vectors. The softmax function ensures that the attention weights sum to 1.

Activation Layer

The layers are reduced into 512 and a pyramid-like shape. The major role of this layer is the reduction and the activation function used here is the ReLU. The activation function is responsible in a neural network for converting the total weighted input from the node into the node or output activation for this input [62]. The equation of the ReLU function is represented as: where x is the neuron’s input value.

Droupout Layer

One of the techniques for neural network regularization is known as dropout. It prevents overfitting due to a section of neurons being randomly ‘dropped out’ from the forward pass during training. This dropout is made with some probability and is applied independently to every neuron in a layer. In our proposed model, we used a dropout rate of 0.3, which means 30% of randomly selected neurons were disabled during model training [63].

Dense Layer

The Dense Layer is one of the most important neural network architectures, providing flexibility and expressiveness to learn from the data. This ability makes all neurons in this layer connect to the previous layer, hence helping to model complex relationships and thus quite indispensable in deep learning. In a dense layer, each neuron is connected to every input feature, which is what makes it highly expressive, hence able to learn complex patterns [64].

Output Layer

The output layer here is named as ‘Prediction’. The output of the dense layer is fed as input and the output is obtained as a binary value of either masked or non-masked. The output of this layer is fed directly as output to the model [65]. The softmax function is defined as:

This function takes a K-dimensional input vector z and returns a K-dimensional vector of values within the range 0 to 1 that sum to 1.

3.3 Explainable AI

By making machine learning models more transparent and interpretable, XAI (Explainable Artificial Intelligence) methods like Grad-CAM, LIME, and SHAP can be vital in healthcare for COVID-19 detection. Building trust in AI-based diagnostic systems is essential, and these techniques help researchers and clinicians understand why a particular prediction was made. The XAI methods can help identify COVID-19 in medical images and patient data by creating graphical representations of model decisions. Along with facilitating a more precise diagnosis, this also aids in the discovery of possible disease-related biomarkers or patterns. Ensuring patient safety, minimizing errors, and fostering collaboration between AI systems and healthcare professionals are crucial in a field as critical as healthcare, where decisions directly impact patient outcomes. Having explainable AI is a must.

3.3.3 SHAP as XAI

By illuminating the model’s reasoning behind medical image classification decisions, SHAP (SHapley Additive exPlanations) [67] can greatly help the healthcare industry’s search for COVID-19. We use SHAP to explain the model’s predictions on a picture of possible COVID-19 symptoms. The SHAP values help medical professionals see which parts of the picture matter for diagnosis because they show how much each pixel contributed to the final product.

For AI-based diagnostic systems to gain confidence from users, their decisions must be interpretable so that doctors can validate them using their knowledge. In addition, SHAP can aid in the identification of particular radiological patterns or characteristics indicative of COVID-19, allowing for faster and more precise diagnoses. SHAP’s interpretability is essential to guarantee patient safety and improve the efficacy of diagnostic procedures in the fast-paced and high-stakes healthcare environment.

• Let f : ℝ^d →ℝ be the function representing the model’s prediction, where d is the number of features. The SHAP value for feature i in the context of a specific prediction x is given by:

Where x_S represents a subset of features excluding feature i, The sum is over all subsets S of features excluding i, |S| denotes the number of features in subset S, f (x_S ∪ {i}) is the model’s prediction when including feature i in the subset S, f (x_S) is the model’s prediction without including feature i in the subset S.

3.4 Proposed Model Training

The features extracted from the image are provided as input to the convolutional neural network layers. Here, we use the ResNet-50 deep convolutional neural network as our base model, and this network aids model training with its 50 levels of depth. The base model is not trained with any predefined weights. The shape of the input is defined as the image size of the processed image, which is 224*224 pixels. The base model’s output is put through global average pooling. The ReLU activation function is used as the model’s input, and the Softmax function is used as the output. The model is then optimized with the Adam optimizer with a learning rate of 0.003; the loss here is considered to be the categorical cross-entropy, and the evaluation metrics of every training epoch are calculated with accuracy. The model is then trained with the train and validation dataset with a batch size of 32, and the epochs considered here are 150 and 100.

3.5 Environment Setup

The software was implemented using Python3. The GPU was selected over the CPU for its superior speed and efficiency in training deep-learning models. The experiments were performed on an Apple Macbook Pro utilizing an Apple silicon M2 Max GPU.

3.6 Evaluation metrics

We evaluated the performance of our proposed model using a range of metrics, including precision, accuracy, F1 score, recall/sensitivity, specificity, Cohen’s kappa score, and Area Under the Curve (AUC). The confusion matrix, which includes the values for True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN), was utilized to derive these metrics [73]. By applying these metrics, we comprehensively evaluated the model’s ability to detect instances accurately. The model’s efficiency was determined using the following formula:

• Precision: It quantifies the precision of the favorable forecasts. It can be characterized by-

• Sensitivity/Recall: It determines the proportion of true positives correctly identified. It can be characterized by-

• Specificity: It determines the proportion of true negatives correctly identified. It can be characterized by-

• Accuracy: Accuracy is a metric that quantifies the extent to which a model is right. It can be characterized by-

• F1 Score: This measure is the reciprocal of the arithmetic mean of precision and recall. It can be characterized by-

• Cohen-Kappa Score:Cohen’s Kappa Score [74] enumerates the level of agreement between two raters or classifiers, using a scale ranging from -1 to 1 (1 = perfect agreement, 0 = chance agreement, ¡0 = less than chance). In our deep learning COVID-19 research, we will use Cohen’s Kappa to assess the agreement between our model’s predictions and the actual diagnoses of patients in order to accurately and reliably classify them as either “COVID-19 positive” or “COVID-19 negative”.

  P_o represents the observed agreement, while P_e denotes for the expected agreement resulting from chance.

• Area Under the Curve (AUC): The area under the curve (AUC) is a fundamental notion that represents the integral of a curve concerning the x-axis within a specific range. The AUC-ROC is a performance measure used to assess the accuracy of binary classification models. The Receiver Operating Characteristic (ROC) Curve at this location illustrates the correlation between sensitivity and 1-specificity at various thresholds. The AUC-ROC varies between 0 and 1. A value of 0.5 signifies random guessing, while 1 implies perfect discrimination. Values between 0.5 and 1 denote performance that is superior to chance. Values below 0.5 indicate performance that is worse than random. ROC analysis evaluates the precision of medical diagnostic tests in differentiating between diseased and non-diseased conditions in clinical epidemiology [75].

4.1 Performance on CT Scan dataset

We first experimented on the CT scan image dataset to evaluate our proposed model. To evaluate our model performance, we utilized a total of seven evaluation metrics that are Precision(Pre), Sensitivity(Sen), Specificity(Spe), Accuracy(ACC), F1-Score(F1-Sc), Cohen-Kappa Score(Co-Ka), and Area Under the Curve (AUC). In the model performance Table 3, we have used the short form of the evaluation metrics. The model’s accuracy represents the percentage of accurately classifying the images in the test dataset set. The model is trained with a sample of 1587 samples from the training dataset, validated with 397 samples, and tested with 497 sample images, which are split in proportion to the 60:20:20 ratio. The graph depicted in Figure 5 illustrates the progression of training accuracy, validation accuracy, training loss, and validation loss throughout each epoch of model training. At the beginning of training, during the first few epochs, both training and validation accuracies are poor, thus proving that the model does not train well at the beginning. Increasing the number of epochs quickly boosts the training accuracy to a high value, close to 1, indicating that it learns and fits the training data effectively. First, the validation accuracy is volatile, then it also grows and converges to a high value, one slightly lower than the model training accuracy.

• Download figure
• Open in new tab

Fig 5.

Proposed model accuracy and loss curve for CT-Scan Dataset

Probably due to either model initialization, learning rate, or noise in the data, it might be that the validation accuracy fluctuation occurs during the few epochs at the beginning. It tends towards the final stabilization near the training accuracy, indicating a good generalization to validation data with no significant model overfitting. The high accuracy of both the train data and the validation data proves to be quite good performance coverage. There is no overfitting against the training data when the training and validation accuracies have very little difference concerning the model; therefore, it is likely to perform well on unseen data.

A confusion matrix will help so much in knowing the performance of a classification model, contrasting true labels vs. predicted labels. In Figure 6, the Y-axis shows the true categories of the data. These categories are “COVID” and “non-COVID”, while the X-axis represents the categories predicted by the model. Top left, True Positives: 252 instances where the model correctly predicted “COVID”. Top-right: 1 instance of times where it has mispredicted “COVID” when it was actually “non-COVID”. Bottom left: 1 for the number of times the model predicted “non-COVID” where it is “COVID”.

• Download figure
• Open in new tab

Fig 6.

Proposed model confusion matrix and ROC curve for CT Scan Dataset

Bottom-right: True Negatives 243 cases where the model rightly predicted “non-COVID”. The performance of the model in classifying COVID-19 and non-COVID cases is very good, with very high accuracy, precision, recall, and high specificity. This means very minimal false positives and zero false negatives, which proves the model is very reliable for identifying true cases of COVID-19 without missing any. High values in the true positive and true negative cells and near-zero values in the false positive and false negative cells demonstrate that this is a very strong model.

Figure 6 represents the Receiver Operating Characteristic Curve (ROC) for a Classification Model. In this Figure, the X-axis represents the false positive rate, that is, the proportion of actual negatives that are misclassified as positives; the Y-axis represents the true positive rate, that is, the proportion of actual positives correctly classified as positives. The ROCs of both classes, 0 (cyan) and class 1 (orange), have an AUC of 1.00, which is a perfect classification with no false positives or false negatives. Also, the micro-average ROC curve is pink dashed with an AUC of 1.00, suggesting that the performance is uniform across classes. The random classifier is represented by the black dashed diagonal, where TPR=FPR and its AUC is 0.5. By contrast, ROC curves for the model under evaluation are well above this line, reflecting a high degree of discrimination. The ideal AUC scores with perfect separations between the classes show very good capability of the model in class discrimination, though such results may call for further investigation to check for overfitting in real-world applications. The combination of these curves presents a clear depiction of the model’s ability to achieve flawless sensitivity and specificity.

Table 2 shows the classification reports of the proposed model with three different optimizers. Among these optimizers, the Adam optimizer shows comparatively better results than the other optimizers. Table 3 shows performance metrics for various deep learning models, which have been optimized using different optimization algorithms on several evaluation criteria. The proposed model achieved the highest performance in all optimization metrics, mainly in Adam optimizers. It is followed that the metrics of precisions, sensitivity, specificity, accuracy, F1-Score, Cohen’s kappa, and AUC are highest for the proposed model are 99.60%, 99.59%, 99.60%, 99.60%, 99.60%, 99.16%, and 99.60% respectively. Among the three optimizers, RMSProp and SGD optimizers achieved the same sensitivity value of 99.55%. From Figure 7, we can easily compare the results for different optimizers individually for the CT scan dataset. This overall performance shows the proposed model to be of superior capability in predicting accurately and reliably on the desired outcomes.

• Download figure
• Open in new tab

Fig 7.

Comparison Bar Chart for different Evaluation Metrics for CT Scan Image dataset results with corresponding Models

4.2 Performance on X-ray dataset

For the second study, we have utilized the X-ray image dataset to classify COVID-19. In this dataset, we have included three class images, including COVID-19, pneumonia, and normal. We have used 12,576 images for the entire dataset, of which 80 percent have been used for training. Also, 20 percent of images from training data were used for the model validation. The remaining 20 percent have been utilized for the model testing, which was new for the model.

Figure 8 describes the performance of the proposed model on the chest X-ray dataset at 100 epochs, where one is about accuracy and the other one is about loss during training and validation. The training accuracy curve shows that the training accuracy starts low but increases rapidly to a value close to 1.0 (or 100%) by epoch 20, thus suggesting that the model can learn the training data very well. The validation accuracy curve is more volatile, especially at early epochs; it converged at about epoch 100. The validation accuracy also increased by nearly 1.0, indicating good generalization performance. These big swings in validation accuracy in the first epochs likely result from model adjustments, changes in learning rate, or data variability. With time, these fluctuations decrease. One can notice that the training loss begins high and reduces very fast initially, leveling near 0 at epoch 20 in the training loss curve. This means the model is efficiently reducing errors in the training data. The validation loss curve also illustrates quite a significant fluctuation, more so in the early epochs. Still, these fluctuations decrease over time, and the loss finally rests at a low value around epoch 100. The high and fluctuating validation loss in the initial epochs could result from overfitting or the model being fit to the distribution of validation data.

• Download figure
• Open in new tab

Fig 8.

Proposed model accuracy and loss curve for X-ray Image Dataset

The fluctuations decrease, finally stabilizing at a low loss, so it must improve model performance and generalize better. The accuracy and loss plots indicate that the model converges with increasing training. The stabilization of training and validation accuracy close to 1.0, with loss reduction close to 0, may be viewed as proof of the model learning effectively and generalizing well. Instability of validation metrics, as manifested through accuracy and loss at the beginning, is quite a common phenomenon, denoting adjustment of parameters by the model. The eventual stabilization could, therefore, be taken to imply that these adjustments are practical in nature. High final accuracy and low final loss in both the training and validation sets imply that the model does not overfit and generalizes well to new data. These figures demonstrate that the model trains well, learns the data properly, and generalizes well, which is evidenced by the fact that training and validation metrics converge.

In Figure 9, the left figure represents the confusion matrix, which summarizes how well the multiclass classification model did on the three classes: COVID-19, pneumonia, and normal. It also perfectly represents how well the model differentiates between these three classes based on actual labels versus predicted labels. From this figure, 842, 824, and 821 were instances where models correctly predicted Covid-19, pneumonia, and normal, respectively. High true positive rates of the proposed model suggest that most instances of COVID-19, pneumonia, and normal cases are classified correctly. A high number of true positives in each category evidences this. The model misclassifies very few, hence highly reliable. Specifically, only 2 case of COVID-19 is miss-classified as Pneumonia and 1 as Normal; only 3 of Pneumonia are misclassified as COVID-19 and 5 as Normal; and only 17 of Normal are misclassified as COVID-19 and 1 as Pneumonia. This low miss-classification number within all categories proves the strength and efficiency of the model in differentiating between different respiratory conditions. This is strongly supported by the confusion matrix, which shows that the classification model is quite effective and reliable. It provides excellent performances in each class, COVID-19, pneumonia, and normal, with great true positive rates and negligible cases of false positives and false negatives. This means that this model is one such measure through assessment metrics for model performance. This also proves that the model is very apt at making accurate diagnoses of these conditions from only the input data itself.

• Download figure
• Open in new tab

Fig 9.

Proposed model confusion matrix for X-ray Image Dataset

In Figure 9, the figure of the Receiver Operating Characteristic curve is a graph showing the diagnostic ability of the classification model to make distinctions among three different classes: COVID-19, Pneumonia, and Normal. The ROC curve indicates the tradeoff between the TPR and the FPR at various threshold settings. The dashed diagonal line from the point (0,0) to the point (1,1) represents a random classifier. Any ROC curve that runs above this line suggests a model performing greater than a random chance. The ROC curve is plotted for three classes altogether representing class 0, class 1, and class 2 belonging to Covid-19, Pneumonia, and Normal cases, respectively. Cyan, orange, and blue curves represent the ROC curves corresponding to each class. These curves plot the relationship between TPR and FPR for various thresholds. The micro-average ROC curve, shown in pink dashed line, aggregates the performance across all classes. This is useful in the case of multi-class classification tasks. For each ROC curve, the AUC, or Area Under the Curve, is given in the legend. The AUC gives a single number summarizing the performance of the model. In the following plot, the AUC for all three classes is 1.00, which means the model is perfectly separating the classes. The three classes are perfectly classified with AUCs of 1.00, and there is complete separation between true positives and false positives, where no class cases are superimposed in their distribution. Actually, all three classes’ ROCs lie almost completely on the top-left corner of the plot, which is the ideal position a classifier can achieve. The micro-average ROC curve also illustrates an AUC of 1.00, reflecting the aggregate performance across all classes and confirming the overall strong performance of the model on the task of multi-class classification. From the receiver operating characteristic curves, it is evident that this model is quite promising for modularity in sensitivity-sensitivity trade-offs when predicting these conditions from a given dataset. This is a good pointer to the model’s reliability and that it can be deployed in the real world for diagnostic purposes since it can confidently distinguish the three respiratory conditions.

Table 4 shows the classification reports of the proposed model with three different optimizers for the X-ray dataset. In chest X-ray classification reports, among these optimizers, the Adam optimizer shows comparatively better results than the other optimizers. Table 5 shows the classification results prepared for various models over an X-ray image dataset. It provides an overall comparison of the performance of each model that applies three variously used optimization algorithms — Adam, RMSProp, and SGD — against multiple evaluation metrics. The table shows the full comparison of multiple models or optimizers and how each combination performs over an X-ray image dataset. Above, it can be noticed that the proposed model is that for which the Adam optimizer has overall performance with precision 98.97%, sensitivity 99.41%, specificity 99.40%, accuracy 99.40%, F1-Score 98.97%, Cohen’s Kappa 98.45%, and AUC 99.22%. Also, the RMSProp optimizer turned out to have very high performance in terms of specificity, and other metrics a bit lower than Adam on some metrics. The SGD optimizer also showed good performance, which was very close to Adam’s results. From Figure 10, we can easily compare the results for different optimizers individually for X-ray image dataset performance. The proposed model always performs best compared to all metrics and optimization algorithms, especially based on Adam. Generally, Adam provides the best performance across most models, showing higher precision, sensitivity, specificity, accuracy, F1-Score, Cohen’s Kappa, and AUC. Once again, RMSProp and SGD perform very competitively in a few models. In a few models, Adam fails to surpass SGD’s performance, where ResNet-50, InceptionV3, and DenseNet121 are found to report very good performance with each optimizer, which manifests and prove prepotency, efficiency, and robustness for handling and dealing with the dataset. With very near-to-perfect performance metrics recorded, it can now be confirmed that the proposed model is very effective and reliable for classifying the X-ray images.

• Download figure
• Open in new tab

Fig 10.

Comparison bar chart for different optimizers for chest x-ray dataset results with corresponding models

4.3 Proposed model with explainable AI

Grad-CAM enhances interpretability by highlighting the most important features or areas in a model’s decision-making process; therefore, this enhances better transparency and reliability of a model’s decisions. Figure 11 and Figure 12 show the application of Grad-CAM in CT scan images and chest X-ray images, respectively. The area is highlighted for consideration during diagnosis. Original CT scans are shown on the left, depicting cross-sectional views of the human body. Regions can be underlined that are affected by abnormalities such as lesions or infections. Grad-CAM heatmaps on the right overlay these images with color gradients that show areas of interest to be most relevant according to the deep learning model for its predictions. Warmer indicates yellow to red indicates regions of high importance, while cooler colors represent areas of lesser importance. First set: In this, the model underlined great areas within the lungs, reinforcing that the attention of the model is on possible abnormalities related to the disease, such as fibrosis or tumors. Second set: This unmasks another focus where the underlined heatmap does in other anatomical areas, probably referring to other pathological features. Figure 12 shows chest X-ray images; from the heatmap derived from the first pair of images, it would appear that this model focuses on the upper regions of both lungs, which might point toward an abnormality that could be fibrosis or consolidation. The second pair of images shows a more focused area of interest in the middle zone of the lung, which may indicate a suspicious lesion or infection. The third pair focuses strongly on the model in the right lung with red and yellow colors, which stand for a conspicuous feature that might be a pathological one. This visualization underlines Grad-CAM’s utility in enhancing the interpretability of deep learning models: giving medical practitioners insight into how the model interprets the complex diagnostic data and ensuring that focus is clinically relevant and aligned with human expertise.

• Download figure
• Open in new tab

Fig 11.

GradCAM-based visualization for the proposed model for CT-Scan image

• Download figure
• Open in new tab

Fig 12.

GradCAM-based visualization for the proposed model for X-ray image

LIME helps interpret the model by slightly changing the input data and observing how the predictions change, highlighting the areas that contribute most to the model’s decision. Figures use Local Interpretable Model-agnostic Explanations (LIME) to explain the model’s prediction on CT scans and X-ray images. The far left image in Figure 13 is the original CT Scan with the prediction from the model. A prediction value of 1.00 indicates a very high confidence in the prediction—COVID-19 in this case. Next image with all superpixels: This CT scan is segmented into superpixels, contiguous regions of similar pixel intensities. LIME uses such superpixels to perturb the image and see how much the model cares about changes in these regions. The positive superpixels image emphasizes the superpixels that positively contribute to the model’s prediction. These are very fundamental areas on which the model builds a lot of confidence in predicting the class. Here, different colors indicate the degree of importance; warmer colors, like red, stand for large positive contributions. The heatmap shows a gradient of contributions by several superpixels. There is a color scale that signals the strength of their contribution. The scale ranges from negative to positive contributions. Here, warmer colors show the most important positive contributions, while cooler colors like blue are for negative or unimportant contributions. This is how the X-ray image could be explained in the same way. Positive Superpixel images indicate areas that most positively influence model predictions. For both CT scans and X-rays, regions highlighted in warm colors, like red, are important for a model’s decision. The Heatmap provides an exact view of how each superpixel contributes to the prediction, with the color intensity reflecting the strength of contribution. In this case, on the CT scan image, superpixels surrounding the lungs and specific patterns within the lungs show up as strong contributors to the model’s prediction. The highlighted regions in the Positive Superpixels and Heatmap images mark areas with possible abnormalities the model has engrafted to the predicted class. In Figure 14, in the case of the X-ray image, just like in the CT scan, superpixels within the lung regions are highly important. Indeed, the Positive Superpixels and Heatmap images reveal this: the model focused on patterns or anomalies within the lungs characteristic of the predicted class. LIME aids in building trust in the model’s predictions by providing a visual explanation of which parts of the image influence the decision. This could be especially helpful for medical practitioners to check if the model concentrates on clinically relevant areas.

• Download figure
• Open in new tab

Fig 13.

LIME-based visualization for the proposed model for CT scan image

• Download figure
• Open in new tab

Fig 14.

LIME-based visualization for the proposed model for X-ray image

Shapley’s Additive explanations are based on a game-theoretic approach that forces each feature to receive an important value for a particular prediction from the machine learning model. In Figure 15, here is one of the figures containing three images: one original CT scan image for a patient’s lungs, the other two being the SHAP values for two different classes, non-COVID and COVID. The input image serves as a reference for the areas being analyzed for the presence of COVID-19 or non-COVID conditions. The middle picture presents the SHAP values overlay on top of the CT scan for the non-COVID class. In the picture, blue regions correspond to a negative SHAP value and thus contribute to the model prediction of the image as non-COVID. The red areas are those parts where SHAP values are positive, contributing less towards the non-COVID prediction. The color intensity corresponds to the magnitude of SHAP values, with darker shades meaning higher absolute values. The right image projects the SHAP values onto the CT scan for the COVID class. In the image, red regions indicate positive SHAP values, and they contribute to the model predicting the image as COVID.

• Download figure
• Open in new tab

Fig 15.

SHAP-based visualization for the proposed model for CT scan image

In contrast, the blue region has a negative SHAP value, contributing less to COVID prediction. The intensities of the colors represent the magnitude of the SHAP values; the darker shades indicate higher absolute values. The color bar gives a scale for interpreting the SHAP values. The farther away the SHAP value from zero, the stronger the contribution of that region to the prediction (-0.0002 or 0.0002). The middle and right images identify which areas of the CT scan are very important to have the model’s predictions for non-COVID and COVID, respectively. For non-COVID, areas marked in blue (negative SHAP values) play a very important role in influencing the prediction toward non-COVID. For COVID, areas colored red (positive SHAP values) have a strong influence on the COVID prediction. By comparing the SHAP value overlays, one can see how different parts of the CT scan impact the model’s predictions for non-COVID vs COVID. If a region is red in the COVID-SHAP image and blue in the non-COVID-SHAP image, the combination makes that a very strong COVID indicator. SHAP values explain how the model works by showing the areas most influential in the classification of the CT scan as either COVID or non-COVID. This may be useful in understanding the strengths and weaknesses of the model.

Figure 16 contains four images: an x-ray image and its corresponding SHAP values for three different classes. We can explain our model with the same information for the three classes’ data as we explained for the CT scan images. Here the middle three images help to indicate which parts of the x-ray are most important to the model predictions for COVID-19, Pneumonia, and normal, respectively. In the COVID-19, red areas show a very strong positive influence of SHAP values on the prediction towards COVID-19. For Pneumonia, areas in red have positive SHAP values that strongly influence the prediction of Pneumonia. It means for Normal, areas in red marked with positive SHAP values strongly influence the prediction towards Normal.

• Download figure
• Open in new tab

Fig 16.

SHAP-based visualization for the proposed model for X-ray image

The SHAP figure properly visualizes how CT scan images in non-COVID and COVID conditions and X-rays in COVID-19, Pneumonia, and Normal conditions contribute different regions to classification. Areas of importance are highlighted with positive and negative SHAP values for explanation, clearly showing how the model made its predictions to interpret and trust its decisions in a clinical setting.

4.4 Model validation using User Interface (UI)

A user interface is developed according to the provided paradigm for the real-time validation of our suggested framework. The interface is developed using Gradio, an open-source Python package. This study utilized Python version 3.12.7 and Gradio version 2.9.2. Gradio facilitates the rapid creation of customizable web-based Graphical User Interfaces (GUIs) for deploying machine learning models or specific functions. This assists us in constructing a prototype of the deployed model to evaluate its performance in the user environment. The Gradio Interface fundamentally requires three parameters: the function defining the classification model, the input components, and the output components. Upon execution, a web page that includes the user interface is generated. The web page’s URL is shareable and can be disseminated publicly. Since this is publicly available, any individual with a browser can utilize it, provided the host device is operational. Figure 17 depicts the user interface wherein an image is uploaded and submitted for classification, yielding an output accompanied by the confidence level. This serves as user feedback for the model’s enhancement. To erase the current image, the ‘Clear’ button is activated. Various CT scans and chest X-ray images from different classes have been uploaded and tested in real-time, as illustrated in Figure 18. The model demonstrates 100% confidence in identifying CT scan pictures as COVID-19 or normal and chest X-ray images as COVID-19, pneumonia, or normal. Consequently, the suggested model exhibits significant performance in a real-time setting. The user can modify the uploaded image by cropping, rotating, or introducing noise to observe the variations in output.

• Download figure
• Open in new tab

Fig 17.

A Python Gradio user interface was designed to assess the proposed model’s real-time prediction capability.

• Download figure
• Open in new tab

Fig 18.

Gradio user interface (UI) for CT Scan and Chest X-ray image prediction.