A Drift-Oriented Adaptive Framework for Concept Drift Detection in Large-Scale Internet-of-Medical-Things Data Streams

4.1 Concept Drift Defination

An IoMT medical data stream consists of data instances that are continually changing and arriving sequentially [19]. In medical data streams, newly received data instances may have a pattern that differs from the preceding data. This occurrence is known as concept drift [20]. The Internet of Medical Things (IoMT) is a network of interconnected smart medical gadgets and applications that work together to improve smart healthcare. The continuous generation of streaming data from these devices raises concept drift challenges affecting model learning performance [21]. As stated in [21], Concept drift refers to a phenomenon observed in medical data streams, wherein the combined probability distribution Prob_t(X,y) of the input variable X and its corresponding class data y ∈ Y undergoes notable changes between successive time intervals, t and t + 1 can be denoted by [22]: There are two categories of concept drift that exist: virtual drift and real drift. These distinctions are demonstrated in Figure 3(a). Virtual drift takes place when the marginal distribution Prob(X) alters but the subsequent distribution Prob(y│X) remains unchanged. This means that Prob_t(X) is not equal to Prob_t+1(X), but Prob_t(y│X) is still equal to Prob_t+1(y|X). Since changes in Prob(X) have no effect on categorization decisions, this kind of drift is known as virtual drift [23].

• Download figure
• Open in new tab

Figure 3(a).

Visual representations of virtual and real drift derived from the initial distributions.

When concept drift influences the decision boundary, it’s termed as real drift. In the initial form of real drift, the alteration occurs in the subsequent distribution Prob(y│X), while the marginal distribution Prob(X) remains consistent. This indicates that Prob_t (X) remains equal to Prob_t+1(X), while Prob_t(y│X) diverges from Prob_t+1(y|X). Such a form of drift, influencing the decision boundary, can result in a decline in the accuracy of the learning process.

Another type of real drift is a hybrid of the prior two, including changes in both the marginal distribution Prob(X) and the subsequent distribution Prob(y│X). In this case, Prob_t(X) varies from Prob_t+1(X), but Prob_t(y│X) does not align with Prob_t+1 (y|X).

Moreover, concept drift can be classified into four distinct categories, namely: (i) sudden/abrupt drift, (ii) gradual drift, (iii) incremental drift, and (iv) recurring drift. Figure 3(b) illustrates the various categories of concept drift.

• Download figure
• Open in new tab

Figure 3(b).

Types of concept drift [21].

Sudden / Abrupt drift happens when an event occurs inside a very small window size, causing an immediate fall in learned performance. Gradual drift can occur over a wide range of window sizes and is characterized by frequent shifts between two notions. Incremental drift is characterized by a linear shift that results in a gradual drop in learned performance. When an old data notion emerges after a certain period of time, this is referred to as recurring drift.

4.2 Concept Drift Detection Techniques

Concept drift detection is the term for the methods and procedures used to detect significant changes and measure concept drift. Evaluation of IoMT streaming frequently confronts concept drift issues due to the non-stationary IoMT environments, where the data distributions evolve over time. The existence of concept drift problems frequently decreases the performance and effectiveness of IoMT models and leads to serious security problems [24]. In general, techniques for identifying concept drift generally include four important steps. The initial stage involves data retrieval, encompassing diverse techniques for sampling data from the flowing data stream. Subsequently, the data modelling stage involves creating a representation of both past and current data distributions. Following that, the third step focuses on comparing and quantifying differences between historical and contemporary data distributions. Lastly, the fourth stage comprises performing a hypothesis test to assess whether the measured dissimilarity exceeds a preset level of significance as presented in Figure 4. Drift detection techniques are broadly categorized into two primary groups: distribution-oriented methods and performance-oriented methods.

• Download figure
• Open in new tab

Figure 4.

A general framework for concept drift detection.

4.3 Concept Drift Adaptation Techniques

After drift detection, it is important to efficiently manage the detected drift so that the learning model is able to adapt to the newly discovered samples. As a solution to the drift adaptation challenge, three different methods can be used: adaptive algorithms, incremental learning, and ensemble learning.

5.1 Overview of the system

The architecture of proposed framework for detecting and adapting the concept drift in IoMT is shown in Figure 5. It has four main phases: dynamically preprocessing the IoMT data, adaptive feature selection using drift-based techniques, learning base models, and developing an online ensemble model. In the initial stage, data preprocessing techniques such as data balancing and normalization are employed to enhance the quality of the incoming IoMT data streams. Based on specific performance and efficiency requirements, an adaptive under-sampling or oversampling technique is automatically chosen. In addition, incoming IoMT data streams are automatically preprocessed by utilizing either dynamic min-max or z-score scaling approaches for data normalization.

• Download figure
• Open in new tab

Figure 5.

Architecture of Proposed Framework.

The second phase encompasses two pivotal steps: online base model learning and adaptive model selection (AMS). During the online base model learning stage, six base online learners are trained using preprocessed data streams. Specifically, for this study, we opted for two Adaptive Random Forest models, each incorporating a distinct drift detection mechanism: Adaptive Window (ADWIN) and Early Drift Detection Methods (EDDM). Designated as leader models, these two were selected. Additionally, we chose the top-performing models from four widely utilized online learning methods: extremely fast decision tree (EFDT), leverage bagging (LB), streaming random patches (SRP), and online passive aggressive (OPA). In the presence of concept drift, the adaptive model selection (AMS) module dynamically adjusts by reassessing and selecting appropriate base models that align with the updated data distributions. Subsequently, within a defined timeframe, these chosen learning models undergo updates with the incorporation of new concept instances. In the final stage, the proposed weighted method is employed to combine the outputs of the selected base learners for the online model ensemble, taking into consideration their prediction probabilities and real-time error rates.

5.2 Adaptive Data Preprocessing in the Context of Internet of Medical Things (IoMT)

Data preprocessing primarily seeks to improve the overall quality of data streams, with the ultimate goal of improving model learning performance. IoMT data streams present two possible issues related to data quality: imbalances in class distribution and variations in feature ranges. In order to resolve these issues, data sampling and normalization approaches can be employed. In dynamic IoMT systems, the distributions of classes and the ranges of features undergo continual changes, resulting in dynamic variables that exhibit significant fluctuations over time. Therefore, online data stream analysis should incorporate adaptable data preprocessing, as opposed to the static pipelines used in traditional machine learning.

5.3 Drift oriented Adaptive Feature Selection (DA-FS) method

The main objective of feature selection is to provide revised information with the most effective input characteristics to improve learning performance since the initial attributes are frequently not the most suitable ones for training an effective machine learning model. Additionally, the process of feature selection can also enhance the effectiveness of model learning by removing characteristics that are irrelevant or noisy. The approach introduced in this study, known as Drift-oriented Adaptive Feature Selection (DA-FS), employs a combination of two feature selection methods: variance threshold and select-k-best. The Variance Threshold acts as a feature selector, removing low-variance characteristics from the dataset that have no significant impact on the modelling process [53]. The sample’s variance is determined employing the following formula given a data sample with n observations; Here, n represents the count of analyzed samples, a_i stands for an input attribute, and indicates the average value of a_i. The variance threshold approach can be employed to help eliminate characteristics whose variance is less than a certain threshold. A small variance suggests that the associated attribute lacks informative value, as it maintains identical values across most data instances. The efficiency of model learning can be improved by excluding certain low-variance features. This approach can be applied to unsupervised learning models since it allows attribute selection without predetermined labels.

The select-k-best method is a well-known method for feature selection. In this process, the significance of each input attribute is determined by computing the correlation between that attribute and the target variable and selecting the top k attributes with the highest importance scores. The Pearson correlation coefficient can be used to derive feature importance scores, which are used to calculate the relevance of features. This frequently used measure, which quantifies the correlations between two variables, is expressed as follows: In this equation, r symbolizes the Pearson Correlation Coefficient. The variables, a_i represent input attributes, b_i stands for the target variable, and as well as denote the mean values of the input attribute and target.

The value of ‘r’ can vary from ‘-1’ to ‘+1’. A value of ‘0’ denotes the absence of connection between the two variables. Indicating that a rise in one variable causes an increase in the other, a number greater than 0 denotes a positive correlation between the variables. In contrast, a value below ‘0’ denotes a negative correlation, implying that a rise in one measure corresponds to a drop in the other. Hence, this metric may quantify the significance of the association among each characteristic and the target variable, facilitating the evaluation and comparison of the significance of various attributes.

Algorithm 1:

Drift oriented Adaptive Feature Selection (DA-FS)

• Download figure
• Open in new tab

Algorithm 1 provides a detailed overview of the key steps in the DA-FS process. In the initial phase, the algorithm uses the variance threshold VF₁, and the select-k-best method KF₂, to extract the primary best possible attribute set UF’from the training dataset (Set of train). Subsequently, this initial UF’is employed to adaptively update the attribute list for each arriving data sample ai originating from the online test dataset (Set of test). Incorporating both ADWIN and EDDM drift detectors, an adaptive feature re-selection process is employed. When a drift is identified by ADWIN and EDDM, the variance threshold and select-k-best feature selection approaches are triggered. Subsequently, they undergo a process of relearning recent data samples within a defined time interval RS’, considering these as the novel concept samples. This relearning process results in the generation of an updated optimal feature set, denoted as UF’. To reduce the frequency of feature re-selection, the feature set update occurs only when both drift detectors signal the presence of concept drift. During the ongoing process of analyzing data streams in real-time, this method of selecting features in response to detecting concept drift is systematically executed whenever a concept drift event is detected. The underlying principle of DA-FS’s drift adaptation functionality is grounded in the idea that as concept drift occurs and data distribution shifts, the most appropriate feature set will also undergo changes.

5.4 Base Model learning and Adaptive Model Selection (AMS)

We employ adaptive and lightweight online learning methods for the purpose of learning the base model based on data streams. The initial step involves creating fundamental base learners using a limited-scale training dataset. During the construction of the base learners, two main drift detection methods (EDDM & ADWIN) are used along with state-of-the-art drift adaptation method ARF to construct a robust ensemble model [45]. In contrast to EDDM, which analyzes the standard deviation of the distance between classification errors and alerts to a potential drift when this distance rises, where ADWIN compares the standard deviation of data between two dynamically adjusted sliding frames. There are two base learners, namely ARF-ADWIN and ARF-EDDM the following are the reasons for selecting each of them:

All of the models mentioned are online ensemble models that are well-suited for adapting to concept drift in IoMT data streams. They are all developed with several HTs, which are effective incremental learning base models.

• The Adaptive Random Forest (ARF) [48] stands out as a proficient approach to online learning. This approach, which is based on Hoeffding Trees (HT) [41], offers a substantial benefit since it can effectively and reliably analyse data streams in a variety of circumstances.

• The Adaptive Random Forest demonstrates that it can effectively manage changes in data streams while maintaining reliable prediction ability. This is accomplished by combining an adaptive mechanism with several Hoeffding Trees to maximize their potential. Our framework employs ARF-ADWIN and ARF-EDDM as the fundamental classifiers to effortlessly incorporate the powerful features of ARF.

• While ADWIN is better at identifying gradual drifts, EDDM is more effective at spotting sudden drift. Both techniques are combined in the suggested ensemble model to efficiently find both kinds of drift.

• As one of the major benefits of the selected models, ARF-ADWIN and ARF-EDDM is their capability to provide robust performance with shorter execution times.

• In order to increase the potential for learning performance improvements, an effective ensemble model has to ensure a wide range of unique foundational learners. On the other hand, an ensemble with little variation frequently exhibits performance closely resembles that of a particular foundational learner.

5.5 Online Ensemble Framework for Drift Adaptation: AEF-CDA

The study introduces an Adaptive Ensemble Framework for Concept Drift Adaptation (AEF-CDA) for integrating foundational learners into IoT data stream analytics. In contrast to the fixed or predefined weights frequently employed in numerous existing ensemble techniques, AEF-CDA dynamically assigns weights to the foundational learners, modifying them in real-time in accordance with their performance. The target class predicted using can be expressed by the following mathematical equation when a data stream D = { (a₁, b₁),……(a_n, b_n) }, is taken into consideration, and the target variable has c distinct classes, b ∈ 1,. ….,c, for each input a. According to the suggested framework, L_j stands for a foundational learner, where k denotes the total number of foundational learners, and in this case, k equals 4. Additionally, p_j (y_j = i | L_j, x) signifies the prediction probability of class value i for data sample x when utilizing L_j as the jth foundational learner. Lastly, w_j represents the weight assigned to each base learner L_j. The real-time rate of error is determined after the data samples have been analyzed by splitting the total number of analyzed samples by the number of samples that were erroneously classified. The weight assigned to each foundational learner, abbreviated as w_j is determined by finding the reciprocal of its associated real-time rate of error, which is denoted as: where ∈ serves as a minor constant to prevent the number in the denominator from being zero. If base learner’s Er._rt →0 for a, it is given a very high weight of , as it is already capable of making excellent predictions.

The weighting function in consideration can be thought of as an enhanced variant of the weighting function employed in the AUE approach. To enhance the precision of drift adaptation, the AEF-CDA model, as proposed, opts for the utilization of real-time error rates for computing base learner weights. Typically, the complexity of base learners plays a pivotal role in determining the time complexity of ensemble models. Given the relatively small values of n (the number of c class values in the target variable) and k (the number of base learners), AEF-CDA boasts a time complexity of merely O(nck).

In order to achieve effective drift detection and adaptation, the proposed AEF-CDA framework incorporates two foundational online learners including: ARF-ADWIN and ARF-EDDM. These two models have been designated as the leader models. Furthermore, we chose the two best performing models from four extensively used online learning methods: extremely fast decision tree (EFDT), leverage bagging (LB), streaming random patches (SRP), and online passive aggressive (OPA). ARF-ADWIN and ARF-EDDM are also employed to build the foundational learners in the suggested framework since they are the most advanced, high-performing drift adaptation techniques. Moreover, to obtain an optimized ensemble model, Particle Swarm Optimization (PSO) is used for hyperparameter tunning [54]. In the context of HPO problems, particle swarm optimization (PSO) is the most widely used metaheuristic algorithm. PSO algorithms use interaction among particles to identify and upgrade the existing global optimal point in each iteration until the ultimate optimum is identified [55].

The suggested AEF-CDA approach offers the following benefits over previous ensemble methods:

• When there is a disparity in class, the AEF-CDA strategy performs exceptionally well. In such challenging-to-forecast states, the model might prioritize minority classes and perform better if it weights predictions according to their confidence probability.

• This proposed AEF-CDA uses a reciprocal weighting function, which surpasses existing ensemble methods including AUE.

• The proposed framework relies on the confidence probabilities provided by all base classifiers for each class, which is deemed more reliable and flexible adaptable compared to numerous other ensemble learning methods that employ strict majority voting. To prevent making less effective decisions, this framework takes into account the uncertainty associated with each base classifier’s prediction for each data sample.

• It is possible to maintain high predictive performance while reducing computational challenges using the ensemble model. Additionally, the ensemble model can be used to process large amounts of data quickly, making it ideal for applications like internet of medical things (IoMT) and real-time analytics.

• ARF can be changed with other, higher-performing drift adaptation techniques if they are developed in the future in order to build a more effective ensemble model utilizing the same AEF-CDA approach.

6.1 Experimental Setup

The execution of this framework utilized the Python 3.8 programming language and was carried out in Google Colaboratory. The implementation involved extending the Scikit-Multiflow framework [56] on a machine equipped with an i5-8350U processor and 8 GB of memory, serving as the central server machine for IoMT. Table 3 provides the insights of dataset employed for experiments.

The suggested approach is evaluated using three datasets. The first dataset, known as WUSTL EHMS 2020 Dataset for (IoMT) Cybersecurity Research [57]. The EHMS testbed is segmented into four components: medical sensors, gateway, network, and control with visualization. The flow of data initiates from the sensors affixed to the patient’s body, transmitting to the gateway. Subsequently, the gateway forwards the data to the server for visualization, facilitated by the switch and router. A potential threat involves the interception of this data before it reaches the server. To counteract such threats, the Intrusion Detection System (IDS) takes charge of capturing real-time network flow traffic, as well as detecting abnormalities in the patient’s biometric data. The dataset comprises of 44 features, encompassing 35 network flow metrics, eight biometric features related to patients, and one feature designated for labeling purposes.

IoTID20 dataset [58], was produced utilizing both legitimate and malicious IoT devices to gather data analysis on IoT network traffic. The analysis of flow-based and packet data yielded 83 network properties, which were then used to identify and detect cyberattacks. These elements include a wide range of information, including active and idle times, the length of flows, and the total number of packets delivered and received in both forward and backward directions.

CICIDS2017, provided by the Canadian Institute of Cybersecurity (CIC), is the name of the second dataset used in the study [59]. It contains the most recent cyberattack scenarios that were accessible at the time. The attack patterns contained within the dataset evolved over time as a result of the launch of numerous attacks throughout various time periods to create the CICIDS2017 dataset. In the CICIDS2017 dataset, this led to several instances of concept drift.

To assess the model, a representative subset of the IoTID20 dataset, comprising 6,252 records, and a representative sample from the CICIDS2017 dataset, comprising 28,303 records, were chosen through the utilization of the k-means clustering sampling technique. Both the datasets, IoTID20 and the CICIDS2017 datasets show a significant class imbalance. IoTID20 has a normal-to-abnormal instance ratio of 94% to 6%, but CICIDS2017 has an abnormal-to-normal instance ratio of 80% to 20%. This feature makes it possible to assess how well the model performs on imbalanced datasets.

6.2 Validation

In this study, we treated the datasets as binary datasets with two labels: “normal” and “abnormal” since the primary objective of concept drift detection and anomaly detection systems is to classify between normal states and cyber-attacks. Utilizing hold-out and prequential validations, the proposed framework is assessed. As a component of the holdout evaluation, the initial model training is carried out utilizing the initial 10% of the data, with the subsequent 90% of the data reserved for online testing [60].

Within the process of prequential validation, alternatively known as test-and-train validation, each input sample from the online test set serves dual purposes. Initially, it functions to test the learning model, and subsequently, it contributes to the training or updating of the model. Given the imbalanced nature of both datasets utilized in this study, the performance assessment of the proposed concept drift detection and anomaly detection framework incorporates five metrics: accuracy, precision, recall, f1-score, and execution time. These metrics serve as evaluative measures to estimate the framework’s effectiveness in identifying concept drift and anomalies within the datasets.

6.3 Results and Discussion

The efficacy and superiority of the AEF-CDA method in addressing concept drift within the realm of online learning are emphasized through the presented figures and tables. Specifically, Table 4 provide a comprehensive comparative analysis between the proposed AEF-CDA technique and other state-of-the-art online learning approaches. In the experimentation phase using the WUSTL EHMS-IoMT dataset, six instances of concept drift were identified, as illustrated in Figure 6.

• Download figure
• Open in new tab

Figure 6.

Proposed framework on WUSTL EHMS-IoMT dataset.

The results demonstrate that the AEF-CDA method, as introduced in this study, outperformed all other evaluated online learning models. It achieved the highest accuracy, reaching 99.64%, precision at 99.39%, recall at 99.96%, and an F1-score at 98.57% as shown in Figure 7. These findings underscore the superior performance of the AEF-CDA technique in effectively detecting and adapting to concept drift in online learning scenarios.

• Download figure
• Open in new tab

Figure 7.

Comparative Analysis of proposed AEF-CDA model performance on WUSTL EHMS-IoMT dataset.

Based on the findings presented in Table 5, the proposed AEF-CDA approach showcases superior performance compared to all other models considered, with notable advantages in accuracy, precision, recall, and F1-Score. In the IoTID20 dataset experiment, two minor drifts were observed at the experiment’s onset, as depicted in Figure 8. All applied methods demonstrated the ability to promptly adapt and respond effectively, albeit with varying degrees of adaptability.

• Download figure
• Open in new tab

Figure 8.

Proposed framework on IoTID20 dataset.

Notably, the AEF-CDA framework stands out among the assessed methods, achieving the highest accuracy at 99.14%, as illustrated in Figure 9. In contrast, when comparing AEF-CDA to the two fundamental learners (ARF-ADWIN and ARF-EDDM), their accuracy ranged between 98.26% to 98.06%. These results emphasize the superior adaptability and performance of the AEF-CDA approach in handling concept drift in the context of the IoTID20 dataset experiment.

• Download figure
• Open in new tab

Figure 9.

Comparative Analysis of proposed AEF-CDA model performance on IoTID20 dataset.

In the experimentation phase on the CICIDS2017 dataset, six instances of concept drift were identified, as depicted in Figure 10. Notably, the first three drifts (drifts 1, 3, and 5) were sudden, while drifts 2, 4, and 6 exhibited a gradual transition. Drawing insights from both the IoTID20 dataset and the proposed AEF-CDA method, foundational learners ARF-ADWIN and ARF-EDDM demonstrated swift adaptability to drifts, maintaining high accuracy in the CICIDS2017 dataset.

• Download figure
• Open in new tab

Figure 10.

Proposed framework on CICIDS2017 dataset.

AEF-CDA emerges as the standout performer with a 99.12% accuracy rate, as illustrated in Figure 11 and supported by Table 6.This places AEF-CDA at the leading approach, confirming its position as the superior approach. Among the four follower models (OPA, SRP, EFDT, and LB), SRP and LB demonstrate notable proficiency. However, the accuracies of OPA, SRP, EFDT, and LB showcase a considerable range, spanning from 97.08% to 98.85%. These outcomes emphasize the effectiveness of AEF-CDA and highlight SRP and LB as particularly adept follower models in managing concept drift in the CICIDS2017 dataset experiment.

• Download figure
• Open in new tab

Figure 11.

Comparative Analysis of proposed AEF-CDA model performance on CICIDS2017 dataset.

The suggested AEF-CDA approach uses min-max sampling to maintain high accuracy on a sampled subset despite the rapid generation of IoT data streams. With this approach, real-time data analytics are made easier, and the model learning speed is increased. Additionally, the SRP drift adaptation model can be replaced with another drift adaptation model with less complexity to reduce the average execution time of the proposed AEF-CDA approach. But when compared to other cutting-edge drift adaptation methods for IoT anomaly detection, the AEF-CDA approach now in use can obtain the best accuracy and F1-score.