Integration of pharmacists’ knowledge into a predictive model for teicoplanin dose planning

2.1 Patients’ characteristics

We enrolled patients who received teicoplanin between August 2019 and April 2022 at Nagoya University Hospital. During the study period, 1165 patients received teicoplanin treatment. Of these, 751 patients were excluded because of the following reasons: age <18 years (n = 546), undergoing peritoneal dialysis or hemodialysis (n = 109), receiving TDM pharmacists’ intervention after the initial dose (n = 33), resuming teicoplanin treatment within 7 days (n = 31), on extracorporeal membrane oxygenation (ECMO) (n = 16), missing data (n = 9), patient immobility (n = 6), or receiving a single dose for surgical prophylaxis (n = 1). The remaining 414 patients were divided into two groups based on whether they received TDM pharmacist intervention (intervention group) or not (nonintervention group). Consequently, 158 and 256 patients were assigned to the intervention and nonintervention groups, respectively (Figure 1 and Tables S1 and S2).

• Download figure
• Open in new tab

Figure 1. Flow of case selection in the study.

2.2 Evaluation of therapeutic drug monitoring pharmacists’ dose planning

We first evaluated the clinical significance of TDM pharmacists’ intervention in teicoplanin treatment. Eighteen of the 158 patients in the intervention group and 37 of the 256 patients in the nonintervention group were excluded from the analysis because of the lack of concentration measurement or description of the sampling time. For the remaining 140 patients in the intervention group, we further excluded the patients with deviation from the TDM pharmacists’ regimen (n = 17), changing regimen after the initial dose (n = 1), and blood sampling after the completion of therapy (n = 1).

For the remaining 121 and 219 patients in the intervention and nonintervention groups, respectively, we assessed whether blood samples were collected at appropriate time points, that is, 18 h after the last dose (2). The incidence rates of inappropriate blood sampling were 5.0% (6/121) and 16.8% (36/219) in the intervention and nonintervention groups, respectively, indicating the contribution of TDM pharmacists’ intervention to proper trough blood sampling (Figure 1 and Table 1). Patients with inappropriate blood sampling in the intervention group (n = 6) were due to the lack of TDM pharmacists’ recommendation on the timing for sampling.

Next, we excluded these patients with inappropriate blood sampling and then analyzed whether TDM pharmacists’ dose planning contributes to target attainment (15–30 mg/L). As shown in Table 2, TDM pharmacist intervention clearly increased the target attainment rate at initial TDM (74.8% [86/115] in the intervention group vs. 57.9% [106/183] in the nonintervention group, p = 0.004), accompanied by a decrease in subtherapeutic ranges (14.8% [17/115] in the intervention group vs. 37.2% [68/183] in the nonintervention group, Table 2 and Tables S3 and S4). However, we observed a large bias in intensive care unit (ICU) and high care unit (HCU) admissions between the two groups (24.3% [28/115] in the intervention group vs. 55.7% [102/183] in the nonintervention group, p <1.0 × 10⁻⁶, Table S5). Thus, we performed subgroup analyses and found no significant increase in target attainment in patients with an ICU/HCU stay (60.7% [17/28] in the intervention group vs. 57.8% [59/102] in the nonintervention group, p = 0.832, Table 2). This reflects the difficulty in predicting the pharmacokinetics of ICU/HCU stay, where frequent and dramatic changes in the patients’ clinical status are often observed. For patients without an ICU/HCU stay, we found a marked increase in the target attainment rate in the intervention group (79.3% [69/87] in the intervention group vs. 58.0% [47/81] in the nonintervention group, p = 0.004, Table 2). Because there were systemic differences in baseline characteristics between the two groups, especially creatinine clearance (CL_CR) (p = 0.009, Figure S1 and Table 3), we applied propensity score matching to reduce the effects of confounding. The results showed that the increase was also found and remained significant after propensity score matching (83.3% [40/48] in the intervention group vs. 60.4% [29/48] in the nonintervention group, p = 0.013, Table 4 and Table S6).

Overall, these results validate the TDM pharmacists’ dose planning, at least for patients without ICU/HCU stay, and underscore the importance of experts’ (e.g., TDM pharmacists) intervention for effective and appropriate initial treatment with teicoplanin.

2.3 Machine learning (ML) model to determine teicoplanin initial dose

Given the increased target achievement by TDM pharmacists’ dose planning of teicoplanin for patients without an ICU/HCU stay, we next sought to build an ML model that tailored dosing regimens at the levels of TDM pharmacists. Toward this end, we trained a two-layer neural network with the dataset of TDM pharmacists’ dose planning (n = 118, Figure S2 and Table S7). Input variables were selected based on the considered information while dose planning: the covariates for teicoplanin pharmacokinetics (age, BW, BMI, CL_CR, and serum albumin level) and the timing of dose planning (day of the week [T1–T7] and time of the day [T0]). The latter comes from our practice of adjusting the dosing regimen, such that the initial TDM is performed on weekdays.

We divided eligible patients into a training group and a testing group at a 94:24 ratio (approximately 80:20). Subsequently, the neural network model to predict TDM pharmacists’ dose planning was trained on the patients in the training group. The prediction accuracies for the loading and maintenance doses (i.e., identical dosing as TDM pharmacists) were both 100% on the training dataset, indicating that the model learned how TDM pharmacists decide dosing regimens. However, in the testing trial, the model scored significantly lower prediction accuracies of 54.2% and 62.5% for the loading and maintenance doses, respectively (Figure 2A, B, left and Table S9). These results indicate that overfitting occurs, which is most likely attributable to the small datasets (17). Because the current dataset consisted of dose planning by multiple pharmacists, each with varying levels of expertise, possible heterogeneity in dose planning among pharmacists may also have contributed to the decrease in accuracy. Next, we retrospectively analyzed the target attainment by ML dose planning using the Bayesian method (2). The original target attainment rate in the testing group was 81.0% (17/21). Importantly, the rate was expected to slightly increase when ML was applied (95.3% [20/21], Figure 2C, left and Table 5). We hypothesized that ML complemented the weaknesses of suboptimal TDM pharmacists by learning policies from other pharmacists. Taken together, these results indicate that although dose planning tends to differ from that of TDM pharmacists, ML is competent for tailoring teicoplanin dosing as TDM pharmacists.

• Download figure
• Open in new tab

Figure 2.

(A) Learning curves of the neural network through training and testing using dataset from the intervention group. (B) Cumulative dose of teicoplanin before initial TDM in TDM pharmacists and ML regimens. (Left) TDM pharmacists vs. ML trained with cases with intervention (TDM pharmacists ML). (Right) TDM pharmacists vs. ML trained with cases without intervention (non-TDM pharmacists ML). Line indicates reference line of y = x. (C) Serum concentration at initial blood sampling with TDM pharmacists and ML regimens. (Left) TDM pharmacists vs. ML trained with cases with intervention (TDM pharmacists ML). (Right) TDM pharmacists vs. ML trained with cases without intervention (non-TDM pharmacists ML). Gray area indicates therapeutic windows (15–30 mg/L).

As a control, we also trained the ML using dose planning by non-TDM pharmacists (i.e., the cases without intervention, Table S8) and then analyzed the target attainment by this ML. As expected from the poor target achievement (Table 2), ML trained on cases without intervention failed to maintain the target attainment of TDM pharmacists. The target attainment was estimated to decrease from 81.0% (17/21) to 52.4% (11/21) (Figure 2C, right, Table 5, and Tables S9 and S10). This decrease was attributed to the subtherapeutic doses of teicoplanin; the incidence of subtherapeutic exposure was estimated to increase from 4.8% (1/21) to 47.6% (10/21) (Figure 2B, right, Table 5, and Tables S9 and S10). This result mirrors the propensity of non-TDM pharmacists to underdose (Tables 2 and 4). Taken together, these results suggest that ML model training with the TDM pharmacists’ dataset augments model performance in tailoring the teicoplanin dose.

2.4 Evaluation of the ML on the external validation dataset

Finally, we investigated whether ML improved target trough attainment. Toward this end, we retrained the ML with whole dataset of TDM pharmacists’ dose planning (n = 118, Table S7) to improve the prediction. Subsequently, we applied retrained ML to patients in the nonintervention group without an ICU/HCU stay who underwent TDM (n = 81, Figure 1 and Table S8). For both loading and maintenance doses, the retrained ML scored 100% accuracy on the training dataset (Figure S3). We also calculated feature importance, which is proportional to the contribution of the features to the ML decision. For the loading dose, BW was the most important feature for the ML output (i.e., dose determination). For the maintenance dose, BW, BMI, and CL_CR represented major contributions to the ML output. For both loading and maintenance doses, the timing of dose planning (T0–T7) also contributed to the ML decision, which is reminiscent of our practice of adjusting dose planning to perform TDM on weekdays (Figure 3A). The small roles of T6 and T7 (dose planning on Saturday and Sunday, respectively) in dose planning were likely due to the lack of data (both n = 1) to optimize the weight parameters (Table S7).

• Download figure
• Open in new tab

Figure 3.

(A) Permutation feature importance for loading dose (top) and maintenance dose (bottom) decision. (B) Serum concentration at initial blood sampling with non-TDM pharmacists and ML regimens. Blue line and blue dashed lines indicate regression line and 95% confidence interval, respectively. (C) Cumulative dose of teicoplanin before initial TDM with non-TDM pharmacists and ML regimens. Line indicates reference line of y = x.

The target attainment rate in the nonintervention group was 58.0% (47/81) (Table 2). Notably, if ML dosing regimens were applied, target attainment rates increased from 58.0% to 72.8% (59/81, p = 0.047, Figure 3B and Table 6). This increase was mainly attributed to the enhanced dose regimen, which prevented subtherapeutic drug exposure; when applying the ML, subtherapeutic levels at initial TDM were expected to decrease from 37.0% (30/81) to 22.2% (18/81) (Figure 3C and Table 6). Meanwhile, notably, enhanced dose planning by ML was accompanied by an incidence of overexposure (e.g., case no. 182, Tables S11 and S12). Therefore, ML may increase the risk of adverse effects of teicoplanin, such as thrombocytopenia, which is observed when trough concentrations exceed 40 mg/L (2, 18, 19).

Altogether, similar to the dose planning by the TDM pharmacists, the current ML is expected to play a role in tailoring dose planning, which contributes to early therapeutic exposure and consequently leads to treatment success. The model shown in Figure 3 is freely available at https://github.com/Matsuzaki-T/TEIC_study.git.

4.1 Study participants

This was a single-center, retrospective, observational study of hospitalized patients who received intravenous teicoplanin between August 2019 and April 2022 at Nagoya University Hospital. TDM pharmacists were defined as pharmacists dedicated to TDM. Patients who commenced teicoplanin treatment with TDM expert-recommended dose regimens during the study period were also included. The exclusion criteria were as follows: patients aged <18 years; undergoing peritoneal dialysis or hemodialysis (including continuous hemodiafiltration); undergoing ECMO; receiving intervention by experts after the initial dose; restarting teicoplanin treatment within 7 days; who were immobile, which indicates decreased muscle mass and may cause overestimation of CL_CR; with surgical antibiotic prophylaxis; and with missing data on sex, age, BW, BMI, and serum albumin and creatinine levels.

4.2 Comparison of appropriate blood samplings and trough concentrations between the intervention and nonintervention groups

Eligible patients with serum teicoplanin concentrations were enrolled to evaluate the dose planning by TDM pharmacists. In this analysis, the patients were divided according to whether they received dose planning by TDM pharmacists.

Patients with intervention were excluded if the time of blood sampling was not described, protocol deviation occurred, dosing regimen was changed by TDM pharmacists after the initial dose, or blood samples were collected after the completion of teicoplanin treatment. Because blood samples for monitoring teicoplanin trough concentrations should be collected at least 18 h after the last dose (2), we monitored the timing of blood sampling. In cases where initial TDM was performed at the recommended time (18 h after the last dose), the target trough achievement (15–30 mg/L) in the intervention group was compared with that in the nonintervention group.

Propensity scores were calculated using logistic modeling, with TDM pharmacists’ dose planning as the dependent variable. Independent variables included age, BW, BMI, serum albumin level, CL_CR calculated using the Cockcroft–Gault equation (28), and timing of dose planning, which were used as input variables in ML construction. The patients were matched 1:1 using the nearest-neighbor technique, with a caliper distance limited to 10% of the standard deviation of the pooled propensity scores.

4.3 Building of the neural network model

The dataset used in this study included clinical and routine laboratory data, initial dosing regimens, and serum teicoplanin concentration at initial TDM (if measured). Age, BW, BMI, serum albumin level, and CL_CR were used as features to predict the initial dosing regimens (loading and maintenance doses). We also used the timing of the dose planning (day of the week [T1–T7] and times of day [T0]) as input variables because TDM pharmacists consider dosing regimens so that initial TDM is performed on weekdays as possible. The dataset (n = 118) was divided into training and test datasets in a 94:24 ratio (approximately 80:20). Numeric input variables (age, BW, BMI, serum albumin level, CL_CR, and time of day [T0]) in the training data were normalized using the following equation:

where x_k,i is a value of parameter k in sample i, x_k,i,norm is a normalized value of parameter k in sample i, µ_k is a mean of parameter k, and σ_k is a standard deviation of parameter k.

We applied the same scaling to the input variables in the testing data as in the training data to avoid shifting the distribution of the data. For ICU/HCU stay, the variable takes 1 for ICU/HCU stay and 0 otherwise. On each day of the week, the variables were binarized using one-hot encoding, where T1, T2, T3, T4, T5, T6, and T7 represented Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, and Sunday, respectively. Output variables (loading and maintenance doses) were binarized using one-hot encoding.

We built a two-layer neural network model for dose planning, as described elsewhere, using Python (29, 30). The structures of the network for the loading and maintenance doses were the same. Briefly, the network was composed of an input layer, a hidden layer, and an output layer (Figure S2). The activation functions in the hidden and output layers were both sigmoid. The number of hidden neurons was set to 15, which was determined by the sum of 2/3 of the size of the input layer neurons (n = 13) and output layer neurons for the maintenance dose (n = 7), as previously proposed (31). Because adding the number of output layer neurons to the loading dose (n = 14) did not result in increased predictive accuracy (data not shown), we set the same number of hidden neurons in both ML.

We trained the neural network to minimize empirical loss over the training data. In this study, we employed cross-entropy loss as the loss function L, which was parameterized by the weight matrices W1, W2, b1, and b2 (Figure S2):

where X is a matrix of the input variables from each training dataset x, N indicates the number of training data points, and f(x) indicates the score for the correct class.

We optimized each parameter in the model using stochastic gradient descent with an adaptive learning rate using AdaGrad (32):

where θ is a parameter of weight matrix and η is a learning rate (in this study, 0.1).

4.4 Feature importance analysis

In the training process, the permutation feature importance (Breiman–Cutler importance) was calculated, as described in previous studies (33, 34). In this process, a single feature value was randomly shuffled while keeping the other input variables constant. Subsequently, decreases in the prediction accuracy, which indicates how the feature contributes to the M decision, were collected for each input variable. We repeated this process 20 times to measure the mean decrease in the prediction accuracy for each feature:

where PI_i(permutation importance) indicates the permutation importance of feature i, N is the number of repetitions (in this study, 20), PA_original indicates the original prediction accuracy, and PA_i,j indicates the prediction accuracy upon shuffling feature i.

4.5 Estimation of trough concentration with the regimen using the ML model

Data, including serum teicoplanin concentrations at appropriate time points (18 h after the last dose), were included in this analysis. If the ML dosing regimen by the ML was identical to the original dosing regimen, the measured serum teicoplanin concentration was defined as the trough concentration with the ML dosing regimen. Otherwise, the serum teicoplanin concentration in the ML regimen was estimated using Bayesian estimation under the following assumptions:

• (i) If the cumulative number of doses was the same in the original and ML regimens, only the dose was changed.

• (ii) If the ML recommended an additional loading dose, additional loading doses were added 8 h after the last dose and every 8 h thereafter.

• (iii) If the ML did not recommend a maintenance dose, blood sampling was performed 18 h after the last dose.

• (iv) If the ML recommended maintenance dose, every 24 h maintenance dose was assumed to be administered at 9:00 a.m. (at least 8 h after the last dose), followed by TDM sampling at 9:00 a.m. the next day.

4.6 Statistical analyses

We used the Mann–Whitney U test to evaluate continuous data. For categorical data, we used Fisher’s exact test or the chi-squared test. All statistical tests were two-tailed, and p values < 0.05 were considered statistically significant. Statistical analyses were performed using the Python Statistics module.