External Validation, Re-Calibration, and Extension of a Prediction Model of Early Acute Kidney Injury in Critically Ill Children using Multi-Center Data

Background Acute kidney injury (AKI) is common among children with critical illness and is associated with high morbidity and mortality. Risk prediction models designed for clinical decision support implementation offer an opportunity to identify and proactively mitigate AKI risks. Existing models have been primarily validated on single-center data, owing partly to the lack of appropriately detailed multicenter datasets.

Objective To determine the accuracy of a single-center model to predict new AKI at 72 hours of ICU admission across two multicenter datasets and extend this model to improve prediction accuracy while maintaining acceptable alert burden.

Derivation and Validation Cohorts We separately derived models in two datasets: PEDSNET-VPS, created through the linkage of PEDSnet electronic health record (EHR) extraction with Virtual Pediatric Systems (VPS); and the PICU Data Collaborative dataset, created through EHR extraction and harmonization from eight participating institutions. Derivation datasets comprised temporal and location-specific spit of these datasets (80%), while the holdout test split comprised the remaining (20%).

Prediction Model We recalibrated an existing single-center model and measured discrimination and accuracy. We then add features guided by precision and recall measures. All features were available at 12 hours of ICU admission. We measure discrimination and accuracy at multiple cut-points and identify the features contributing most to the risk score.

Results In two datasets comprising 186,540 ICU admissions, we report an incidence of early AKI of 2.2 – 2.7%. Initial recalibration of an existing single-center model demonstrated poor discrimination (AUROC 0.60 – 0.78). Following the addition of new features, we report higher AUROC values of 0.79 - 0.80 and AUPRC values of 0.13 – 0.21 in both datasets. We report accuracy at several cutpoints as well as cross-validate between datasets.

Conclusions In this first use of two new multicenter datasets, we report improved discrimination and accuracy in a model designed specifically for implementation, balancing sensitivity and precision to predict patients at risk for AKI development.