Early prediction of impending septic shock in children using age-adjusted Sepsis-3 criteria
===========================================================================================

* Ran Liu
* Joseph L. Greenstein
* James C. Fackler
* Jules Bergmann
* Melania M. Bembea
* Raimond L. Winslow

## Abstract

Sepsis is a syndrome which afflicts both adults and children, with many disease courses and diverse outcomes. Understanding of sepsis pathophysiology has changed over time; the Sepsis-3 criteria define sepsis in adults as organ dysfunction, quantified by SOFA score, caused by dysregulated immune response to infection. However, pediatric consensus definitions still utilize the SIRS-based Sepsis-2 criteria, though individual groups have attempted to adapt the Sepsis-3 criteria for children. We evaluate age-adjusted Sepsis-3 criteria on 2,384 pediatric patients admitted to the Johns Hopkins PICU, and apply previously-published methods for early prediction of septic shock. We obtain best early prediction performance of 0.96 AUC, 49.9% overall PPV, and a 5.8-hour median EWT using Sepsis-3 labels based on age-adjusted SOFA score. Through analyses of risk score evolution over time, we corroborate our past finding of an abrupt transition preceding onset of septic shock in children, and are able to stratify pediatric sepsis patients using their first post-threshold-crossing risk score.

## Introduction

Sepsis is a leading cause of in-hospital mortality for both adults and children worldwide1,2. Kumar et al. showed that every hour of delayed treatment increased mortality in adult patients by ∼8%3. Weiss et al. found this same association in pediatric sepsis patients, and that delayed treatment remains common4.

Developing effective guidelines for diagnosis and treatment of sepsis is difficult because sepsis is not a single disease. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3)5 reflect the most recent understanding of sepsis as organ dysfunction caused by dysregulated immune response to infection. However, consensus definitions for pediatric sepsis were last updated in 20056, modeled closely along the Sepsis-2 definitions for adults7. Recently, studies attempting to adapt the adult Sepsis-3 criteria for children have been conducted by individual groups8-10, and an international expert group published guidelines for the treatment of septic shock and its associated organ dysfunction11.

Several computational approaches for early prediction of sepsis and septic shock using electronic health record (EHR) data have been developed with the aim of reducing treatment delays in adult patients12-14. We developed a method for predicting an oncoming transition from sepsis to septic shock based on the hypothesis that there exists a physiologically distinct state of sepsis, which we term the “pre-shock state”, and that entry into this state presages the impending onset of septic shock13. Through analyses of the temporal evolution of patient state, we previously stratified adult sepsis patients by outcome and interventions received, and discovered that entry into the pre-shock state was marked by a rapid shift in patient physiology occurring within a 30-60 minute timeframe15.

In this study, we evaluate age-adapted Sepsis-3 criteria, and apply our previously-published method for early prediction of septic shock on a population of patients admitted to an academic, quaternary center pediatric intensive care unit. We corroborate our past finding of an abrupt transition preceding onset of septic shock in children and are able to stratify sepsis patients using their first post-threshold-crossing risk score.

## Methods

### Study Population

We conducted a retrospective observational cohort study of all patients admitted to the Johns Hopkins Pediatric Intensive Care Unit on or after July 1, 2016, and with discharge prior to January 31, 2018. Patients aged 18 and older were excluded.

### Data Extraction and Processing

Raw data for the Sepsis Quality Improvement Program (SEQUIP) dataset was sourced from an EHR data report and included patient demographics, encounter diagnosis codes, ADT (admit discharge transfer codes with patient room/bed assignments), provider-entered flowsheets (which included nurse-validated vitals and respiratory therapist-validated ventilator settings and measurements), a subset of laboratory results, medication orders, and medication administrations.

The SEQUIP dataset contains data from 2,527 distinct patients admitted to the ICU and 3,125 hospital admissions. Excluding patients aged 18 and older yields our analysis set of 2,384 patients and 2,923 hospital admissions, with an overall sepsis prevalence of 15.88%, as determined by age-adjusted Sepsis-3 criteria8,9. Data entries in SEQUIP are comprised of timestamp-value pairs, with ID numbers for patients, hospital admissions, and ICU stays, as well as an ID indicating the feature associated with each value. Complete details of entries corresponding to each physiological variable used in this study are given in the Supplement. 70% of patients were randomly sampled into the training set, and the remaining 30% reserved for testing.

### Labeling Clinical States

Suspected infection was determined using ICD-10 codes. Though Seymour et al.16 recommend determining suspected infection from orders of antibiotics and blood cultures, the availability of blood culture data is limited in this dataset, and thus we cannot use this criterion to determine suspected infection. Angus et al. specify ICD-9 codes indicative of suspected infection, used in several prior studies of sepsis2,12-14. Mapping between ICD-9 and ICD-10 codes was performed based on the code descriptions (see Supplement, *Infection Criteria*). Comorbidities (Table S2) were computed according to the criteria for the Pediatric Complex Chronic Condition Classification using ICD-10 codes17,18.

The Goldstein consensus criteria were evaluated using EHR data from SEQUIP6. According to these criteria, sepsis is defined as suspected infection and two or more age-adjusted SIRS criteria (Supplement, *Labeling Clinical States*). Septic shock is defined as sepsis with cardiovascular dysfunction. Labels were re-evaluated each time there was a new observation of clinical data; missing values were imputed using the last observation carried forward, and if no prior observations of a feature were available, the patient was assumed to be within normal ranges for that variable.

According to the Sepsis-3 criteria, sepsis is defined as organ dysfunction consequent to suspected infection. We determine organ dysfunction as a 2-point rise in age-adjusted SOFA score, as defined by Matics et al.9 by using the PELOD-2 cutoffs for mean arterial pressure (MAP) and creatinine for cardiovascular and liver SOFA, respectively, and by either a 2-point rise, or a 6-point rise in PELOD-2. Septic shock patients are those with sepsis who were adequately fluid resuscitated, were administered vasopressors, and exhibited a serum lactate concentration greater than 2 mmol/L. Adequate fluid resuscitation was determined using the 2020 SSC pediatric guidelines11, and is defined as 40 mL/kg of fluids in the past 3 hours, or having attained the fluid resuscitation target of MAP at the 5th percentile or higher for age, estimated as 1.5 x age in years + 40 mmHg19.

### Risk Modeling and Prediction

Risk models were built according to methods previously described, using 26 clinical features extracted from EHR data13. MAP and heart rate (HR) were normalized to percentile values by age19,20. In order to characterize the pre-shock state, XGBoost and GLM regression models were trained using data from the sepsis clinical state in patients who do not develop septic shock and data from a time window spanning 100 minutes prior to septic shock onset to 1 minute prior to septic shock onset in septic shock patients. Lasso regularization was used for feature selection in GLM models.

For early prediction of septic shock, each patient’s risk score is calculated for each time at which there is EHR data from the beginning of their entries until septic shock onset. Prediction occurs at the first time at which a patient’s risk score exceeds the threshold value. The optimal detection threshold is determined from the training set as the value of the threshold corresponding to the point on the ROC curve closest to the upper left-hand corner. Early warning time (EWT) is defined as the difference from this time until time of septic shock onset. Confidence intervals for performance criteria were estimated using 100 bootstrap iterations.

### Stratification of Sepsis Patients

Stratification of sepsis patients by risk score trajectories was performed as described previously15. Risk score trajectories for each patient were computed by applying the XGBoost model at each hour in the time window surrounding time of early prediction using the most recently observed values of each feature. Spectral clustering was applied to these trajectories in order to stratify patients into clusters with similar risk trajectories in the time window following time of early prediction21 (Supplement, *Spectral Clustering*).

## Results

### Baseline Statistics

We applied to the SEQUIP dataset four sets of diagnostic criteria for diagnosing sepsis and septic shock. These criteria were the 2005 Goldstein consensus criteria and three others evaluated by Schlapbach et al.8. Using these labels, we computed the prevalence and mortality rate in each cohort (Table 2). Depending on the criteria used, the prevalence of sepsis varies between 5.5% to 35%, and the prevalence of septic shock ranges from 2% to almost 20%. We find that sepsis patients labeled using age-adjusted Sepsis-3 criteria have greater mortality than those labeled using the Goldstein criteria. When using age-adjusted SOFA scores to determine clinical state labels, mortality in both sepsis cohorts (sepsis without shock and septic shock patients) is higher than in the corresponding cohort determined using the SIRS-based Goldstein criteria. When organ dysfunction is defined by an increase of 6 points or greater in PELOD-2 score, this results in the lowest prevalence of sepsis, and the highest mortality in both sepsis cohorts. The Goldstein criteria result in the highest prevalence of sepsis, but the lowest mortality in both sepsis cohorts.

View this table:
[Table 1:](http://medrxiv.org/content/early/2020/12/02/2020.11.30.20241430/T1)

Table 1: 
Baseline statistics of SEQUIP dataset. Sepsis cohorts are determined using age-adjusted SOFA score8.

View this table:
[Table 2:](http://medrxiv.org/content/early/2020/12/02/2020.11.30.20241430/T2)

Table 2: 
Comparison of diagnostic criteria: hospital admission counts and proportion of all admissions represented by each cohort in parentheses **(A)**, and mortality in each cohort **(B)**.

### Early Prediction of Septic Shock

Figure 1 shows risk score trajectories using XGBoost22 from a patient who does (Figure 1A) and a patient who does not (Figure 1B) develop septic shock.

![Figure 1:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/12/02/2020.11.30.20241430/F1.medium.gif)

[Figure 1:](http://medrxiv.org/content/early/2020/12/02/2020.11.30.20241430/F1)

Figure 1: 
Example risk trajectories for (A) a patient who developed septic shock and (B) a non-shock sepsis patient. Threshold for early prediction is indicated by the red horizontal line, and time of septic shock onset is indicated by the blue vertical line.

The performance of our method for early prediction of septic shock was evaluated on the SEQUIP dataset using two machine learning methods, GLM23 and XGBoost22. Figure 2 shows the receiver operating characteristic (ROC) curves and performance metrics, for clinical labels produced using the pediatric Sepsis-3 criteria with age-adjusted SOFA scores. We evaluated performance in early prediction using all sets of clinical criteria, and found that best performance overall was obtained with labels using age-adjusted SOFA scores (Figure S2). Of the two machine learning methods, XGBoost yields greater performance, with 0.96 AUC and 49.9% average PPV, with a shorter median EWT of 5.8 hours compared to 0.92 AUC, 41.0% PPV, and 10.5 hours median EWT when using GLM.

![Figure 2:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/12/02/2020.11.30.20241430/F2.medium.gif)

[Figure 2:](http://medrxiv.org/content/early/2020/12/02/2020.11.30.20241430/F2)

Figure 2: 
Performance of early prediction of septic shock in pediatric patients: (A) ROC curves and 90% confidence intervals, (B) area under the ROC curve (AUC), positive predictive value (PPV), sensitivity, and specificity for GLM and XGBoost where error bars indicate 90% confidence intervals, and (C) early warning time (EWT) for GLM and XGBoost where error bars indicate 90% confidence intervals.

Both XGBoost and GLM models are interpretable by examining the importance of each feature in the models. Feature importance is given in Table S8. XGBoost is a decision tree gradient boosting method. For XGBoost, gain is the average relative increase in performance after the addition of a feature to a model. Coverage is the relative frequency at which decision tree nodes which split on a particular feature are reached. Frequency is the relative frequency at which a feature appears in decision trees. For GLM, exponentiated coefficients directly explain the magnitude of the contribution of each feature to the risk score and can be interpreted as odds ratios. For example, the exponentiated coefficient of lactate is 2.14. Therefore, a patient with serum lactate concentration 1 standard deviation above the population mean is approximately twice as likely to develop septic shock than a patient with average serum lactate.

### Stratification of Sepsis Patients

As previously published13, our approach to early prediction allows for the calculation of a patient-specific PPV. This is performed based on the first value of risk score that exceeds the threshold. These values were binned into quartiles, and PPV was computed in each bin, yielding an estimate of the probability that a prediction of impending septic shock onset for a patient whose risk score falls into the specified range is a true positive (Table S9). In higher quartiles, the likelihood that predictions are true positives is greater than in lower quartiles, with PPV as large as 80%.

We repeated our analyses of risk score trajectories for stratification of sepsis patients15 on the SEQUIP dataset. Spectral clustering21 of risk score trajectories in the window surrounding early prediction yielded two clusters (Figure 3). Patient risk trajectories are largely indistinguishable prior to the time of early prediction. Risk scores increases abruptly at the time of threshold crossing for all patients, and then clusters diverge subsequent to that. Clusters stratify by the prevalence of septic shock, time to septic shock onset, and the proportion of patients who are adequately fluid resuscitated prior to time of early warning (Table S10). Differences in mortality between these clusters, however, are not statistically significant.

![Figure 3:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/12/02/2020.11.30.20241430/F3.medium.gif)

[Figure 3:](http://medrxiv.org/content/early/2020/12/02/2020.11.30.20241430/F3)

Figure 3: 
Clusters of risk score trajectories in the window surrounding time of early warning.

## Discussion

### Pediatric Sepsis Criteria

The shortcomings of SIRS-based criteria for sepsis are well-known. The presence of SIRS is not specific for infection, and in adults, over 90% of intensive care patients meet the criteria for SIRS24-26. The Sepsis-3 criteria for adults redefined sepsis as infection resulting in organ dysfunction, determined by an increase of at least 2 points in SOFA score27,28. Consequently, there has been strong interest in redefining pediatric sepsis on the basis of organ dysfunction. Shortly after the publication of the Sepsis-3 criteria, Leclerc et al. suggested the use of PELOD-2 scores in children with suspected infection10, and Matics et al. suggested the use of an age-adjusted SOFA score. Our findings support the usage of pediatric Sepsis-3 criteria based on an age-adjusted SOFA score, as these criteria achieve the greatest performance in early prediction, with an AUC of 0.96 and overall PPV of 49.9%. Furthermore, we corroborate the findings of other groups that age-adjusted SOFA and PELOD-2 have greater validity in stratifying patients by metrics of disease severity than the SIRS-based Goldstein criteria (Figure S4)8.

### Prediction Method

Just as in adults, the goal of early prediction of septic shock in pediatric patients is to provide clinicians with a time window of intervention for patients at high risk of deteriorating from sepsis to septic shock, enabling more timely treatment and possibly preventing development of shock. The premise of our method for early prediction of septic shock is the existence of a pre-shock state. In this study, we validate our previously published methodology by demonstrating the role of this pre-shock state in early prediction. We achieve high performance of 0.96 AUC, 49.9% overall PPV, and a 5.8-hour median early warning time. Model performance using XGBoost22 is higher than that in our prior study on data from adult patients, and is higher than that obtained using GLM23. XGBoost uses gradient boosting of decision trees, and thus is able to learn nonlinear associations between features and risk. This is likely the source of its improved performance compared to GLM, which is consistent with our previous results.

In our previous work, we introduced the notion of patient-specific positive predictive value, where patients were stratified into 10 bins based on their first post-threshold crossing value of risk13. Due to the much smaller number of patients in the SEQUIP dataset compared to the MIMIC-III and eICU datasets (2,650 vs 38,418 vs 139,367), and the lower prevalence of sepsis and septic shock in the pediatric population, we cannot achieve the same level of resolution in estimating patient-specific PPV, as there would be too few patients to estimate the PPV in each stratum. Therefore, we stratified positive predictions into quartiles. However, we demonstrate that our method for estimating the confidence of a positive prediction remains applicable in pediatric sepsis patients. Being able to estimate the confidence level of a prediction enables clinicians to act only on higher-confidence predictions, or take different courses of action informed with the likelihood that a patient develops septic shock.

### Stratification of Patients

Previously, using spectral clustering of patient risk score trajectories, we further elucidated the pre-shock state, finding that entry into the state was marked by a rapid transition from low to high risk. Prior to entry, sepsis patient physiology was indistinguishable between the low and high risk clusters of patients15. However, after the occurrence of this event, patient risk trajectories diverged, and stratified patients by risk of septic shock, mortality, time to septic shock onset, and treatments received. We did not find the difference in mortality between clusters to be statistically significant in pediatric patients; this is likely due to the relatively small size of our data set. The limited number of patients in our dataset also causes further difficulties in applying clustering to risk trajectories. If we cluster patients based on risk trajectories in the 12 hours following early prediction, then the space over which we are clustering is 13-dimensional; such a space is sparsely populated by the roughly 200 trajectories which exceed the prediction threshold 21. Nonetheless, this does not invalidate our finding that a very rapid transition in risk score at the time of threshold crossing occurs in pediatric as well as adult sepsis patients, and that risk score trajectories are able to stratify pediatric sepsis patients by risk of septic shock, time to shock onset, and treatments received. These findings support our postulation of the pre-shock state, and that the entry into septic shock is extremely rapid in both adult and pediatric sepsis patients. The rapid nature of the transition further indicates the necessity of automated methods for the detection and prediction of septic shock.

### Model Interpretation

It is possible to determine the relative importance of features in both XGBoost and GLM models. Both models share 4 out of their top 5 features in common: lactate, respiratory SOFA, PaO2, and BUN. Lactate is the most important feature in both models, as was true in our previous study of adult sepsis patients. These findings align with existing literature on the pathophysiology of sepsis and septic shock. Elevated serum lactate indicates reduced tissue perfusion, and predicts mortality in patients with infections29,30. Increased respiratory SOFA and abnormal PaO2 are both associated with respiratory dysfunction27. BUN levels are associated with kidney function, which is known to be affected in pediatric sepsis31.

### Limitations

We algorithmically determine sepsis and septic shock labels according to the Sepsis-3 consensus definitions, using age-adjusted SOFA score or PELOD-2 in order to determine organ dysfunction, and the Surviving Sepsis Campaign32 guidelines to determine when patients are adequately fluid resuscitated. Therefore, limitations inherent to these labels are also limitations of the study. For example, the Infectious Disease Society of America points out that determining septic shock in adults using adequate fluid resuscitation as a criterion results in some ambiguity and disagreement in interpretation as to the exact time of onset33. Deutschman further remarks that the Sepsis-3 criteria may not precisely reflect the pathophysiology of sepsis, which may become life-threatening through mechanisms other than organ dysfunction34. These limitations would also apply in our analysis of age-adjusted Sepsis-3 in pediatric sepsis patients. Completeness of data records in the EHR may also influence the accuracy of our labels (e.g. availability of culture and antibiotic data to determine suspected infection), as well as the accuracy of our predictions. Furthermore, our dataset is limited to a relatively small set of patients from a single center of care. While we corroborate results obtained by other research groups in other cohorts of patients, greater validity could be achieved via a multi-center analysis encompassing a greater number of patients treated within diverse settings.

## Supporting information

Supplementary Materials [[supplements/241430_file02.docx]](pending:yes)

## Data Availability

Data referred to in this manuscript are not publicly available.

*   Received November 30, 2020.
*   Revision received November 30, 2020.
*   Accepted December 2, 2020.


*   © 2020, Posted by Cold Spring Harbor Laboratory

This pre-print is available under a Creative Commons License (Attribution 4.0 International), CC BY 4.0, as described at [http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/)

## References

1.  1.Watson RS, Carcillo JA, Linde-Zwirble WT, Clermont G, Lidicker J, Angus DC. The Epidemiology of Severe Sepsis in Children in the United States. American Journal of Respiratory and Critical Care Medicine. 2003;167(5):695–701.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1164/rccm.200207-682OC&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=12433670&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F12%2F02%2F2020.11.30.20241430.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000181195300007&link_type=ISI) 

2.  2.Angus DC, Linde-Zwirble WT, Lidicker J, Clermont G, Carcillo J, Pinsky MR. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med. 2001;29(7):1303–1310.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/00003246-200107000-00002&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=11445675&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F12%2F02%2F2020.11.30.20241430.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000169944400002&link_type=ISI) 

3.  3.Kumar A, Roberts D, Wood KE, et al. Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock*. Critical Care Medicine. 2006;34(6):1589–1596.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/01.CCM.0000217961.75225.E9&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=16625125&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F12%2F02%2F2020.11.30.20241430.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000237884300001&link_type=ISI) 

4.  4.Weiss SL, Fitzgerald JC, Balamuth F, et al. Delayed Antimicrobial Therapy Increases Mortality and Organ Dysfunction Duration in Pediatric Sepsis*. Critical Care Medicine. 2014;42(11):2409–2417.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/CCM.0000000000000509&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25148597&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F12%2F02%2F2020.11.30.20241430.atom) 

5.  5.Singer M, Deutschman CS, Seymour CW, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). Jama. 2016;315(8):801.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jama.2016.0287&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26903338&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F12%2F02%2F2020.11.30.20241430.atom) 

6.  6.Goldstein B, Giroir B, Randolph A. International pediatric sepsis consensus conference: Definitions for sepsis and organ dysfunction in pediatrics*. Pediatric Critical Care Medicine. 2005;6(1):2–8.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/01.PCC.0000149131.72248.E6&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=15636651&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F12%2F02%2F2020.11.30.20241430.atom) 

7.  7.Levy MM, Fink MP, Marshall JC, et al. 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Critical Care Medicine. 2003;31(4):1250–1256.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/01.CCM.0000050454.01978.3B&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=12682500&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F12%2F02%2F2020.11.30.20241430.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000182411900038&link_type=ISI) 

8.  8.Schlapbach LJ, Straney L, Bellomo R, MacLaren G, Pilcher D. Prognostic accuracy of age-adapted SOFA, SIRS, PELOD-2, and qSOFA for in-hospital mortality among children with suspected infection admitted to the intensive care unit. Intensive Care Medicine. 2017;44(2):179–188.
    
    

9.  9.Matics TJ, Sanchez-Pinto LN. Adaptation and Validation of a Pediatric Sequential Organ Failure Assessment Score and Evaluation of the Sepsis-3 Definitions in Critically Ill Children. JAMA Pediatrics. 2017;171(10):e172352.
    
    

10. 10.Leclerc F, Duhamel A, Deken V, Grandbastien B, Leteurtre S. Can the Pediatric Logistic Organ Dysfunction-2 Score on Day 1 Be Used in Clinical Criteria for Sepsis in Children?*. Pediatric Critical Care Medicine. 2017;18(8):758–763.
    
    

11. 11.Weiss SL, Peters MJ, Alhazzani W, et al. Surviving Sepsis Campaign International Guidelines for the Management of Septic Shock and Sepsis-Associated Organ Dysfunction in Children. Pediatric Critical Care Medicine. 2020;21(2):e52–e106.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F12%2F02%2F2020.11.30.20241430.atom) 

12. 12.Henry KE, Hager DN, Pronovost PJ, Saria S. A targeted real-time early warning score (TREWScore) for septic shock. Science Translational Medicine. 2015;7(299):299ra122–299ra122.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTE6InNjaXRyYW5zbWVkIjtzOjU6InJlc2lkIjtzOjE0OiI3LzI5OS8yOTlyYTEyMiI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIwLzEyLzAyLzIwMjAuMTEuMzAuMjAyNDE0MzAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

13. 13.Liu R, Greenstein JL, Granite SJ, et al. Data-driven discovery of a novel sepsis pre-shock state predicts impending septic shock in the ICU. Scientific Reports. 2019;9(1).
    
    

14. 14.Calvert JS, Price DA, Chettipally UK, et al. A computational approach to early sepsis detection. Computers in Biology and Medicine. 2016;74:69–73.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.compbiomed.2016.05.003&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=27208704&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F12%2F02%2F2020.11.30.20241430.atom) 

15. 15.Liu R, Greenstein JL, Fackler JC, Bembea MM, Winslow RL. Spectral clustering of risk score trajectories stratifies sepsis patients by clinical outcome and interventions received. eLife. 2020;9.
    
    

16. 16.Seymour CW, Liu VX, Iwashyna TJ, et al. Assessment of Clinical Criteria for Sepsis. Jama. 2016;315(8):762.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jama.2016.0288&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26903335&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F12%2F02%2F2020.11.30.20241430.atom) 

17. 17.Feinstein JA, Russell S, DeWitt PE, Feudtner C, Dai D, Bennett TD. R Package for Pediatric Complex Chronic Condition Classification. JAMA Pediatrics. 2018;172(6):596.
    
    

18. 18.Feudtner C, Feinstein JA, Zhong W, Hall M, Dai D. Pediatric complex chronic conditions classification system version 2: updated for ICD-10 and complex medical technology dependence and transplantation. BMC Pediatrics. 2014;14(1).
    
    

19. 19.Haque IU, Zaritsky AL. Analysis of the evidence for the lower limit of systolic and mean arterial pressure in children. Pediatric Critical Care Medicine. 2007;8(2):138–144.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/01.PCC.0000257039.32593.DC&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=17273118&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F12%2F02%2F2020.11.30.20241430.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000244768300007&link_type=ISI) 

20. 20.Bryan S,  Larose MS-P, Campbell N, Clarke J, Tremblay MS. Resting blood pressure and heart rate measurement in the Canadian Health Measures Survey, cycle 1. Health Reports. 2010;21(1):71.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21269014&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F12%2F02%2F2020.11.30.20241430.atom) 

21. 21.Ng AY, Jordan MI, Weiss Y. On spectral clustering: Analysis and an algorithm. Paper presented at: Advances in neural information processing systems2002.
    
    

22. 22.Chen T, Guestrin C. Xgboost: A scalable tree boosting system. Paper presented at: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining2016.
    
    

23. 23.Tibshirani R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B (Methodological). 1996:267–288.
    
    

24. 24.Vincent J-L, Opal SM, Marshall JC, Tracey KJ. Sepsis definitions: time for change. The Lancet. 2013;381(9868):774–775.
    
    

25. 25.Sprung CL, Sakr Y, Vincent J-L, et al. An evaluation of systemic inflammatory response syndrome signs in the Sepsis Occurrence in Acutely ill Patients (SOAP) study. Intensive Care Medicine. 2006;32(3):421–427.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s00134-005-0039-8&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=16479382&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F12%2F02%2F2020.11.30.20241430.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000235546900010&link_type=ISI) 

26. 26.Kawasaki T. Update on pediatric sepsis: a review. Journal of Intensive Care. 2017;(1).
    
    

27. 27.Vincent JL, Moreno R, Takala J, et al. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. Intensive Care Med. 1996;22(7):707–710.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/BF01709751&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=8844239&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F12%2F02%2F2020.11.30.20241430.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1996UZ45200017&link_type=ISI) 

28. 28.Donnelly JP, Safford MM, Shapiro NI, Baddley JW, Wang HE. Application of the Third International Consensus Definitions for Sepsis (Sepsis-3) Classification: a retrospective population-based cohort study. The Lancet Infectious Diseases. 2017;17(6):661–670.
    
    

29. 29.Mikkelsen ME, Miltiades AN, Gaieski DF, et al. Serum lactate is associated with mortality in severe sepsis independent of organ failure and shock*. Critical Care Medicine. 2009;37(5):1670–1677.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/CCM.0b013e31819fcf68&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19325467&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F12%2F02%2F2020.11.30.20241430.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000265368900017&link_type=ISI) 

30. 30.Trzeciak S, Dellinger RP, Chansky ME, et al. Serum lactate as a predictor of mortality in patients with infection. Intensive Care Medicine. 2007;33(6):970–977.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s00134-007-0563-9&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=17431582&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F12%2F02%2F2020.11.30.20241430.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000247155500005&link_type=ISI) 

31. 31.Miao H, Shi J, Wang C, et al. Continuous Renal Replacement Therapy in Pediatric Severe Sepsis. Critical Care Medicine. 2019;47(10):e806–e813.
    
    

32. 32.Rhodes A, Evans LE, Alhazzani W, et al. Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock: 2016. Intensive Care Medicine. 2017;43(3):304–377.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s00134-017-4683-6&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=28101605&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F12%2F02%2F2020.11.30.20241430.atom) 

33. 33.Klompas M, Winslow DL, Strich JR, et al. Infectious Diseases Society of America Position Paper: Recommended Revisions to the National Severe Sepsis and Septic Shock Early Management Bundle (SEP-1) Sepsis Quality Measure. Clinical Infectious Diseases. 2020.
    
    

34. 34.Deutschman CS. Imprecise Medicine. Critical Care Medicine. 2016;44(5):857–858.