ABSTRACT
Rationale SeptiCyte RAPID, a molecular test distinguishing sepsis from non-infectious systemic inflammation, has potential clinical utility.
Objectives Clinical validation of SeptiCyte RAPID, based on testing retrospective (banked) and prospectively collected patient samples.
Methods Testing retrospective (banked) and prospective samples from adult patients in ICU either with systemic inflammatory response syndrome (SIRS) or suspected of sepsis, with test results compared to “gold standard” clinical evaluation by a blinded three physician external panel.
Measurements and Main Results The cartridge-based SeptiCyte RAPID assay accepts a PAXgene blood RNA sample and provides sample-to-answer processing in ∼1 hour. The test output (SeptiScore, range 0-15) falls into four interpretation bands, with higher scores indicating higher probabilities of sepsis. SeptiCyte RAPID performance is comparable to that for SeptiCyte LAB, with Area Under the ROC Curve (AUC) ranging from 0.82 – 0.85, negative predictive value 0.91 (sensitivity 0.94) for SeptiScores between 0.1 and 5.0 (Band 1, lowest risk of sepsis), and positive predictive value 0.81 (specificity 0.90) for SeptiScores between 7.4 and 15 (Band 4, highest risk of sepsis). For ninety percent of blood culture confirmed sepsis cases, SeptiCyte RAPID indicated an elevated (Band 3 or 4) risk of sepsis. In multivariable analysis, SeptiScore was the most important variable for sepsis diagnosis. A likelihood ratio method was developed to estimate the post-test probability of sepsis for individual patients, when combining the SeptiScore with additional clinical parameters.
Conclusions This study validates SeptiCyte RAPID for differentiating patients with sepsis vs. SIRS, on the first day of ICU admission.
INTRODUCTION
Sepsis is an important and expensive global health problem with high morbidity and mortality (1). According to the World Health Organization, more than 11 million people die from sepsis worldwide annually, comprising more than the deaths caused by all cancers combined (2). Sepsis is the leading cause of death in U.S. hospitals is ranked as the most expensive disease state to manage for admitted patients (3), with total annual costs of treatment and rehabilitation estimated at $62 billion (4). Before the COVID-19 pandemic, at least 1.7 million adults in the United States developed sepsis annually (5). During the pandemic, this problem was greatly exacerbated, as most COVID-19 deaths are due to viral and/or bacterial sepsis (6). Up to 20% of all deaths worldwide can be attributed to sepsis on an annual basis (1).
Early identification of sepsis and implementation of treatment bundles have been shown to improve outcomes for sepsis patients (7). However, early identification of sepsis can be difficult for many reasons. In the early stages of the disease, patients present with clinical signs that are common to many other non-infectious conditions (8). Furthermore, sepsis does not have a diagnostic gold standard since blood cultures lack timeliness (9) and are negative in 30-50% of retrospectively diagnosed cases (10, 11). In addition, unnecessary administration of broad-spectrum antibiotic therapy for presumed sepsis may adversely contribute to the growing problem of multidrug resistant organisms and carries the risk of adverse drug reactions (12).
We previously reported on the development and validation of a diagnostic assay (SeptiCyte LAB) to differentiate patients with sepsis from those with non-infectious systemic inflammation, providing a probability of sepsis based on measurement of four host immune response biomarkers (13, 14). Here we present data on the clinical validation of SeptiCyte RAPID, a simplified and improved cartridge-based version of this assay. SeptiCyte RAPID assay addresses a clinical need for more rapid and accurate differentiation of sepsis from non-infectious systemic inflammatory response syndrome (SIRS) within a clinically actionable (∼1 hour) time frame. Some of these results have been presented earlier in the form of an abstract (15).
METHODS
Study Cohorts
Clinical validation of SeptiCyte RAPID used PAXgene blood RNA samples from retrospective (N=356) and prospective (N=63) patient cohorts.The retrospective cohort was drawn from the observational MARS and VENUS trials (www.clinicaltrials.gov NCT01905033 and NCT02127502) which have been previously described (14). The retrospective cohort comprised 80% of the 447 patients used for the 510(k) clearance of SeptiCyte LAB for which duplicate banked PAXgene blood RNA samples remained available.
The prospective cohort consisted of 63 critically ill adult subjects enrolled in an observational trial (NEPTUNE, www.clinicaltrials.gov NCT05469048) between the dates May 26, 2020 - April 25, 2021 at Emory University / Grady Memorial Hospital (Atlanta GA), Rush University Medical Center (Chicago IL) and University of Southern California Medical Center (with 2 separate sites, Keck Hospital of USC, and LAC+USC Medical Center, in Los Angeles CA). The study protocol was approved by each institutional review board (see below). All patients (or legally authorized representatives) provided signed informed consent. Subjects were considered for inclusion if adult (≥18 years old), exhibited two or more SIRS criteria, and received an ICU admission order. Subjects were “suspected of sepsis” if microbiological tests were ordered within 24 hours of the ICU admission order. Subjects were excluded if therapeutic antibiotic treatment was initiated >24 hours before ICU admission order. As much as possible, enrollments were consecutive; however, during the COVID-19 pandemic, screening did not happen every day, and initial consenting sometimes resulted in later refusal to participate. PAXgene blood samples were collected within 24 hours of ICU admission order, and run fresh, in real time on Idylla instruments installed at the sites. Clinical data were collected as described in (14).
Ethics approvals for the MARS and VENUS trials (retrospective cohort) are referenced in Miller et al. (2018). Ethics approvals for the NEPTUNE trial (prospective cohort) are as follows: # IRB00115400 (Emory University); # 00-115400 (Grady Memorial Hospital); # 19101603-IRB01 (Rush University Medical Center), and # HS-19-0884-CR001 (University of Southern California Medical Center). The retrospective and prospective trials comply with the CONSORT checklist, and a CONSORT flow diagram is provided in the Online Data Supplement, Section 1.
Reference Method and Calculations
Clinical performance of SeptiCyte RAPID was determined by comparison to retrospective physician diagnoses (RPD), as described in (14) and Online Data Supplement, Section 2.
Sepsis probability as a function of RAPID SeptiScore was calculated by a “sliding window” approach (Online Data Supplement, Section 3).The continuous range of sepsis probabilities was also parsed into four bands, with higher sepsis probabilities correlating with higher SeptiScores. Band boundaries were pre-defined, based on an independent set of 195 clinical samples from the MARS consortium, to give 90% sensitivity for binarization at the Band 1/2 cutoff (SeptiScore 4.95) and 80% specificity for binarization at the Band 3/4 cutoff (SeptiScore 7.45), using site clinical adjudications as Ground Truth values. The intermediate zone between the Band 1/2 and Band 3/4 cutoffs was divided in half to define the Band 2/3 cutoff (SeptiScore 6.15).
Statistical methods for additional data analyses, including combining the SeptiScore with other clinical variables, are described in the Online Data Supplement, Sections 3-8.
RESULTS
Demographic and Clinical Characteristics
The demographics of the population from the combined retrospective and prospective clinical validation cohorts (n=419) are presented in Table 1. Sepsis, SIRS or indeterminate diagnoses were assigned by consensus RPD. When compared to patients with SIRS, those patients who were adjudicated as having sepsis tended to be older (p = 0.016). There were no significant differences by sex or race/ethnicity.
Patients originated from a variety of locations within the hospital systems in our study. When coming from the ED, post-anesthesia unit or post-operating room, a higher proportion of patients were found to have SIRS (as opposed to being septic), while the opposite was found for patients coming from hospital wards.
Clinical characteristics of the study population are presented in Table 2. The highly statistically significant culture results are of particular note: positive blood cultures, positive urine cultures, and dual-positive cultures (blood+urine) were very highly associated with RPD diagnoses of sepsis, as opposed to SIRS, with p-values of 2.7 x 10-12, 3.1 x 10-6, and 9.0 x 10-4 respectively.
SeptiCyte RAPID Performance: Primary Analyses
Test and Device Description
SeptiCyte RAPID is an in-vitro diagnostic test for simultaneous amplification and detection of two RNA transcripts (PLA2G7 and PLAC8) in human blood samples.The test is run on a near-to-patient platform, Idylla, manufactured by Biocartis NV (Mechelen, Belgium). The SeptiCyte RAPID test is performed by pipetting 0.9 mL of PAXgene-stabilized blood (corresponding to 0.24 mL of drawn blood) into a custom cartridge which performs all assay steps including sample extraction/purification and RT-qPCR for the detection and relative quantification of the PLAC8 and PLA2G7 mRNA targets. Test results are presented automatically through a software-generated report which includes a quantitative probability score (SeptiScore, range 0-15), calculated by combining the RT-qPCR Cq values measured for PLAC8 and PLA2G7. The test has a hands-on time of ∼2 min and a turnaround time of ∼1 hour.
Comparison to Predicate
A strong correlation of SeptiScores between SeptiCyte LAB and SeptiCyte RAPID, based upon the 356 clinical samples from the retrospective cohort run on both platforms, was found (Pearson’s sample correlation coefficient r = 0.88) as shown in Figure 1. In serial dilution experiments, SeptiCyte RAPID scores were verified to be independent of the white blood cell (WBC) count across an input range of 25 to 25,000 WBC/μl.
ROC Curve Analyses
Receiver operating characteristic (ROC) curve analyses were performed on samples from the combined retrospective and prospective cohorts (Figure 2A). Performance estimates ranged from AUC 0.82-0.85 (depending upon RPD method) and were statistically indistinguishable both from each other, and from SeptiCyte LAB AUC values (14). ROC analyses were also performed on the prospective cohort alone (Figure 2B), with AUCs ranging from 0.86-0.95 and again being statistically indistinguishable from each other. Finally, a comparison of the ROC curves from Figures 2A vs. 2B showed an absence of significant differences, except for the unanimous RPD curves in the two figures, for which deLong’s test gave p = 0.013.
Sepsis Probability Distributions
Patients with higher SeptiScores have higher probabilities of sepsis (Figure 3A). To assist in interpretation and downstream clinical decision making, we also report SeptiScores in terms of four probability bands (Figure 3B) as for the predicate, SeptiCyte LAB. A strong correlation was found between the banding schemes for SeptiCyte LAB and SeptiCyte RAPID (Figure 4).
Table 3 presents SeptiScores for the complete (retrospective + prospective) cohort, parsed into the four probability bands. Assignment of sepsis or SIRS was by consensus RPD. The probability and likelihood ratio of sepsis increases with increasing SeptiScore, with band-averaged sepsis probabilities of 9% (Band 1), 21% (Band 2), 42% (Band 3), and 81% (Band 4).
SeptiCyte RAPID Clinical Performance: Secondary Analyses
Chi-squared analyses (Table 4) indicated that, after sorting patients into the four SeptiScore bands, no significant differences were found with respect to baseline demographic criteria (sex, age, race or ethnicity). Positive culture results (blood, urine, and blood/urine double positive) were more predominant in the higher SeptiScore bands (especially Band 4), with chi-squared analyses giving p-values of 7.2 x 10-6, 0.023 and 0.005 for the observed distributions of these positive results into Band 4 vs. pooled Bands 1+2+3. Of the positive results from blood, urine, and blood+urine culture, the percentages falling in Band 4 were 63%, 52% and 73% respectively, and the percentages falling in either Band 3 or Band 4 were 83%, 64% and 82% respectively. Of the 41 patients with blood culture positive results, only one had a SeptiScore in band 1, and this patient was diagnosed as SIRS by RPD with the blood culture result (coagulase negative Staphylococcus) being considered a potential contaminant.
Multivariable analysis
We asked whether SeptiCyte RAPID provides diagnostic clinical utility for discriminating between sepsis and SIRS patients, beyond that provided by other clinical variables and laboratory assessments available at first day of ICU admission. Fourteen variables in addition to SeptiCyte RAPID were examined (see Online Data Supplement, Section 6). We evaluated all 32,767 possible combinations of the fifteen variables, and performance was assessed by AUC against consensus RPD. Procalcitonin was included in this analysis, as well as lactate which is rapid and commonly used parameter to determine which patients should receive sepsis treatment bundles (16, 17). Figure 5 shows the results of the analysis. With respect to individual variables, SeptiScore was found to rank highest by AUC. When combinations of clinical variables were considered, those containing SeptiScore were found to have higher AUCs than all combinations lacking SeptiScore.
Sepsis probability adjustments
The preceding multivariable analysis suggests that SeptiCyte RAPID might be combined with other clinical variables to boost the overall performance at discriminating between sepsis and SIRS (see the red distribution of AUCs in Figure 5). Therefore, we investigated which additional clinical variables could be used in combination with SeptiCyte RAPID to improve diagnostic performance (post-test probability of sepsis).
Additional clinical variables of importance were identified through a machine learning (Random Forest) analysis of our clinical validation data (Online Data Supplement, Section 7). Assurance of the results was gained through a literature search and by consulting with practising sepsis experts. We then combined the identified significant variables in a sequential likelihood ratio analysis (18, 19). The pre-versus post-test sepsis probabilities for six example patients are shown in Figure 6. These patients were chosen based on Forced RPD that represented three SIRS (band 1-2) and three sepsis (band 4) patients. For band 1-2 patients the post-test probability of sepsis was decreased, while for band 4 patients the post-test probability of sepsis was increased. Additional detail on the calculations for these six patients are given in Table 5 and Online Data Supplement, Section 8.
DISCUSSION
We have presented data validating the use of SeptiCyte RAPID for differentiating sepsis from SIRS in critically ill adult patients, within a clinically actionable time frame. The test is a cartridge-based, fully integrated assay measuring the transcription levels of the host immune response biomarkers PLAC8 and PLA2G7. The test provides a quantitative score, the SeptiScore, ranging from 0-15 with higher scores indicating higher probabilities of sepsis. Sepsis probabilities can be interpreted either in terms of a continuum from 0% to 100% (Figure 3A), or quantized into four “probability bands” (Figure 3B). The latter interpretation, however, may result in a loss of diagnostic resolution, compared to interpretation along a continuous sepsis probability scale. This is because the sepsis probability is not constant within each band, but rather increases from the lower edge to the upper edge of each band.
Patients in this study came from the ED, post-anesthesia unit, post-operating rooms and wards, and were tested in the ICU. For the full (retrospective + prospective) cohort, diagnostic performance of SeptiCyte RAPID was equivalent to that previously reported for the predicate device, SeptiCyte LAB (14) with AUC ranging from 0.81 – 0.84 depending upon comparator method (RPD). Using consensus RPD as reference, patients with Band 1 SeptiScores had an estimated 9% probability of sepsis, while those with Band 4 SeptiScores had a much higher estimated probability of sepsis (∼81%). For the prospective cohort, diagnostic performance of SeptiCyte RAPID based on AUC was higher than for the full cohort, ranging from 0.86 – 0.98 depending upon RPD method.With consensus RPD, patients with Band 1 SeptiScores were estimated to have a ∼0% average probability of sepsis, while patients with Band 4 SeptiScores were estimated to have an average sepsis probability of ∼83%.
We note that the comparator method used (RPD) is imperfect. In our complete (n=419) validation dataset, three expert clinicians failed to reach either a unanimous or consensus diagnosis for 41/419 (9.8%) of patients. The use of an imperfect comparator sets an upper limit to the measurable diagnostic performance of a new test (20). This observation mirrors the real-life reality that the diagnosis of sepsis is often difficult, even through a retrospective review that includes all the clinical data and outcome information.
In a multivariable analysis, we examined all possible combinations of SeptiScore and up to 14 additional clinical or laboratory variables, including lactate and PCT. We found that SeptiScore alone had greater performance than any combination of variables without SeptiScore, for differentiating sepsis vs. SIRS (Figure 5). However, our analysis also indicated it should be possible to moderately boost the performance of SeptiCyte RAPID by combining the SeptiScore value with other clinical parameters. Accordingly, we developed a likelihood ratio approach for updating post-test sepsis probabilities, through the sequential application of likelihood ratios calculated with additional clinical parameters (Figure 6 and Online Data Supplement, Section 8). We believe this approach may have utility for patients with intermediate SeptiScore values (in Band 2 or Band 3).
In 2017 the Surviving Sepsis Campaign (SSC) published updated sepsis management guidelines with the objective of guiding and improving the care of sepsis (21). These guidelines and the introduction of the sepsis core measures by the United States Centers for Medicare and Medicaid Services inadvertently led to encouraging physicians to initiate early broad-spectrum antibiotics in patients who were at low risk for sepsis (22). The recent updated Surviving Sepsis Campaign guidelines (2021) has attempted to address this problem by dividing the patients into 3 groups (low, intermediate, and high) based on sepsis probability and have recommended appropriate evaluation and treatment of these groups (23). SeptiCyte RAPID aligns well with these guidelines, and could have a role in supporting their implementation and in contributing to their futher refinement and evolution.
The SSC low sepsis probability group has, under previous guidelines, been treated early resulting in poor antibiotic stewardship. The new guidelines recommend deferring antibiotics, and monitoring and evaluate for other etiologies that may underlie the presenting symptoms. A SeptiScore <5 (Band 1) with a sensitivity of 0.94 in this group of patients would support the deferring of antibiotics thereby facilitating antibiotic and diagnostic stewardship.
For patients falling in the SSC intermediate sepsis probability (without shock) group, the SSC recommendation is a rapid assessment of infectious versus non-infectious cause of the illness. Septicyte RAPID, with a one-hour turnaround time, rapidly differentiates infectious positive from infection negative systemic inflammation. The presence of elevated SeptiScores in this group would provide confirming evidence for infectious etiology, and could enable appropriate early antibiotic administration 24 to 48 hours before any positive microbiological results are available. The resulting SeptiScore turnaround time of ∼1 hour would also aid in meeting the recommend 3 hour initiation of antibiotics for this group.
Those patients with high sepsis probability per the SSC guidelines, differentiated by the presence of shock, should appropriately be treated within 1 hour of recognition. SeptiScores >7.2 in this patient group, with specificity 0.9 or greater, would confirm the high probability of sepsis and the continuation of antibiotics.
We note several limitations to the present study. We have focused only on adult patients within 24 hours of ICU admission, so generalization to other patient cohorts not been established. We have not conducted serial sampling to measure variation in SeptiCyte RAPID scores as patients move into, through, and out of ICU. We have, however, previously reported strong diagnostic performance of SeptiCyte LAB in children (24) and the high correlation between SeptiCyte LAB and SeptiCyte RAPID (Figure 1) suggests that an equivalently strong performance in children will be found for SeptiCyte RAPID.
SeptiScores falling in Bands 2 or 3 do not, by themselves, provide definitive conclusions (i.e. very high or low probabilities) regarding absence or presence of sepsis. For SeptiScores in this range, it may be necessary to combine the readings with data from other clinical parameters, to adjust the inferred sepsis probabilities (see Table 5 and Figure 6).
During the discovery and initial validation of the PLAC8 and PLA2G7 biomarkers used in SeptiCyte RAPID (13, 14), patients with a broad range of co-morbidities were examined, including septic patients with confirmed bacterial, viral, fungal infections and malignancies, and non-septic patients with non-infectious systemic inflammation of varying etiologies. The validation cohort also included use of a broad range of prescribed medications such as immunosuppressants, anti-neoplastic drugs, antithrombotics, corticosteroids and statins. To our knowledge, SeptiCyte RAPID results are unaffected by these factors. However, there may be other specific conditions and treatments we have not yet examined, that may affect SeptiCyte RAPID performance.
We have shown that the Limit of Quantitation (LoQ) and Limit of Detection (LoD) of SeptiCyte RAPID is 25 WBC/μL of blood, and we have validated the test to an upper limit of 25,000 WBC/uL. This dynamic range of SeptiCyte RAPID is broad and extends well below and above the 4,000-11,000 WBC/uL normal reference range. However, we have not yet completed an evaluation of the assay on severely neutropenic patients. It also is possible that SeptiCyte RAPID scores might be skewed by selective leukopenias due to disease or medications, for example T cell depletion in HIV / AIDS. However, it is known from single cell sequencing studies that both PLAC8 and PLA2G7 are expressed across a range of different white cell types (25) which would mitigate the effect of a selective leukopenia.
The time between injecting a blood sample into the SeptiCyte RAPID cartridge and generating a test report is only ∼1 hour. However, by the time patient blood is drawn and delivered e.g. to a STAT lab, the total time between blood draw and presentation of the test result to an attending clinician is likely to be 1.5-2 hours. Although this is longer than requirements to implement a 1-hour sepsis bundle, SeptiCyte RAPID would clearly provide timely information with respect to implementation of a 3-hour sepsis bundle, which has been shown to have a low level of compliance in US hospitals (26, 27). Compliance with a 3-hour sepsis bundle in patients with severe sepsis and septic shock has been shown to improve survival and reduce overall costs (Leisman et al., 2017).
CONCLUSION
SeptiCyte RAPID addresses a clinical need for more rapid and accurate differentiation of sepsis from non-infectious systemic inflammation conditions by providing a probability of sepsis within a timeframe that would allow clinicians to take meaningful action. SeptiCyte RAPID may provide clinical utility through better patient management decisions, appropriate implementation and timing of sepsis bundles, the timing and choice of therapies, downstream antibiotic and diagnostic stewardship. Finally, the test may also enable a more appropriate selection or stratification of patients in clinical trials for sepsis pharmacological agents and therapeutic medical devices.
Data Availability
Data produced in the present study are available upon reasonable request to the authors.
Abbreviations
- %A
- Percent Agreement
- ANOVA
- Analysis of Variance
- AUC
- Area Under (Receiver Operator Characteristic) Curve
- CAD
- Coronary artery disease
- CKD
- Chronic kidney disease
- CPAP
- Continuous positive airway pressure
- Cq
- Quantification Cycle (for PCR)
- CRP
- C-Reactive Protein
- ED
- Emergency Department
- ICU
- Intensive Care Unit
- IQR
- Interquartile Range
- Ku
- Unweighted inter-observer kappa
- Kw
- Linearly weighted inter-observer kappa
- LR
- Likelihood Ratio
- PCR
- Polymerase Chain Reaction
- PCT
- Procalcitonin
- PLA2G7
- Phospholipase A2, Group VII
- PLAC8
- Placenta associated 8
- ROC
- Receiver Operator Characteristic (curve)
- RPD
- Retrospective Physician Diagnosis
- RT-qPCR
- Reverse Transcription - Quantitative Polymerase Chain Reaction
- SIRS
- Systemic Inflammatory Response Syndrome
- SSC
- Surviving Sepsis Campaign
- Tukey HSD
- Tukey Honestly Significant Difference
- WBC
- White Blood Cell
ACKNOWLEDGEMENTS
The authors thank the clinical study coordinators of the NEPTUNE study: Joyce D. Brown (Rush University), Liliacna Jara (University of Southern California), Leona Wells and Maya C. Whaley (Grady Memorial Hospital). The authors also thank the laboratory technical staff at each institution,at Reach Bio (Seattle WA), and at Biocartis (Mechelen, Belgium).
Footnotes
This article has an Online Data Supplement, which is accessible online and linked to the article.
Sources of Support: This study was supported by Immunexpress Inc.
Table E5, Online Data Supplement: In the previously uploaded version of this table,the patients were incorrectly labeled. This error has now been corrected.