International Comparisons of Harmonized Laboratory Value Trajectories to Predict Severe COVID-19: Leveraging the 4CE Collaborative Across 342 Hospitals and 6 Countries: A Retrospective Cohort Study

Cohort identification

We included all patients hospitalized at participating 4CE sites with an admission date from 7 days before to 14 days after the date of their first reverse transcription polymerase chain reaction (PCR)-confirmed SARS-CoV-2 positive test result. The first admission date within this 21-day time window was considered the index admission date. Throughout this work, “days since admission” refers to this index date.

Participating sites

Data were available from 36,447 patients from 342 hospitals (affiliated with 46 sites) across six countries: France, Germany, Italy, Singapore, Spain, and the United States. See eTABLE 2 for details about participating sites. Several sites collected data from multiple hospitals. In the United States, 170 medical centers of the US Department of Veterans Affairs were grouped into 18 regional divisions called Veterans Integrated Service Networks.

Patient and Public Involvement

Patients and the public were not involved in the design, conduct, or reporting, or dissemination plans of the research.

Severe COVID-19

We categorized the primary patient outcome as “ever-severe” or “never-severe” based on whether the patients, at any time during their hospitalization, progressed to severe disease—even if they later recovered from COVID-19. We defined “ever-severe” COVID-19 using the validated 4CE severity criteria based on the presence of at least one of the following codes in the EHR during the hospitalization: an order for any medication in two broad classes (intravenous anesthetics or cardiac inotropes), a collected partial pressure of arterial oxygen (PaO2) laboratory test (regardless of the result), an International Classification of Disease (ICD)-10 diagnosis code for acute respiratory distress syndrome or ventilator-associated pneumonia, or an ICD-10 procedure code for insertion of endotracheal tube or invasive mechanical ventilation [8]. A more complete description is available in eTABLE 3. The full list of codes is available in [8] and in the consortium’s GitHub [9].

We chose this definition of severity because of the availability of standardized coded data describing medications, laboratory tests, diagnoses, and procedures. In contrast, other measures of disease severity, including requirement for mechanical ventilation or intensive care and death, are often unavailable in the EHR or are inconsistently recorded and are not easily ascertained. As an example, admission to traditional ICU spaces was not an effective measure of ICU admission when surge conditions required non-traditional placement.

Local data collection and central data aggregation

Following our prior approach [5], each contributing site executed queries on local database systems containing EHR data to generate six tables in comma-separated values (CSV) format, containing aggregate counts and statistics on their respective patient cohorts: DailyCounts, ClinicalCourse, Demographics, Labs, Diagnoses, and Medications (see eTABLE 4 for descriptions of the files and Figure 1). Sites uploaded their six CSV files to a central 4CE Data Upload Tool, which performed several validation steps, which included variable consistency and range and count evaluation. The results of validation were either a report of any errors or the ability to save the files to a private shared folder. Throughout this process, patient-level files remained at each site and were not centrally shared. A key advantage of executing queries locally and sharing only aggregate data was that sites were able to obtain institutional approval more rapidly. Further, this approach facilitated analysis at the site, country, and region by reducing barriers to federated data generation and aggregation at each level.

• Download figure
• Open in new tab

FIGURE 1.

Each site generated six data tables (comma-separated files) containing aggregate counts and statistics on their individual level data: 1) demographic breakdowns, 2) clinical course shown as counts and disposition from index date, 3) daily counts of patients and their disposition, 4) daily diagnosis counts and 5.) daily medication counts 6) daily trajectories of lab tests. These aggregate descriptive files without individual level data were provided to the consortium for extensive quality-assurance steps (see Methods).

Most sites used the open-source i2b2 (Informatics for Integrating Biology and the Bedside) software platform to obtain the data. More than 200 organizations worldwide use i2b2 for purposes that include identifying participants for clinical trials, drug safety monitoring, and clinical and epidemiological research. Those 4CE sites with i2b2 used database scripts to directly query their i2b2 repository, calculate the counts and statistics, and export the data files. The 4CE sites without i2b2 used the Observational Medical Outcomes Partnership (OMOP) Common Data Model or their own clinical data warehouse solutions (e.g., Epic Caboodle) and querying tools to create the required files.

Selection of laboratory tests and medications

We focused on 18 laboratory tests associated with worse outcomes in patients with COVID-19 based on prior reports [6]. We provided each site with a single standard Logical Objects, Identifiers, Names and Codes (LOINC) identifier for each test, but sites often needed to map tests to additional LOINC or custom codes within their EHR. We addressed barriers that arose during initial efforts to extract these laboratory values by stratifying region-specific laboratory test types to reduce extraction errors and enable standardization. For example, D-dimer was extracted with both fibrinogen-equivalent unit and D-dimer unit measurements. We converted all results to D-dimer units for subsequent analyses.

Quality control

After file upload, the online validation process checked the file and column names, column orders, data types, code values and ranges, and duplicated records. Then, we ran an R script for the following additional quality control checks: consistency of the total counts of ever-severe and never-severe cases across all datasets within each site, consistency between the 3-digit diagnosis codes and the ICD dictionary, and consistency of the range of laboratory data from each site with data observed from all sites. We contacted a site for site-specific quality control if the site laboratory values were consistently lower or higher than the other sites or otherwise implausible.

Statistical analysis

We estimated the country-level daily incidence of new patients hospitalized with COVID-19 during the study period from January 23, 2020 to September 29, 2020. Specifically, for each country, we summed the daily incidence of new patients hospitalized with COVID-19 at each site within that country per 100,000 people of the country and multiplied this by an adjustment factor, defined as the ratio between the country’s overall inpatient discharge rate and the overall inpatient discharge rate of all 4CE sites in that country irrespective of COVID-19 status. We then reported the adjusted 7-day average incidence of new COVID-19 hospitalizations per 100,000 of the country population.

We assessed the performance of laboratory values in predicting severity. We summarized the severity risk among different demographic subgroups by country based on the DerSimonian and Laird random effects meta-analysis of the site-specific risk estimates [10]. For each laboratory test, we first compared the mean values over time among the ever-severe group to those in the never-severe group. To account for cross-site variabilities and mitigate confounding bias, we summarized the overall mean trajectory over time via random effects meta-analyses and only included sites that had measurements for patients in both the ever-severe and never-severe groups.

For each laboratory test, we assessed the ability of laboratory results measured at sequential days following the admission date to predict severity risk. We quantified the discriminatory capacity of each laboratory result by estimating the area under the receiver operating characteristic curve (AUC), assuming that the laboratory values within the ever-severe group or the never-severe group at each site followed either a normal distribution or a log-normal distribution. The AUC estimate quantified the probability that the laboratory result of a randomly selected patient in the ever-severe group was sufficiently different from that of a randomly selected patient in the never-severe group. For each laboratory value measured on a given day, we obtained AUC estimates within each site and combined them across sites using meta-analysis (see Supplementary Materials). In addition, we estimated site-level AUCs as the probability that the average laboratory value in an ever-severe group selected randomly from all sites is higher than the average laboratory value in a never-severe group selected randomly from all sites. Site-level AUC assessed the degree to which each laboratory value is different on average in the ever-severe group compared to the never-severe group.

Recent literature suggested that combining multiple laboratory measurements may yield better severity predictions. We chose to validate a published composite severity risk score, calculated as a weighted average of several laboratory measurements, in comparison to our individual laboratory predictions. We chose [7] given the extremely high AUC, recent publication, and ease of application in our data set.

Characteristics of the Study Population

In this study population of 36,447 patients, the incidence of hospitalization for COVID-19 largely tracked with population dynamics of COVID-19 cases [11] across different countries during the initial pandemic period (FIGURE 2). Both the COVID-19 case rate and the COVID-19 hospitalization rate dropped significantly from the first peak in April 2020. While hospitalization rates and incidence of severe disease remained relatively low for all countries, case rates increased in France, United States, and Spain and Singapore after June 2020. Hospitalization rate and incidence of ever-severe disease in Spain and Singapore increased after August 2020.

• Download figure
• Open in new tab

FIGURE 2.

Adjusted 7-day average new hospitalization rate and rate of ever-severe disease per 100,000 people by country based on 4CE contributors along with 95% confidence intervals compared with 7-day average new case rates collected by Johns Hopkins Center for Systems Science and Engineering (JHU CSSE).

Consistent with prior studies [4][12], the study population of patients hospitalized with COVID-19 showed a higher prevalence of men and older populations. See Table 1 for demographic characteristics among ever-severe and never-severe patients and percentages among age group, race/ethnicity, and sex. In our sample, men in the 50-69 age group made up a plurality (25%) of admitted cases, were the most likely to have ever-severe disease. International comparisons were consistent and showed across six countries that older age was associated with increased risk of severe disease (eFIGURE 1). In the United States, where race/ethnicity data were available, minority patients had a higher number of COVID-19 hospitalizations and proportion of severe disease relative to the White reference group.

Predicting Severity from Individual Laboratory Values at Admission

We pooled admission laboratory measures in ever-severe and never-severe patients across all sites (eTABLE 5). After log transformation, differences in admission laboratory values were significant for all laboratory tests (FIGURE 3). For most laboratory studies, differences in initial laboratory values widened between ever-severe and never-severe cases during the first few days after admission. For example, average CRP at admission across all sites was 124 mg/dL (95% CI of 111-135) for ever-severe patients and 71 mg/dL (95% CI of 57-84) for never-severe patients. Initial CRP values predicted severe disease with an AUC of 0.66 (95% CI of 0.61-0.71). CRP was the best laboratory test for distinguishing severe patients at admission in our study, with a threshold value of 91 mg/dL. Additional laboratory studies with potential ability to identify severe disease at admission were D-dimer (AUC = 0.57, 95% CI = 0.50-0.63), albumin (AUC = 0.61, 95% CI = 0.55-0.66), LDH (AUC = 0.64, 95% CI = 0.59-0.69), and procalcitonin (AUC = 0.62, 95% CI = 0.56-0.68).

Predicting Severity using Individual Laboratory Values after Admission

Given the potential value of laboratory values to better predict patients who transition to severe disease, we next examined the predictive performance of the laboratory values during the hospitalization (FIGURE 3, eFIGURE 2). Laboratory values exhibited distinct trajectories over time. For instance, CRP rapidly fell after an initial peak shortly after admission. In contrast, neutrophil count rose gradually during the hospitalization and ultimately plateaued. Confidence intervals widened over time for all laboratory values due to decreasing sample size with prolonged hospitalization.

We calculated AUC (and 95% CI) of six laboratory tests that distinguished ever-severe and never-severe patients over 14 days. Albumin and CRP were the best predictors of ever-severe disease. Compared to laboratory values on admission, prediction models based on CRP values over the 7 days after admission only marginally improved prediction. Maximum AUC for CRP and albumin was 0.66 and 0.68, respectively. Procalcitonin, neutrophil count, and LDH also had an AUC greater than 0.6. Thus, the utility of most laboratory tests in predicting severe disease from measurements subsequent to admission was minimal. One notable exception was D-dimer. Its performance in predicting severe disease improved from AUC=0.56 (95% CI of 0.50 - 0.63) at admission to AUC=0.68 (95% CI of 0.58 - 0.77) during the later stage of the hospitalization, suggesting that initial elevations of the disease may be less valuable for predicting ever-severe disease.

Validating Existing Risk Scores

Next, we performed a subgroup analysis to assess whether a combination of laboratory tests would improve prediction of severe disease over each individual laboratory test. Specifically, we chose to validate a published algorithm [7] that included age, D-dimer, CRP, and lymphocyte count. For the three laboratory tests, we used site-level average values. FIGURE 4 shows the performance of the algorithm to identify severe cases over the first 14 days of hospitalization. For the subset of 11 sites where we had all necessary admission laboratory data, we found that the algorithm had an AUC of 0.91 at day 0, which was similar to the published AUC of 0.881. Similar to models based on individual laboratory tests, prediction performance improved marginally in the first couple days after admission and then gradually deteriorated over time. Such changes in performance underlined the need for prediction systems that explicitly address early versus later periods.

• Download figure
• Open in new tab

FIGURE 4.

Site level AUC of the risk score compared to the individual laboratory tests.

Evaluating Geographical Differences in Severity Prediction

Given the unique multi-national nature of the 4CE consortium, we compared performance of the best performing laboratory tests (CRP, albumin, LDH, neutrophil count, D-dimer, and procalcitonin) over time between continents, after grouping North American and European sites (FIGURE 5). Overall, the predictive performance was similar between the sites from the two continents at admission (e.g., for CRP at North American sites: AUC=0.63, 95% CI 0.56 - 0.72; European sites: AUC=0.67, 95% CI 0.63 - 0.70) and over time (e.g., for CRP at North American sites: AUC=0.65, 95% CI 0.56 - 0.74; European sites: AUC=0.68, 95% CI 0.65 - 0.72). We observed a similar pattern for other laboratory tests (eFIGURE4). While variation in laboratory results was notably greater in North America than Europe at all time points, the average results between these two continents were surprisingly similar. These results indicate that it is feasible to use national datasets for severity prediction.

• Download figure
• Open in new tab

FIGURE 5.

Patient level AUC of six selected laboratory tests stratified by regions.

Statement of Principal findings

This study builds upon the growing literature of COVID-19 severity prediction to report key findings pertaining to the optimal timing of inpatient laboratory results to predict a severe course of disease. We additionally used an international cohort to validate the generalizability of these models. First, we identified severe patients using a previously validated algorithm within the 4CE consortium. For patients hospitalized for COVID-19, we identified six laboratory tests with the highest predictive power to identify severe disease: CRP, LDH, procalcitonin, D-dimer, neutrophil count, and albumin. These laboratory tests are a subset of a list of tests previously associated with worse outcomes in patients with COVID-19.[6] Second, with the exception of D-dimer, laboratory values at the time of admission showed comparable predictive performance to values measured 7 days after admission. These findings suggest the predictive performance of the identified individual laboratory tests is not significantly improved after admission. Third, we reported laboratory value thresholds for these multiple laboratory tests to predict transitions to severe disease which should be validated with individual-level patient data in future work. Finally, we report important comparisons of laboratory studies between North American and European sites.

Strengths and weaknesses

The nature and construction of the 4CE consortium offers a number of key strengths and requires acknowledgement of important limitations. The consortium approach enabled the pooling of laboratory values across 342 hospitals with diverse healthcare practices. This showed that site-level (within-site and between-site) differences were greater than country-level differences. Interestingly, the severity predictive performance (AUC) of each laboratory study is remarkably consistent between North America and Europe. Despite the differences in the composition of ever-severe and never-severe patients across sites and countries, the directionality of laboratory values and the threshold for ever-severe disease shared many similarities. The design of the consortium and these analyses highlight that these findings are unlikely to be site-specific or the result of health care system dynamics; the predictive nature of identified lab tests is more likely to generalize. Additionally, given the diversity of sites, the findings are unlikely to be biased by a majority population demographic.

The federated nature of analyses presented several additional limitations the consortium acknowledged and took measures to mitigate. First, EHR data have intrinsic noise, variable levels and causes of missing data, and policy effects on available documentation that will result in differences between sites. By leveraging a federated system of common EHR data elements and capturing site-level heterogeneity to identify patterns across hospitals and countries, the 4CE consortium is uniquely positioned to identify international differences in patient trajectories and hospital care that affect patient and population-level outcomes. To mitigate the intrinsic noisiness of EHR data, we performed extensive and iterative quality controls to address potential imprecision and incompleteness of datasets. Careful work with collaborators uncovered and addressed site-specific variations in data extraction and incomplete mapping of local EHR codes to desired data elements. Second, we used aggregate rather than patient-level data from each site for the current analysis in order to minimize privacy concerns and expedite the collaboration.

Aggregation enabled rapid sharing of data across sites and countries. As such, we could not assess the prediction performance of a composite risk score at patient level but instead report the predictive accuracy at the site level. To address this concern, we used comprehensive statistical tools to perform meta-analyses across sites. Third, the severity algorithm can accurately identify patients hospitalized with COVID-19 who develop severe disease, but it cannot presently identify the clinical trajectory that leads up to such a transition. Given the limitations of the severity algorithm that we previously validated, [8] we could not differentiate between patients who presented in a severe state and those who transitioned to severe disease over time. Our clinical experience and patient-level data from several institutions suggested that some patients were severe at admission while other patients transitioned to severe disease within a week of admission, sometimes suddenly. Thus, post-admission laboratory values might not capture the transition to severe disease.

Unanswered questions and future research

We plan future analyses of patient-level data at each site to fully assess the temporal trend of individual and composite laboratory tests throughout the disease course. We have established a platform of harmonized data that is enabling our consortium to study questions related to coagulopathy, acute renal failure, pediatric symptomology, neurological sequelae, and many other areas of interest of our members. As the pandemic evolves, we are also examining how the data collection assumptions made in the first wave are faring when presentations of those with chronic COVID-19 disease is now present within our health systems. Our future work related to severity prediction will optimize risk scores based on laboratory values and other clinical features to capture temporal changes in patient population and clinical care. Additionally, we are evaluating the case mix changes for hospitals over time that may influence identification and type of severe COVID-19 disease.

Meaning of the study: possible explanations and implications for clinicians and policymakers

We make several noteworthy observations. First, the key laboratory values emerging from our study represent the combination of acute inflammatory changes (CRP, LDH, procalcitonin) and underlying physiology (albumin). Both baseline health state and acuity of physiologic response to severe viral infection predict the risk of severe COVID-19. Second, the relatively low AUCs are likely due to the large variation within each site for a given laboratory value. When used in combination, prediction performance was dramatically improved. Third, the performance of the combined algorithm was relatively stable--and did not improve--over the first days of hospitalization. Stability over time implies that laboratory values at admission can be used to differentiate those patients likely to develop severe disease almost as well as models from subsequent days of hospitalization. As such, the current study examines the timing of laboratory tests in distinguishing patients who develop severe disease from those who do not.

Our systematic approach and extensive quality control allowed us to aggregate EHR data across sites and countries in a collaborative fashion at scale. The value of this work arises from leveraging these cross-country differences and similarities. While there may be dramatic differences in the outcomes of patients based on the complex interplay of comorbid conditions, natural history of the disease, and healthcare system dynamics, this fundamental similarity highlights a truism of disease: that there are clear and consistent patterns to the way humans react to disease. As we continue to build our collaborative and pursue future patient-level analyses, we will seek to uncover more subtle patterns and the role healthcare systems have in defining such perturbations.