Summary
We sought to develop and validate a multivariable prediction model of death in Mexican patients with COVID-19, by using demographic and patient history predictors. We conducted a national retrospective cohort in two different sets of patients from the Mexican COVID-19 Epidemiologic Surveillance Study. To develop the model, we included 264,026 patients tested for SARS-CoV-2 between February 28 and May 30, 2020. To validate the model, 592,160 patients studied between June 1 and July 23, 2020 were included. Patients with a positive RT-PCR for SARS-CoV-2 and complete unduplicated data were eligible. Demographic and patient history variables were analyzed through Multivariable Cox regression models to evaluate predictors to be included in the prognostic scoring system called PH-Covid19. 83,779 patients were included to develop the model; 100,000, to validate the model. Eight predictors (age, sex, diabetes, COPD, immunosuppression, hypertension, obesity, and CKD) were included in the PH-Covid19 scoring system (range of values: −2 to 25 points). The predictive model has a discrimination of death of 0.8 (95%CI:0.796-0.804). The PH-Covid19 scoring system was developed and validated in Mexican patients to aid clinicians to stratify patients with COVID-19 at risk of fatal outcomes, allowing for better and efficient use of resources.
Introduction
The coronavirus disease (COVID-19) pandemic is the global phenomenon which is shaping modern societies in the year 2020, a reason why the severe acute respiratory coronavirus 2 (SARS-CoV-2) has been named the once-in-a-century pathogen that scientists and global leaders had been worrying for [1]. Demographic and patient history risk factors for fatal outcomes in patients with COVID-19 have been characterized in large national cohorts [2–5], and broadly include old age, sex (men), comorbidities, deprivation (a correlate of poverty), and belonging to certain ethnic groups. Other clinical, radiological, and laboratory parameters at presentation have also been studied as risk factors for disease progression and death [5,6].
Low and middle-income countries often suffer from inadequate healthcare due to the lack of equipment, poor organization, and scarce qualified healthcare professionals. Thus, what works in high-income countries may not work in low-income countries [7]. Mexico has been one of the most affected countries by COVID-19, and disparate differences in patient outcomes have been noted be related to inequalities in healthcare access. Therefore, there is a pressing need to develop accessible and simple tools to aid clinicians providing medical attention in the most unfavored regions of Mexico.
Several diagnostic and prognostic models have been developed to be used in patients with COVID-19 [8]. However, most models include laboratory and radiographic variables which would be nearly impossible to collect in low-resource settings. Furthermore, these models have seldom been validated, are often inadequately reported, or are overfitted due to a large predictor to outcome ratio [8–10], reasons that may limit their usefulness in real-world settings. Developing and validating models that only require demographic and patient history data, by using large national or multinational cohorts, may be a way to overcome these shortcomings to provide useful tools to clinicians in low-resource regions.
Therefore, we sought to develop and validate a multivariable prediction model of death in Mexican patients with COVID-19, by using demographic and patient history predictors.
Method
Study Design
We conducted a national retrospective cohort in two different sets of patients from the Mexican COVID-19 Epidemiologic Surveillance Study [11] to develop and validate a multivariable prediction model of death in Mexican patients with COVID-19. Patient history variables were used as predictors of death as the outcome of interest. Blind assessment was not required since these are objective variables unlikely subjected to bias.
To develop the model, we included 264,026 patients studied between 28 February and 30 May, 2020. All patients with a positive RT-PCR for SARS-CoV-2 were included to maximize the power and generalizability of results. Patients with incomplete data were excluded, whereas patients with the same demographic, clinical, and follow-up variables were considered duplicated and only one entry was kept.
To validate the model, we included 592,160 patients studied between 1 June and 23 July, 2020. Only patients with a positive RT-PCR for SARS-CoV-2 and complete unduplicated data were included to validate the model. We further performed simple random sampling of positive cases to increase statistical power in approximately 15% with respect to the sample used for developing the model.
Source of Data
Data is collected and regularly updated by the Mexican Secretariat of Health and is available in the Open Data platform of the Federal Government of Mexico [12]. A historical repository of individual datasets starting on 12 April 2020 is available through the General Directorate of Epidemiology [13]. Patients who met criteria of suspected COVID-19 case and were subsequently tested for SARS-CoV-2 were included in the study, starting on late February 2020 when the first suspected cases arrived in Mexico. Two diagnostic strategies are outlined in the National COVID-19 Epidemiologic Surveillance Plan [11]: 1. testing of 10% of ambulatory patients with mild symptoms of respiratory disease and 100% of patients with respiratory distress at evaluation in one of the 475 monitoring units of viral respiratory disease (USMER, for its acronym in Spanish) which are strategically distributed to be representative of the Mexican population, and 2. testing 100% of patients who meet diagnostic criteria of Severe Acute Respiratory Infection (defined as shortness of breath, temperature ≥38 °C, cough, and ≥1 of the following: chest pain, tachypnea, or acute respiratory distress syndrome) who seek medical attention in non-USMER units.
Healthcare professionals collecting a diagnostic specimen are required to fill out a format containing demographic and patient history variables. Follow-up of all suspected COVID-19 cases is registered by accredited hospital epidemiologists (inpatients) and the responsible healthcare professional of every Local Health Jurisdiction (ambulatory patients), who ultimately upload data into the Respiratory Diseases Epidemiologic Surveillance System. Results of diagnostic RT-PCR for SARS-CoV-2 are directly uploaded by the diagnostic facility; accreditation of diagnostic procedures by the Mexican Institute of Diagnostics and Epidemiological Reference is required to upload results. Reporting of deaths is obligatory and must be done in less than 48 hours after occurrence. One caveat to this reporting method is that patients who are tested more than once in different jurisdictions may be duplicated. No variables that could lead to identification of patients are provided in datasets. Thus, the only way to eliminate duplications is through matching of cases with equal demographic and clinical variables. Specific information of treatments is not released.
Variables provided in the datasets are: origin (USMER, non-USMER), healthcare provider, state, birthplace, place of residency, nationality, indigenous language speaker, migratory status, type of medical attention (hospitalization/ambulatory), admission date, symptom onset date, invasive mechanical ventilation (intubation [yes/no]), admission to ICU (yes/no), pneumonia (yes/no), date of death, contact with confirmed COVID-19 cases (yes/no), SARS-CoV-2 RT-PCR result (positive, negative, or pending), age, sex, current pregnancy, and the following comorbidities (yes/no): diabetes, hypertension, obesity, cardiovascular disease (CVD), chronic kidney disease (CKD), immunosuppression, asthma, chronic obstructive pulmonary disease (COPD), and smoking.
Statistical Analysis
Descriptive data were calculated and are provided as frequencies, percentages, or mean with standard deviation (SD). Characteristics of patients in the model development and validation cohorts were compared through Student’s t-test or X2 A Cox regression model was applied to predict the risk of death. The risk of death was assessed through univariate analysis of the following variables: age, sex, current pregnancy, diabetes, COPD, asthma, immunosuppression, hypertension, CVD, obesity, CKD, smoking, and time from symptom onset to medical attention. Risk factors associated with death which were statistically significant (p< 0.05) were included to develop the multivariate regression model. All variables with a level of significance p< 0.1 were considered in the Cox regression model by using the Enter method. Variables that kept a level of significance p< 0.05 were used in the final model, which was evaluated through Harrell’s C-statistic to determine its discrimination of death.
A scoring system was developed in accordance to the model proposed by Sullivan LM, et al. [14] Each risk factor was organized into categories and the reference value was determined as follows. Age was entered into the Cox regression model as a continuous variable and further categorized into 10 sets of years (< 20, 20-29, 30-39, 40-49, 50-59, 60-69, 70-79, 80-89, 90-99, and > 90); the midpoint between the nine values of each category was set as the reference value (Wij). The reference for age as a risk factor (WiREF) was set in the 20-29 years category since this group had the lowest mortality in a previous study performed in Mexican patients with COVID-19 [3]. The rest of the risk factors were modeled as dummy variables, setting the absence of comorbidities and being woman (sex) as the reference values. Each WiREF was subtracted from Wij and multiplied by the regression coefficient (βi) of the risk factor to determine units of regression of distancing for every risk factor in the reference category βi(Wij - WiREF). A constant B, defined as the constant increase in units of risk for each 5-year increase, was obtained by multiplying βi • 5. Values for B were 0.25 in this study. Scores for each category were obtained with the equation βi(Wij – WiREF)B. For every point in the risk score, the estimated risk of death (p) was calculated with the Cox proportional hazards regression analysis:
Where: S0(t) is the average survival according to mean values of every risk factor; is substituted with each value of the risk score, times the B constant, plus the reference age value according to the βi of age; and
is the sum of every βi times the proportion or mean value of every risk factor.
To validate the model, we calculated the risk score for every patient and applied the Cox proportional hazards regression analysis. Estimated risks were obtained from both the scoring system and the observed risk in the regression analysis. The values obtained for patients in the validation cohort were distributed in percentiles (1 to 99) to determine the punctuation categories. The estimated and observed risks were compared in each category of punctuation (percentiles 25, 50, 75, and 99).
The Kaplan-Meier analysis was performed to determine survival in each category of the scoring system, and a Cox regression analysis, to determine increases in the lethality risk for each category.
The association between the risk score and the probability of other adverse events (hospitalization, invasive mechanical ventilation, pneumonia, and admission to ICU) were also studied. The frequencies of each adverse event for every category in the scoring system were quantified and a binomial logistic regression analysis was performed to determine the risk of each adverse event according to the risk score; logarithms of the odds ratios (OR) were graphed to stablish the scoring value at which risk for every adverse event is increased.
All statistical analyses were performed using the SPSS software v.21 and the R statistical software v.3.6.2; figures were created in GraphPad Prism v.6. A value of p< 0.05 was used to stablish statistical significance.
Results
Out of 264,026 patients in the model development cohort, 84,627 had a positive RT-PCR for SARS-CoV-2, 140,553 were negative, and 38,846 had pending results. After exclusion of patients with incomplete data and duplicated registries, 83,779 patients with a positive test were included to develop the model. Among the 592,160 patients in the model validation cohort, 256,488 patients had a positive result, 253,447 were negative, and 82,225 had unreported results. After excluding duplicated and incomplete registries, and random sampling of positive cases, 100,000 patients were included to validate the model. Descriptive values of demographic characteristics, patient history, and outcomes of patients (survivors and non survivors) in both cohorts are provided in Table 1.
Demographic characteristics, patient history data, and outcomes in the model development and validation cohorts.
Variables included in the Cox regression model are presented in Table 2. Eight risk factors were included in the model; the only quantitative variable was age. Age, diabetes, and CKD were associated with the greatest increases in death. The predictive model has a discrimination of 0.8 (95%CI:0.796-0.804) and an average survival of 0.903 with the mean values for every risk factor. The Patient History COVID-19 (PH-Covid19) scoring system assigns a score to every risk factor ultimately included in the predictive model (Table 3); the sum of scores for all risk factors included ranges from −2 to 25 points. Predicted probabilities of death in patients with a positive test for SARS-CoV-2 for every possible total value in the scoring system range from 0.74% to 99.82% (Table 4).
Risk factors associated with death in Mexican patients with a positive diagnostic test for SARS-CoV-2 (model development cohort)
The PH-Covid19 risk score to predict death in patients with COVID-19
Estimated risk of death according to every possible score in the PHCovid19 score, in Mexican patients with a positive test for SARS-CoV-2
Baseline characteristics and outcomes of patients in the validation cohort were statistically significantly different from those in the model development cohort, except for time from symptom onset to medical attention (Table 1). Results of the Cox regression model applied to the validation cohort are provided in Supplementary Table S1. Calculated scores in the validation cohort reflected the following distribution according to percentiles: −2 to 2 points, percentile 1 to 25; 3 to 5 points, percentile 25-50; 6 to 8 points, percentile 51 to 75; and 9 to 15 points, percentile 76 to 99. Patients > 99 percentile were considered as extreme values. Estimated risks and the observed risks of death obtained from the Cox proportional hazards regression analysis, in relation to the scores in each percentile, are presented in Fig. 1. The estimated and observed risks were similar for each group and were strongly correlated (r = 0.98, R2 = 0.96, p< 0.0001).
In the survival comparison between categories generated according to percentiles (Fig. 2), survival was decreased with increasing risk scores (−2 to 2 points, 99.6%; 3 to 5 points, 98.6%; 6 to 8 points; 95.7%, and 9 to 15 points, 84.3%, p< 0.0001 for all categories). A subsequent Cox regression analysis was performed to determine the risk of death for each individual category compared to the category comprising the lower percentile (−2 to 2 points): 2 to 5 points, HR:3.54 (95-CI:2.85-4.39); 6 to 8 points, HR:10.67 (95%CI:8.78-12.98); 9 to 15 points, HR:41.9 (95%CI:34.7-50.7); and > 15 points, HR:87.6, (95%CI:70.1-109.5). In accordance with this, risk groups were derived from the scoring system: Low (−2 to 2 points), Medium-Low (3 to 5 points), Medium (6 to 8 points), Medium-High (9 to 15 points), and High (>15 points).
The prevalence for every adverse event in each category of the scoring system are provided in Fig. 3; results of the logistic regression analysis to determine the risk of adverse events for each risk group in the PH-Covid19 scoring system are given in Supplementary Table S2. Tendencies of risk increasement for each adverse event with augmenting scores (Supplementary Figure S1) reflect that risk for any adverse event starts at a value of 5 points.
Discussion
We have developed and validated the PH-Covid19 score, a multivariable prediction model of death in Mexican patients with COVID-19, by using different datasets from the Mexican COVID-19 Epidemiologic Surveillance Study. This scoring system has been created to aid clinicians working in resource-strained conditions to early stratify patients with COVID-19 according to their risk of fatal outcomes, without the need to perform laboratory or imaging studies.
Similar to other studies, we found that older age is the main risk factor for dying from COVID-19, with every 10-year increase associated with the largest increases in hazard ratio [2,5,15]; in the PH-Covid19 score, 10-year increases added two points, starting from 30 years, and being < 20 years subtracted 2 points. Sex (men), which increases two points in our score, was correlated with death in our study and others [2,3,5,6,15]. Diabetes and CKD resulted in two-point increases in the score. Other studies had similar findings to ours [3,4,6,15]; one study found an uncertain increased risk attributable to diabetes (HR:1.14, 95%CI:0.91-1.43) in critically-ill patients [5]; in another study, uncontrolled diabetes further increased the risk (HR:1.95, 95%CI:1.83–2.08) with respect to controlled diabetes (HR:1.31, 95%CI:1.24-1.37), while CKD stages 4-5 caused a greater increase (HR:2.52, 95%CI:2.33-2.72) than CKD stages 3a-3b (HR:2.52, 95%CI:2.33-2.72) [2]. Obesity had the third strongest association with death, resulting in a 1-point increase in our scoring system; other studies have had similar findings [3–15]. Adjusted risk of death by obesity occurred with gradual increases in higher obesity classes2 and no evident or clear risk was identified in another study for class I and II obesity, whereas class III obesity was significantly correlated with death[5]. This last study, however, included only patients admitted to ICUs who had a high prevalence of comorbidities, which are independent risk factors for ICU admission [3].
Hypertension was a risk factor for death similar to other studies [3,4], and resulted in a 1-point increase in the scoring system. In one large study, adjusted risk of death for hypertension adjusted by all covariates apparently reduced the risk (HR:0.89, 95%CI:0.85–0.93), likely reflecting an artificial reduction of risk driven by diabetes and obesity since age and sex adjusted risk increased the risk (HR:1.09, 95%CI:1.05–1.14)[2]. COPD also increased 1 point in our model, being also associated to death in other studies [2–4,15].
Results from our validation did not provide different patient history predictors which could enhance the performance of our model. Thus, model updating was not required. Some of our findings raise concerns regarding quality and access to healthcare in Mexico. 10.3-13.5% of patients who died in our cohorts did not receive in-hospital care at any moment of the disease, only 15.5- 18.3% of patients were admitted to an ICU before dying, and only 21.3-28.1% of patients who died were intubated; conversely, intubation in survivors was unusually low (0.8-1.3%). In other studies of hospitalizedonly patients, 53-72% of non-survivors were admitted to an ICU and 51-59% received invasive mechanical ventilation [6,16,17]. Observed mortality in COVID-19 patients under 60 years is lower when access to healthcare is not a limitation [18]; non-survivors in our cohorts were younger (mean age 60-61.4 years) than those in other studies (67-80 years) [5,6,15,19].
Patients diagnosed with pneumonia in both cohorts were 12.8-21% among survivors and 77.4-87.5% of non-survivors. These numbers are low compared to the prevalence of chest CT-scan abnormalities which occur in 67.3-70.8% of asymptomatic/pre-symptomatic patients [20,21], 95.5% of patients with mild COVID-19 [22], and 98% of all COVID-19 patients included in a meta-analysis [23]. Chest X-ray, on the other hand, may be normal in up to 63% of patients with early COVID-19 pneumonia [24]. Nonetheless, the low proportion of pneumonia in non-survivors suggests non-optimal diagnosis of pneumonia may be occurring in Mexico. The lack of an operational definition may have contributed since clinicians could have defined pneumonia differently based on clinical and/or radiographical findings. Other possibilities should be explored, including knowledge of Mexican clinicians on how to diagnose pneumonia and access to radiological studies during the pandemic in low-resource settings.
Three prognostic COVID-19 models have been developed in Mexican patients. The LOW-HARM model is a 100-point scoring system calculated by inputting patient history and laboratory values, setting 65 points as the cut-off value of greater discrimination of risk of death [25]. Prediction of death (0.81, 95%CI:0.77–0.84) is similar to the PH-Covid19 scoring system (0.80, CI95%:0796-0.804), which advantageously only requires patient history predictors. Another scoring system uses age (cut-off 65 years), comorbidities, and pneumonia to predict death [26]. This model was accurate at predicting death and other adverse events but has the limitation of not accounting for the large increases in risk for every 10-year category or similar. Also, the model by Bello-Chavoya, et al. was developed using one dataset of the Mexican Epidemiologic Surveillance Study, which unfortunately had no operational definition for pneumonia as discussed earlier. A third model was developed and validated to predict the risk of admission to ICU; the ABC-GOALS was developed in three versions: clinical, clinical + laboratory, and clinical + laboratory + imaging predictors [27].
In one systematic review of existing prediction models for diagnosis and prognosis of COVID-19, the use of any of the reviewed models was discouraged since, out of 91 diagnostic and 50 prognostic models, all were at high risk of bias due to methodological constraints and poor reporting [8]. Predictive models of death (eight) often excluded patients that had not developed the outcome of interest, did not account for censorship, inadequately reported discrimination and calibration of the model, and had a high risk of bias according to PROBAST evaluation, despite authors claiming good global performances of their models. We have addressed these concerns in our development and validation of the PH-Covid19 scoring system.
Another strength of our study is that we were able to perform a type 3 analysis according to TRIPOD by using individual datasets to develop and validate our model; this design allows for external validation of the performance of a model [28]. It is worth highlighting that sample sizes in both cohorts include thousands of patients, which adds robustness to our model.
One limitation of our study is that certain diseases (cancer, hematological malignancies, and neurologic diseases) and specific states of a disease (obesity class, former or current smoker, and control of diabetes, hypertension, and asthma) which increase the risk of dying from COVID-19 [2] were not studied since they are not provided in the datasets. However, our model accounts for the main risk factors associated with death in patients with COVID-19, and not requiring inputting specific disease states makes it easier to be used by clinicians while minimizing the risk of not having enough data to use the score with precision.
Another limitation is that the epidemiologic surveillance strategy in Mexico allows testing of only 10% of ambulatory patients. Furthermore, the operational definition of suspected COVID-19 case used in Mexico until August 24, 2020 had a low sensitivity (58.2%), but a high specificity (63.7%) compared to that used by the CDC (85.8% and 25.8%, respectively) [29]. Altogether, this means that our cohorts may include very few patients with asymptomatic COVID-19 and less patients with mild COVID-19 compared to other national datasets with higher testing rates. Since it was not possible to determine the exact number of patients with mild disease in our cohorts, we can only indirectly suggest that mild-disease patients could comprise around 71.5 and 84.4% of patients according to the fraction of non-hospitalized patients who survived, a number high enough to permit the use of this score in patients with mild-to-severe COVID-19.
The PH-Covid19 score was created to be used in limited-resource settings where access to laboratory and imaging studies may be restricted. In places where this is not a limitation and for patients who are likely to have already been admitted to hospital (critical patients), other prognostic models may have a better performance than ours. However, clinicians should consider that most models have not been validated before deciding to use any COVID-19 diagnostic or prognostic model.
The PH-Covid19 score uses patient history predictors which are frequently known at the first contact with a patient, or can be interrogated rapidly, to predict death in patients with COVID-19. This score was developed and validated in Mexican patients to be used in low-resource settings where obtaining laboratory and radiographic studies may not be immediately possible. This score will aid clinicians to stratify patients with COVID-19 at risk of fatal outcomes to use healthcare resources more efficiently.
Data Availability
all data is free for your consultation
Ethical disclosures
This article is a retrospective study of an open-source dataset of patients in Mexico. The Ministry of Health of Mexico approved the recollection of information.
Declarations of interest
None
Funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Acknowledgements
J M-G would like to thank “Dirección General de Calidad y Educación en Salud” for supporting his participation in “Programa Nacional de Servicio Social en Investigación en Salud”.
Footnotes
Funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.