Abstract
Background Causal interpretation of findings from existing epidemiological studies on long-term clinical outcomes of coronavirus disease 2019 (COVID-19) may be limited by the choice of comparator (control) group.
Objective We compare two approaches to control group selection (based on requirement for negative SARS-CoV-2 test for eligibility) in long-term clinical outcomes after COVID-19 in patients with history of heart failure (HF).
Design Retrospective cohort study using data from February 1, 2020 to July 31, 2021. Setting: Veteran Health Administration (VHA).
Participants We studied two cohorts of Veterans with COVID-19 and history of HF which selected comparison group using two different approaches. In Cohort I, Veterans with HF who tested for positive for SARS-CoV-2 were age, sex, and race matched to Veterans with no evidence of COVID-19 in 1:5 ratio. In Cohort II Veterans with HF who tested positive for SARS-CoV-2 were age, sex, and race matched with Veterans with HF who tested negative for SARS-CoV-2 within +/-15 days of the positive test date within the same VHA facility.
Exposure COVID-19 as determined by a positive SARS-CoV-2 test.
Main Outcomes and Measures 1-year all-cause mortality and hospital admissions beyond the first 30 days after COVID-19 diagnosis. Adjusted hazard ratios (HRs) accounting for comorbidity and 95% confidence intervals were calculated.
Results Cohort I comprised 13,722 Veterans with HF with COVID-19 (mean [SD] age 72.0 [10.2] years, 2.4% female, 71.1% White) and 60,956 matched controls not known to have COVID-19. Cohort II comprised 6,725 Veterans with HF with COVID-19 (mean [SD] age 72.5 [7.5] years, 0.1% female, 80.8% White) and 6,726 matched controls with negative SARS-CoV-2 test. The adjusted HRs for 1-year mortality and hospital admission beyond the first 30 days after diagnosis of COVID-19 were 1.40 (1.32-1.49) and 1.34 (1.28-1.41), respectively, in analysis of Cohort-I (where the comparator group was not required to test negative for SARS-CoV-2). However, in Cohort-II (using the second comparator group specifying negative SARS-CoV-2 test for eligibility), the associations were markedly attenuated; adjusted HRs 1.05 (0.95-1.17) and 1.07 (0.96-1.19), respectively.
Conclusions We found significant attenuation of associations between COVID-19 and long-term risk of mortality and hospital admissions beyond the first 30 days among patient with existing HF, when comparing with a control group selected based on a negative SARS-CoV-2 test versus control group not known to have COVID-19. The findings have implications for the design of studies of long-term CVD (and non-CVD) outcome of COVID-19.
Matter Arising (Letter Format)
We read the manuscript on long-term cardiovascular outcomes of Coronavirus disease 2019 (COVID-19) by Al-Aly et al published in Nature on 22 April, 2021, with great interest.1 In a retrospective cohort study using data from the Veterans Health Administration (VHA), the authors showed that “a substantial burden of health loss that spans pulmonary and several extrapulmonary organ systems is experienced by patients who survive after the acute phase of COVID-19.” The authors have also recently published another article in Nature Medicine on a wide array of long-term cardiovascular outcomes among survivors of COVID-19 using a similar approach.2 For these analyses, the authors sampled a comparison group of “VHA users who did not have a positive test for COVID-19”. They did not require all individuals in the comparator group to test negative for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Based on some of our own analyses of a smaller subset of VHA data, we believe Al-Aly et al.’s study may benefit from an additional stability analysis, specifically requiring that all individuals in the comparison group were also tested for SARS-CoV-2 and negative. This allows for the eligibility criterion of having a SARS-CoV-2 test is uniformly applied to both exposure groups, i.e., the COVID-19 and the non-COVID-19 comparator groups.
Al-Aly et al used rigorous methods to control for confounders. They first identified a set of pre-defined and empirically identified covariates, then included these in a regression model to estimate propensity scores. These propensity scores were then used to adjust for confounding through inverse probability of treatment weights. The analytic approach to address measured confounding was sound.3 However, as the authors rightly acknowledge in their limitations, residual confounding remains a challenge when making causal inferences using observational data. We are concerned that the design of Al-Aly and colleagues’ study may have resulted in intractable residual (unmeasured) confounding by not requiring the exposed (COVID-19) and unexposed (non-COVID-19 comparator) groups to be exchangeable in their testing status. We suggest that those being tested for SARS-CoV-2 (whether positive or negative) are fundamentally different from the population of Veterans who are not tested. Further, we propose that only those who were tested for SARS-CoV-2 and who were negative should be sampled as the comparison group, and that time zero for all eligible Veterans should be the date of the test (or, if using discrete time, the time in which the test occurred). Al-Aly et al. implicitly required that only the COVID-19 group be tested, thus differentially applying an eligibility criterion to the exposed but not the unexposed group. The unexposed group was therefore a population of Veterans among whom an unknown fraction was tested.
Our ongoing analyses of a smaller subset of VHA data suggest that Veterans who were tested for SARS-CoV-2 were markedly different from the broader source VHA population, irrespective of whether they tested positive or negative for SARS-CoV-2. Veterans who were tested for SARS-CoV-2 may have a different distribution of socioeconomic factors (e.g., job type, income, living condition)4,5and behavioral factors (e.g., knowledge of prevention methods, risk behavior, health seeking behavior)5,6 from those who were not tested, some of which may be unmeasured. They may also have a different burden of illness that is inadequately captured by the information in the VHA databases. The data presented here provide some evidence that a better approach to address confounding may be to sample the unexposed group from among those individuals who tested negative for SARS-CoV-2, thereby applying the testing eligibility criterion uniformly regardless of exposure status. To demonstrate this, among Veterans with heart failure (HF), we identified those who developed COVID-19 between 2/1/2020 and 7/31/2021. The COVID-19 group (HF patients testing positive for SARS-CoV-2) were then compared to two different comparator groups of Veterans with HF selected using two strategies: Comparator group 1 - Veterans without a known diagnosis of COVID-19 were matched on age, sex and race to those with positive SARS-CoV-2 test, in a 1:5 ratio (‘Cohort-1’). For the exposed individuals in Cohort-1, time zero was the date of the test. For the unexposed individuals, time zero was assigned based on the distribution of the dates of time zero in the exposed group. Comparator group 2 – Veterans who tested negative for SARS-CoV-2 were age-, sex-, race-matched to COVID-19 group in 1:1 ratio, within the same facility within +/- 15 days of the SARS-CoV-2 positive date (‘Cohort-2’). Time zero in the second comparator group was defined by the test date of SARS-CoV-2. The Comparator group 2 sampling strategy allowed us to account for geographic and time variation in the COVID-19 epidemic and, most importantly, allowed us to ensure that all individuals received a SARS-CoV-2 test regardless of their exposure status (COVID-19 or non-COVID-19 comparator). Cohort 1 (based on the first comparator selection strategy) comprised 13,722 HF patients testing positive for SARS-CoV-2 and a comparator group of 60,956 HF patients with “no evidence” of COVID-19. Cohort-2 (based on the second comparator selection strategy) comprised of 6725 HF patients testing positive for SARS-CoV-2 and a comparator group of 6766 HF patients who definitively tested negative for SARS-CoV-2 (Table).
As shown in the Table, the adjusted hazard ratios (HRs) for 1-year mortality and hospital admission beyond the first 30 days after diagnosis of COVID-19 were 1.40 (1.32-1.49) and 1.34 (1.28-1.41), respectively, when using the first comparator group (i.e., analysis of Cohort-1). However, in Cohort-2, the associations were markedly attenuated – adjusted HRs 1.05 (0.95-1.17) and 1.07 (0.96-1.19), respectively. This may suggest the presence of residual confounding in the first set of analyses despite adjusting for a wide array of covariates, possibly due to unmeasured or inaccurately measured confounders. As shown in the Figure the standardized mean differences (SMDs) comparing covariates between the COVID-19 group and the comparator group were all ≤0.1 for Cohort-2 except for EF categories (SMD = 0.11) while for Cohort-1, 14 variables had SMD >0.1, indicating that the exposed and unexposed groups were more balanced in terms of the measured covariates in Cohort-2. As these covariates are known to be associated with the outcome of interest, this suggests that in Cohort-2 confounding may have been mitigated through the study design decision to require all individuals (both exposed and unexposed) to have received a SARS-CoV-2 test. Those tested for SARS-CoV-2, even when testing negative, had higher comorbidity burden (Supplement Table), and had higher hospitalization and mortality rates (Table) compared to those who were not tested. It is possible that differences in unmeasured or inadequately measured covariates may follow the same pattern as observed for measured covariates.
Sampling exposed and unexposed individuals with the same eligibility criteria, including SARS-CoV-2 testing status, appears to potentially account (through design) for some residual confounding. In addition, specifying negative SARS-CoV-2 testing for eligibility to comparator group is important to reduce bias arising from misclassification of the exposure. It is estimated that at least 40% SARS-CoV-2 infections are asymptomatic,7 and since the comparator group inclusion criteria by Al-Aly et al did not require a negative for SARS-CoV-2, these asymptomatic SARS-CoV-2 cases would be included in their study as controls resulting in non-random misclassification of the exposure. If one makes a reasonable assumption that individuals with asymptomatic SARS-CoV-2 infection are less likely to develop long-term sequala (compared to those with symptomatic infection), then this non-random misclassification will cause a bias away from the null resulting in associations that are stronger than the “true” relationship existing in the population. The assumption that asymptomatic SARS-CoV-2 infections (compared to symptomatic infections) are less likely to be associated with long-term clinical outcomes is consistent with the observation by Al-Aly et al of a graded increase in hazard ratios of long-term outcomes comparing COVID-19 cases who, (i) were not hospitalized, (ii) were hospitalized, or (iii) were admitted to intensive care unit, with their control group, as shown in Figure 3 of their article in Nature.1
Another explanation for the differences in the results between the two approaches might be differences in the distributions of effect measure modifiers introduced by selecting a different comparator group. Hence, the differences in the estimates that we observed may not necessarily represent an improved confounding control; rather, it may represent different overall distributions of effect measure modifiers between the cohorts. It is also possible that our sampling strategy of selecting the comparator individuals within the same facility and within the same fine strata of calendar time as the individuals with COVID-19, rather than our requirement that both the exposed and unexposed group should meet the same eligibility SARS-CoV-2 testing criterion, could be more responsible for any reductions in residual confounding or selection bias.
Nonetheless, when studying long-term COVID-19 outcomes, we believe the approach demonstrated here of sampling the unexposed comparator individuals from among those tested for SARS-CoV-2 at the same time and place as the COVID-19 exposed group allows the exposed and unexposed groups to be more exchangeable in their covariate distributions even before any statistical adjustment. We hope that Al-Aly et al. will consider this additional stability analysis to shed additional light on their findings. A similar comparator group selection strategy has been implemented in the design of at least one other currently on-going study of long-term CVD outcomes among survivors of COVID-19.8 If the results of Al-Aly et al. persist after applying the eligibility criteria (including SARS-CoV-2 testing) to both exposed and unexposed groups uniformly, that information will be a very important contribution to scientific knowledge. If the results are not robust to this change, that will guide future investigators to avoid sampling a mixture of individuals who were and were not tested for SARS-CoV-2 as an unexposed comparator group.
Data Availability
Based on restrictions in the Data Use Agreements used in this study, the authors are unable to make a data set available. Methodology questions may be directed to the corresponding author.
Acknowledgement
The reported/outlined here was supported by the Department of Veterans Affairs, Veterans Health Administration, VISN 1 Career Development Award to SE. SE is also funded by Center for Aids Research, The Rhode Island Foundation, and Lifespan Cardiovascular Institute. JLR, WW, SE, and LJ were supported by the VA Health Services Research and Development Center of Innovation in Long Term Services and Supports (CIN 13-4193; C19-20-213). ARZ was supported in part by grants from the National Institute on Aging (R21AG061632, R01AG065722, RF1AG061221, R24AG064025, and R01AG062492). GC was supported by NIGMS P20GM103652, VA CSR&D grant I01CX001892, and NHLBI R01HL148727. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government.