Abstract
Background Randomised trials conducted before 2021 indicated that vitamin D supplementation (VDS) was able to prevent severe COVID-19 and acute respiratory infections (ARI). However, these health benefits were not confirmed by larger randomised trials published after 2021.
Objective To examine the characteristics of randomised trials on VDS to COVID-19 patients and admission to intensive care unit (ICU), and on VDS for the prevention of ARI.
Method A systematic search retrieved randomised trials on VDS to COVID-19 patients and admission to ICU. Data on VDS and ARI were extracted from the meta-analysis of Jolliffe et al., 2021. The associations between VDS vs no VDS, and admission to ICU were evaluated using random effect models. Meta-analyses were done for all trials and by groups trial size. Publication bias was assessed using the LFK index (no bias if index between -1 and +1) and the Trim and Fill method.
Results Nine trials on VDS for preventing admission to ICU were identified. The summary odds ratio (SOR) was 0.61 (95%CI: 0.39-0.95) for all trials, 0.34 (0.13-0.93) for trials including 50 to <106 patients and 0.88 (0.62-1.24) for trials including 106 to 548 patients (effect modification: p=0.04). The LFK index was -3.79, and after Trim and Fill, the SOR was 0.80 (0.40-1.61). The SOR for the 37 trials on VDS for ARI prevention was 0.92 (0.86-0.99) for all trials, 0.69 (0.57-0.83) for trials including 25 to <248 patients and 0.98 (0.94-1.03) for trials including 248 to 16,000 patients (effect modification p=0.0001). The LFK index was -3.11, and after Trim and Fill, the SOR was 0.96 (0.88-1.05).
Conclusion Strong publication bias affected randomised trials on VDS for the prevention of severe COVID-19 and of ARI. Systematic reviews should beware of small-size randomised trials that generally exaggerate health benefits.
Introduction
When the COVID-19 pandemic struck the World in 2019-2021, many thought that vitamin D supplementation (VDS) was an option for reducing the risk of COVID-19 and progression to severe disease [1–3]. The VDS option was nurtured by a wealth of data.
First, laboratory studies have documented the involvement of physiologically active forms of vitamin D in immunological mechanisms dealing with infectious agents [4, 5] [6, 7].
Second, the serum concentration of 25-hydroxyvitamin D (hereafter s-25OHD) is usually considered reflecting individual vitamin D status. Numerous observational studies done before and after the pandemic onset found first that patients with low s-25OHD were at higher risk of severe acute respiratory infections (ARI) and severe COVID-19 [8–10], and second that COVID-19 patients taking VDS had less severe disease course [11].
Third, a meta-analysis of aggregated data from 37 trials conducted between 2006 and 2019 found that VDS was associated with a 8% (95% confidence interval [CI]: 1% to 14%) reduction in the risk of ARI [12].
Fourth, randomised trials published in 2020 and 2021 obtained results compatible with a favourable effect of VDS on COVID-19 outcomes [13].
However, the promises of the VDS option were tempered by the publication in 2022 of large size randomised trials that found no effect of VDS on the incidence and severity of COVID-19 [14–16]. In addition, six Mendelian randomisation studies examined the risk of incidence and severity of COVID-19 in patients whose s-25OHD is lower than average because of their genetic background. Patients with naturally low s-25OHD had no greater risk of COVID-19 incidence or severity than patients with higher s-25OHD [17–22].
The disillusionment caused by Mendelian and large-size randomised trials called for a review of the thread of data which resulted in considering VDS as an option for the treatment of COVID-19. This article is a systematic review with meta-analysis of randomised trials on VDS for the prevention of severe COVID-19 in patients infected with the SARS-CoV-2 virus. This article also revisited the meta-analysis of randomised trials of VDS done before 2020 for the prevention of ARI [12].
Methods
This review was not registered. A scoping review of the literature showed that the most robust outcome indicating severe COVID-19 disease was the admission to intensive care unit (ICU). Other possible outcomes were less robust (e.g., admission to hospital), or possibly influenced by subjectivity (e.g., self-reported dyspnoea, resolution of symptoms, duration of hospitalisation).
We performed a systematic search restricted to PubMed of randomised trials published since January 2000 on VDS and outcomes of COVID-19 patients aged 18 years or more. The search was done in two successive steps (S1 Text). We also searched references cited in review articles.
Studies were selected if (i) they reported on randomised trials that compared rates of admission to ICU or COVID-19 patients receiving VDS to patients not receiving VDS, (ii) VDS was the unique intervention, (iii) COVID-19 patients enrolled in trials were not taking VDS before trial enrolment, (iv) comparison groups included COVID-19 patients who did not receive any type of VDS during trial courses, and (v) published in English language. All forms of vitamin D were considered (ergocalciferol (vitamin D2), cholecalciferol (vitamin D3), calcifediol (25-hydroxycholecalciferol), calcitriol (1α,25-dihydroxycholecalciferol)).
Key study characteristics and data of selected studies were extracted by PA and PM, and independently verified by GD and OD. Extracted information were first author, publication year, patient numbers by randomization group, intervention, comparison groups (i.e., whether the VDS intervention was tested against a placebo, or no placebo), and numbers of patients with outcome by randomization group. Key data on patient numbers and outcomes (i.e., odds ratios [OR] and 95% CI) included in the meta-analysis of Jolliffe et al., 2021 [12] were extracted by PA and PM.
For trials in COVID-19 patients, risks of admission to ICU were computed as odds ratios with 95% CI following a per-protocol approach. Hence, statistical analyses were done using data from patients who took the VDS or the placebo (when applicable) according to trial protocols.
Standard errors (SE) of odds ratios (OR) were derived from the equation SQRT(1/a+1/b+1/c+1/d), where a and b are the numbers of patients admitted to ICU in the VDS and in the comparison group, respectively, and c and d are the numbers of patients not admitted to ICU in the VDS and in the comparison group, respectively.
For the meta-analysis on VDS of COVID-19 patients and admission in ICU, we used random-effect models to calculate summary odds ratios and 95% CI of admission to ICU of COVID-19 patients.
Random-effect models necessitate the estimation of the between-study variance τ2. Various methods exist for computing an estimator of the between-study variance τ2 and for estimating its confidence interval. Mathematical simulations have shown that the widely used Der Simonian-Laird estimator is known to be negatively biased in scenarios with small studies and in scenarios with a rare binary outcome [23].
Because most randomised trials found in the systematic search were of small and medium size, i.e., from about 40 to 400 patients enrolled, we opted to use the REML
(Restricted Maximum Likelihood) estimator, and the REML estimator combined with the HKSJ (Hartung-Knapp-Sidik-Jonkman) method for estimating the 95% CI [24, 25].
For the meta-analysis on VDS for ARI prevention [12], we computed pooled estimate applying the Der Simonian-Laird method in order to reproduce the original analysis.
Then for both meta-analyses, statistical heterogeneity was evaluated through the I2 index: a value <50% was considered indicating an absence of statistically significant between-study heterogeneity. To investigate the between-study heterogeneity and the stability of the pooled estimate, we carried out a sensitivity analysis by excluding from the meta-analysis one of the trials at a time.
We evaluated the presence of possible publication bias, using Egger’s regression test, and the Makaskill test. We also used the method proposed by Furuya-Kanamori et al. [26], which displays in a Doi plot the weight of each trial (i.e., the sample size of each trial relative to the size of all trials combined, expressed as a Z-score) against the natural logarithm of odds ratios reported by trials. The LFK index derived from the Doi plot provides a quantitative estimate of the asymmetry of odds ratios included in the meta-analysis. A LFK index of zero indicates an absence of asymmetry. Asymmetry, and thus the possibility of publication bias, is suggested by a LFK index less than -1 (studies not in favour of VDS less likely to be published) or greater than +1 (studies in favour of VDS less likely to be published).
We performed an exploratory analysis of publication bias using the Trim and Fill method [27], which provides summary odds ratios adjusted for publication bias.
We studied the effect of the sample size through a subgroup analysis, by dividing the trials based on their sample sizes (lower or greater than the median), and including in our statistical models the trial size as a possible effect modifier.
For trials on VDS in COVID-19 patients and admission to ICU, we assessed seven sources of bias following Cochrane’s rules. We attributed to each source a value +1 if the risk of bias was low (L), 0 if it was undetermined (U) and -1 if it was high (H). Then we created a summary score 1 which was the summation of values attributed to the seven sources. A summary score 2 was the summation of sources with low risk of bias (i.e., sources with +1 value).
To evaluate the influence of the risk of bias on results, we fitted meta-regression models to verify whether the quality scores were effect modifiers.
All reported p-values were two sided and p<0.05 was considered statistically significant. Meta-analyses were carried out by using the R-Studio software (R version 4.1.1).
Results
VDS to COVID-19 patients and admission to ICU
The literature search is depicted in Fig 1, S1 Table. Three articles [28–30] were found during hand search of review articles. After full article reading, nine articles [16, 28, 31–37] were selected for review and 14 articles were excluded. Reasons for exclusion were: high vs. low dose VDS [38–40], no specific numbers of patients who were admitted in ICU [14, 15, 30, 41–45], failure of randomization [29], and not a randomised trial [46, 47].
Characteristics of the nine selected randomised trials are summarized in Table 1. The vitamin D compound were cholecalciferol (5 trials), calcifediol (2 trials), or calcitriol (1 trial). VDS dosages and regimens were variable with daily intake in six trials and bolus administration in three trials. Four trials were placebo controlled and five trials were open-label. The numbers of COVID-19 patients enrolled in trials ranged from 50 to 548. The odds ratios for admission in ICU ranged from 0.02 to 1.06, with wide 95% CIs. The risk of bias analysis is detailed in S2 Table. Five trials had a low risk of bias, and four had a high risk of bias.
The meta-analysis of the nine trials based on the RMLE estimator resulted in a summary odds ratio of 0.61 (95% CI: 0.39-0.95) (Fig 2, Table 2), with low heterogeneity (I2=35%), suggesting that overall COVID-19 patients taking VDS would have a substantially decreased risk of admission to ICU.
The Egger’ and the modified Macaskill tests indicated significant publication bias (Table 2). The display of odds ratios from the nine trials in a Doi plot (Fig 3a) showed a distribution markedly stretched to the left, witnessing a negative publication bias, i.e., trials not favouring a reduction of admissions to ICU associated with VDS were less likely to be published than trials favouring a reduction of admissions to ICU associated with VDS. Of note, the smaller the trial size (i.e., the higher the Z-score), the greater the risk reductions, i.e., ln(OR) moving towards negative values far from zero.
The LFK index of -3.79, well below -1.0, quantifies the strong imbalance by which trials in favour of VDS had a much greater probability to be published than trials in favour of VDS.
The Trim and Fill method resulted in the addition of four open circles on the funnel plot (Fig 3b). These open circles are imputations of results of four hypothetical smaller-size trials not in favour of VDS that should have been published for completely excluding the possibility of publication bias. After application of the Trim and Fill method, the summary odds ratio increased to 0.80 (95% CI: 0.40-1.61), which was no longer significant.
Exclusion of one trial at a time from the meta-analysis showed that exclusion of Entrenas-Castillo et al, 2020 [37] of the meta-analysis ended up in a non-significant summary odds ratio of 0.78, with a drastic drop in heterogeneity between results of the eight other trials (I2=4%).
After stratification of trials according to numbers of patients enrolled (Table 2), the summary odds ratios were 0.34 (95% CI: 0.13-0.93) for the five trials including 50 to 105 COVID-19 patients, and 0.88 (95% CI: 0.62-1.24) for the four trials including 106 to 548 patients. The trial size had a significant modification effect on summary odds ratios (p-value = 0.04).
A meta-regression model showed that the two risk of bias scores (Table 1) did not significantly modify meta-analysis results: p-values were 0.46 for score 1 and 0.34 for score 2.
Use of the RMLE estimator with Hartung-Knapp-Sidik-Jonkman (HKSJ) method for the 95% CI resulted in a summary odds ratio of 0.61 (95% CI: 0.32-1.16). If the summary odds ratio remained the same, the 95% CI was wider, from 0.56 (=0.95-0.39) when the RMLE estimator only was used to 0.72 (=1.16-9.32) when the RMLE estimator and the HKSJ methods were used. Because of the wider 95% CI, the risk of admission to ICU was no longer significant. All other meta-analysis results remained equivalent or close
to results when the RMLE estimator only was used, but owing to the larger confidence interval, the effect modification with trial size was no longer significant.
VDS for ARI prevention
The 37 trials on VDS for the prevention of ARI included in the publication of Jolliffe et al, 2021 [12] enrolled 25 to 16,000 subjects (median of 247). The meta-analysis of the 37 trials resulted in a summary odds ratio of 0.92 (95% CI: 0.86-0.99), suggesting a beneficial effect of VDS on the risk of ARI. But the Egger and the modified Macaskill tests provided evidence for significant publication bias (Table 3).
The display of odds ratios from the 37 trials in a Doi plot (Fig 4a) showed a distribution markedly stretched to the left, witnessing a negative publication bias, i.e., trials not favouring a reduction of ARI associated with VDS were less likely to be published than trials favouring a reduction of ARI associated with VDS. The smaller the trial size (i.e., the higher the Z-Score), the greater the reduction of the risk of ARI, i.e., ln(OR) moving towards negative values far from zero.
The LFK index of -3.11, well below -1.0, quantifies the strong imbalance by which trials in favour of VDS had a much greater probability to be published than trials not in favour of VDS.
The Trim and Fill method resulted in the addition of eight open-circle points on the funnel plot (Fig 4b). These open-circle points are imputed results of hypothetical smaller-size trials not in favour of VDS that should have been published for completely excluding the possibility of publication bias. After application of the Trim and Fill method, the summary odds ratio increased to 0.96 (95% CI: 0.88-1.95), which was no longer significant.
The meta-analysis of the 19 trials including 25 to 247 patients obtained a summary odds ratio of 0.69 (95% CI: 0.57-0.83) suggesting marked protective effect of VDS. Instead, the meta-analysis of the 18 trials including 250 to 16,000 patients or more resulted in a summary odds ratio of 0.98 (95% CI: 0.94-1.03) indicating no protective effect of VDS. The trial size had a significant modification effect on summary odds ratios (p-value = 0.0001).
Discussion
Our review on VDS for COVID-19 patients and the risk of admission to ICU, and on VDS for the prevention of ARI shows that the majority of small-size randomised trials obtained results in sharp contrast with results of medium-to-large-size randomised trials. Results of medium-size randomised trials in the COVID-19 setting are in agreement with the six aforementioned Mendelian randomization studies that examined the risk of incidence and severity of COVID-19 according to s-25OHD. They
also agree with the negative results of large-size randomised trials conducted before 2020 on high-dose VDS for critically ill, vitamin D deficient patients admitted in intensive care unit [48, 49].
Hence publication bias was probably present, meaning that a number of small-size trials that obtained odds ratio not in favour of VDS were not published because either authors did not consider that their work was worth being submitted to medical journals, or because submissions were turned down by medical journals.
When the COVID-19 pandemic expanded, many new open-source resources were created for rapid dissemination of research findings, with the consequence of more lenience on the quality of research [50].
Jolliffe et al., 2021 [12] considered that virtually all 37 trials were at low risk of bias, which is surprising in view of the contrasting results between small-size and large-size trials. Although in the original publication a funnel plot highlighted a left-sided asymmetry with Egger’s test having a p-value of 0.007, the authors considered that these results “might reflect heterogeneity of effect across trials, or publication bias arising from omission of small trials showing non-protective effects of vitamin D supplementation from the meta-analysis”.
The interpretation of small-size trial effects is notoriously tricky. Overrepresentation of small-size trials with substantial effect on outcome can be due to publication bias and to
other reasons like poorer quality of small-size trials compared to large-size trials, or selective reporting of results [51–54].
Systematic reviews with meta-analyses of randomised trials are considered the optimal methodology for evaluating treatment efficacy. However, our study shows that, regardless of being the result of publication bias or poor-quality research, small-size randomised trials can lead to overestimate the efficacy of interventions [54, 55]. The undesirable effects of small-size trials have been reported for a variety of therapeutic areas [56], including critical care [57].
Study limitations
Our study has several limitations. The study unfolded with the uncovering of the effects of trial sizes on the associations between VDS and acute respiratory infections or COVID-19 severity. In this regard, the study did not follow a protocol written before looking at the data. We reused data from Jolliffe et al, 2021 without a literature search for more recent publication on randomised trials testing VDS for acute respiratory infections. The literature search on VDS for COVID-19 was restricted to the Pubmed and reference lists of review articles published in 2022. However, we believe that more extensive literature searches were not likely to find other publications relevant to this work.
Low vitamin D status and reverse causation
The negative results of medium-size trials on VDS are in striking contrast with the common observation that low s-25OHD is associated with more severe COVID-19 and death. This contrast underscores that low s-25OHD would have no detrimental role in COVID-19 but would rather be the consequence of two sets of factors (i.e., reverse causation). First, s-25OHD seems to behave like a negative surface reactant whose serum concentration fall during inflammatory states, just like albumin, transferrin and other compounds [41] [58] [59–61]. Second, patient’s characteristics and medical history known to precipitate severe COVID-19, like age, diabetes, obesity, frailty, cardiovascular conditions, are also known risk factors for low s-25OHD [62].
The associations between low s-25OHD and COVID-19 severity echoes the many observational studies showing that patients with low s-25OHD are at higher risk of acute and chronic disease, including cardiovascular diseases, diabetes, and fractures, and of premature death [62, 63]. However, randomised trials and their meta-analyses have failed to confirm that VDS could prevent or treat any of these diseases and prolong life-expectancy [64–67].
Conclusions
Our review showed that the alleged protection of VDS against severe COVID-19 was the consequence of small-size randomised trials published before and in the early stages of the COVID-19 pandemic. If in 2022, it was recognised that VDS brought no benefit to COVID-19 patients [68, 69], mechanistic data, observational studies and small-size randomised trials done before have contributed to treating many COVID-19 patients with VDS. Even if VDS does not cause serious side effect, the same thread of laboratory and observational studies, and small size randomised trials could lead to the adoption of poorly effective but unsafe preventive or therapeutic interventions, until larger randomised trials would discourage their use.
Supporting information
S1 Text. Keywords used for literature search in Pubmed and Prisma Flow Chart. S1 Table. PRISMA 2020 checklist
S2 Table. Risk of bias in randomised trials on VDS of COVID-19 patients and admission to ICU
Data Availability
All relevant data are within the manuscript and its Supporting Information files. Articles from which data were extracted are publicly available and cited in references.