Results Availability and Timeliness of Registered COVID-19 Clinical Trials: A Cross-Sectional Study

Trial Population

The World Health Organization (WHO) International Clinical Trials Registry Platform (ICTRP) maintains a list of registered COVID-19 studies across 18 global registries (Table 1). This database was downloaded on 30 June 2020 and processed to clean and standardise certain data elements across various registries (e.g., phase, study type). Known cross-registrations, taken from the COVID-19 TrialsTracker project (run by NJD), were collapsed to single trial record, preferring a ClinicalTrials.gov entry when available, unless another registration is deemed more complete by the study team, per protocol.

Inclusion/Exclusion Criteria

Inclusion and exclusion criteria for included trials were assessed twice. First using automated methods on data fields extracted directly from the ICTRP and registries between 30 June 2020 and 5 July 2020, and then via a manual assessment of each trial passing automated screening. We note two additional post hoc exclusion criteria that arose from coding reconciliation discussions.

Inclusion Criteria:

• Trial assessed an intervention for treatment or prevention of COVID-19 infection and subsequent acute disease.

• Trial included a completion date, or primary completion date, on or before 30 June 2020.

Exclusion Criteria:

• Registration was found, at any time, to indicate that the trial was withdrawn before enrollment and therefore never occurred.

• Registration prior to 1 January 2020.

• Trial on symptomatic treatment of COVID-19 disease side-effects only (post hoc)

• Trial on rehabilitation after acute disease (post hoc)

Search Strategy

Outcomes

Protocol Deviations

In addition to the post hoc sensitivity analyses described above, we report the following protocol deviations. We decided to delay manual searches in order to provide all completed trials at least 6 weeks to report (i.e., by 15 August 2020). This exact cutoff was not specified in our protocol. For our first round of searches, we began searches 15 weeks following our cutoff due to extensive development and piloting of our search and data extraction methods. We plan to maintain this 6-week search buffer as a minimum for results searches moving forward.

We also noted that we would continuously check inter-rater reliability for each reviewer pair. This was difficult and impractical to implement into our eventual workflow for assessing results availability. Therefore we developed an alternate system in which each reviewer-pair would attempt to reconcile any differences on study inclusion or results availability and categorisation through consensus. If consensus could not be reached, the issue was referred to the larger study team for discussion, a final consensus decision, and if necessary a specific rule addressing similar situations moving forward. We created two post hoc exclusion criteria based on these adjudications as detailed above. For future data extraction tasks (e.g., determining directionality of results, extracting intervention information) we may reintroduce inter-rater reliability calculations as necessary.

Software, Code, and Data

ICTRP and registry data collection, management, and parts of the analysis were performed in Python 3.8.1 (Python Software Foundation). Additional data preparation and analysis was performed in R version 4.0.2 (R Foundation for Statistical Computing). All code and data for this project is available on GitHub.[17,18] Manual data extraction was conducted via Qualtrics,[19] and reviewer reconciliation via Numbat Systematic Review Manager.[20]

Patient and Public Involvement

Patients and the public were not formally involved in developing this study.

Per-Protocol Analysis

Among the 285 trials registered as completed by 30 June 2020, we located 41 trials (14%) with results spread across 47 individual results publications, available by 15 August 2020. Details on the screening of included results are included in Appendix A. The breakdown of results by dissemination route is detailed in Figure 2. Figure 3 shows the reporting of results by registry.

• Download figure
• Open in new tab

Figure 2

• Download figure
• Open in new tab

Figure 3*

*The count of registrations is inclusive of known cross-registrations for included trials, therefore the denominators will sum to >285

Figures 4a-c shows cumulative incidence plots for time-to-publication and 95% CIs. Trials with a publication date prior to the available completion date (across all results, n = 8 of 41 trials, 19.5%) were considered reported at time 0 (except for the Aalen-Johanssen plot, which used nominal offsets to break ties at 0). If a full results publication included a different completion date (i.e., last patient, last visit), this replaced the registered completion date in the dataset (n = 4 of 41 trials, 9.8%). We show the cumulative incidence of a) first publication across any dissemination route, b) earliest journal publication, and c) earliest preprint publication. The medians for all cumulative incidence plots were undefined as no curve crossed 50%. The cumulative incidence of any results availability, accounting for censorship, surpassed 20% at 119 days from completion.

• Download figure
• Open in new tab

Figure 4a-c

As only two trials reported results to a registry, we did not generate a plot for this route. Summary results for these two trials were published on ChiCTR 3 and 20 days following completion. For the 4 trials that had both a preprint and a journal publication, the median time from preprint publication to journal publication was 24.5 days (IQR: 14.2, 36.2). For the 21 trials that had only a preprint, preprints had been published a median of 90 days (IQR: 76, 136) by completion of follow-up.

Trials with only a preprint (n = 21) published the preprint with a median of 32 days (IQR: 10, 53), whereas trials with both a preprint and a journal article (n = 4) published the preprint with a median of 31 days (IQR: 19.5, 31); trials with only a journal article (n = 14) published the article with a median of 45.5 days (IQR: 0.8, 83.5), whereas trials with both a preprint and a journal article published the article with a median of 46 days (IQR: 24.2, 67.2). Due to the small number of trials with both preprint and journal publications, we did not undertake any statistical comparisons.

Sensitivity Analyses

Each sensitivity analysis changed only a single aspect of the main analysis, the changes were not cumulatively applied. First, we recorded a new completion date for 48 trials during manual searches, 33 of which moved the completion date post-June 2020. Using this updated trial population (n = 252) shows a reporting percentage of 16% with a minimum of 6 weeks of follow-up. Second, restricting our original sample to only those trials that reached full completion by 30 June 2020 showed 38 of 212 (18%) trials reported with a minimum of 6 weeks of follow-up. Third, adding interim results to our per-protocol findings yielded only one additional results publication for a reporting percentage of 15%.

Lastly, extending our six-week minimum follow-up window to three-months (i.e., counting any results reported by 1 October 2020), aligned with when the searches for this phase of our project actually began, adds an additional 17 results for a reporting rate of 54/285 (19%). Figure 5 includes a cumulative incidence plot for results from expanded follow-up time.

• Download figure
• Open in new tab

Figure 5

Summary of Findings

We identified 285 registered trials for the treatment or prevention of COVID-19 completed during the first six months of the pandemic. Of these trials, 14% had a result available in either a preprint, journal article, or posted to a clinical trials registry after a minimum follow-up time of six weeks. Sensitivity analyses using alternate dates and definitions of results for assessments did not appreciably change the reporting percentage. Extending the minimum follow-up time to three months yielded 19% of registered trials with results. Few new publications were seen beyond 120 days of follow-up. Preprints have played an important role in results dissemination with the most unique results of any route. These preprint results do not appear to be rapidly making it into the peer reviewed literature as just four preprints in our sample had a matched journal publication. There was low use (5%) of clinical trial registries for results dissemination.

Strengths and Limitations

This work had a number of strengths. We used a broad definition of interventional clinical trials, beyond just randomized controlled trials, and included the full ICTRP database in our potential population. Searches for publications were performed using multiple strategies and databases, with dual-coded manual verification throughout. Our findings were also robust to a number of alternate assumptions for judging completion and results availability.

This work also has limitations. Missing, incomplete, inaccurate, or outdated data all likely impacted our sample to varying degrees that are difficult to quantify. Deviations from best practices for discoverability (e.g., inclusion of registration IDs in abstracts and metadata), publications in non-indexed journals or preprint servers, and articles not available in English language may have made some results difficult to locate despite our extensive search strategy. Lastly, our methods may under-represent some aspects of how results have been shared during the pandemic. For instance, large adaptive trials like the RECOVERY trial will essentially contain numerous trials under one broad registration, produce numerous publications, and will only be represented as a single datapoint in our analysis that remains ongoing until the end of follow-up for the final arm.

Research in Context

Registered and reported COVID-19 trials have been well-described in the literature.[7,21–25] Our results add to these examinations of the design, focus, and outcomes of the COVID-19 research response. We are aware of one similar analysis to date that attempted to quantify the dissemination of registered COVID-19 clinical trials. Janiaud and colleagues searched a single database for results of all registered COVID-19 RCTs from the first 100 days of 2020 and conducted outreach to trial contacts in October 2020. They located results for 53 of the 516 trials (10.3%) but did not match results to only registered completed trials. Of the 516 trials in their sample, 155 (30%) had not started or were discontinued, per either the registry or trial team outreach.[26]

Reporting for COVID-19 trials may be accelerated compared to other trial populations even if the overall reporting percentage remains very low. A meta-analysis from 2014 showed a pooled reporting percent of registered trials across studies of 54.2% with a minimum of 24 months of follow-up but with substantial heterogeneity. Reporting percentages in the individual studies, often examining trials in a single specialty, ranged from 23% to 76%. In five studies with time to publication data showed a 30% pooled probability of reporting at 24 months of follow-up.[27] Subsequent large examinations of reporting registered trials from academic medical centers in the US and Germany showed 35.9% and 39% of results published in the literature within two years of completion and overall reporting did not surpass 20% in either study until well after six months of follow-up.[15,28] Low reporting rates have also been seen in previous pandemics; vaccine trials during the H1N1 pandemic showed <20% probability of reporting of even within 5 months of completion.[6]

Preprints grew modestly during prior outbreaks;[8] however their rise has been substantial since the COVID-19 pandemic began, including for trials.[10,29] Preprints represented the most popular dissemination route, and often the only route, in which results were available in our sample. The low conversion of preprints to journal articles is consistent with other research on preprints during COVID-19.[30,31] The time to journal publication for preprinted trials requires additional follow-up and consideration alongside emerging evidence on the relationship between preprints and the peer-reviewed COVID-19 academic literature.[11]

Implications for Policy and Practice

The deluge of trial registrations also reflects the fragmentation of the global COVID-19 response. As others have noted, this rush to register research without coordination or consideration of the work of others may compromise progress towards answering crucial questions.[32] A disparate research environment may lead to considerable research waste as unnecessary duplication, disjointed outcomes, and competition for participants slows evidence generation.[33–36] It is essential that we understand the COVID-19 research response and how and why it evolved as it did.

A complete assessment of COVID-19 trial results dissemination would require accurate registry data on both trial completion and the overall status of the trial. The availability of completion dates in trials considered ranged from never (e.g., REBEC), to mixed (e.g., ANZCTR), to always (e.g., ClinicalTrials.gov). Other registries, like the EUCTR, make completion dates available only retrospectively. Despite being a CONSORT element, explicit dates for trial completion are rarely included in clinical trial publications, which would aid checks for accuracy and provide context for results.[37] Issues with missing, incomplete, and outdated data on registries has been detailed in prior investigations.[38–40] Registries were also rarely used as a dissemination route for clinical trial results. The US National Institutes of Health have since called for the rapid publication of results on ClinicalTrials.gov, the largest global registry, and instituted expedited reviews of results submissions moving forward.[41] Our future work will examine whether this has led to more results availability on registries.

It is difficult to conclude from these preliminary results whether our findings show true potential for publication bias from unpublished results, poor management of trial registrations, or some combination of both. The stakes of a global pandemic only amplify the importance of minimizing reporting issues that may impact evidence synthesis, guideline development, and ultimately clinical practice. Failure to update registry entries compromises an important tool for transparency and accountability in the COVID-19 research response. Searching for or anticipating results that may simply never come and/or should not be expected hampers crucial efforts to collect and examine the evidence around COVID-19.

Future Research

Data collection for Phase 2 of the DIRECCT project is currently underway, extending our results to trials completed at any time in 2020. The continued examination of publication rates over time will help place the COVID-19 results reporting data in further context. In our final Phase, we plan to expand our data collection to include outreach to authors and study teams, as well as to examine factors associated with the timely reporting of COVID-19 clinical trials. Future studies may also wish to examine additional forms of dissemination, such as press releases or other grey literature findings. We hope that our openly available dataset of results matched to registrations will be useful to other groups investigating the COVID research response. Our data forms one part of a robust landscape of curation, examination, and synthesis of COVID-19 research and literature, through platforms like COVID-NMA, Cochrane, COVID-evidence, and Epistimonikos.[12,42–44] These efforts allow for rapid evidence synthesis and the ability to efficiently examine pandemic research output in order to inform strategies for improved management and coordination of the global scientific community both during global health emergencies and otherwise.