Missing clinical trial data: the evidence gap in the safety of potential COVID-19 drugs

Background

Coronavirus disease 2019 (COVID-19) is a pandemic infection caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Its global spread has been rapid and unprecedented, at the time of writing 24.5 million confirmed cases have been reported with 832,378 deaths across 188 countries (1). Currently, treatment options for COVID-19 are limited. However, several drugs developed for other indications have shown promising results against SARS-CoV-2 in vitro, in animal models, or in compassionate use trials (2,3). Many of these drugs are now being experimentally repurposed for COVID-19 or are undergoing clinical trials in humans (4). Such candidates include nitazoxanide, remdesivir, favipiravir, lopinavir, darunavir, hydroxychloroquine, chloroquine, and ivermectin amongst others (5).

Investigations into the efficacy of these experimental treatments for COVID-19 are ongoing. Meanwhile, in response to positive media coverage, some speculative rather than evidence-based, governments are stockpiling vast supplies of these treatments in anticipation of their licensing for COVID-19. Furthermore, national regulatory institutions, meant to safeguard against the unsafe use of drugs, are under increasing pressure to relax approval standards to accelerate market-entry for COVID-19 treatments. For example, on April 27th 76 2020 the U.S. Food and Drug Administration (FDA) approved the antimalarial drug hydroxychloroquine for emergency treatment of COVID-19 with unknown optimal dosage and duration of treatment (6). However, the efficacy of hydroxychloroquine is still under question (7). Furthermore, it is cardiotoxic at the higher doses which may be indicated for COVID-19, causing QT prolongation leading to ventricular tachycardia and death (8). Care must be taken not to lose the rigorous safety standards usually stipulated for pharmaceuticals, even during a pandemic, to avoid unnecessary morbidity and mortality worldwide.

As many of these drugs have been used widely in other indications for years, there should be substantive information on safety, tolerability and pharmacokinetics available in the public domain, including public trial registries. Yet, whilst the International Committee of Medical Journal Editors (ICMJE) policy requires prospective registration of interventional studies on a WHO primary registry or on ClinicalTrials.gov, the ICMJE does not currently require researchers to report summary results on these registries before academic publication (9). However, according to the FDA Amendment Act 2007, the responsible party for some applicable clinical trials that are registered on ClinicalTrials.gov must report results to a public register within twelve months of the primary completion date, or in some cases risk a fine of $11,569 for every day results are delayed (10,11). Failure to share clinical trial results publicly can additionally have far-reaching consequences for health and public expenditure, as illustrated by the widespread stockpiling and prescription of oseltamivir (Tamiflu) during the swine flu outbreak in 2009, despite a lack of evidence on safety and efficacy (12). When the clinical study reports for this therapeutic were finally made publicly accessible, the benefits of the product turned out to be exaggerated and misrepresented in the journal publications compared to the underlying data. Following the same H1N1 swine flu outbreak a novel vaccine (Pandemrix) was rapidly rolled out, but it only emerged seven years later that the manufacturer had failed to disclose internal pharmacovigilance data showing narcolepsy to be a rare side-effect of recipients of the vaccine(13). Clinical trial transparency is therefore vital in order to maximise and unify the sharing of safety information and data on efficacy, during this pandemic and beyond.

Public clinical trial registries are an important tool for transparent collaborative research. On these registries, safety and efficacy data can be uploaded freely, shortly after completion of the study, and protocol and data collection methods are still quality assessed (14,15). In contrast, academic publication may stall for significant periods of time and can be costly and selective, with time-intensive writing and review processes. Furthermore, clinical trial registry data is available free of charge and can be pooled without concern for silent outcome switching or publication bias (16-18). Indeed, inclusion of unpublished study results from clinical trial registries in meta-analysis may provide important additional information on adverse events and more precise risk estimates than looking at journal publications alone (19). Moreover, academic publications often fail to disclose all information on adverse events occurring during clinical trials, and a systematic review comparing the journal publication with unpublished documents has suggested in 75% of the studies reviewed, the number of named adverse events was lower in published medical literature (20). This is especially important during the COVID-19 pandemic, as interest in potential treatments for COVID-19 generates even greater incentive than normal for the publication of studies with positive results only. Without rapid sharing of datasets for drugs that may be repurposed for COVID-19, secondary analyses of safety data will be arduous and often incomplete (14,21). This may slow down the biomedical innovation process and could lead to preventable side effects occurring in vulnerable patients if safety information remains missing. As the pharmaceutical pipeline is accelerated to address the COVID-19 pandemic, enhancing clinical trial transparency is more important than ever.

In this rapid review of ClinicalTrials.gov, we aimed to determine the scale of unpublished clinical trial results which may hinder safety reviews of repurposed drugs for COVID-19. We reviewed the number of completed or terminated trials that have not reported results for an extensive, list of medications being repurposed for COVID-19, looking at all previous indications for these drugs. Specifically, we searched for any trial results that have not been made available to the public, with no results published on either the ClinicalTrial.gov registry or in the academic literature.

Methods

We selected 19 potential treatments for coronavirus diseases based on information found in potential COVID-19 treatment reviews (5,22,23). The drugs assessed were pirfenidone, hydroxychloroquine, azithromycin, favipiravir, oseltamivir, sarilumab, tocilizumab, remdesivir, leflunomide, interferon-alpha, lopinavir-ritonavir, darunavir-ritonavir, baloxivir marboxil, umifenovir, interferon-beta, sofosbuvir, nitazoxanide, APN01 and ivermectin (Table 1). Synonyms and chemical names for these drugs were taken from pubchem.ncbi.nlm.nih.gov (Appendix 1) (24).

The U.S. clinical trials registry (ClinicalTrials.gov), was searched for all trials that listed these drugs as an intervention. Results of the search were downloaded on 4^th April 2020 (25). Numbers of trials with and without results on the registry were recorded. For trials without results, trial status was determined (completed, ongoing, suspended, terminated or withdrawn). Trials listing ‘primary completion date’ in the future were counted as ongoing, if no primary completion date was available then the study completion date was used. Listed trial status was used to identify terminated, suspended and withdrawn trials.

For all trials without results on ClinicalTrials.gov, a three-step process was followed between 4^th – 27^th April 2020 to determine whether results were reported elsewhere through academic publication (Figure 1).

1. Publications automatically indexed by clinical trial identifier (NCT number) on ClinicalTrials.gov were screened and included based on criteria below. If multiple publications were listed, the earliest dated publication was selected.

2. If results were not available on the registry, the NCT number was used to search and screen academic publications in PubMed.

3. If the PubMed search did not retrieve an academic publication, an additional search was conducted in Google Scholar using the following search terms in succession: clinical trial identifier; listed title; intervention name with primary investigator’s name. For each search, the first twenty results were screened.

• Download figure
• Open in new tab

Figure 1:

Flow chart depicting the methodology used to search for and identify relevant publications for each of the trials listed on CT.gov

If a publication did not include the clinical trial identifier, it was cross-referenced with the primary investigator, study design, intervention and outcomes listed on ClinicalTrials.gov to assess relevance. We excluded publications that had fewer than 500 words, as well as conference abstracts, posters, presentations, and non-English texts.

Publications of results were recorded by PMID, DOI, and publication date. Trials with results on ClinicalTrials.gov combined with those with a journal publication gave a total number of trials with results where results were found in the public domain. This allowed approximation of registered trials without results.

Additionally, overdue trials were calculated as any completed trial with no result on the registry and a primary completion date before 18^th April 2019, 395 days prior to final analysis (1 year + 30-day grace period). This is the standard outlined in the FDAAA 2007 and used as a reference throughout this study despite not all included trials being covered by the law (10). This is also consistent with international ethical standards for timely results dissemination set by the World Health Organisation (26).

A second review was conducted by a different researcher on 10% of trials for each drug to check concordance between reviewers. The protocol during the second review remained unchanged and researchers were blinded to the results of the first review. A random number generator was used to select trials for second review. Concordance was calculated by simple percent agreement on a results judgement; above 80% was deemed acceptable from consulting experts and in line with recommendations in the literature (27). Results published between the dates of first and second review (29^th April – 9^th May 2020) were not counted in the concordance.

Results

Nineteen drugs were screened, encompassing a total of 4553 clinical trials registered on ClinicalTrials.gov (Table 1). We excluded 799, with 661 ongoing (primary completion date in the future) and 138 with a trial status of suspended or withdrawn. Figure 2 shows the number of trials found on ClinicalTrials.gov, those excluded from this analysis, and the final results status of all included trials. All recorded percentages in text are in relation to the 3754 completed trials, seen in Table 2.

• Download figure
• Open in new tab

Figure 2:

Flow diagram displaying the numbers of registered trials identified on CT.gov and the proportion of these which ultimately have no results available. The number of trials which had results available from various sources or had been withdrawn/suspended or trails for which results were not yet due are also shown.

In sum, our protocol revealed 2238 (59.6%) completed trials had published results either on the registry or in the academic literature (Table 2). Of these, 1172 (31.2%) completed trials had tabular results on ClinicalTrials.gov (Table 2). A further 1066 (28.4%) completed trials had results from the literature search, but did not report results on ClinicalTrials.gov (Table 2). Across the 19 drugs which may be repurposed for the treatment of COVID-19, 1516 (40.4%) of completed clinical trials listed on ClinicalTrials.gov were missing results. Figure 3 shows the proportions of trial results available on ClinicalTrials.gov, available in the literature, and those with no results available.

• Download figure
• Open in new tab

Figure 3:

Bar chart displaying the proportion of trial results available across all potential COVID-19 interventions - categorised into those registered fully on CT.gov, those with results available elsewhere and those with no results available.

Of the 3754 completed studies, 2379 (63.4%) were without results on ClinicalTrials.gov outside of the 395-day timeframe mandated by the FDAAA 2007 (although not all trials may have been covered by the FDAAA 2007). 1008 (26.9%) of these had published results in the academic literature (10).

In the blinded second review of 341 (10%) trials, the same results status was found 83.6% of the time meeting our 80-100% threshold for acceptable concordance in searches.

Discussion

Of the completed clinical trials for existing drugs that may be repurposed for COVID-19, 40.4% did not report results on either ClinicalTrials.gov or through academic publication (Table 2). This shows a large gap in the evidence base potentially regarding the adverse effects of these drugs, limiting attempts to comprehensively review their safety before potential global distribution for the COVID-19 pandemic. The 2238 (59.6%) completed studies with available results were comprised of 1172 (31.2%) with results on the registry and 1066 (28.4%) without results on the registry, but with results from a standardised search of the literature (Table 2). Furthermore, 2379 (63.4%) studies without registry results were outside of the 395 day timeframe for results publication as mandated by the FDAAA 2007 (10), although 1008 (26.9%) of these had results published in academic literature, they still failed to upload their results on ClinicalTrials.gov. Not all trials include in this study are covered by the FDAAA 2007, but the twelve month deadline remains an important benchmark for good scientific practice (26). With 40.4% of clinical trial results unavailable for potential COVID-19 treatments, the data for clinical decision making regarding the safety of these therapeutics are limited. If any drug with an incomplete evidence base is used during the pandemic, even in compassionate use programmes, there is a risk of avoidable harm being done because of missing adverse safety data. An evidence gap was revealed for drugs which have had extensive media coverage such as hydroxychloroquine (37.0% without results), favipiravir (77.8%) and lopinavir (40.5%) (28-30). These drugs are currently being used in COVID-19 patients and clinical trials across the globe, sometimes in novel regimens and doses (7,31,32). Clinicians currently have few treatment options available, but with greater transparency and proactiveness from trial sponsors regarding the posting of results, there would be less risk of unforeseen adverse outcomes, especially in the treatment of mild-moderate COVID-19 as in the PIONEER trial (31).

It is recognised that there are sources of safety data outside of clinical trial registries, for example pharmacovigilance registers, however these databases are not always free to access and publicly available. In contrast with clinical trial registries, there is often no obligation for all adverse events to be reported by trial sponsors. Public health decision-makers, guideline developers, clinicians, and patients therefore rely on clinical trial registries, systematic reviews, and meta-analyses to inform treatment decisions. Evidence gaps and publication bias therefore have the potential to influence clinical practice and drug usage worldwide, particularly in a treatment landscape as changeable as the COVID-19 pandemic. If clinical decisions are based on incomplete evidence, this can result in avoidable morbidity and mortality, especially if unsafe drugs or ineffective treatments are given on a large scale. Sponsors and researchers alike carry an ethical responsibility towards clinical trial participants, who consent to participate in research in order to contribute to scientific understanding and improved clinical practice, to make results publicly available (33).

Our study reveals an important evidence gap regarding existing pharmaceuticals potentially being repurposed for COVID-19. However, the proportion of studies with results available in the academic literature is an approximation and there are several limitations to our study. Firstly, our trial population was limited only to those registered on ClinicalTrials.gov. While ClinicalTrials.gov is the largest registry in the world, with over 340,000 registrations as of writing and an order of magnitude greater than the next largest journal, additional trials on these therapies may have been registered elsewhere. However, it is unlikely that these would be expected to report at a different rate than those registered to ClinicalTrials.gov. Secondly, our strategy for locating publications included only those listed on ClinicalTrials.gov and identified through searches on PubMed and Google Scholar, which are open-access resources that should cover a majority of published clinical research. While including proprietary databases like Scopus or Ovid may have located some additional publications, we do not believe this would have substantially impacted our overall results (34). Thirdly, we are aware that trials that were not registered at all or published in non-English language journals without inclusion of the NCT number would not have been captured by our methodology. Finally, searcher heterogeneity and difficulty identifying results publication in the academic literature limits accuracy in any manual publication search, however, our search strategy was standardised and produced a high level of agreement between assessors (83.6%) in a check of a 10% random sample. However, any discordance between reviewers only reveals the inherent difficulties in finding results for the drugs in question, especially if the trial identification number was not included in line with CONSORT standards (35).

Our findings add to the existing evidence of the dearth of clinical trial reporting on public registries. This analysis investigated clinical trials of existing drugs currently being considered for use for COVID-19. However, given the diversity of drug classes included in this report, findings are likely to be representative of many pharmaceuticals. This presents a major problem for researchers attempting to summarise safety and efficacy by pooling trial data (36). Since academic publications often summarise key findings only, secondary research efforts are impinged by the incomplete publishing of all trial outcomes. ClinicalTrials.gov, in contrast, provides a forum to share complete safety and efficacy data reports, as well as facilitating consistent data reporting in a timely manner (14,15). Prior research has shown that results reported to ClincialTrials.gov were often more complete, especially for safety data, when compared to matched journal publications (16-18). However, the availability of this data depends on researchers registering trials and uploading results in a timely manner, within twelve months of the primary completion date.

The International Committee of Medical Journal Editors (ICMJE) and the editorial offices of medical journals could play an important role in improving the lack of timely results posting on clinical trial registries by demanding submission of a link to summary results on public registries before academic publication, although this may mean that the publication bias of positive, ‘publishable’ results may trickle down to reporting on public registries as well. Furthermore, public funders and institutional publication funds could demand that trial sponsors post their results before allocating funding for academic (open-access) publication. These funding bodies could also deny individual sponsors funding if they have in the past violated clinical trial reporting rules (37). At the very least, journals should conform to the CONSORT statement in ensuring that registry identification numbers are clearly indicated in the abstract, full-text, and meta-data of published clinical trials in order promote discoverability and record linkage between registries and publications (35). Finally, clinical trial sponsors, such as universities, hospitals, public research institutions, and pharmaceutical companies, should themselves work towards improving their institutional clinical trial reporting performance by making use of available resources that provide detailed step-by-step instructions as to how to go about this task (11). Especially during the COVID-19 pandemic, it is of great importance that trials sponsors release summary results on these registries retrospectively to inform decision making around existing treatments being re-purposed for COVID-19.

Conclusions

Overall, our findings reveal a significant evidence gap for the safety of drugs being repurposed for COVID-19. This uncertainty could cause a large burden of extra morbidity in the global pandemic. We recommend caution in experimental drug use for non-severe disease and urge trial sponsors to report missing results retrospectively. Medicine during the COVID-19 pandemic cannot be evidence-based if a large proportion of the evidence is missing.

List of Abbreviations

COVID-19

Corona Virus Disease 2019

CONSORT

Consolidated Standards of Reporting Trials

FDAAA2007

FDA Amendment Act 2007

ICMJE

International Committee of Medical Journal Editors

NCT Number

ClinicalTrials.gov identifier

PMID

PubMed ID

Declarations