Abstract
Background Prior studies suggest that clinical trials are often hampered by problems in design, conduct and reporting that limit their uptake in clinical practice. We have described “informativeness” as the ability of a trial to guide clinical, policy or research decisions. Little is known about the proportion of initiated trials that inform clinical practice.
Methods We created a cohort of randomized interventional clinical trials in three disease areas (ischemic heart disease, diabetes mellitus and lung cancer), that were initiated between 1 January 2009 and 31 December 2010 using ClinicalTrials.gov. We restricted inclusion to trials aimed at answering a clinical question related to the treatment or prevention of disease. Our primary outcome was the proportion of clinical trials fulfilling four conditions of informativeness: importance of the clinical question, trial design, feasibility, and reporting of results.
Results Our study included 125 clinical trials. The proportion meeting four conditions for informativeness was 26.4% (95% CI 18.9 – 35.0). Sixty-seven percent of participants were enrolled in informative trials. The proportion of informative trials did not differ significantly between our three disease areas.
Conclusions Our results suggest that the majority of clinical trials designed to guide clinical practice possess features that may compromise their ability to do so. This highlights opportunities to improve the scientific vetting of clinical research.
Funding This study was funded by the Fonds de recherche Santé Québec postdoctoral research grant (NH). This funding body was not involved in study design, conduct or reporting.
Introduction
The ultimate goal of clinical research is to produce evidence that supports clinical and policy decisions. Numerous analyses suggest that a substantial proportion of clinical trials aimed at informing clinical practice are marred by flaws in design, execution, analysis and reporting.1–8 The initial research response to COVID-19 illustrated the fact that existing oversight mechanisms fail to prevent the initiation of flawed trials.9 While unexpected events can stymie well-conceived and implemented studies, trials that have features rendering them unlikely to inform clinical practice may do harm by misleading potential participants of their benefits, and by diverting patient-participants from otherwise informative research efforts.10
We have previously described five conditions that trials should fulfill to support clinical or policy decision-making.10,11 First, trials must ask an important and clinically relevant question that is not yet resolved. Second, trials must be designed to provide a meaningful answer to that question. Third, trials must be feasible, with achievable enrollment goals and timely primary outcome completion. Fourth, outcomes must be analyzed in ways that support valid interpretation. Last, trial results must be made accessible in a timely fashion.
In what follows, we created surrogate measures for four conditions of informativeness: trial importance, design quality, feasibility, and reporting (the fifth condition, analytical integrity, did not lend itself to objective, dichotomous assessment, and is not assessed below). We then evaluated the proportion of “clinically directed randomized controlled trials” in three common disease areas meeting these four conditions. This information can be used to help healthcare and research systems identify studies in need of further scrutiny, thereby improving the impact of their research.
Methods
Overview of Approach
We created a cohort of randomized, interventional clinical trials in three broad disease areas that are representative of the clinical research enterprise and that have a significant impact on patient morbidity and mortality: ischemic heart disease, diabetes mellitus and lung cancer. We restricted inclusion to trials that appeared to be aimed at informing clinical practice by selecting trials with a stated purpose of treatment or prevention of disease and with a primary clinical outcome or appropriate surrogate. We established milestones that could serve as objectively verifiable surrogates for four conditions of informativeness. Trials in our sample were then tracked forward to assess the proportion attaining each informativeness condition. “Informative trials” were trials that fulfilled all four conditions of informativeness.
Surrogate Measures for Four Conditions of Informativeness
We formulated surrogate measures for each condition of informativeness. Measures were chosen based on i) close correspondence with each informativeness condition; ii) objective and reproducible dichotomous scoring; and iii) feasibility of assessment. The four surrogates of informativeness, described in greater detail below, were as follows: trial importance (determined by citation of reported trial results in high quality clinical synthesizing documents; the premise of this surrogate is that these documents focus on questions of clinical importance); trial design quality (assessed using a modified Cochrane risk of bias (ROB) tool which is designed to identify threats to study internal validity); trial feasibility (established based on ability to achieve adequate participant enrollment and timely primary outcome completion); and reporting (based on accessibility of primary outcome results via deposition on ClinicalTrials.gov or in journal publications).
Clinical Trial Sampling
We identified all trials registered on ClinicalTrials.gov in our three disease areas with a start date from 1 January 2009 to 31 December 2010 inclusive (eMethods 1 – search criteria). Our time range provided a minimum of nine years of follow-up for maturation toward trial completion and fulfillment of all four surrogates of informativeness. Trials were downloaded from ClinicalTrials.gov on 15 May 2020. We updated trial status and enrollment for all trials meeting our inclusion criteria on 6 October 2021.
We included randomized trials i) evaluating interventions of any type; ii) aimed at the treatment or prevention of ischemic heart disease, diabetes mellitus or lung cancer; iii) with at least one site in the United States (most of which will thus have a regulatory requirement for results reporting);12 and iv) interventions that were FDA approved, that advanced to FDA approval, or interventions not subject to FDA approval (e.g. cardiac rehabilitation). We did not include trials that we deemed unlikely to be targeted at informing clinical practice by excluding: i) studies that exclusively evaluated safety, diagnostic or screening interventions; and ii) early phase trials (phase 0 or phase 1) (eMethods 2 – inclusion/exclusion criteria; Figure 1 – flow diagram for trial inclusion; eMethods 3 – flow diagrams by disease; eMethods 4 – assessment of regulatory approval status). Phase 2 trials were included in our study as they are frequently used to inform both clinical and regulatory decision-making, particularly in cancer, where over one quarter of recent FDA cancer drug approvals were based on the results of phase 1/2 or phase 2 clinical trials.13 Trials were independently screened and assessed for eligibility by two authors (NH & HM), with disagreements resolved by a third reviewer (JK).
Scoring Conditions of Informativeness
Two authors (NH & HM) independently scored all trials for the surrogate measures of the four conditions of informativeness (eTable 1; eMethods 5). Disagreements were resolved by a third reviewer (JK). Because of their logical relationship among surrogate measures (e.g. citation in a high quality clinical synthesizing documents cannot be assessed unless trial results are available) and workflow (e.g. risk of bias information is often available in systematic reviews), conditions were scored sequentially. Trials not meeting one condition were not advanced for evaluation of subsequent conditions. The order of scoring was as follows: i) feasibility; ii) reporting; iii) importance; and iv) design. Trials meeting all four conditions were deemed informative; trials failing on any condition possessed features that compromised their informativeness.
Our assessments of informativeness began by evaluating feasibility based on timely trial completion and patient-participant recruitment success. Terminated trials were deemed infeasible if the reason for termination in the ClinicalTrials.gov registration record involved accrual, feasibility, funding or another non-scientific reason (as opposed to termination due to accumulated scientific data suggesting early efficacy, futility or toxicity) (eMethods 6 – classification of reason for termination). Completed trials were deemed to have not fulfilled feasibility if final participant enrollment was less than 85% of expected enrollment as listed in the final registration record prior to study start, thus reflecting a substantial loss of statistical power for the primary outcome.14 Trials that were ongoing were categorized as infeasible if they had already surpassed double the intended time for primary completion, which was calculated by subtracting the intended primary completion date (as stated in the final registration record prior to study start) from the trial start date, then multiplying by two.
We next assessed results reporting by determining whether primary outcome results were publicly available. Trials were categorized as reported if they either had primary outcome results available on ClinicalTrials.gov or in a publication (eMethods 7 – methodology for publication search). When more than one publication presented primary outcome results, the earliest published report was identified and advanced to the next step of assessment. ClinicalTrials.gov results reporting and publication search were updated in October 2021 for those trials previously deemed to have not met the criteria for reporting.
Importance was scored by determining whether trial results were included in a high-quality review document designed to inform medical decision-making. To credit trials with being informative even if they produced negative results, trials were first assessed for inclusion in the results of a high-quality systematic review (SR), given that SR citation practices are results neutral. We assessed for trial results citation in a Cochrane SR, Agency for Healthcare Research and Quality (AHRQ) SR or in an SR deemed of high quality based on a modified AMSTAR score (eMethods 8 – SR search strategy and quality assessment). Trials not cited in the results of high-quality SRs were evaluated for inclusion in a high-quality CPG; remaining uncited trials were then assessed for inclusion in an UpToDate review article15 (eMethods 9 – CPG search strategy, CPG quality assessment and point-of-care medical database search). Trials cited in high-quality review documents were deemed to have fulfilled the importance condition. Assessment of importance was updated in October 2021 for all trials previously deemed to have not met the criterion for importance.
Finally, design was assessed by determining whether studies were at elevated risk of bias, using a modified Cochrane Risk of Bias (ROB) tool16 (eMethods 10). When available, ROB scores were extracted directly from high-quality SRs identified during the assessment of trial importance. When unavailable, ROB scores were independently performed by two authors (NH & HM), with disagreements resolved by a third reviewer (JK). Trials were deemed to have fulfilled the design condition of informativeness if all ROB elements were deemed to be of low risk of bias, or a majority were low risk of bias with a minority of elements deemed to be of unclear risk of bias.
Statistical Analysis
Our primary outcome was the proportion of trials that met all four conditions of trial informativeness. We provided a 95% binomial confidence interval for our primary outcome. We performed a sensitivity analysis on our primary outcome excluding small, pilot-type studies that would not have been designed to inform clinical decision-making. These were identified based on an anticipated participant enrollment below the lowest quartile of target enrollment for our cohort of trials. Due to concern that phase 2 trials are less likely to inform clinical practice than trials of a higher phase, we performed a second sensitivity analysis on our primary outcome excluding phase 1/2 and phase 2 trials.
As secondary outcomes, we estimated the proportion of trial participants who were enrolled in informative trials, as well as the proportion of informative trials in each of our three disease areas. We also report the proportion of trials advancing across each condition of informativeness. We provided 95% binomial confidence intervals for these secondary outcomes.
We compared the proportion of informative trials between disease categories and by trial sponsor using the Chi-square test (chisq.test function in R) and provided binomial confidence intervals for each stratum. We used the fisher.test function in R to perform a two-sided Fisher’s Exact test assessing the proportion of informative trials by type of intervention and trial phase and provided exact confidence intervals for each. We calculated inter-rater agreement rates using Cohen’s kappa (eTable 2). We defined p < 0.05 as statistically significant. All analyses were performed using R version 4.0.2.17
Our study was not subject to Institutional Review Board approval, as it relied on publicly accessible data. The study protocol was prospectively registered on Open Science Framework;18 deviations and amendments to the study protocol are detailed in eMethods 11. The code19 and data set18 used in this analysis are available online. This study follows the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines for cohort studies (eMethods 12).
Results
Over half of the 125 interventional trials in our cohort were studies of drug or biologic interventions (77 trials; 61.6%). The majority were Phase 2 (24 trials, 19.2%) or Phase 3 trials (50 trials, 40.0%). Trial status was “Completed” in 99 of 125 trials (79.2%) and “Terminated” in 15 trials (12.0%) (Table 1).
Our primary outcome, the proportion of trials that informed clinical practice, was 26.4% (95% CI 18.9 – 35.0) (Figure 2). As a sensitivity analysis, we re-analyzed our primary outcome excluding the 35 trials in the lowest quartile for target enrollment. This resulted in a proportion of informative trials of 35.6% (95% CI 25.7 – 46.3). We performed a second sensitivity analysis on our primary outcome excluding phase 1/2 and phase 2 trials. This resulted in 30.7% (95% CI 21.9 – 40.7) of trials meeting 4 conditions of informativeness.
A total of 193,839 participants were enrolled in the 125 trials in our cohort, of which 129,973 (67.1% (95% CI 66.8 – 67.3)) were enrolled in informative trials. The proportion of ischemic heart disease trials that was informative was 27.5% (95% CI 14.6 – 43.9); the proportion for diabetes mellitus trials was 31.6% (95% CI 19.9 – 45.2), and the proportion for lung cancer was 14.3% (95% CI 4.0 – 32.7) (Figure 3). Proportions did not vary significantly by disease area (p value = 0.23) (Figure 4). Each surrogate measure contributed considerably to the stepwise decline in the proportion of informative trials (eTable 3).
Studies sponsored by industry were significantly more likely to fulfill all four conditions of informativeness than those not sponsored by industry (50.0% vs. 6.0%, p value < 0.001)(Figure 4). Using the two-sided Fisher’s exact test, there was a non-random association between trial phase and informativeness, and type of intervention and informativeness (Figure 4).
Discussion
This study provides the first assessment of the proportion of randomized trials fulfilling four key conditions of informativeness. In our analysis, just over one fourth of trials demonstrated adequacy for study feasibility, reporting, importance, and design. The remaining 73.6% contained a limitation in design, conduct or reporting that compromised their ability to inform clinical decision-making.
Certain shortcomings of clinical trials are a result of experimenting in a dynamic real-world environment and cannot be entirely avoided. Clinical trials are difficult to plan, and there may be defensible reasons for falling short of some conditions. For example, changes in medical practice may render a research question irrelevant to clinical practice; an emerging viral pandemic might lead to under-recruitment. However, our findings underscore the major challenges sponsors and clinical investigators confront in fulfilling the scientific and ethical warrant for enrolling patient-participants in randomized trials. The goal should be to address foreseeable limitations in trial design, conduct or reporting. For example, increased oversight by research funders, including requirements for landscape analysis of completed and ongoing clinical trials to ensure trials are addressing important questions, and the provision of independent scientific review to highlight vulnerabilities in trial design,20 are measures that can be implemented to increase the likelihood that trials will be informative. Many methodological weaknesses in trial design can be corrected at minor cost.8
The proportion of informative trials did not differ significantly between ischemic heart disease, diabetes mellitus and lung cancer, indicating shared challenges in design, implementation, and reporting. Our study also demonstrated that each condition of informativeness goes unfulfilled in roughly equal proportions (eTable 2), suggesting that vigilance is required throughout the life cycle of a trial. Our estimates for the fraction of studies fulfilling criteria for recruitment feasibility are in line with prior studies.14,21–23 The fraction of trials at low risk of bias is similar to prior estimates.8,24,25 Our estimate for the fraction of studies fulfilling reporting requirements (90.0%) is in line with prior studies that evaluated both ClinicalTrials.gov results deposition and publication,26,27 both of which were deemed acceptable means of results reporting in our study. To our knowledge, our study is the first to apply these conditions jointly to a sample of trials, in addition to assessing importance via citation in clinical synthesizing documents.
Our results also indicate that certain types of trials may be at greater risk for having their informativeness compromised. Phase 4 trials fared worse than Phase 3 trials, with only 2 of 19 fulfilling all 4 conditions of informativeness (eTable 4). Trials sponsored by industry funders were far more likely to fulfill all four conditions than those with non-industry sponsorship (50.0% vs. 6.0%). This is in keeping with prior research demonstrating greater recruitment challenges for non-industry funded trials,14 in addition to diminished compliance with timely results reporting on ClinicalTrials.gov.28
These results suggest that funding bodies and academic medical centers may not provide adequate resources for fulfilling the clinical mission of the trials they support. Several recent initiatives aim at improving various aspects of informativeness, including increased consideration given to the importance and clinical relevance of the research question, the evidentiary basis for proposed research, study registration and reporting, by many funders.29 The implementation of new frameworks, such as INQUIRE, developed to guide academic institutions in addressing waste in research, including assessments of research design, feasibility, transparency, relevance, and internal and external validity, if widely adopted, may lead to further improvements in research quality.30 The SARS-CoV-2 pandemic has highlighted both the susceptibility of our clinical research enterprise to substandard trials, while also showing what is possible with robust research vetting, coordination and collaboration.31
Limitations
Our study should be interpreted considering several limitations. First, our measures for each condition of informativeness are proxies for the concepts they represent. For example, scoring trial importance required citation in a clinical synthesizing document. This measure may have erroneously classified some informative trials as at risk of being uninformative (e.g. trials that evaluate disease management in niche populations that are not addressed in practice guidelines or systematic reviews). It may also have misclassified some trials as informative (e.g. trials addressing already resolved clinical hypotheses, which might nevertheless be cited in systematic reviews). To the former, none of the 18 trials not fulfilling the importance condition involved niche populations (eTable 5). We also acknowledge that some trials may inform clinical practice despite failing our criteria. The DAPT Study (NCT00977938) was a large Phase 4 study that was deemed at high risk of bias in several high-quality systematic reviews.32,33 However, this study has had an important impact on the clinical management of antiplatelet therapy following drug-eluting stent placement.34 Our metrics are best understood as capturing factors that seriously (but not fatally) compromise a trial’s prospects of informing practice, and that are rectifiable. Second, we applied strict inclusion/exclusion criteria when identifying out cohort of “clinically directed randomized controlled trials,” thus limiting generalizability to other types of trials, including those involving diagnostics or interventions that do not advance to FDA approval. The latter would require different criteria, given their primary goal of informing regulatory decision-making. Third, we used a longitudinal approach, since conditions like publication or citation are only fulfilled after a study is completed. Changes in research practices or policy occurring over the last decade might produce different estimates for the proportion of randomized trials that are informative.
Conclusions
Trial volunteers are generally told that their participation will advance clinical practice. However, one third (33%) of patient-participants in our study were enrolled in trials that possessed at least one feature that compromised their goal of informing clinical practice. Sponsors and investigators often face unforeseeable challenges, and trials with flaws in design and implementation occasionally uncover actionable insights. Nevertheless, research systems and oversight should address persistent barriers to fulfilling the societal mission of clinical research.
Data Availability
Our manuscript relies on publicly available data from ClinicalTrials.gov. Our data set is available online on Open Science Framework (DOI 10.17605/OSF.IO/3EGKU) (reference 18).
Supplementary Material
eMethods 1 – ClinicalTrials.gov search criteria
eMethods 2 – Trial Inclusion and Exclusion criteria
eMethods 3 – Flow Diagrams for Each Disease
eMethods 4 – Assessment of Regulatory Approval Status
eTable 1 – Addressing 4 Conditions for Informative Clinical Trials
eMethods 5 – Flow Chart of Informative Criteria Assessment
eMethods 6 – Classification of Reason for Termination
eMethods 7 – Methodology for Publication Search
eMethods 8 – Systematic Review Citation Search Strategy and Quality Assessment
eMethods 9 – Clinical Practice Guideline and Point-of-Care Medical Database Search Strategies and Quality Assessment
eMethods 10 – Operationalization of modified Cochrane Risk of Bias score
eMethods 11 – Deviations to the Study Protocol
eMethods 12 – STROBE Checklist for Cohort Studies
eTable 2 – Inter-rater Agreement Rates
eTable 3 – Proportion of Trials Meeting Each Criterion for Informativeness
eTable 4 – Phase 4 Trials Not Meeting 4 Informativeness Criteria
eTable 5 – Trials Not Cited in Clinical Review Documents
eMethods 1 – ClinicalTrials.gov search criteria
Condition or disease search terms:
ISCHEMIC HEART DISEASE: coronary artery disease OR coronary disease OR coronary heart disease OR coronary occlusion OR acute coronary syndrome OR myocardial ischemia OR angina pectoris OR angina, stable OR angina, unstable OR myocardial infarction OR ischemic heart disease
LUNG CANCER: Lung cancer OR lung neoplasm OR lung carcinoma OR non-small-cell lung cancer OR non-small-cell lung carcinoma OR small cell lung cancer OR small cell lung carcinoma OR lung tumor OR lung tumour
DIABETES: diabetes mellitus OR diabetes
Study type: “Interventional Studies (Clinical Trials)”
Recruitment status: “Recruiting, Completed, Suspended, Terminated, Active not recruiting, enrolling by invitation and unknown status”
Study start: 2009-01-01 to 2010-12-31
eMethods 2 – Trial Inclusion and Exclusion criteria
Inclusion criteria:
- Primary outcome = clinical decision-related outcome: including mortality, morbidity, quality of life, functional status, need for further interventions, or an appropriate surrogate measures (for example, ejection fraction, Hgb A1c, or progression free survival in ischemic heart disease, diabetes mellitus and lung cancer respectively)
- Intervention of any type (drug, device, behavioral, surgical or other) directed towards the treatment or prevention of ischemic heart disease/lung cancer/diabetes mellitus (and not side-effects of the disease or complications from disease treatment)
- Trials with a US site
- Randomized trials
- Multi-arm trials
- Trials of interventions subject to FDA regulations and are FDA approved prior to trial start
- Trials of interventions not subject to FDA regulations
- Trials of interventions subject to FDA regulations, FDA approved post trial start, and have at least 5 years of follow-up since FDA approval, to allow for ample time for results incorporation into systematic reviews/clinical practice guidelines/UpToDate
Exclusion criteria:
- Exclusively evaluating safety, diagnostic or screening interventions
- Exclusion of phase 0 or phase 1 trials (given these early phase trials would be unlikely to inform clinical decision-making)
- Exclusion of extension studies with primary aim to enable continued access to drug (and without additional post-marketing surveillance outcomes)
- Indeterminate trials, which are ongoing trials have not surpassed double the allotted time for primary outcome completion (as first stated in the historical clinicaltrials.gov registration record)
eMethods 3 – Flow Diagrams for Each Disease
Flow Diagram for Ischemic Heart Disease Interventional Trials
Flow Diagram for Diabetes Mellitus Interventional Trials
Flow Diagram for Lung Cancer Interventional Trials
eMethods 4 – Assessment of Regulatory Approval Status
Two authors (NH & HM) independently evaluated all eligible trials for regulatory approval status using Drugs@FDA1 for drug and biological interventions and the 510(k) Premarket Notification website for devices.2 Interventions were classified into one of 3 categories: i) FDA approved prior to trial start (drug, biological or device interventions approved for any use by the time of trial start); ii) FDA approved at least 5 years ago ((drug, biological or device interventions approved for any use prior to October 31, 2016); and, iii) interventions not subject to FDA approval.
eTable 1 – Addressing the 4 Conditions for Informative Clinical Trials
eMethods 5 – Flow Chart of Informative Criteria Asses
eMethods 6 – Classification of Reason for Termination
eMethods 7 – Methodology for Publication Search
The search for publications was independently performed by two authors (NH & HM) and included an evaluation of publication links provided on ClinicalTrials.gov, as well as directed searches on Google Scholar, Scopus and Medline using a combination of the unique trial registration number (NCT number), surname of the principal investigator, indication, intervention, phase and study design. Publication identity was confirmed by comparing trial arms, sample size, intervention details, comparators and sponsor with the registration record. A published abstract was not counted as a full publication. Publication search was repeated in October 2021 by NH & HM for those trials without publications when first assessed.
eMethods 8 – Systematic Review Citation Search Strategy and Quality Assessment
Assessment of citation of trial results in high quality systematic reviews (SRs) was independently performed by two authors (NH & HM). This first involved a search for sources that are well known for producing high quality SRs: Cochrane SRs on the Cochrane Database of Systematic Reviews4 and Agency for Healthcare Research and Quality (AHRQ) SRs 5. If trials were not included in Cochrane or AHRQ reviews, additional SRs for published studies were identified using the Scopus database 6 citation analysis search function or via Google Scholar 7 for unpublished studies. SRs identified through Scopus or Google Scholar that included trial results in review results were assessed for quality using a modified AMSTAR scoring system:
Operationalization of modified AMSTAR8,9 scoring system
Trials were deemed to have fulfilled the importance criterion if they were cited in the results of a Cochrane SR, an AHRQ SR, or an SR achieving a modified AMSTAR score of greater or equal to 3 out of 5.
Assessment of citation of trial results in SRs was repeated in October 2021 by NH & HM for those trials without an informative citation when first assessed.
eMethods 9 – Clinical Practice Guideline and Point-of-Care Medical Database Search Strategies and Quality Assessment
Assessment of citation of trial results in Clinical Practice Guidelines (CPGs) was independently performed by two authors (NH & HM). CPGs were identified via Scopus 6 citation analysis for published studies or via Google Scholar 7 for unpublished trials. Quality of CPGs was assessed using a modified AGREE II scoring system:
Operationalization of modified AGREE II 10 scoring system
Trials were deemed to have fulfilled criteria for importance if they were cited in the results of a high-quality CPG. The remaining uncited trials were assessed for inclusion in a point-of-care medical database article by two authors (NH & HM). Using disease and intervention keywords, we searched UpToDate11 to identify any articles citing the remaining trials.
Assessment of citation of trial results in Clinical Practice Guidelines (CPGs) was repeated in October 2021 by NH & HM for those trials without an informative citation when first assessed.
eMethods 10 – Operationalization of modified Cochrane Risk of Bias score
We employed a modified 2011 version of the Cochrane Risk of Bias Assessment Tool (detailed criteria for judging risk of bias provided in Table 8.5d of the Cochrane Handbook for Systematic Reviews of Interventions version 5.1) 12. When available, ROB scores were extracted directly from high-quality SRs identified during the assessment of trial importance. When not available, assessment was independently carried out by two authors (NH & HM), with differences resolved by a third (JK). Assessment included the following elements: i) random sequence generation; ii) allocation concealment; iii) blinding of participants and personnel; iv) blinding of outcome assessment; v) incomplete outcome data; and, vi) selective reporting. Trials were deemed of sufficient design quality if all elements were deemed to be “low risk of bias” or if a minority of elements were deemed of “unclear risk” and the remaining were “low risk.” Any “high risk” of bias element equated with poor trial design.
Of the 63 trials assessed for trial design, 36 ROB scores were extracted directly from SRs, the remaining 27 trials were assessed by the study team.
eMethods 11 – Deviations to the Study Protocol
For our feasibility assessment, we first evaluated feasibility of goal patient enrollment and planned date of primary completion based on the first record available on ClinicalTrials.gov (this was independently double-coded). However, we repeated this assessment using the final registration record prior to trial start date as this was felt to provide a better evaluation of feasibility, allowing investigators to adjust enrollment plans and primary outcome timeline prior to trial start. The latter method was single coded and did not produce any change in the results of our feasibility assessment.
We performed two additional sensitivity analyses on the primary outcome: i) excluding all trials in the lower quartile of goal patient enrollment; and, ii) excluding all phase 1/2 and phase 2 trials from the assessment.
We excluded evaluation of primary outcome integrity from our assessment of trial informativeness, given concerns that there can be scientifically valid reasons for altering a primary outcome. For example, a primary outcome might be changed due to evolving clinical practice or in response to new data from outside the trial.
We excluded trials of interventions that were subject to FDA regulations, but were never approved for any indication, or were FDA approved after trial start, but did not have 5 years of follow-up time from 31 October 31 2016 post-approval to allow for enough time for trial results to be included in systematic reviews/clinical practice guidelines/UpToDate.
We excluded Indeterminate trials from our cohort, defined as ongoing trials that have not surpassed double the allotted time for primary outcome completion (as first stated in the historical clinicaltrials.gov registration record)
eMethods 12 – STROBE Checklist for Cohort Studies
eTable 2 – Inter-rater Agreement Rates
eTable 3 – Proportion of Trials Meeting Each Criterion for Informativeness
eTable 4 – Phase 4 Trials Not Meeting All 4 Informativeness Criteria
eTable 5 – Trials Not Cited in Clinical Review Documents
Footnotes
Conflicts of Interest: JK received consulting fees from Amylyx Inc. Authors declare no other potential conflicts of interest.