Comparing the clinical trial characteristics of industry-funded trials and non–industry-funded trials

INTRODUCTION

In 1948, the Medical Research Council’s trial of streptomycin for the treatment of pulmonary tuberculosis became the first published randomized controlled trial (RCT) in medicine.¹ The RCT design is now considered to be the gold standard for generating rigorous and reliable data to inform clinical decision making.² Randomization removes selection bias and, assuming the trial is sufficiently large, balances measured and unmeasured covariates.^3,4 Randomization can also ensure that patients have a consistently defined study start date, which prevents immortal time bias.⁵ When investigators conducting RCTs blind participants, clinicians, and researchers, the risk of ascertainment bias and observer bias are minimized, which helps to prevent time-varying confounding.³ Maintaining the rigor of a well-designed and well-conducted RCT requires considerable time and financial investment. The average cost of an RCT varies depending on the disease area, but in the United States the cost of Phase 3 RCTs range, on average, from $12 million to $53 million.⁶ These costs are often prohibitive for academic investigators.

The potential for bias in industry-sponsored trials has been well documented, and has led to important questions about whether corporate entities with a financial interest in the outcome of trials should be so closely involved in the research.⁷ Concerns have been raised that industry-sponsored trials may be terminated early due to financial reasons rather than scientific or ethical reasons, and that academic investigators who receive corporate funding may be incentivized to bias the analysis and reporting of trial results.⁷ Further, there have been reports of industry choosing to selectively present positive results and withhold negative results. Among physicians, these documented instances have fostered mistrust in industry-sponsored trials.⁸ One well-cited example is GlaxoSmithKline’s marketing of Paxil (paroxetine) for adolescent major depression.

The initial study publication in 2001 indicated that paroxetine for treatment of adolescent major depression was generally well tolerated, and demonstrated significantly greater improvement compared with placebo.⁹ However, a 2015 post-publication analysis of the original raw data revealed that GlaxoSmithKline had manipulated the data and selectively downplayed the harms of the drug: in fact, paroxetine is no better than placebo for any prespecified primary or secondary efficacy outcome, and use could result in serious side effects including self-injury and suicide.¹⁰

There is consequently—and understandably—a perception among clinicians that RCTs funded by industry may be of lower quality. In a 2012 study, Kesselheim et al.¹¹ created a series of abstracts for hypothetical clinical trials and randomly assigned to each the designation of either pharmaceutical company funding, federal funding, or no funding. Board-certified internists were then asked to evaluate the quality of the hypothetical trials. Despite the content of the abstract being otherwise identical, physicians who received the version indicating funding by a pharmaceutical company perceived the methodologic quality more negatively than physicians evaluating the exact same abstract with an indication of either federal funding or no funding.

Systematic analysis of how trial characteristics differ between industry-funded and non–industry-funded trials may offer insight into the extent to which pre-conceived notions of study quality based on funder are accurate. Our objective was to compare study characteristics of RCTs funded by industry to those of RCTs not funded by industry.

METHODS

Study Population

We reviewed all RCTs published in the New England Journal of Medicine (NEJM), the Journal of the American Medical Association (JAMA), and Lancet between January 1, 2015 and December 31, 2019. These journals were selected because they have the highest impact factors among general medicine peer-reviewed journals, and they commonly publish RCTs. We chose to stop data collection at the year 2019 (inclusive) because the advent of the COVID-19 pandemic may have changed the landscape of RCT publishing. Excluding trials after 2019 also afforded at least two years of follow-up in the evaluation of post-publication metrics for each RCT. Using MEDLINE, we identified all articles published in these journals during the study period and then excluded review articles, research letters, letters to the editors, and editorials. The title and abstract of the remaining articles were reviewed independently by three study members (MF, UP, AR) to identify and exclude any remaining non-randomized studies and duplicate publications. Disagreements were resolved through consensus. A detailed outline of the study selection process is provided in Figure 1.

• Download figure
• Open in new tab

Figure 1.

Study Flow Diagram

NEJM = New England Journal of Medicine, JAMA = Journal of the American Medical Association, RCT = randomized controlled trial

RESULTS

We identified 1533 RCTs published in NEJM, JAMA and Lancet between January 1, 2015, and December 31, 2019. Most of the RCTs were trials of medications. Approximately one-third were double-blind, approximately 40% had a surrogate outcome as their primary endpoint, and approximately two-thirds had a positive primary outcome (Table 1). Of all the RCTs, 697 were funded by industry while 835 were not; one study lacked funding details altogether.

Studies funded by industry were more likely to be related to cardiovascular disease or oncology, and less likely to be related to infectious disease (Table 1). The median sample size was 557 (IQR: 230-1369) for industry-funded trials and 648 (IQR: 301, 1916) for non–industry-funded trials. Compared to non–industry-funded trials, industry-funded trials were more likely to be double-blind, more likely to include a placebo, and more likely to have a surrogate primary outcome (Table 1). The primary outcome was positive for 82% of industry-funded RCTs compared to 54% of non–industry-funded RCTs.

We found that the concluding statements of trials funded by industry were more likely to have greater positive sentiment than those not funded by industry (n=246, 35% vs n=208, 25%), which aligns with the fact that industry-funded trials were more likely to have positive findings. Our findings were robust across two different natural language processing (NLP) methods (Table 2).

Industry-funded studies received twice as many citations as non–industry-funded studies (n=286 [IQR: 140-562] vs n=145 [IQR: 76-266], p < 0.01). AltMetric scores were similar for all included studies, regardless of their funding source (n=229 [IQR: 122-468] for industry-funded; n=226 [IQR: 119-441] for non–industry-funded, p=0.2). Industry-funded studies were also more likely to post their results on ClinicalTrials.gov compared to non–industry-funded studies (78% vs 41%).

DISCUSSION

In this study of RCTs published in NEJM, JAMA, and Lancet between 2015 and 2019, we identified 697 RCTs that were funded by industry and 835 that were not. Trials funded by industry were more likely to be blinded, more likely to be placebo controlled, and more likely to post trial results on ClinicalTrials.gov. These data suggest that there are key metrics of trial quality that may be better conducted in industry-funded trials compared to non–industry-funded trials, adding important and often overlooked nuance to the broader discussion about industry-funded research.

The greatest strengths of RCTs are that selection bias is removed and, assuming the study is large enough, both measured and unmeasured confounders are balanced. However, without blinding it is almost impossible to prevent time varying confounding or ascertainment bias, both of which can directly affect the internal validity of a study, especially when the primary outcome is subjective or prone to measurement error. Trials funded by industry may have been more likely to include blinding and a placebo for several reasons. First, industry is less likely to perform pragmatic trials that test an intervention in real-world practice; this absence of real-world conditions makes it more feasible to include double-blinding and a placebo control.

Second, industry entities manufacture not only the drug but also the matching placebo, facilitating easy access. Third, placebos are expensive, and industry-funded trials are typically better resourced than non–industry-funded trials.^7,15 Fourth, there is often little incentive for a pharmaceutical company to use an available medication as their comparator, as opposed to a placebo; importantly, this can be problematic in cases where the use of an active comparator instead of placebo would have generated more clinically important data.

Another important consideration is the primary outcome itself. We observed that industry-sponsored trials were more likely to utilize surrogate outcomes, which are arguably less clinically relevant to patients and clinicians. And while we did observe that industry-funded trials were smaller than non–industry-funded trials, the absolute difference was modest (i.e., 67 participants).

We found that industry-funded trials are more likely to make their results publicly available on ClinicalTrials.gov. Our results align with a recent study of 4209 trials registered on ClinicalTrials.gov which identified that while two-thirds of the registered trials posted their results, the odds of this occurring were three-fold higher for industry-sponsored trials compared with non–industry sponsored trials.¹⁶ Similarly, Anderson et al. (2015) found that adherence to legal obligations (outlined by the Food and Drug Administration Amendments Act) for timely reporting of results within 12 months of trial completion was higher among industry-sponsored trials than NIH funded or academic institution funded trials.¹⁷ This observation might reflect that industry-funded RCTs are more likely to be for new molecules, which legally obligates the investigators to adhere to stricter regulatory reporting.

While reporting compliance appears to be higher among industry-funded trials, it is important to highlight prior literature documenting publication bias among this group. There are instances of industry-funded trials not publishing at all if the trial was negative, or choosing to present positive results while withholding negative results.^18–22 Consider the previously cited example of Paxil by GlaxoSmithKline, which was prescribed to adolescents for years before it was confirmed to not only be ineffective, but to cause serious side effect including self-injury and suicide.¹⁰ Or the example of Roche’s Tamiflu (oseltamivir): the initial results from industry-funded trials indicated it was effective in reducing hospitalizations and serious complications from influenza, which resulted in governments stockpiling Tamiflu in advance of potential influenza outbreaks.²³ However, a Cochrane review subsequently identified that 60% of patient data from the randomized, placebo controlled, phase 3 treatment trials were never published, and that the exclusion of these data significantly changed the findings; in reality, oseltamivir was less effective in reducing complications of influenza than previously stated.²⁴

While our study found that trials funded by industry were more likely to have a positive outcome, there are multiple reasons why this might be the case. Publication bias may explain why there was a higher number of positive industry-funded trials; if results aren’t positive, researchers are less likely to pursue publication. Industry-funded trials might be more likely to be positive because such studies focus their resources on drugs and devices that are more likely to work based on earlier phase studies, or involve comparison against placebo rather than an established effective medication. We also found that the concluding statements of trials funded by industry were more likely to have greater positive sentiment than those not funded by industry, which aligns with the greater likelihood of industry-funded trials to have positive outcomes. Our study did not find evidence to either refute or support the claim that industry-sponsored research chooses to withhold negative results.

One limitation of our study is that we lacked data on publication bias because we only included studies that were published. Another limitation is that we focused on three of the highest impact medical journals, and thus our results may not apply to RCTs published in other medical journals. Finally, we lacked data on other aspects of trial design that can directly affect internal validity, including concealment of allocation, number of study sites (e.g., single-centre compared to multi-centre) adherence, statistical analysis (e.g., intention to treat), and loss to follow up.

Taken in sum, our findings suggest that industry-funded RCTs published in high impact journals (i.e., NEJM, JAMA, Lancet) are more likely to be blinded, more likely to include a placebo, and more likely to post trial results on ClinicalTrials.gov. Industry funding has been instrumental to major trials with clinical importance, so a culture of systemically undervaluing such trials is concerning. Our findings emphasize the importance of evaluating the quality of an RCT based on its methodological rigour, not its funder type. Our observations also raise questions about how we can empower non–industry-funded research to improve performance in some of these metrics of trial quality. Further research is needed to determine trial characteristics of (a) industry-funded research that is not published and (b) industry-sponsored research that is published in other journals, as the findings from this current study may not be generalizable to these other areas.