fMRI as an outcome measure in clinical trials: A systematic review in clinicaltrials.gov

Introduction

FMRI is the most powerful and dominant imaging technique in living human brain by now that have entered a variety of branches of today’s clinical research to apply new advances in clinical practice. Speaking of clinical trials, fMRI’s involvement in different domains is so acknowledged that in the current NIH case studies regarding their 2014 definition of clinical trials, there are specific case studies for fMRI in 6 variations trying to elucidate the distinction of the different roles of fMRI in clinical trials as a clinical measurement tool and also as an intervention (NIH, 2018, case #18” a-f”).

In our expectation, the role of fMRI in clinical trials would hopefully be as a biomarker by definition, that is, “a characteristic that is objectively measured and evaluated as an indicator of normal biological processes, pathogenic processes, or pharmacologic responses to a therapeutic intervention” (Biomarkers Definitions Working Group, 2001). To be more specific, fMRI is supposed to be used in order to facilitate development of new interventions by objectively monitoring functional changes in brain, hypothetically targeted by the interventions. For example, changes in specific brain activity after taking pharmacological agents are detected by fMRI to evaluate the effect of treatment in early phases in drug trials. In this regard, fMRI would be promising for accelerating the development of new treatment; In drug development for mental health disorders, urgent need for new treatment was address by an NIMH program, Fast-Fail trials (FAST), utilizing target-engagement biomarkers including fMRI as one of the potential functional biomarker (https://www.nimh.nih.gov/research-priorities/research-initiatives/fast-fast-fail-trials.shtml). FMRI also might play an exploratory role in clinical trials to providing more objective evidence base for submission to regulators, e.g. in later phases of drug trials (Carmichael et al., 2017). In this study we examined fMRI as an outcome measure, but speaking more broadly, by NIH definition of intervention—manipulation of the subject or subject’s environment for the purpose of modifying one or more health-related biomedical or behavioral processes and/or endpoints—it seems that fMRI can also be considered as an intervention in clinical trials in certain conditions (e.g. pre-surgical mapping) (NIH, 2018, case #18e).

Nonetheless, for fMRI to become a powerful biomarker in clinical trials, still some technical and logistical measurements need to be taken in order to make the results reliable and valid. There is a high range of false positive results in fMRI activations (Wager et al., 2009) and the vast choice of analysis are one of the main reasons (Poldrack et al., 2017); Carp (2012) showed that 6,912 unique analysis pipelines are possible for one single dataset which leads to 34,560 significance maps that varies in activation strength, location, and extent. Registration of fMRI methodology before the study or pre-registration which restricts “research’s degree of freedom”—a term coined by Simon (2011), meaning the flexibility of researcher at methodological level— is one of suggested logistical measurements to deal with false positive results (Button et al., 2013; Munafò et al., 2017; Poldrack et al., 2017; Carmichael et al., 2017). Clinical trials’ protocol registration in publicly available clinical trial registries is mandatory by laws and policies (https://clinicaltrials.gov/ct2/about-site/history). ClinicalTrials.gov is the largest clinical trials registry with over 200,000 registered studies. The registry was first established to make possible the public access to clinical trials for participation purposes, and as the need for transparency and reproducibility in science emerges, it extended its goals to also serve as a registry for better monitoring and tracking of clinical trials (Zarin & Keselman, 2007). As the time passed, the site tightened its rules and modified the protocol registration’s required information for more accurate and complete registration. However, for fMRI in clinical trials, the best practices for pre-registration needs to be well-defined through a detailed checklist. If following that checklist becomes obligatory by policy makers and granting agencies as a module in the protocol registration in formal clinical trial registries, like clinicaltrials.gov, hopefully it will result in more valid and replicable fMRI outcomes in clinical trials in long term.

In this study, we completed a systematic review to investigate the scope of fMRI clinical trials in clinicaltrials.gov. We extracted the available data for characteristics of fMRI clinical trials. For categorizing the large number of tasks in task-based fMRI trials, we chose NIMH Research Domain Criteria (RDoC) construct definition (https://www.nimh.nih.gov/research-priorities/rdoc/index.shtml). As a potential pre-registration role that clinicaltrials.gov could play in future for an fMRI specific data element, we also explored the extent of available pre-registered data in clinicaltrials.gov and provided few recommendations.

Method

Search strategy and study selection

We searched clinicaltirals.gov on 10/13/2018 for potential clinical trials that used task-based or resting-state fMRI as their outcome measure by using the search term fMRI. Clinicaltrials.gov is a registry that includes more than clinical trials (or interventional studies). Hence, the search was then restricted to clinical trials by applying one of the potential search filters “study type: Interventional”. For further analysis, we downloaded the results in a tab-separated values format and transferred it to an excel table to fix the dataset for later analysis. The dataset encompassed eligible trials’ protocol mandatory required information (data elements) which was submitted to the clinicaltrials.gov registry by sponsor or principle investigator at the start of the study. In order to include completed or ongoing clinical trials we excluded Suspended, Terminated, Withdrawn, and Unknown study states. We also excluded the following studies by manually reviewing each trial’s record page:

• Trials that the outcome measures were recording data of real-time neuro-feedback fMRI to reduce methodological complexities

• Trials with empty (not recorded) outcome measures

• Trials that didn’t report fMRI in their outcome measures either as a consequence of incomplete reporting, or in other cases e.g. if fMRI was used as a part of an intervention in the trial, like “brain surgery based on pre-surgical planning with fMRI”

• Explicitly stated Not BOLD fMRI trials e.g. perfusion fMRI

• Non-relevant trials that appeared in our result as a result of similarity of keywords

Trials’ selection process is outlined in the flow diagram (Figure 1). To follow the best practices in reporting, we applied relevant ‘Preferred Reporting Items for Systematic Reviews and Meta-analyses’ PRISMA guidelines. See Checklist S1 for PRISMA checklist of items.

• Download figure
• Open in new tab

Figure 1.

Inclusion flowchart of fMRI trials registered in clinicaltrials.gov on or before 10/13/2018

Result

The ClinicalTrials.gov registry includes 2026 entries for interventional studies with fMRI as a keyword. Among these, we identified 1386 completed (46%) or ongoing clinical trials (54%) that used BOLD fMRI as an outcome measure. The reasons for exclusion of 640 interventional trials are listed in Figure 1. Among completed trials 84% didn’t report their results yet. In 33% of trials fMRI was the only primary outcome (Table 1). 50% of trials had less than 50 participants and 6% more than 200. 82% of fMRI trials were started after 2011. The most frequent intervention was drug (29%). 57% trials had parallel assignment design and 20% were designed for cross over assignment (Table 2). More details on the characteristics of trials can be found in Table 1 and Table 2.

Forty-two percent of trials registered task-based fMRI, 19% resting state, and 12% of trials registered both as the outcome measures. There were 963 tasks in 753 trials that reported task-based fMRI as a part of their outcome measures. 20% of trials used more than one task (Table S1 provides details of the number of tasks in these trials). Positive valence systems (19%), cognitive control (15%), perception (10%), and systems for social processes (10%) were the most frequent domains/subdomains of tasks based on RDoC (Table 3). Food cue reactivity task, pain perception task, n-back task and monetary incentive delay task were recruited in more than 25 registered trials (Table 4). You can find more details on the fMRI tasks in each domain in Table S2. The trend of use in the most frequently recruited fMRI tasks is also reported in Table 5. We found that 28% of trials have reported at least one region of interest. We also included some of the best pre-registered data samples of pre-processing and analysis plan that was found in our clinical trials data-set in Table S3. The dataset is also shared in the Supplementary Material (Supplementary_DataSet).

Discussion

In order to develop an understanding of fMRI usage in clinical trials as an outcome measure, we systematically reviewed the fundamental characteristics of eligible trials registered in the ClinicalTrials.gov before 2019. During the last 20 years, fMRI as an outcome measure have grown rapidly such that about half of the total trials with fMRI as an outcome measure were registered after the year 2016. 54% of fMRI trials are still ongoing and 84 % of the completed trials didn’t report their result yet so we still do not have a good methodological representation of these ongoing or completed trials in the published literature. This is the first report to give an overview about the current status in the field based on the available data in clinicaltrials.gov. In about one third of the included trials in this systematic review, fMRI was the only primary outcome measure and the “Primary Purpose” of half of the trials was reported as “treatment”. 29% of trials examined a drug intervention. 70% of trials had randomized allocation. Triple and Quadruple blindness overall made up 25 percent of trials. Cognitive systems (46%) based on RDoC was the most frequent domain of tasks, followed by positive valence systems (19%), systems for social processing (10%), and sensorimotor systems (5%). Emotional processing tasks, cue reactivity tasks, pain perception tasks, N-back task, emotional faces processing task, motor tasks, and Go/no-go task was the most frequent tasks that used in these trials (please see supplementary material 2 for more details). fMRI statistical analysis details in registered trials is scarce and less than one third of trials registered at least one region of interest for their analysis.

We reviewed all registered materials for each trial to extract fMRI specific information that might be reported in different sections such as outcome measures, study arms, detailed description. In about 25 percent of trials the rest or task-based fMRI types were not specified. In about 10 percent of task-based fMRI trials there was not any clear indication (even of task-domain). Task’s name as well as reporting model was varied according to the following categories: 1) Conventional name mentioned explicitly (e.g., monetary incentive delay task) 2) Task was introduced in a general way (e.g., decision-making task, cognitive task) 3) The procedure was indicated (e.g., listen to baby’s cry, exposure to auditory and visual food cues, luminous stimulation) 4) Task’s name was not mentioned but the task’s details was described 5) evaluation of brain response to stimuli in study implicated the task (e.g. brain activity in response to noxious stimuli, as assessed by fMRI). We organized the data to be consistent and placed them in tabular format by categorizing the task domains according to the RDoC definition.

In an effort to search for pre-registered data of fMRI specification, we had a plan to collect details of registered fMRI information including data design, data acquisition, data pre-processing, and analysis plan, but the minute amount of available data and the lack of similarity in reporting style posted restrictions on this procedure so that we were unable to take our pre-registration classification any further (See Table S3). Even for basic details like regions of interest (ROI), less than one third of trials registered at least one ROI for their analysis.

At this point there is sufficient evidence that higher standards of pre-registration information are needed to help transparency and reproducibility of research, which would lead to bias reduction. Overall, as Munafo et al. (2017) have mentioned, study design, primary outcome(s), and analysis plans are what should be pre-specified for a study in the strongest form of pre-registration. There are also some previous efforts to recommend essentials for an fMRI study pre-registration; editors of “Neuroimaging: Clinical” referred to the existence possibility of prediction in clinical neuroimaging studies and pre-registration of ROIs as a specific suggestion against SHARKing. “This is common in clinical trials and there is no reason that strong predictions cannot be defined in clinical neuroimaging studies. Pre-registration of ROIs may be particularly useful as it guards against SHARKing “(Roiser et al., 2016). Poldrack and colleagues (2017) have indicated “planned sample size, specific analysis tools to be used, specification of predicted outcomes, and definition of any specific ROIs or localizer strategies that will be used for analysis” as the details of an fMRI study that should be pre-registered. Particularly in clinical trials —to extend Carmichael et al., (2017) statement is helpful (even though it is initially recommended for drug development trials): “fMRI methodology should be held to the same standard as other clinical endpoints, namely, methods must be pre-specified and fixed for the duration of the study. This pre-specification should include a thorough description of task design and implementation, image acquisition and quality control, data preprocessing, ROI definition, model estimation, and endpoint calculation (Schwarz et al. 2011a, b)”.

The current study is the first one that tried to provide an overview about registered trials with fMRI as an outcome measure in clinicaltrials.gov to the best of our knowledge. It should be mentioned that ClinicalTrials.gov is not the sole registry for all clinical trials in the world, so our database is a limited subset of all fMRI clinical trials. The use of fMRI as an outcome measures in clinical trials is growing fast, imposing the requirement for a considerable amount of registration standardization for fMRI specific information to make the accumulating data useful for researchers and assure the validation of results. Based on this systematic review, we suggest following actions:

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.