ABSTRACT
Background Randomized-clinical trials (RCTs) are the gold-standard for comparing health care interventions, but can be limited by early termination, feasibility issues, and prolonged time to trial reporting. In contrast, adaptive clinical trials (ACTs), which are defined by pre-planned modifications and analyses that occur after starting patient recruitment, are gaining popularity as they can streamline trial design and time to reporting. As adaptive methodologies continue to be adopted by researchers, it will be critical to develop a risk-of-bias tool that evaluates the unique methodological features of ACTs so that their quality can be improved and standardized for the future. In our proposed methodological review, we plan to develop a list of risk-of-bias items for ACTs to develop a candidate instrument.
Methods and analysis We will perform a systematic database search to capture: (1) ACTs published in any discipline of medicine and/or surgery; and (2) studies that have proposed or reviewed items pertaining to methodological risk, bias, and/or quality in ACTs. We will perform a comprehensive search of citation databases, such as Ovid MEDLINE, EMBASE, CENTRAL, the Cochrane library, and Web of Science, in addition to multiple grey literature sources to capture published and unpublished literature related to ACTs and studies evaluating the methodological quality of ACTs. We will also search methodological registries for any risk of bias tools for ACTs. All screening and review stages will be performed in duplicate with a third senior author serving as arbitrator for any discrepancies. Included ACTs will be analyzed in a descriptive manner, and we will perform regression analysis to identify factors associated with poor reporting quality and high risk of bias. We will also perform a risk of bias assessment of ACTs using the Cochrane risk-of-bias 2.0 tool and we will assess reporting quality using the CONSORT-ACE tool. These assessments will be performed independently and in duplicate. This will be done to help generate risk of bias concepts, themes, and items that can be included in the candidate tool. For all studies of methodological quality and risk of bias, we will extract all pertinent bias items and/or tools. We will combine conceptually similar items in a descriptive manner and classify them as referring to bias or to other aspects of methodological quality, such as reporting. We will plan to generate pertinent risk of bias items and fields and finally, a candidate tool that will undergo further refinement, testing, and validation in future development stages.
Ethics and dissemination This review does not require ethics approval as human subjects are not involved. As mentioned previously, this study is the first step in developing a tool to evaluate the risk of bias and methodological quality of ACTs.
INTRODUCTION
Evidence-based medicine has revolutionized the development of clinical practice guidelines and decision making in healthcare [1]. Randomized-controlled trials (RCTs) are the gold-standard for comparing the effectiveness and safety of novel healthcare interventions [2]. Conventional RCTs, however, can be burdened by high costs, early termination due to feasibility issues, and an overly rigid design that does not permit adjustments for unforeseen challenges [3]. These issues are amplified in surgical trials and as such, the annual number of published surgical trials remains stagnant [4, 5]. As a response to these challenges, researchers have begun using adaptive trial designs, which allow for dynamic protocol changes after beginning patient recruitment. Adaptive clinical trials (ACTs) Adaptive clinical trial (ACTs) are a novel approach may help to streamline patient recruitment, drop poorly performing treatment arms, and combine clinical trial stages, thereby, reducing unnecessary trial costs, decreasing sample size, and accelerating time to reporting [6]. For instance, the TAILoR trial of telmisartan in HIV employed an interim analysis at half maximal patient recruitment and dropped the most ineffective medication dosage group based on a pre-specified efficacy threshold [7]. Adaptive methodologies helped to streamline the clinical trial process during COVID-19 to optimize the number of therapies evaluated and reduce uncertainties related to the natural history of the disease and the number of patients necessary for each trial [8]. Adopting ACTs when prolonged RCTs are impractical may also reduce needed funding, thereby overcoming barriers to conducting trials in developing nations [9]. Stakeholders, however, report that adaptive trial designs remain nebulous with practical barriers, including high bias potential, ethical concerns, and a lack of knowledge dissemination amongst trialists [10].
In 2020, CONSORT published an extension for adaptive trials to guide ACT reporting [11]. These guidelines ACT-specific methodological components such as pre-planned interim analyses and sample size estimation (and re-estimation) descriptions [11]. In conjunction with CONSORT reporting guidelines, a validated risk-of-bias tool developed in a similar manner to the Cochrane risk-of-bias 2.0 tool [12], may improve the design of ACTs and the quality of future meta-analyses combing ACTs. Risk-of-bias tools are designed for specific study designs (e.g. RCTs) and help to promote methodological transparency and reproducibility while minimizing bias, so results can be accurately interpreted and soundly applied to patient care. For conventional RCTs, there exist several tools and checklists to guide reporting or evaluate quality and bias risk (Table 1) [12-25]. There is no existing risk-of-bias tool to evaluate the methodological limitations of ACTs, which is of particular importance due to the potential for ACTs to be impacted by bias if not soundly designed [26]. It is for this reason that we have decided to embark upon creating a novel risk-of-bias tool to improve the quality of future ACTs and ACT meta-analyses.
Our proposed methodological review has two main objectives: (1) to describe ACTs in medicine and surgery and evaluate them using the CONSORT ACE guidelines, so that we can brainstorm novel risk of bias items and (2) to search and identify any current risk of bias items pertaining to ACTs. We will develop our risk-of-bias tool for ACTs in accordance with the framework described by Whiting et al. (2017) [27].
METHODS
Study design
We present a protocol to describe the rationale for performing a methodological review of ACTs and generate a list of bias items related to ACTs. We will follow the methodological framework proposed by Whiting et al. (2017) and Sanderson et al. (2007) [27, 28]. This protocol was written with guidance from a methodology review protocol published by Lunny and colleagues, who set out to create a novel risk-of-bias tool for network meta-analyses [29]. As described by Lunny and colleagues, subsequent steps in creating a risk-of-bias tool will include (1) a multi-round Delphi survey and panel to select, refine, and compile bias items into a single tool; (2) a pilot test to further refine the draft tool; and (3) a knowledge translation strategy to disseminate the final risk-of-bias tool [29]. These steps will be addressed in future studies as we progress through this framework in developing this proposed ACT risk-of-bias instrument. We did perform this systematic review following the PRISMA-P checklist [30].
Eligibility
There will be three types of studies included in this scoping review. Study type 1 will be ACTs published in the medical and/or surgical literature. A review of all ACTs investigating medical and surgical interventions will facilitate a comprehensive description of the type and quality of ACTs. With regards to the review of all published ACTs, we will perform our search to take place after that of Purjak et al. (2022) who performed a scoping review of all RCTs with adaptive designs and evaluated reporting quality [31]. Study type 2 will be studies that describe items related to bias, reporting, or methodological quality of ACTs in any discipline. We will retain all items related to methodological bias and/or reporting as they may be able to be translated into a risk-of-bias tool. Study type 3 will be studies that assess the methodological quality, or risk of bias, of ACTs using criteria that focus on methodological features specific to ACTs. Study type 1 articles will be categorized and analyzed separately, while study types 2 and 3 will be analyzed with the goal of collating bias items. All relevant inclusion and exclusion criteria are listed in Table 2.
We will include all articles with any publication status and written in any language. In cases where the co-authors are not fluent or review authors are unable to understand the study text, we will utilize Google Translate (Mountain View, CA, USA). If we identify a systematic review or meta-analysis of ACTs, we will not include the systematic review in the final ACT synthesis, but instead any primary ACTs that meet inclusion criteria.
Search Strategy
All databases to be used in this review were selected with guidance from Lunny et al. (2021) [29]. We will search all databases with no language or publication type limits. We will search the following databases: MEDLINE (Ovid), CINAHL, EMBASE (Ovid), the Cochrane library, the Cochrane Central Register of Controlled Trials (CENTRAL), Web of Science, BIOSIS, Derwent Innovations Database, and KCI. We will also search clinical trials registries including clinicaltrials.gov and the WHO International Clinical Trials Registry Platform (ICTRP). All searches for peer-review ACTs will be performed from the end date of the scoping review of ACTs published by Purja and colleagues [31]. We will search the following grey literature databases and resources: the EQUATOR network, dissertation abstracts, websites of evidence synthesis organizations (e.g., Campbell Collaboration, Cochrane Multiple Treatments Group, CADTH, NICE-DSU, Health Technology Assessment International (HTAi), Pharmaceutical Benefits Advisory Committee, Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen, European Network for Health Technology Assessment, Guidelines International Network, ISPOR, International Network of Agencies for Health Technology Assessment, and JBI), and methods collections (e.g., Cochrane Methodology Register, AHRQ Effective Healthcare Programme). We will also search LIGHTS and LATITUDES (https://www.latitudes-network.org/), which are two methodological registries that capture guidance and validity assessment tools, respectively [32]. All online registries will be searched using the following terms “adaptive clinical trial”, “bias”, and/or risk-of-bias (ROB)”. The words found within the titles, abstracts, and MeSH terms of relevant articles were used to develop focused search strategies for each database. Reference lists of studies found will also be searched for additional papers to be included. The MEDLINE search will be validated for 10 studies identified by senior authors prior to screening. Eligibility screening will only begin after these 10 trials are identified from the search strategy. All database search strategies are described in Appendix 2.
The search strategy will be generated by two authors (P.S. and D.O.) alongside a librarian specialist. It will be generated and reviewed in accordance with PRESS (Peer Review Electronic Search Strategies) guidelines [33]. Any concerns with search strategy generation will be raised with a senior methodologist (M.B.). The database search will be conducted without limitations to publication type, status, language, or date to identify existing tools or articles.
Screening and data extraction
First, we will pilot eligibility criteria in Microsoft Excel (Redmond, WA, USA) by evaluating a sample of 25 citations amongst two independent reviewers. If high agreement is achieved (≥70%), then we will continue to abstract screening with two reviewers. If less than high agreement is achieved, then the eligibility criteria will be re-examined, and additional teaching sessions will be provided to reviewers. All screening and full-text review will be conducted using the web-based application, Covidence (http://www.covidence.org; Melbourne, Australia). Study titles and abstracts will then be assessed for relevance and eligibility. Any disagreements identified during these screening and review stages will be resolved via discussion until consensus is reached. A third senior reviewer will arbitrate if screening or full-text review disagreements cannot be resolved.
A data extraction form will be generated using Microsoft Excel and piloted by reviewers on five included studies. Two authors will extract data from all included studies. We will first categorize all sources based on our eligibility criteria and we will extract author details, publication year, and study type. For all ACTs, we will extract the following information: type of adaptive design, patient details (e.g., number of enrolled/analyzed patients, eligibility criteria, patient demographics), pre-specified interim analyses and adaptive decisions, sample size calculations, treatment group details (e.g., randomization plan, allocation sequence generation, number of treatment groups, blinding details and maintenance), outcome details (e.g., primary and secondary outcome measures, statistical analysis plan), results (e.g., baseline patient data, reported outcomes, final statistical analysis, patients lost to follow-up), and any methodological limitations identified by the authors. All risk of bias analyses will be performed by two independent reviewers using the Cochrane ROB 2.0 tool. We will also perform an assessment of reporting quality using the CONSORT ACE guidelines. All disagreements will be resolved via discussion and if consensus is not reached, then a senior reviewer (P.S.) will arbitrate the final decision. From this process of risk of bias and reporting quality assessment of published ACTs, we will brainstorm novel risk of bias items and fields.
For all studies that identified bias items, tools, or quantified methodological bias in ACTs, we will generate the following list of headings: type of tool (e.g., tool, scale, checklist, or domain-based tool); scope of the tool; number of items within the tool; domains within the tool; whether the item relates to reporting or methodological quality; ratings of items and domains within the tool; methods used to develop the tool and the availability of an “explanation and elaboration”. These fields were all derived from Lunny et al. (2021) and Page et al. (2018) [29, 34]. Data will be extracted on items that are relevant to ACTs and all items will initially be extracted verbatim.
Data analysis and reporting
We will plan to split the analysis based upon study design: (1) ACTs will be collated and described based upon all extracted fields and (2) all studies evaluating methodological quality, and/or proposing bias tools or items will be collated and items will be grouped accordingly. Firstly, we will divide ACTs based upon their respective discipline and year of publication. We will perform descriptive analyses (i.e., proportions, mean, ranges, standard deviations), where applicable, for all extracted data fields.
Next, we will categorize all studies of methodological quality and risk-of-bias tools/items in ACTs. These studies will again undergo descriptive analyses based upon previously extracted fields. All bias items will all be mapped to corresponding domains within the CONSORT-ACE guidelines, as this is the only known quality tool specific to ACT features. If no risk-of-bias tools or relevant items are identified in our review, then we will plan to hypothesize items and formulate candidate tool. We will develop a candidate tool that will undergo refinement by the authors, and in subsequent steps will be further evaluated and refined using a Delphi consensus method prior to undergoing validation and testing. All statistical analyses will be performed using R software (version 4.3.2).
Public and public involvement
Patient or the public were not involved in the design of this research protocol.
DISCUSSION
A comprehensive risk-of-bias tool is needed to facilitate the publication of high-quality ACTs and any future pooled analyses of ACTs. A recent scoping review of ACTs has demonstrated poor reporting based on the CONSORT-ACE guidelines [31]. We are, therefore, going to use the framework developed by Whiting and colleagues to develop a new risk-of-bias tool for randomized trials with adaptive design features [27]. The framework developed by Whiting and colleagues has been used to develop the QUADAS tool for diagnostic test studies and a risk-of-bias tool for network meta-analyses. As AI networks and computational technologies continue to be optimized, their role in generating adaptive trial paradigms and performing statistical simulations may revolutionize the future of trial design and medical innovation [35]. We, must, therefore ensure that methodological tools are developed at a similar pace so that novel trial designs and adaptive paradigms are standardized, transparent, reproducible, and interpretable for the future.
Data Availability
No datasets were generated or analysed during the current study. All relevant data from this study will be made available upon study completion.
Contributors
PS, TM, HZ, MA, MKG, MB, and JP conceived the research question. PS, CL, DO, EO, TM, AT, and KN composed the manuscript. All authors edited and helped to draft the manuscript. The final version of this manuscript was approved by all authors. All authors have agreed to be held accountable for this work.
Funding
None
Competing interests
None
Patient consent for publication
None
Acknowledgements
We thank Dr. Edward Mills for his helpful manuscript comments and suggestions.
APPENDICES
Appendix 1. Database search strategies
Footnotes
Conflicts of interest: None
Funding: None
REFERENCES
