2. Abstract
Single-arm trials supplemented with external comparator arm(s) (ECA) derived from real-world data are sometimes used when randomized trials are infeasible. However, due to data sharing restrictions, privacy/security concerns, or for logistical reasons, patient-level real-world data may not be available to researchers for analysis. Instead, it may be possible to use generative models to construct synthetic data from the real-world dataset that can then be freely shared with researchers. Although the use of generative models and synthetic data is gaining prominence, the extent to which a synthetic data ECA can replace original data while preserving patient privacy in small samples is unclear.
Objective To compare the efficacy of nivolumab + ipilimumab combination therapy (“experimental arm”) versus nivolumab monotherapy (“control arm”) in patients with metastatic non-small cell lung cancer (mNSCLC) using real-world data from two real-world databases (“original ECA”), and synthetic data versions of these datasets (“synthetic ECA”), with the aim of validating synthetic data for use in ECA analysis.
Study design Non-randomized analyses of treatment efficacy comparing the experimental arm to the (i) original ECA and (ii) synthetic ECA, with baseline confounding adjustment.
Data sources The experimental arm is from the Lung-MAP no-match substudy S1400I (NCT02785952) provided by National Clinical Trials Network (NCTN) in the United States. The real-world data source for the ECA is data from population-based oncology data from the Canadian province of Alberta, and from Nordic countries in Europe, specifically Denmark and Norway.
3. Amendments and updates
4. Milestones
5. Rationale and background
Clinical researchers often encounter challenges in accessing patient-level real-world data for external comparator arms (ECAs) in single-arm trials, attributed to data sharing restrictions, privacy concerns, and logistical constraints. To address this issue, an emerging approach involves utilizing generative models to construct synthetic data that mirrors the statistical properties of real-world datasets. This strategy enables the creation of alternative comparator arms, which can be freely shared and analyzed, circumventing obstacles associated with data availability. However, the extent to which synthetic data faithfully reproduces the characteristics of the original real-world dataset remains uncertain, raising questions about its reliability and validity in the context of single-arm trials.
This research aims to explore the viability of synthetic data in external comparator arms by assessing its fidelity compared to real-world data. The study delves into the current landscape of single-arm trials, emphasizing the limitations in accessing patient-level real-world data and elucidating the role of synthetic data in overcoming these challenges. By investigating the potential benefits and limitations of synthetic data generation methods, the research seeks to provide valuable insights into the reliability of study outcomes derived from ECAs constructed using generative models, thereby contributing to the ongoing discourse on innovative methodologies for enhancing the robustness of single-arm trials amidst data constraints.
The following are details about the Lung-MAP trial used in this study:
What is known about the condition: Squamous cell lung carcinoma is a histological subtype of NSCLC that originates in the squamous cells lining the airways of the lungs. Historically, 25-30% of all cases of NSCLC are squamous cell carcinoma although these percentages can vary regionally and may change over time due to factors such as changes in smoking patterns. Compared to other NSCLC subtypes, such as adenocarcinoma, the presence of actionable genetic variants is less common and there are fewer targeted therapies available for squamous cell advanced/metastatic NSCLC.
What is known about the exposure of interest: Patients diagnosed with squamous mNSCLC often receive Platinum-based chemotherapy regimens as front-line systemic therapy. Following progression, patients will most often receive therapy with immune checkpoint inhibitors that target either the PD-(L)1 pathway, such as nivolumab and atezolizumab, or CTLA-4, such as ipilimumab. The Lung-MAP S1400I trial (NCT02785952) compared overall survival in United States patients with recurrent stage IV squamous NSCLC randomized to receive either nivolumab monotherapy or nivolumab + ipilimumab combination therapy and found no significant difference in mortality rates between these groups.
Note: Since FDA approval in October 2018, combination therapy with pembrolizumab + chemotherapy has gradually replaced chemotherapy as first-line systemic treatment for patients with squamous aNSCLC. The NCT02785952 study was performed between 2016-2018.
Gaps in knowledge: While the proposed research addresses the innovative use of synthetic data in constructing ECAs for single-arm trials, several gaps in knowledge exist:
1. What metrics should be used to evaluate utility and privacy risks of synthetic ECA compared to the original,
2. What validation strategies should be used to ensure that the synthetic ECA captures inferential results compared to the original ECA
3. What best practices should be followed to ensure reliable synthetic ECA are generated.
The concept of using generative models to simulate data for “synthetic patients” to create a “virtual cohort” for the purposes of emulating a randomised trial has similarities to existing work in in silico clinical trials.
What is the expected contribution of this study? This study aims to answer the gaps in knowledge described above using a case study in mNSCLC, as well as describe limitations of this strategy and long-term implications for wider adoption.
6. Research question and objectives
7. Research methods
7.1. Study design
Research design (e.g. cohort, case-control, etc.): External control arm study (non-randomized comparison of trial and real-world patients)
Rationale for study design choice: For assessing the feasibility of using synthetic data for ECA analysis, an ECA design is most appropriate.
7.2. Study design diagram
7.3. Setting
7.3.1 Context and rationale for definition of time 0 (and other primary time anchors) for entry to the study population
The index date is the date of treatment assignment in the Lung-MAP study NCT02785952. The ECA analysis aims to emulate the same.
7.3.2 Context and rationale for study eligibility criteria
Eligibility criteria for the target trial are identical to those from NCT02785952.
5.4. Variables
7.4.1 Context and rationale for exposure(s) of interest
The treatment groups of interest are the same as those in NCT02785952. For real-world studies, we make the concession that any dose and frequency compatible with observed data is permitted, even if not reported.
Treatment regimens in NCT02785952:
● Nivolumab administered intravenously at a dose of 3 mg/kg every 14 days
● Nivolumab administered intravenously at a dose of 3 mg/kg every 14 days + ipilimumab given at 1 mg/kg on day 1 of every third cycle
Algorithm to define duration of exposure effect:
n/a
7.4.2 Context and rationale for outcome(s) of interest
Overall survival defined as time from index date to death from any cause is the primary and sole outcome of interest. Overall survival is the most important clinical outcome in aNSCLC.
7.4.3 Context and rationale for follow up
The maximum length of follow-up in the NCT02785952 data cut available to us is approximately 40 months.Click here to enter text.
7.4.4 Context and rationale for covariates (confounding variables and effect modifiers, e.g. risk factors, comorbidities, comedications)
An assumption for unbiased estimation of the treatment effect is adjustment for all confounders of the treatment effect. However, not all risk factors for survival are measured across the data sources. Furthermore, it is impossible to adjust for all risk factors even if they were measured. We only adjust for measured confounding variables in this study.
The following is a tentative list of measured risk factors for overall survival we will attempt to adjust for. Variables marked with an asterisk are missing for the majority of patients in the real-world database.
7.5. Data analysis
7.5.1 Context and rationale for analysis plan
The inferential data analysis plan is described conceptually, and the context or rationale for the choices are provided in this section.
No sensitivity analyses are planned.
7.6. Data sources
7.6.1 Context and rationale for data sources
Reason for selection: Selection of data sources was based on availability of individual-level data. NCT02785952 was selected from Project Data Sphere. The criteria for selecting a suitable clinical trial dataset for this study were as follows: (i) a lung cancer indication, (ii) a randomized trial with both treatment arms available, (iii) sample size >100 and (iv) testing a non-chemotherapy regimen of clinical importance that is approved for use and commonly administered. NCT02785952 was the only trial on Project Data Sphere that fit these criteria. aNSCLC was chosen because the study authors have substantial experience in this disease setting, and because it is a common indication for drug development, and therefore for regulatory approvals and health technology assessments.
Subsequently, three real-world data sources were identified:
● Administrative cancer data from the province of Alberta, Canada
● Danish national cancer registry
● Norwegian national cancer register
Strengths of data source(s): Patient-level data is available for important risk factors and overall survival.
Limitations of data source(s): The original ADaM/SDTM dataset is not available, and therefore we only have derived variables in some cases for the trial. No longitudinal data on risk factors or subsequent therapies is available. Some variables, notably ECOG scores, are not available in the real-world data.
Data source provenance/curation: Not available for the trial dataset. However, the distribution of baseline characteristics and Kaplan-Meier estimates match those from the trial publication (not shown here). Both the Canadian and Nordic databases collect cancer data at a national population level and are well-documented and have previously been used for epidemiologic and comparative analyses.
7.7. Data management
Patient-level data from NCT02785952 has been provided by the trial sponsor in a deidentified format. A single copy of this dataset will be stored on a local password-protected computer. Programming code will be stored and backed up in a private and secured cloud repository.
7.8. Quality control
For NCT02785952, we have a limited dataset provided by the trial sponsor. Data from Alberta, Canada and Nordic countries are both validated and analyzed by an expert deeply familiar with the data and its limitations. Although double programming will not be performed, steps will be taken to ensure that there are no programming errors for data processing and analysis that can affect the accuracy of the results, for example, through sanity checks and visualization of intermediate results/outputs in the analysis.
7.9. Study size and feasibility
Because this is an exploratory study, sample size calculations were not formally performed. The primary objective of this study is to compare results between original ECA and synthetic ECA used for ECA analysis, rather than producing unbiased estimates of the treatment effect (which are available from the trial and known). Therefore, the study should be feasible regardless of sample size or adequacy of confounding control. However, sensitivity analyses will be performed for unmeasured confounding.
Not applicable. This is an exploratory study and no formal power calculations were performed.
8. Limitation of the methods
The following is a discussion of potential limitations of the study design, and analytic methods, including issues relating to confounding, bias, generalisability, and random error:
1. Random error – NCT02785952 is a relatively small trial (125 + 127 patients) and therefore the results may have low precision. This limitation exists for any study with small samples, and is a common concern in external control arm studies. Therefore, it may be difficult to disentangle a small difference in results between the original ECA and synthetic ECA if it exists.
2. Inability to emulate the target trial perfectly – Due to lack of data availability, there are limitations for perfectly emulating eligibility criteria from the trial. There is also the possibility that there are other unknown differences in variable recording and derivation across the two datasets. However, the primary objective of this study is to compare results between original ECA and synthetic ECA used for ECA analysis, rather than producing unbiased estimates of the treatment effect, and therefore this limitation may be discussed by comparing ECA results with those from the randomized trial.
3. Generalizability of the results – Our results may not generalize to other settings or other datasets. We will not assert that they do. Instead, we will use this case study to describe best practices for future benchmarking studies or applications using synthetic data for ECA analyses.
9. Protection of human subjects
n/a. This study uses deidentified data.
10. Reporting of adverse events
n/a
12. Appendices
n/a
Footnotes
Research question & Objectives
To compare the efficacy of
Nivolumab + ipilimumab combination therapy versus
Nivolumab monotherapy
in patients with metastatic non-small cell lung cancer using real-world data and synthetic data derived from the same, with the aim of validating synthetic data for use in external control arm analysis.
Contributors
Primary investigator contact information:
Alind Gupta (alind.gupta{at}utoronto.ca)
Contributor names:
Luke Segars
David Singletary
Paul Arora
Johan Liseth Hansen
Kirk Geale
Anmol Arora
Manuel Gomes
Sreeram Ramagopalan
Winson Cheung
Study registration
Site: n/a
Identifier: n/a
Sponsor
Organization: n/a
Contact: n/a
Conflict of interest
None
11. References
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
Subject Area
Reviews and Context
0
Comment
0
TRIP Peer Reviews
0
Community Reviews
0
Automated Services
0
Blogs/Media
Author Videos