Comparing effect estimates in randomized trials and observational studies from the same population: an application to percutaneous coronary intervention
========================================================================================================================================================

* Anthony A Matthews
* Karolina Szummer
* Issa J Dahabreh
* Bertil Lindahl
* David Erlinge
* Maria Feychting
* Tomas Jernberg
* Anita Berglund
* Miguel A Hernán

## Abstract

**Background** The ability for real world data to deliver similar results as a trial that asks the same question about the risks or benefits of a clinical intervention can be restricted not only by lack of randomization, but also limited information on eligibility criteria and outcomes. To understand when results from observational studies and randomized trials are comparable, we carried out an observational emulation of a target trial designed to ask similar questions as the VALIDATE randomized trial. VALIDATE compared the effect of bivalirudin and heparin during percutaneous coronary intervention on the risk of death, myocardial infarction, and bleeding across Sweden.

**Methods** We specified the protocol of a target trial similar to the VALIDATE trial protocol, then emulated the target trial in the period before the trial took place using data from the SWEDEHEART registry; the same registry in which the trial was undertaken.

**Results** The target trial emulation and the VALIDATE trial both estimated no difference in the effect of bivalirudin and heparin on the risk of death or myocardial infarction by 180 days: emulation risk ratio for death 1.21 (0.88, 1.54); VALIDATE hazard ratio for death 1.05 (0.78, 1.41). The observational data, however, could not capture less severe cases of bleeding, resulting in an inability to define a bleeding outcome like the trial, and could not account for intractable confounding early in follow-up (risk ratio for death by 14 days 1.85 (0.95, 3.63)).

**Conclusion** Using real world data to emulate a target trial can deliver accurate long-term effect estimates. Yet, even with rich observational data, it is not always possible to estimate the short-term effect of interventions, or the effect on outcomes for which data are not routinely collected. If registries included information on reasons for treatment decisions, researchers may be better positioned to identify important confounders.

Keywords
*   Target trial emulation
*   randomized controlled trials
*   methodology
*   observational studies
*   registry
*   SWEDEHEART

## Background

Observational analyses of routinely collected healthcare data, a form of “real world evidence”, are often used to evaluate the benefits and risks of clinical interventions for acute coronary syndromes. Most criticisms of these observational analyses revolve around the lack of randomized assignment of the treatment strategies under comparison, which may result in confounded effect estimates.1-3 Despite this limitation, routinely collected healthcare data can be used to explicitly emulate target trials, as recent applications in diverse clinical areas have shown.4-8

Routinely collected observational data can only be used to emulate pragmatic trials, but not trials with placebo control and blind treatment assignment, or those that rely on detailed data collection to determine eligibility or ascertain outcomes.9 Specifically, the lack of detailed information on eligibility criteria and outcomes restricts the type of causal questions that can be answered using real world data, a point that is not always emphasized in discussions about the topic.4

To study the strengths and limitations of observational analyses that emulate a trial, a near ideal scenario is to compare a registry-based randomized trial versus the emulation of a target trial with the same protocol as the registry-based randomized trial using observational data from the same registry.10 This comparison ensures that the causal question is asked in the same population and health care setting.11 The Swedish Web System for Enhancement and Development of Evidence-based Care in Heart Disease Evaluated According to Recommended Therapies (SWEDEHEART) is a national quality registry of myocardial infarction, coronary intervention, and heart surgery, which includes longitudinal information on demographic and clinical characteristics, use of therapeutic and preventive services, diagnostic procedures, and various measures of health care utilization for the whole of Sweden.12 By using the registry to enroll participants, randomize interventions, and report outcomes, SWEDEHEART can be used to run randomized trials nested within the registry.13 An example of a registry-based randomized trial within SWEDEHEART is the VALIDATE trial, which compared two anticoagulation interventions, bivalirudin and heparin, over 180 days in patients with acute myocardial infarction undergoing percutaneous coronary intervention in Sweden.14

Here, we use the observational data from the SWEDEHEART registry to emulate a target trial designed to answer similar questions as the VALIDATE trial, and we outline conditions under which one can successfully design a target trial and carry out an observational emulation of this target trial. We describe the VALIDATE randomized trial and the observational data from the SWEDEHEART registry, the specification of the target trial, and the emulation procedures. Our example illustrates the opportunities and limitations of healthcare registries to emulate a target trial.

## VALIDATE: the index randomized trial

### Trial design and analysis

VALIDATE was a multicenter, randomized, controlled, open-label clinical trial carried out between June 2014 and September 2016. Individuals in the SWEDEHEART registry were eligible to participate if they were admitted to hospital with a diagnosis of ST-segment elevated myocardial infarction (STEMI) or non-ST-segment elevated myocardial infarction (NSTEMI), urgent percutaneous coronary intervention was planned in one of 25 of the 29 percutaneous coronary intervention centers in Sweden, and some other eligibility criteria were met (see Table 1). Individuals who accepted the invitation to participate were randomly assigned to receive either intravenous bivalirudin (0.75 mg per kg) or intraarterial unfractionated heparin (5,000 U/ml) under percutaneous coronary intervention. The primary end point was the composite of death from any cause, myocardial infarction, or major bleeding by 180 days. The individual components of the composite event were also assessed. Research nurses screened for clinical end-point events by contacting the patients or first-degree relatives by telephone 7 days and 180 days after percutaneous coronary intervention. If the patient or relatives could not be contacted after the nurses had placed repeated telephone calls and mailed a letter, information was collected through review of hospital records. The intention-to-treat analysis, described in detail elsewhere, relied on the comparison of 180-day risk differences and hazard ratios.14

View this table:
[Table 1.](http://medrxiv.org/content/early/2021/02/03/2021.02.01.21250739/T1)

Table 1. Description of VALIDATE randomized trial, target trial, and target trial emulation using SWEDEHEART register

### Trial results

A total of 6,006 patients underwent randomization in the VALIDATE trial, 3,004 were assigned to bivalirudin and 3,002 were assigned to heparin. The risk of the composite end point did not differ between the treatment groups 30 days after percutaneous coronary intervention. By 180 days, the composite endpoint occurred in 12.3% of patients (369 of 3004) in the bivalirudin group and in 12.8% (383 of 3002) in the heparin group, with a hazard ratio of 0.96 (95% CI: 0.83, 1.10). Death occurred in 2.9% of patients in the bivalirudin group and in 2.8% in the heparin group (without accounting for censoring), with a hazard ratio of 1.05 (95% CI: 0.78, 1.41). Myocardial infarction occurred in 2.0% of patients in the bivalirudin group and in 2.4% in the heparin group, with a hazard ratio of 0.84 (95% CI: 0.60, 1.19). Major bleeding occurred in 8.6% of patients in both the bivalirudin and heparin groups, with a hazard ratio of 1.00 (95% CI: 0.84, 1.19).

## Specifying and emulating a target trial in the SWEDEHEART registry

### Data sources

SWEDEHEART collects data from all patients hospitalized for acute coronary syndrome or undergoing coronary or valvular intervention for any indication in all relevant hospitals across Sweden.13 The registry was officially launched in 2009 when four existing cardiovascular healthcare quality registries were merged: the Register of Information and Knowledge About Swedish Heart Intensive Care Admissions (RIKSHIA), the Swedish Coronary Angiography and Angioplasty Registry (SCAAR), the Swedish Heart Surgery Register and the National Registry of Secondary Prevention (SEPHIA), and the Swedish Heart Surgery Registry. SWEDEHEART was used to collect information for patients when they were randomized in the VALIDATE trial, hence the target trial was emulated in the same population as the original trial and the data collection process was broadly similar between the two studies. SWEDEHEART is also linked to the Inpatient and Outpatient Register, which records all primary and secondary diagnoses and procedures from inpatient hospitalizations and outpatient specialist care visits across Sweden; the Swedish Cause of Death register, which records all deaths and causes of death; and the Prescribed Drug register, which collects information on all dispensed medications.15-17

The approach for emulating a target trial similar to the VALIDATE trial had two steps: 1) specifying the protocol of the target trial, and 2) emulating the target trial using the observational data from the SWEDEHEART registry. Our target trial has the same protocol as the VALIDATE trial, with exceptions only when the observational data were not adequate to identify the information collected in the trial. Table 1 summarizes the target trial protocol and outlines the emulation procedure described below.

### Eligibility criteria

We identified individuals in the SWEDEHEART registry data that met the target trial’s eligibility criteria. The eligibility criteria for the target trial were the same as the VALIDATE trial with six exceptions. First, the study period was between January 2012 to May 2014, which immediately precedes the period of the VALIDATE trial, as all eligible patients during that period were considered in the VALIDATE trial. Second, no informed consent was asked and hence the target trial could not exclude individuals who would have no been asked or declined participation if asked. Third, the target trial used a proxy measure to indicate known terminal disease with life expectancy less than a year (palliative care; dialysis; dementia; or malignant disease, including pancreatic, lung, liver, stomach, and brain cancer, or mesothelioma). Fourth, some eligibility criteria could not be applied in the target trial due to unavailable data (ongoing bleeding, contraindications to study medication). Fifth, there was no information on dose of heparin prior to percutaneous coronary intervention so we excluded all individuals with any prior heparin. Sixth, individuals that died on the day of percutaneous coronary intervention were excluded and identification of outcomes started from the day after percutaneous coronary intervention as it was not possible to distinguish if outcome events such as myocardial infarction and bleeding occurred before or after percutaneous coronary intervention when the events occurred on the same day as the procedure.

### Treatment strategies

The treatment strategies in the target trial closely mimicked those in the VALIDATE trial and were 1) administration of bivalirudin or 2) administration of heparin during percutaneous coronary intervention. The SWEDHEART registry contains information on the anticoagulant given under percutaneous coronary intervention, but does not contain information on dosage. In the target trial, we therefore assumed that if a patient was given both treatments, then the heparin was low dose, and the treatment strategy was defined as bivalirudin.

### Treatment assignment

We classified eligible individuals in the SWEDEHEART registry into two groups according to the strategy their data were compatible with at baseline. That is, our emulation presumes that all individuals assigned to a treatment strategy ended up receiving it. Our emulation treats individuals as if they were randomly assigned to a treatment strategy conditional on the baseline covariates: severity of myocardial infarction (STEMI/NSTEMI, Killip class, angiography finding), center where percutaneous coronary intervention took place (university hospital/non-university hospital), demographics (sex, age), lifestyle characteristics (weight, smoking), laboratory measurements (GFR, heart rate, systolic blood pressure, diastolic blood pressure), diagnoses (diabetes, severe bleeding, anemia), medications (warfarin, anti-hypertensives, lipid lowering treatment), and prior cardiovascular disease or cardiovascular procedures (myocardial infarction, percutaneous coronary intervention, coronary artery bypass grafting). See Appendix 1 for further details on covariates and their definitions.

### Outcomes

The outcomes in the target trial were the same as the VALIDATE trial and included death from any cause, myocardial infarction, major bleeding events, and a composite of all above outcomes. Death was identified from the Cause of Death register, myocardial infarction from the SWEDEHEART registry, and bleeding from the Inpatient and Outpatient register or SWEDEHEART registry (full outcome definitions in Appendix 2).

### Follow-up

Each individual was followed from the day following percutaneous coronary intervention until the outcome of interest or 180 days, whichever occurred first. In the myocardial infarction and bleeding analyses, individuals were not censored if they died during follow-up, which is valid when estimating the total effect of treatment, but we undertook a sensitivity analysis to explore censoring at death (Sensitivity 10).18 Outcome data on those who migrated out of Sweden were unavailable, but the probability of migration during such a short period is low.

### Causal contrast

In the target trial, the intention-to-treat effect is the effect of being assigned to bivalirudin versus heparin at baseline on the risk of death, myocardial infarction, bleeding, and a composite outcome of all outcomes. Our emulation focuses on an observational analog of the intention-to-treat effect under the assumption that all individuals received the treatment they were assigned to.

### Statistical analysis

We estimated the 14, 30, and 180-day risk (cumulative incidence), risk difference, and risk ratio for each outcome using pooled logistic regression models with a flexible time-varying intercept and product terms between the treatment and time.19 We adjusted for baseline covariates using two alternative approaches: inverse probability weighting and standardization. We truncated the stabilized inverse probability weights at their 99th percentile to prevent outliers with extreme weights from influencing effect estimates. We plotted unweighted and inverse probability weighted Kaplan-Meier survival curves for all outcomes. We used nonparametric bootstrapping with 500 samples to calculate all 95% confidence intervals (CI). Further details on our modelling approach are in Appendix 3.

We repeated the 180-day analysis using inverse probability weighted models stratified by myocardial infarction type (STEMI/NSTEMI) to explore effect modification. Finally, we carried out several sensitivity analyses using inverse probability weighted models to assess the sensitivity of the eligibility criteria, treatment strategies, outcomes, confounders, follow-up, and analysis. Full details on sensitivity analyses are in Appendix 4. All analysis code is available at: [https://github.com/tonymatthews/validate](https://github.com/tonymatthews/validate).

## Results

Figure 1 shows a flowchart of patient selection for the target trial emulation. Table 2 shows the baseline characteristics of the 4,940 eligible patients, of whom 2,634 were given bivalirudin and 2,306 heparin. Compared with those in the heparin group, patients in the bivalirudin group were more likely to be diagnosed with STEMI over NSTEMI before percutaneous coronary intervention (81% vs 17%).

View this table:
[Table 2.](http://medrxiv.org/content/early/2021/02/03/2021.02.01.21250739/T2)

Table 2. Baseline characteristics of eligible individuals for the observational emulation of a target trial of bivalirudin vs. heparin, SWEDEHEART Register, 2012-2014

![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/02/03/2021.02.01.21250739/F1.medium.gif)

[Figure 1.](http://medrxiv.org/content/early/2021/02/03/2021.02.01.21250739/F1)

Figure 1. Flowchart of individuals eligible for the observational emulation of a target trial of bivalirudin vs. heparin, SWEDEHEART Register, 2012-2014

Table 3 shows the estimated 180-day risks, risk differences, and risk ratios obtained via inverse probability weighting and standardization. The inverse probability weighted risk (95% CI) of the composite outcome was 9.3% (8.2%, 10.4%) in the bivalirudin group and 10.0% (8.7%, 11.3%) in the heparin group, which results in a risk difference of −0.7% (−2.5%, 1.1%) and a risk ratio of 0.93 (0.77, 1.12). The risk of death was 4.1% (3.4%, 4.8%) in the bivalirudin group and 3.4% (2.5%, 4.2%) in the heparin group, which results in a risk difference of 0.7% (−0.4%, 1.8%) and risk ratio of 1.21 (0.88, 1.68). The risk of myocardial infarction was 3.0% (2.3%, 3.8%) in the bivalirudin group and 2.8% (2.2%, 3.4%) in the heparin group, which results in a risk difference of 0.2% (−0.8%, 1.2%) and risk ratio of 1.08 (0.76, 1.54). The risk of bleeding was 3.2% (2.5%, 3.9%) in the bivalirudin group and 4.6% (3.7%, 5.6%) in the heparin group, which results in a risk difference of −1.4% (−2.6%, −0.3%) and risk ratio of 0.69 (0.50, 0.95). Standardized estimates were similar.

View this table:
[Table 3.](http://medrxiv.org/content/early/2021/02/03/2021.02.01.21250739/T3)

Table 3. Estimated 180-day risk, risk difference, and risk ratios from the observational emulation* of a target trial of bivalirudin vs. heparin, SWEDEHEART Register, 2012-2014

Figure 2 shows the inverse probability weighted survival curves (unweighted survival curves in Appendix 5). The myocardial infarction-free survival almost overlapped throughout the 180 days of follow-up. However, there was an elevated risk of death in the bivalirudin group compared with the heparin group in the 14 days after percutaneous coronary intervention. The short-term difference in risk of death can also be seen in the estimated risk difference and risk ratio by 14 and 30 days (Appendices 6 and 7).

![Figure 2.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/02/03/2021.02.01.21250739/F2.medium.gif)

[Figure 2.](http://medrxiv.org/content/early/2021/02/03/2021.02.01.21250739/F2)

Figure 2. Inverse-probability weighted survival curves from an observational emulation of a target trial of bivalirudin vs. heparin, SWEDEHEART Register, 2012-2014*
*Adjusted at baseline for severity of myocardial infarction (STEMI/NSTEMI, Killip class, angiography finding), center where percutaneous coronary intervention took place (university hospital/non-university hospital), demographics (sex, age), lifestyle characteristics (weight, smoking), laboratory measurements (GFR, heart rate, systolic blood pressure, diastolic blood pressure), diagnoses (diabetes, severe bleeding, anemia), medications (warfarin, anti-hypertensives, lipid lowering treatment), and prior cardiovascular disease or cardiovascular procedures (myocardial infarction, percutaneous coronary intervention, coronary artery bypass grafting)

Appendix 8 shows the 180-day risks, risk differences, and risk ratios obtained via inverse probability weighting, stratified by myocardial infarction type, and there were no apparent differences between strata. Appendices 9-19 show results from sensitivity analyses; all results were broadly like those estimated in the main analyses.

## Discussion

We emulated a target trial similar to the VALIDATE trial using real world observational data from SWEDEHEART, the same register in which the trial was undertaken. There was broad agreement in the estimates for death and myocardial infarction from the randomized trial and the observational emulation of the target trial; both found little differences in risk between those given heparin and bivalirudin. However, there were some telling differences between the estimates from the VALIDATE trial and its observational emulation.

First, the target trial could not define a bleeding outcome similar to that in the VALIDATE trial, which was ascertained using a combination of phone calls to patients at 7 and 180 days after percutaneous coronary intervention and a review of hospital records from registers. Exclusively using the registers, as was done in the target trial, only allowed us to identify the most severe cases of bleeding. We also attempted to expand the bleeding outcome in a sensitivity analysis (outcome definition in Appendix 4 and results in Appendix 16), but we were still unable to fully capture less severe cases of bleeding. This difference in the definition of bleeding is likely responsible for the differences in estimates for the bleeding outcome and the composite outcome in the VALIDATE and target trial emulation.

Second, the observational estimates showed a risk difference of 1.0% (0.1%, 1.8%) for death after 14 days of follow-up (Figure 2, Appendix 6) whereas the randomized estimates found no discernible difference in mortality throughout the entire follow-up. This almost instantaneous difference in mortality suggests the presence of imbalanced, and unadjusted for, prognostic factors between treatment groups. It is possible, for example, that patients perceived to be at high risk of bleeding, and hence a higher immediate risk of death, may have been more likely to be administered bivalirudin (because bivalirudin was marketed as having a lower risk of bleeding than heparin). Even with rich data from the SWEDEHEART register, we were not able to accurately identify these potentially important characteristics. One way to aid future researchers in identifying important potential confounders would be if registers included detailed information on reasons for making treatment decisions.

The number of individuals randomized in VALIDATE was 6,006 (2,971 individuals either declined, were not asked, or could not be asked to participate), whereas the number of individuals included in the target trial emulation over a recruitment period of similar length was 4,940. This discrepancy is partly due to our inability to emulate the eligibility criteria for the VALIDATE trial. In the VALIDATE trial individuals given >5000U heparin before percutaneous coronary intervention were excluded, but the SWEDEHEART data does not include detail on dosage of heparin administered. We, therefore, excluded all 3,070 individuals given any dose of heparin before percutaneous coronary intervention, of whom some would have received low doses. A sensitivity analysis that did not exclude anyone given heparin before percutaneous coronary intervention (Appendix 12) included 7,652 individuals, and results for death and myocardial infarction by 180 days were broadly similar to the results from primary analyses. The risk difference for death between those given bivalirudin and heparin early in follow-up was, however, reduced in this sensitivity analysis, as shown in the inverse probability weighted Kaplan-Meier survival curves (Appendix 13). This suggests that by excluding those given any prior heparin in the main analysis, rather than only those given high dose heparin as in the trial, we may be selecting a group of individuals given heparin during percutaneous coronary intervention that are at lower risk of death soon after start of follow-up compared to those in the trial. Results from this sensitivity demonstrate the importance of designing a target trail that closely mimics the eligibility criteria of the index trial if aiming for comparable results, and is another possible explanation for the differences in short-term risk of death discussed above.

The SWEDEHEART register contains some of the most complete routinely collected data available worldwide when a patient undergoes percutaneous coronary intervention. Even with high quality data, however, close harmonization of protocols, adjustment for important confounders, and analytic methods appropriate for estimating causal quantities analogous to those estimated in the trial, we have shown that it is not always possible for the target trial emulation to obtain the same results as an index randomized trial due to trials collecting more detailed, study specific information that are not collected routinely. Nevertheless, we have shown that using real world data to emulate a target trial can deliver accurate long-term effect estimates in certain situations. It is, therefore, possible to foresee a system in which data from randomized trials with short-term follow-up are combined with real world data to estimate the long-term effects of certain interventions.20 Estimating the effect from the short randomized trial would overcome the problem of confounding soon after the intervention; we could then use observational data to estimate the effect during the remainder of follow-up and create synthetic survival curves using both data sources. This would enable prompt estimation of the long-term effects of interventions and allow adaptive clinical or regulatory changes to be put quickly into practice. Synergy between randomized trials and real-world data has the potential to revolutionize both the length of time patients can be followed up and the speed at which results from trials reach clinical practice.

## Supporting information

Appendix [[supplements/250739_file02.docx]](pending:yes)

## Data Availability

Pseudonymized personal data were obtained from national Swedish Registry holders after ethical approval and secrecy assessment. According to Swedish laws and regulations, personal sensitive data can only be made available for researchers who fulfil legal requirements for access to personal sensitive data.

## Footnotes

### Funding

This work was supported by grant from the Swedish Research Council (registration number 2018-03028). Issa J Dahabreh is supported by the Patient-Centered Outcomes Research Institute (PCORI, award ME-1502-27794)

### Disclosure

Dr. Matthews, Dr. Szummer, Dr. Lindahl, and Dr. Jernberg have nothing to disclose; Dr. Dahabreh reports grants from PCORI, during the conduct of the study; Dr. Erlinge reports personal fees from AstraZeneca, outside the submitted work; Dr. Feychting, Dr. Berglund, and Dr. Hernan report grants from Swedish Research Council, during the conduct of the study.

### Data availability

Pseudonymized personal data were obtained from national Swedish Registry holders after ethical approval and secrecy assessment. According to Swedish laws and regulations, personal sensitive data can only be made available for researchers who fulfil legal requirements for access to personal sensitive data.

*   Received February 1, 2021.
*   Revision received February 1, 2021.
*   Accepted February 3, 2021.


*   © 2021, Posted by Cold Spring Harbor Laboratory

This pre-print is available under a Creative Commons License (Attribution-NonCommercial 4.0 International), CC BY-NC 4.0, as described at [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/)

## References

1.  1.Gerstein HC, McMurray J, Holman RR. Real-world studies no substitute for RCTs in establishing efficacy. Lancet 2019;393(10168):210–211.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S0140-6736(18)32840-X&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F02%2F03%2F2021.02.01.21250739.atom) 

2.  2.Collins R, Bowman L, Landray M, Peto R. The Magic of Randomization versus the Myth of Real-World Evidence. N Engl J Med 2020;382(7):674–678.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMsb1901642&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32053307&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F02%2F03%2F2021.02.01.21250739.atom) 

3.  3.Bowman L, Baras A, Bombien R, Califf RM, Chen Z, Gale CP, Gaziano JM, Grobbee DE, Maggioni AP, Muse ED, Roden DM, Schroeder S, Wallentin L, Casadei B. Understanding the use of observational and randomized data in cardiovascular medicine. Eur Heart J 2020;41(27):2571–2578.
    
    

4.  4.Booth CM, Karim S, Mackillop WJ. Real-world data: towards achieving the achievable in cancer care. Nat Rev Clin Oncol 2019;16(5):312–325.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41571-019-0167-7&link_type=DOI) 

5.  5.Petito LC, Garcia-Albeniz X, Logan RW, Howlader N, Mariotto AB, Dahabreh IJ, Hernan MA. Estimates of Overall Survival in Patients With Cancer Receiving Different Treatment Regimens: Emulating Hypothetical Target Trials in the Surveillance, Epidemiology, and End Results (SEER)-Medicare Linked Database. JAMA Netw Open 2020;3(3):e200452.
    
    

6.  6.Dickerman BA, Garcia-Albeniz X, Logan RW, Denaxas S, Hernan MA. Avoidable flaws in observational analyses: an application to statins and cancer. Nat Med 2019;25(10):1601–1606.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F02%2F03%2F2021.02.01.21250739.atom) 

7.  7.Lodi S, Phillips A, Lundgren J, Logan R, Sharma S, Cole SR, Babiker A, Law M, Chu H, Byrne D, Horban A, Sterne JAC, Porter K, Sabin C, Costagliola D, Abgrall S, Gill J, Touloumi G, Pacheco AG, van Sighem A, Reiss P, Bucher HC, Montoliu Gimenez A, Jarrin I, Wittkop L, Meyer L, Perez-Hoyos S, Justice A, Neaton JD, Hernan MA, Group ISS, the HIVCC. Effect Estimates in Randomized Trials and Observational Studies: Comparing Apples With Apples. Am J Epidemiol 2019;188(8):1569–1577.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/aje/kwz100&link_type=DOI) 

8.  8.Hernan MA, Alonso A, Logan R, Grodstein F, Michels KB, Willett WC, Manson JE, Robins JM. Observational studies analyzed like randomized experiments: an application to postmenopausal hormone therapy and coronary heart disease. Epidemiology 2008;19(6):766–79.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/EDE.0b013e3181875e61&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=18854702&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F02%2F03%2F2021.02.01.21250739.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000260191700002&link_type=ISI) 

9.  9.Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available. American Journal of Epidemiology 2016;183(8):758–764.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/aje/kwv254&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26994063&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F02%2F03%2F2021.02.01.21250739.atom) 

10. 10.Forbes SP, Dahabreh IJ. Benchmarking Observational Analyses Against Randomized Trials: a Review of Studies Assessing Propensity Score Methods. J Gen Intern Med 2020;35(5):1396–1404.
    
    

11. 11.James S, Rao SV, Granger CB. Registry-based randomized clinical trials--a new clinical trial paradigm. Nat Rev Cardiol 2015;12(5):312–6.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/nrcardio.2015.33&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25781411&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F02%2F03%2F2021.02.01.21250739.atom) 

12. 12.Thygesen LC, Ersboll AK. When the entire population is the sample: strengths and limitations in register-based epidemiology. Eur J Epidemiol 2014;29(8):551–8.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s10654-013-9873-0&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24407880&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F02%2F03%2F2021.02.01.21250739.atom) 

13. 13.Jernberg T, Attebring MF, Hambraeus K, Ivert T, James S, Jeppsson A, Lagerqvist B, Lindahl B, Stenestrand U, Wallentin L. The Swedish Web-system for enhancement and development of evidence-based care in heart disease evaluated according to recommended therapies (SWEDEHEART). Heart 2010;96(20):1617–21.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiaGVhcnRqbmwiO3M6NToicmVzaWQiO3M6MTA6Ijk2LzIwLzE2MTciO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMS8wMi8wMy8yMDIxLjAyLjAxLjIxMjUwNzM5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

14. 14.Erlinge D, Omerovic E, Frobert O, Linder R, Danielewicz M, Hamid M, Swahn E, Henareh L, Wagner H, Hardhammar P, Sjogren I, Stewart J, Grimfjard P, Jensen J, Aasa M, Robertsson L, Lindroos P, Haupt J, Wikstrom H, Ulvenstam A, Bhiladvala P, Lindvall B, Lundin A, Todt T, Ioanes D, Ramunddal T, Kellerth T, Zagozdzon L, Gotberg M, Andersson J, Angeras O, Ostlund O, Lagerqvist B, Held C, Wallentin L, Schersten F, Eriksson P, Koul S, James S. Bivalirudin versus Heparin Monotherapy in Myocardial Infarction. N Engl J Med 2017;377(12):1132–1142.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F02%2F03%2F2021.02.01.21250739.atom) 

15. 15.Ludvigsson JF, Andersson E, Ekbom A, Feychting M, Kim JL, Reuterwall C, Heurgren M, Olausson PO. External review and validation of the Swedish national inpatient register. BMC Public Health 2011;11:450.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1471-2458-11-450&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21658213&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F02%2F03%2F2021.02.01.21250739.atom) 

16. 16.Brooke HL, Talback M, Hornblad J, Johansson LA, Ludvigsson JF, Druid H, Feychting M, Ljung R. The Swedish cause of death register. Eur J Epidemiol 2017;32(9):765–773.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s10654-017-0316-1&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=28983736&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F02%2F03%2F2021.02.01.21250739.atom) 

17. 17.Wettermark B, Hammar N, Fored CM, Leimanis A, Otterblad Olausson P, Bergman U, Persson I, Sundstrom A, Westerholm B, Rosen M. The new Swedish Prescribed Drug Register--opportunities for pharmacoepidemiological research and experience from the first six months. Pharmacoepidemiol Drug Saf 2007;16(7):726–35.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/pds.1294&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=16897791&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F02%2F03%2F2021.02.01.21250739.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000248553400002&link_type=ISI) 

18. 18.Young JG, Stensrud MJ, Tchetgen Tchetgen EJ, Hernan MA. A causal framework for classical statistical estimands in failure-time settings with competing events. Stat Med 2020;39(8):1199–1236.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F02%2F03%2F2021.02.01.21250739.atom) 

19. 19.Hernán MA, Robins JM. Causal Inference: What If: Boca Raton: Chapman & Hall/CRC; 2020.
    
    

20. 20.Dahabreh IJ, Robins JM, Hernan MA. Benchmarking Observational Methods by Comparing Randomized Trials and Their Emulations. Epidemiology 2020;31(5):614–619.