Transfer of Statistical Innovations of the 1990s-2000s in Oncology to the Biomedical Literature
===============================================================================================

* Alexandre Vivot
* Vincent Lévy
* Sylvie Chevret

## ABSTRACT

**Introduction** Innovations in the fields of clinical studies require time to generate and disseminate new knowledge. We aimed to specifically explore lag times between the introduction and widespread use of innovative statistical methods in oncology using the competing risks and phase I model-based clinical trials settings as examples.

**Methods** First, we defined a set of closed articles for each setting based on two princeps papers (Gray, Annals of Statistics 1998 for the competing risks setting and O’Quigley et al., Biometrics 1990 for the phase I setting). Secondly, we retrieved from the web of science all citations of the papers included in these sets. Each journal was classified as applied, semi-applied or methodological.

**Results** A total of 6,727 citations for the competing risks setting and 2,639 citations for the phase I setting were found. Time to reach 25 citations was 6.2 years for the Gray’s paper and 4.5 years for the Fine and Gray paper, while it ranged from 3.4 years up to at least 20.1 years and not reached for 6 papers from the competing risks setting. The vast majority (91%) of the citing papers for the competing risks setting originated from applied journals. In contrast, less than half (44%) of the citing papers for the phase I setting were published in applied journals.

**Conclusion** Statistical innovations in the competing risks setting have been widely diffused in the medical literature unlike the model-based designs for phase I trials, which are still seldom used 30 years after publication.

Key Words
*   Translation of Innovation
*   Biostatistics
*   Clinical trials
*   Oncology

## Introduction

Translational medicine, aiming to expedite the discovery of new diagnostic tools and treatments by using a multi-disciplinary and collaborative approach, has grown interests in the last decade. Moreover, the commonly accepted delay of 17 years for research evidence to reach clinical practice recently appears to decrease [1] as exemplified by the mapping networks of publications and cross-references. [2] However, most translation gaps between knowledge and clinical application that have been investigated, notably in oncology, concerned translation of biological drivers into therapeutic benefits for patients. [3]

For the advancement of clinical research and eventually of patient care, the translation of innovative statistical methods into practice is also of crucial importance, though the average time elapsed for biostatical innovation to reach medical use is less known. In 1994, Altman showed that, while the key survival paper of Kaplan and Meier achieved only six citations in medical journals in the first 10 years after publication, evidence of decreasing lag times between the introduction and widespread use of innovative statistical methods was expected. [4] More than 2 decades later, we wondered whether the reported above decreasing dichotomy between basic or preclinical and clinical research [2] could be also true in the transfer of innovative statistical methods, confirming the Altman’s hypothesis. [4]

We first focused on competing risks methods, the key innovative survival methods of the last decades, [5] common in cancer. Indeed, competing risks exist whenever the probability of the main event (e.g., cancer) is prevented (e.g., by death) or altered (e.g., by transplantation) to occur. In this setting, standard survival methods that ignore the informative censoring of these competing events may result in biased effects of prognostic factors or treatments, [6] illustrating the need for bridging the gap of innovation into practice. Two innovative papers that first proposed a statistical test [7] and a regression model [8] for such data, were selected as the first innovative set.

With the aim of focusing on statistical innovations of potential large expected therapeutic benefits for patients, we secondly considered the setting of phase I clinical trials, which is also of great interest in the oncology area though with a limited diffusion as compared to phase III. [9] We chose as the most influential statistical pioneering approach, the continual reassessment method (CRM) that is, a model-based approach for dose-finding studies where only empiric 3+3 designs (formerly Fibonacci) were long available. We thus included in the second innovative set the original paper of the CRM [10] together with its close proposed modifications (Table 2).

View this table:
[Table 1:](http://medrxiv.org/content/early/2019/06/25/19000638/T1)

Table 1: 
Number of citations retrieved from Web of Science on July 23, 2018

View this table:
[Table 2.](http://medrxiv.org/content/early/2019/06/25/19000638/T2)

Table 2. 
List of selected articles for each innovative set.

The primary objective of this paper was to assess whether Altman’s prediction was fulfilled in the field of oncology using the competing risks and the phase I clinical trials as examples. Secondary objectives were to study the relative importance of the applied and statistical literature in papers citing statistical innovations and to analyze which medical areas the methods are mainly used.

## Methods

### Selection of articles

Based on our knowledge, we *a priori* included the paper by Gray[7] and that by Fine and Gray [8] in the innovative set for the competing risks setting. For the phase I setting, beside the paper by O’Quigley, [10] because numerous modifications of the initial CRM very similar to the initial paper were published, we jointly considered the set of those articles. Thus, we retrieved all papers citing O’Quigley’s paper [10] in the Web of Science on May 9, 2018. We then constructed a co-citations network, where co-citations were defined as links between two papers, both of which cited by the same paper. The relatedness of papers was based on the number of times they have been cited together. This process was used to create the innovative set for the phase I trials setting.

### Citation data

For both settings, we retrieved all citations of the innovative set from the Web of Science on July, 23 2018. Citation data were imported into R software ([https://www.R-project.org/](https://www.R-project.org/)) using the Bibliometrix package. [12] The information retrieved included the title, abstract, date of publication, journal name and category according to the classification by Incites Journal Citation ([https://jcr.incites.thomsonreuters.com](https://jcr.incites.thomsonreuters.com)). We excluded animal studies by searching animal names (cat, dog, mice, mouse, rat, monkey, primate, macaque) in the title. Each citation article was also segregated to one of the three following categories according to the journal of publication: 1) applied journals, 2) semi-applied journals, and 3) methodological journals (Table 3). For citations of the phase I setting, one investigator (AV) manually reviewed the papers with “phase I” in title to check if they were actual dose-finding phase I clinical trials.

View this table:
[Table 3.](http://medrxiv.org/content/early/2019/06/25/19000638/T3)

Table 3. 
List of selected journals for each class.

### Statistical analysis

Analysis was performed for each setting, separately. Summary statistics were reported, either mean (standard deviation) or median [interquartile range]. We plotted the cumulative number and proportions of “applied”, “semi-applied”, and “methodological” fractions of citations 5 over time as a percentage of the citations classified in the three categories of journals defined above. We defined translational gap as the time between the publication of the biostatistical paper and the time it reached 25 citations in applied literature, as defined by Altman; [4] estimated cumulative incidence of translational gap was estimated by the Kaplan Meier approach, due to the administrative censoring of the data on 2018.

To obtain further insight into the medical literature citing these innovative statistical tools, we constructed a citation network using the journal as the statistical unit (rather than the paper). Use of this network, together with clustering techniques, allowed us to analyze which medical areas (using the journal as a proxy) cited the most statistical innovations. VosViewer was used to create maps and clustering analysis. [14]

### Ethical Statement

All methods were carried out in accordance with relevant guidelines and regulations. No informed consent or ethics approval was necessary because this study is based on publicly available data and involved no individual patient data collection or analysis.

## Results

The innovative sets of papers used for both settings are reported in Table 2. They included 2 articles for the competing risks set, and 28 for the phase I set.

### Competing risks set

A total of 6,727 citations (5,863 unique articles), including 2,431 for Gray’s article and 4,296 for the Fine and Gray paper, were found (Table 1). Time to translational gap was 6.2 years for the Gray’s paper and 4.5 years for the Fine and Gray paper. The vast majority (91%) of the citations originated from applied journals, with a sharp and continuous increase in citations over time (Figure 1); in contrast, the citations from methodological and semi-applied journals represented only small percentages (6% and 2%, respectively) of the citing papers over the entire period. This was confirmed by the representation of the network of these journals (Figure 2).

![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2019/06/25/19000638/F1.medium.gif)

[Figure 1.](http://medrxiv.org/content/early/2019/06/25/19000638/F1)

Figure 1. 
Cumulative number of citations for the competing risks setting (Panel A) and the phase I trials setting (Panel B) retrieved from the Web of Science and classified in each category of journals.

![Figure 2.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2019/06/25/19000638/F2.medium.gif)

[Figure 2.](http://medrxiv.org/content/early/2019/06/25/19000638/F2)

Figure 2. 
Citation network of journals for the two innovative sets (Competing risks- Fig 2A, Phase I trials- Fig 2B). For clarity, only journals with at least 5 citations are represented.

Beside a methodological cluster, two major clusters confirmed the two main areas of application of competing risks, namely oncology (e.g., Journal of Clinical Oncology, European Journal of Cancer and Hematology) and hematology (e.g., Blood, Haematologica, and Bone Marrow Transplantation). Other clusters represent other areas of application, such as cardiovascular diseases.

### Phase I clinical trials set

After excluding books and animal studies, a total of 2,639 citations (1,114 unique papers after removing duplicates) for the phase I innovative set were found. The three most cited papers were published by O’Quigley in 1999 (676 citations), Goodman in 1995 (258 citations) and Babb in 1998 (235 citations). Contrarily to the competing risks setting, the translational gap differed across papers, from 3.4 years up to at least 20.1 years since not reached for 6 papers. Overall, it was reached by 10.7% of the set articles at year-5 and by 47.9% at year-10. Less than one half (44%) of citing articles were published in applied journals, more than one third (36%) in statistics journals, and a fifth (20%) in semi-applied journals, with roughly similar rate of citations over time (Figure 1). When we restricted the citations to those articles with “phase I” in the title, only 415 (37%) articles were selected; of these, only 110 were found to be phase I clinical trials after manual reviewing the titles.

Network of the journals of papers citing one of the phase I set is displayed in Figure 2. The right part of the graph represents the methodological cluster containing, for example, Statistics in Medicine and Biometrics. The left part of the graph represents the large group concerning applications or reviews of the CRM in the most important clinical cancer journals (Journal of Clinical Oncology, Clinical Cancer Research, Annals of Oncology, British Journal of Cancer, and Journal of National Cancer Institute); in contrast, non-cancer journals had published only a few studies.

## Discussion

This study aimed to check whether the time lags in the statistical translation process was actually shortened, focusing on two areas with a large potential for clinical research improvements and widely encountered in the oncology literature, namely, the survival methods for competing risks data and the new designs for phase I clinical trials. The answer was bifid, with a short translational gap, defined by the time to reach 25 citations, [4] of about 5 years for the former, but delayed above 10 years for one half of the later. This could be expected given the two statistical innovative sets are used at different stages of clinical studies, with different levels of complexity. Indeed, the first set of competing risks articles deals with method of data analysis, that can easily be performed using modern statistical software without the need for major expertise - although whether such studies are used, performed and interpreted correctly can be another matter. [15] Therefore, such a lag time of about 5 years for using an innovative data analytic method is in agreement with the time to publication after completion of data collection and analysis recently estimated at about 3 years in six journals with high impact factors. [16] In contrast, the second set of innovative methods concern a change in trial design and logistics; thus, beside the time of analysis and publication, the transfer of innovative clinical trial design into the medical literature is obviously impacted by the additional constraints of patient enrollment time, and the follow-up period for the end point. It moreover requires statisticians to engage since the planning phase, [17] up to the analysis of the data, contrary to traditional methods such as the ‘3+3’ design, which can be used without involving statisticians and computer programs and remains the most common choice among clinicians for phase I dose-escalation oncology trials. [18]

Moreover, difference in complexity of both settings was illustrated in terms of citation patterns. Although the competing risks methodology was widely diffused over the medical (in particular, oncology) community, methodology relating to innovative designs for cancer Phase I trials failed to translate easily into practice, consistent with the results of a previous study that showed a very slow transfer of phase I design improvements into clinical practice. [19]

Nevertheless, the two settings share large implications for clinical research, and researchers trying to apply the statistical innovative methods should not be delayed in using the new knowledge. This is notably true in the setting of phase I trials in oncology, where improved selection of patients due to improvements in translational medicine should translate into faster and more precise dose determination. [20] Diffusing more widely into the applied literature and the medical community could be achieved in several ways. First, communication between biostatisticians and clinical colleagues should be improved, although many reviews have been published on these methods in the medical literature, as illustrated by the non-negligible fraction of semi-applied and applied papers that are different from the original clinical trials used in our study. This could be driven through key gateway journals, as suggested by the clusters of citation journals where Journal of Clinical Oncology and Statistics in Medicine appear major players (Figure 2). Concerning competing risks analysis, for which adequate modern methods have been integrated into all modern statistical software, a key step may be to educate clinicians to recognize the settings in which competing risks are of concern and to persuade them of the importance of using those methods. For the design of phase I studies, improvements in providing concrete guidance for designing such trials and facilitating their implementation in practice is still mandatory for bridging the gap between statistical innovation and practical implementation. Model-based designs are cited in the US FDA guidance for industries that are related to adaptive clinical trials as “less understood models,” and the FDA highlights some of their disadvantages. [21]

Our study has some limitations. First, we used citations as a measure of the transfer of knowledge between researchers and physicians. Such citation counts, although not a direct measure of the intrinsic significance of a research idea, provide a measure of statistical technology transfer and of its impact. [4] However, in addition to papers that focus on ranking journals and impact factors [22] many others have considered the citations only qualitatively. [23] Second, we segregated articles into three categories of applied, semi-applied and methodological journals, as previously reported. [5] However, this distinction is somewhat simplistic because the applied fractions of articles published in statistical journals may increase over time; this is reportedly highest for statistical medicine. [5] Third, we focused on two main and distinct topics of biostatistics, and the results might differ for other statistical innovations, such as dynamic prediction modelling, the joint modelling of longitudinal data, and multiple imputation techniques for handling missing data.

### Conclusion

In summary, statistical innovations for the competing risks setting have been widely diffused in the medical literature, especially in oncology and hematology, fulfilling Altman’s prediction about decreasing lag times, unlike the model-based designs for phase I trials, which are still seldom used 30 years after their first publication. However, for both statistical methods, a translational gap remains that needs to be filled before the oncology community can benefit fully from these modern methods.

## Data Availability

Data used in this study are publicly available data. The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request.

## Competing interests

The author(s) declare no competing interests.

## Funding

There was no specific funding for this study.

## Authors’ contributions

SC had the original idea. SC and AV designed the study. AV conducted the literature search, extracted and analyzed data. AV, VL, and SC participated in the interpretation of the data. AV drafted the manuscript. AV, VL, and SC authors reviewed the manuscript and approved the final version.

## Availability of data and material

Data used in this study are publicly available data. The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request.

## Acknowledgements

We wish to thank Dr Lucie Biard for helpful comments on the manuscript. We acknowledge the paid contribution of American Journal Experts (Research Square, Durham, NC, USA) for copyediting.

## Abbreviations

CRM
:   Continual Reassessment Method

*   Received June 20, 2019.
*   Revision received June 20, 2019.
*   Accepted June 25, 2019.


*   © 2019, Posted by Cold Spring Harbor Laboratory

The copyright holder for this pre-print is the author. All rights reserved. The material may not be redistributed, re-used or adapted without the author’s permission.

## References

1.  [1].Green LW, Ottoson JM, García C, Hiatt RA. Diffusion theory and knowledge dissemination, utilization, and integration in public health. Annu Rev Public Health 2009;30:151–74. doi:10.1146/annurev.publhealth.031308.100049.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1146/annurev.publhealth.031308.100049&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19705558&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2019%2F06%2F25%2F19000638.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000268071900010&link_type=ISI) 

2.  [2].Gal D, Thijs B, Glänzel W, Sipido KR. A Changing Landscape in Cardiovascular Research Publication Output. J Am Coll Cardiol 2018;71:1584–9. doi:10.1016/j.jacc.2018.01.073.
    
    [FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6MzoiUERGIjtzOjExOiJqb3VybmFsQ29kZSI7czo0OiJhY2NqIjtzOjU6InJlc2lkIjtzOjEwOiI3MS8xNC8xNTg0IjtzOjQ6ImF0b20iO3M6Mzk6Ii9tZWRyeGl2L2Vhcmx5LzIwMTkvMDYvMjUvMTkwMDA2MzguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

3.  [3].Dalasanur Nagaprashantha L, Adhikari R, Singhal J, Chikara S, Awasthi S, Horne D, et al. Translational opportunities for broad-spectrum natural phytochemicals and targeted agent combinations in breast cancer: Translational strategies for breast cancer prevention and therapy. Int J Cancer 2018;142:658–70. doi:10.1002/ijc.31085.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/ijc.31085&link_type=DOI) 

4.  [4].Altman DG, Goodman SN. Transfer of Technology From Statistical Journals to the Biomedical Literature: Past Trends and Future Predictions. JAMA 1994;272:129–32. doi:10.1001/jama.1994.03520020055015.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jama.1994.03520020055015&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=8015123&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2019%2F06%2F25%2F19000638.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1994NV42400013&link_type=ISI) 

5.  [5].Schell MJ. Identifying Key Statistical Papers From 1985 to 2002 Using Citation Data for Applied Biostatisticians. Am Stat 2010;64:310–7. doi:10.1198/tast.2010.08250.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1198/tast.2010.08250&link_type=DOI) 

6.  [6].Lesko CR, Lau B. Bias Due to Confounders for the Exposure–Competing Risk Relationship. Epidemiology 2017;28:20. doi:10.1097/EDE.0000000000000565.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/EDE.0000000000000565&link_type=DOI) 

7.  [7].Gray RJ. A Class of K-Sample Tests for Comparing the Cumulative Incidence of a Competing Risk. Ann Stat 1988;16:1141–54.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1214/aos/1176350951&link_type=DOI) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1988Q161700019&link_type=ISI) 

8.  [8].Fine JP, Gray RJ. A Proportional Hazards Model for the Subdistribution of a Competing Risk. J Am Stat Assoc 1999;94:496–509. doi:10.1080/01621459.1999.10474144.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/01621459.1999.10474144&link_type=DOI) 

9.  [9].Decullier E, Chan A-W, Chapuis F. Inadequate Dissemination of Phase I Trials: A Retrospective Cohort Study. PLOS Med 2009;6:e1000034. doi:10.1371/journal.pmed.1000034.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pmed.1000034&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19226185&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2019%2F06%2F25%2F19000638.atom) 

10. [10].O’Quigley J, Pepe M, Fisher L. Continual Reassessment Method: A Practical Design for Phase 1 Clinical Trials in Cancer. Biometrics 1990;46:33–48. doi:10.2307/2531628.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2307/2531628&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=2350571&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2019%2F06%2F25%2F19000638.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1990DA02200003&link_type=ISI) 

11. [11].R Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria; 2016.
    
    

12. [12].Aria M, Cuccurullo C. bibliometrix: An R-tool for comprehensive science mapping analysis. J Informetr 2017;11:959–75.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.joi.2017.08.007&link_type=DOI) 

13. [13].Clavirate Analytics. Journal Citations Report 2017. [https://jcr.incites.thomsonreuters.com/](https://jcr.incites.thomsonreuters.com/).
    
    

14. [14].van Eck NJ, Waltman L. Citation-based clustering of publications using CitNetExplorer and VOSviewer. Scientometrics 2017;111:1053–70. doi:10.1007/s11192-017-2300-7.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s11192-017-2300-7&link_type=DOI) 

15. [15].Austin PC, Fine JP. Accounting for competing risks in randomized controlled trials: a review and recommendations for improvement. Stat Med 2017;36:1203–9. doi:10.1002/sim.7215.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/sim.7215&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=28102550&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2019%2F06%2F25%2F19000638.atom) 

16. [16].Welsh J, Lu Y, Dhruva SS, Bikdeli B, Desai NR, Benchetrit L, et al. Age of Data at the Time of Publication of Contemporary Clinical Trials. JAMA Netw Open 2018;1:e181065. doi:10.1001/jamanetworkopen.2018.1065.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jamanetworkopen.2018.1065&link_type=DOI) 

17. [17].Thorlund K, Haggstrom J, Park JJ, Mills EJ. Key design considerations for adaptive clinical trials: a primer for clinicians. BMJ 2018;360:k698. doi:10.1136/bmj.k698.
    
    [FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE2OiIzNjAvbWFyMDhfNy9rNjk4IjtzOjQ6ImF0b20iO3M6Mzk6Ii9tZWRyeGl2L2Vhcmx5LzIwMTkvMDYvMjUvMTkwMDA2MzguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

18. [18].Love SB, Brown S, Weir CJ, Harbron C, Yap C, Gaschler-Markefski B, et al. Embracing model-based designs for dose-finding trials. Br J Cancer 2017;117:332–9. doi:10.1038/bjc.2017.186.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/bjc.2017.186&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2019%2F06%2F25%2F19000638.atom) 

19. [19].Rogatko A, Schoeneck D, Jonas W, Tighiouart M, Khuri FR, Porter A. Translation of innovative designs into phase I trials. J Clin Oncol Off J Am Soc Clin Oncol 2007;25:4982–6. doi:10.1200/JCO.2007.12.1012.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiamNvIjtzOjU6InJlc2lkIjtzOjEwOiIyNS8zMS80OTgyIjtzOjQ6ImF0b20iO3M6Mzk6Ii9tZWRyeGl2L2Vhcmx5LzIwMTkvMDYvMjUvMTkwMDA2MzguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

20. [20].Wong KM, Capasso A, Eckhardt SG. The changing landscape of phase I trials in oncology. Nat Rev Clin Oncol 2016;13:106–17. doi:10.1038/nrclinonc.2015.194.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/nrclinonc.2015.194&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26552953&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2019%2F06%2F25%2F19000638.atom) 

21. [21].US Food and Drug Administration. Guidance for Industry. Adaptive Design Clinical Trials for Drugs and Biologics. 2010.
    
    

22. [22].Varin C, Cattelan M, Firth D. Statistical modelling of citation exchange between statistics journals. J R Stat Soc Ser A Stat Soc 2016;179:1–63. doi:10.1111/rssa.12124.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/rssa.12124&link_type=DOI) 

23. [23].Lang TA, Altman DG. Basic statistical reporting for articles published in biomedical journals: the “Statistical Analyses and Methods in the Published Literature” or the SAMPL Guidelines. Int J Nurs Stud 2015;52:5–9. doi:10.1016/j.ijnurstu.2014.09.006.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.ijnurstu.2014.09.006&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25441757&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2019%2F06%2F25%2F19000638.atom) 

24. [24].Nieminen P, Carpenter J, Rucker G, Schumacher M. The relationship between quality of research and citation frequency. BMC Med Res Methodol 2006;6:42. doi:10.1186/1471-2288-6-42.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1471-2288-6-42&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=16948835&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2019%2F06%2F25%2F19000638.atom)