Up front and open, shrouded in secrecy, or somewhere in between? A Meta Research Systematic Review of Open Science Practices in Sport Medicine Research

Garrett S. Bullock, Patrick Ward, Franco M. Impellizzeri, Stefan Kluzek, Tom Hughes, Charles Hillman, Brian R. Waterman, Kerry Danelson, Kaitlin Henry, Emily Barr, Kelsey Healey, Anu M. Räisänen, Christina Gomez, Garrett Fernandez, Jakob Wolf, Kristen F. Nicholson, Tim Sell, Ryan Zerega, Paula Dhiman, Richard D. Riley, Gary S Collins
doi: https://doi.org/10.1101/2023.03.30.23287959

Abstract

Objective To investigate the extent and qualitatively synthesize open science practices within research published in the top five sports medicine journals from 01 May 2022 and 01 October 2022.

Design Meta-research systematic review

Data Sources MEDLINE

Eligibility Criteria Studies were included if they were published in one of the identified top five sports medicine journals as ranked by Clarivate. Studies were excluded if they were systematic reviews, qualitative research, grey literature, or animal or cadaver models.

Results 243 studies were included. The median number of open science practices met per study was 2, out of a maximum of 12 (Range: 0-8; IQR: 2). 234 studies (96%, 95% CI: 94-99) provided an author conflict of interest statement and 163 (67%, 95% CI: 62-73) reported funding. 21 studies (9%, 95% CI: 5-12) provided open access data. 54 studies (22%, 95% CI: 17-included a data availability statement and 3 (1%, 95% CI: 0-3) made code available. 76 studies (32%, 95% CI: 25-37) had transparent materials and 30 (12%, 95% CI: 8-16) included a reporting guideline. 28 studies (12%, 95% CI: 8-16) were pre-registered. 6 studies (3%, 95% CI: 1-4) published a protocol. 4 studies (2%, 95% CI: 0-3) reported the availability of an analysis plan. 7 studies (3%, 95% CI: 1-5) reported patient and public involvement.

Conclusion Sports medicine open science practices are extremely limited. The least followed practices were sharing code, data, and analysis plans. Without implementing open practices, barriers concerning the ability to aggregate findings and create cumulative science will continue to exist.

What is already known

  • Open science practices provide a mechanism for evaluating and improving the quality and reproducibility of research in a transparent manner, thereby enhancing the benefits to patient outcomes and society at large.

  • Understanding the current open science practices in sport medicine research can assist in identifying where and how sports medicine leadership can raise awareness, and develop strategies for improvement.

What are the new findings

  • No study published in the top five sports medicine journals met all open science practices

  • Studies often only met a small number of open science practices

  • Open science practices that were least met included providing open access code, data sharing, and the availability of an analysis plan.

Introduction

Advances in how we plan and conduct research has created (in principle) opportunities for international collaboration1 and sharing of data that lead to gains in overall scientific knowledge,2-4 that was unimaginable only a few decades ago. While sports medicine and science have paralleled these scientific attainments,5-7 there is still much to learn in terms of preventing detrimental sport medical outcomes and improving athlete health.5

It is highly likely that some progress in sport medicine and science is being stymied as a result of significant and serious barriers that have been recently highlighted in the current literature. For example, numerous systematic literature reviews have consistently demonstrated the use of suboptimal methods and incomplete reporting,8 indicating a risk of adopting findings based on misleading or potentially harmful conclusions.8,9 Methodological flaws and misconduct such as ‘p-hacking’ and hypothesizing after the results are known (HARKing)10 further impede scientific advancements by inflating risk of type 1 error.10,11 Incomplete reporting and a lack of transparency in the scientific design, conduct and reporting of scientific studies, including unavailability of protocols, analysis plans, code, and data enables these practices to continue.

This may also, in combination with poor, or incomplete reporting of final studies, restrict the ability to identify valid findings from well-designed studies or limit the accuracy of aggregated analyses.12,13 In addition, research conduct is often limited in sport because of sample size restrictions that are exerted by using datasets from individual teams or organizations.14 While data sharing initiatives can overcome this barrier, a team’s proprietary data is often strongly protected.15

One overriding principle to help improve transparency and utility in research is through open science.16 Open science is a movement to make all scientific materials and results accessible to all levels of society17 and is a process where scientists openly share protocols and analysis plans, register studies, report results, and share data and code.18 To advance science, and within sports medicine, to continuously improve athlete health, the entire scientific process must be based on the Open Science paradigm and must be “findable, accessible, interoperable, and reusable (FAIR)”.19 This process allows fellow scientists to evaluate, replicate, and confirm previous research from transparent methods, open data and code.16,19

While open science practices have been relatively well adopted in the physical and biological sciences,5 however, these fields do not have the same ethical considerations as the medical sciences.2,5,18 Sports medicine and science has further concerns such as competition between clubs and athlete re-identification which may constitute a barrier to open science uptake.16 Despite these issues, funders and charity organizations increasingly require plans for open science practices to be embedded in grant applications for funded sports medicine research.2,5,18

Adopting open science practices can improve research transparency and credibility and advance athlete care and health at a faster pace compared to opaque (i.e., less transparent) research.16 However, it is not clear as to what extent open science practices are adopted within sports medicine and science research. Understanding the current state can assist academics, practitioners, journal editors, reviewers, and funding bodies in determining where and how they can employ and improve open science practices, potentially accelerating collaboration, methodological transparency, and athlete health outcomes.16,20,21 Therefore, the purpose of this study was to carry out meta-research using a systematic review to investigate the extent and qualitatively synthesize open science practices within research published in the top five sports medicine journals from 01 May 2022 to 01 October 2022.

Methods

Study design

The design of this meta research systematic review was informed by previous work by Hardwicke et al.22 This study was reported using guidelines for reporting methodology research23 and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocol (PRISMA-P).24 Evaluation of open science practice was informed through two sources, Evaluating implementation of the Transparency and Openness Promotion (TOP) guidelines25 and the review by Tennant et al.26 This study was registered on the Open Science Framework (https://osf.io/4amek/). The final draft manuscript was uploaded and made available on a pre-print server prior to peer review.

Relevant party involvement (i.e., Patient and public involvement)

The research question was developed following consultation with several professional groups, including non-academic partners and individuals who had an interest in or were affected by amateur, collegiate and professional sport. These groups included physiotherapists, physicians, sports performance coaches, athletic trainers, as well as statistical and methodological researchers. These groups also met virtually to discuss strategy and study progress, preliminary results and interpretation of findings, and provide input into the plan for dissemination of findings

Equity, Diversity, and Inclusion

After consideration of the necessity to involve relevant parties and collaborators with required expertise, the author team consists of a diverse range of individuals, including students, clinicians, and early, middle, and late career researchers with balance from people whom identify as men and women, different age groups, and nationalities.

Study eligibility criteria

Article inclusion and exclusion criteria are reported in Table 1.

View this table:
Table 1.

Article Inclusion and Exclusion Criteria

Search strategy and journal selection

Sports medicine journals were chosen based on Clarivate journal citation rankings. While these rankings have limitations,27,28 this method was chosen to remove author subjectivity, and therefore prohibited the author team from selectively cherry picking journals. After excluding journals that are focused on systematic reviews (Sports Medicine; Exercise Immunology Review) and qualitative research (Qualitative Research in Sport, Exercise and Health), the top five journals were (1) British Journal of Sports Medicine, (2) Journal of Sport and Health Science, (3) American Journal of Sports Medicine, (4) Medicine Science Sport and Exercise, and (5) Sports Medicine-Open. These five journals were searched through MEDLINE on October 10, 2022 for all articles published over a six-month time period, between May 1, 2022 and October 1, 2022 (Appendix 1).

Study Selection

All reviewers participated in an online training session (led by GB) that provided information for article screening and the data extraction process. A calibration exercise was then performed prior to screening, with all reviewers required to achieve greater than 90% agreement prior to screening. Titles and abstracts were screened independently for eligibility in equal numbers of randomized articles by paired screening groups (PW and FI, TH and CH, KD and KH, EB and KH, AR and CG, GF and JW, TS and RZ). The full-text of eligible studies were then recovered and screened independently by the same screening pairs.29 Title and abstract and full-text study disputes were resolved by consensus within each screening pair. If consensus could not be resolved, the lead author (GB) had final resolution on study inclusion or exclusion. Selected full-text articles were retrieved through university online library portals. If a study could not be retrieved, the authors were contacted to request full text, and, if required, interlibrary loan with the assistance of a librarian was attempted. If a full-text article could not be retrieved, the study was excluded from the review.29-31 All screening was performed in Covidence systematic review software (Veritas Health Innovation, Melbourne, Australia).

Data Extraction

Data were extracted by the same screening pairs (PW and FI, TH and CH, KD and KH, EB and KH, AR and CG, GF and JW, TS and RZ), entered into a customized electronic database, using the recommended practices of The National Institute for Health and Care Excellence evidence tables.32 33 Each pair independently extracted data, with conflicts resolved first by consensus, followed by the lead author (GB). A random sample of three articles from each data extraction team were screened and graded by the study leads (GB, GC) for quality control. Data extraction included author details (e.g., first author surname, title, study design, journal, month of publication, and sport. Open science methods were extracted in accordance with the TOP guidelines,25 with further open science data comprising patient public involvement (Table 2).26 Any articles that were electronic publications ahead of print were extracted, but were not complicit to open science criteria such as disclosing author conflicts or reporting funding. The Open Science data were extracted as a ‘yes’ or ‘no’ for meeting the criteria.

View this table:
Table 2.

Open Science Practices evaluated in the review (*adapted from the TOP guidelines25)

Data Sharing

The reconciled extracted data that form the results in this study are available in the Open Science Framework (https://osf.io/4amek/).

Collating, Summarizing and Reporting the Results

Overall screening agreement and quality control agreement were calculated by Cohen’s Weighted Kappa. Proportions, point estimates and 95% confidence intervals were summarized across all included studies, calculated according to whether or not each criterion for open science practice was met in each study. Data were also stratified according to journal, study design and sport. Data were further summarized through median, range, and interquartile range (IQR) of articles meeting open science practices Due to small sample size and proportions at or around zero, Clopper-Pearson confidence intervals were calculated for proportions.34 A narrative synthesis was performed. All analyses were performed in R 4.02 (R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/). The dplyr package was used for cleaning and calculations.

Code Sharing

Analytical code used to summarize the findings in this paper are available on the Open Science Framework (https://osf.io/4amek/).

Results

A total of 361 titles and abstracts were identified over the 6-month sample period for the five sports medicine journals. After removal of duplicates, the screening process identified 243 studies that met inclusion criteria and so were included in the review (Figure 1). Overall, the Kappa agreement between reviewers for data extraction was 0.86. Random sample quality control agreement was 0.98.

Figure 1.
Figure 1.

PRISMA Flow Diagram

Study Characteristics

Of the 243 included studies, a total of 20 (8%, 95% CI: 5-12) were included from the British Journal of Sports Medicine, 5 (2%, 95% CI: 0-7) from Journal of Sport and Health Science, 112 (46%, 95% CI: 40-53) from the American Journal of Sports Medicine, 85 (35%, 95% CI: 29-41) from Medicine Science Sport and Exercise, and 21 (9%, 95% CI: 5-13) from Sports Medicine-Open.

A total of 94 studies (39%, 95% CI: 33-45) were prospective cohort studies, 58 (24%, 95% CI: 19-30) retrospective cohort, 32 (13%, 95% CI: 9-18) cross-sectional, 29 (12%, 95% CI: 8-17) were randomized controlled trials, 14 (6%, 95% CI: 3-9) case-control, 14 (6%, 95% CI: 3-9) case series, 1 (<1%, 95% CI: 0-2) quasi-experimental, and 1 (<1%, 95% CI: 0-2) economic and decision analysis.

A total of 81 studies (33%, 95% CI: 27-40) investigated general population exercise, 57 (23%, 95% CI: 18-29) multiple sports, 51 (21%, 95% CI: 16-27) general orthopaedic patients, 15 (6%, 95% CI: 3-10) running, 10 (4%, 95% CI: 2-7) baseball, 4 (2%, 95% CI: 1-4) cycling, 4 (2%, 95% CI: 1-4) military, 3 (1%, 95% CI: 0-4) soccer, 3 (1%, 95% CI: 0-4) swimming and diving, 2 (1%, 95% CI: 0-3) football, and 1 (<1%, 95% CI: 0-2) for individual sports of basketball, e-sports, handball, lacrosse, motor sports, netball, occupational population, pregnant athletes, rowers, and skiing.

Evaluation of Overall Open Science Practices

No studies met all open science practices. One study (<0.1%, 95% CI: 0-2) met at least 8 out of 12 open science criteria. The median number of open science practices met per study was 2 (range: 0-8; IQR: 2). Please refer to supplementary data (https://osf.io/4amek/) for individual study evaluations.

A total of 234 (96%, 95% CI: 93-98) reported author conflicts, and 163 (67%, 95% CI: 61-73) provided details on funding. A total of 21 (9%, 95% CI: 5-13) provided open access data through an embedded a link or made data available in the supplementary material. Fifty-four (22%, 95% CI: 17-28) included a data availability statement or signposted where data was available. Of these 54 studies, 39 (72 %, 95% CI: 58-84) reported data was available upon reasonable request, and 15 (28%, 95% CI: 16-42) reported a publicly available site to request data. Three studies (6%, 95% CI: 1-15) provided a link, made available the supplementary material, or highlighted where open access code was available.

Seventy-six studies (32%, 95% CI: 22-34) had fully transparent and available materials and methods. Twenty-eight studies (12%, 95% CI: 8-16) reported following a reporting guideline. Of these, 14 (50%, 95% CI: 31-69) reported the Consolidated Standards of Reporting Trials (CONSORT) guidelines,35 11 (39%, 95% CI: 22-59) reported the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines,36 4 (14%, 95% CI: 4-33) the TRIPOD guidelines,37 and 1 (4%, 95% CI: 0-18) the Checklist for Reporting Results of Internet E-Surveys (CHERRIES) guidelines.38 Twenty eight studies (12%, 95% CI: 8-16) reported preregistration and 6 (3%, 95% CI: 1-5) published a protocol in an open access journal or placed it in an open science repository. Four (2%, 95% CI: 0-4) reported the availability of an analysis plan. No studies (0%, 95% CI: 0-2) were replication studies. Seven studies (3%, 95% CI: 1-6) reported patient and public involvement or citizen science. (Figure 2).

Figure 2.
Figure 2.

Breakdown of Open Science Practice

Figure 3.
Figure 3.

Open Sciences Practices

*Replication is not reported as no studies were replication

COI = Conflict of Interest. AJSM = American Journal of Sports Medicine, BJSM = British Journal of Sports Medicine, JSHS = Journal of Sport and Health Science, MSSE = Medicine and Science in Sport and Exercise, SMO = Sports Medicine-Open.

Open Science Practices by Journal

The median number of open science practices met per article for British Journal of Sports Medicine was 3 (range: 2-8; IQR: 3), the median for Journal of Sport and Health Science was 3 (range: 3-5; IQR: 1), the median for American Journal of Sports Medicine was 1 (range: 1-7; IQR: 1), the median for Medicine Science Sport and Exercise was 2 (range: 0-6; IQR: 1), and the median for Sports Medicine-Open was 4 (range: 3-7; IQR: 1).

Greater than 50% of studies published in each journal reported author conflicts and funding. Less than 40% of studies reported for data citation in each journal, and only two journals (AJSM and MSSE) had any articles report open access code. The use of reporting guidelines was reported in 25% or less of studies published in each journal. Only studies in two journals (AJSM and BJSM) reported the availability of statistical analyses plans. Studies in the British Journal of Sports Medicine were twice as likely to report patient and public involvement (Figure 2A; Appendix 2)

Open Science Practices by Study Design

The median number of open science practices met per study for prospective cohorts was 2 (range: 0-6; IQR: 2), the median for randomized controlled trials was 4 (range: 1-8; IQR: 2), the median for retrospective cohorts was 1 (range: 1-4; IQR: 1), the median for case-controls was 2 (range: 1-7; IQR: 2), the median for cross-sectional studies was 2 (range: 1-5; IQR: 3), the median for case series was 1 (range: 1-6; IQR: 0). Economic and decision analyses and quasi-experimental studies both only included one study.

All study designs had similar percentage in terms of meeting the open science criteria for author conflicts, funding, data transparency, and analysis and code transparency. Randomized controlled trials had four times greater percentage of studies that used reporting guidelines and five times greater percentage for registering a study. Randomized controlled trials had three times greater percentage for reporting availability of a statistical analysis plan, and five times greater percentage for reporting patient and public involvement (Figure 2B; Appendix 2)

Open Science Practices by Sport

The median number of open science practices met per study for general population exercise was 2 (range: 0-8; IQR: 2), the median for multiple sports was 1 (range: 1-5; IQR: 1), the median for general orthopaedic patients was 2 (range: 1-7; IQR: 1), the median for running was 3 (range: 1-5; IQR: 2), and the median for baseball was 1 (range: 1-3; IQR: 1).

All sport, exercise, and orthopaedic population studies demonstrated a similar percentage for meeting open science criterion for author conflicts, funding, data transparency, analysis and code transparency, study registration, analysis plan, and patient and public involvement. Studies that investigated orthopaedic patients had two times greater percentage of using a reporting guideline compared to studies that studied investigated sport and exercise populations (Figure 2C; Appendix 2).

Discussion

The primary finding of this study was that no studies from the top five sports medicine journals met all open science practices. One study met 8 out of 11 open science practices, whereas the median number of open science practices met was only two. Open science practices that were least likely to be upheld were sharing of analysis code, sharing data, and availability of an analysis plan. When stratifying by study design, randomized controlled trials reported adopting the most open science practices criteria, whilst observational studies the least.

Our review revealed that the severely limited use of open science practices which has significant implications for the analysis and impact of research findings on clinical and sports science practice. Current sport medicine literature has generally demonstrated a high risk for poor research practices and publication bias towards selecting ‘statistically significant’ findings,39,40 with over 80% of sports medicine research resulting in confirmed hypotheses.40 This is an inordinate high positive rate, but when considering studies that are of high methodological and reporting quality and of adequately powered studies, this resulted in less than 50% ‘significant’ findings.41,42

This indicates substantial potential for bias, both publication bias as well as potential shortcomings in the design, execution, or analysis resulting in the inability to replicate studies also impedes the generalizability of results.43 In addition, the limited adoption of open science practices makes it challenging to test reproducibility and generalizability of the published results. The importance of study replication has been highlighted previously, whereby an open science initiative replicated 100 psychological studies that reported ‘statistically significant’ results, with only 37% reporting positive results after replication.44 The implausibly high prevalence of statistically significant results is detrimental to evidence-based practice.43 False positives(a ‘statistically significant’ result, when in reality no effect exists) could be used to justify the identification of risk factors or use of potential interventions that clinicians and organizations invest time and resources to implementing, with no effect or possibly a harmful effect. Without improved and consistent open science uptake and research integrity, sports medicine research will continue to be limited.

Of particular importance, our review found that sports medicine and science studies demonstrated a paucity in sharing open access analysis code, data, and availability of analytical plans. Freely accessible statistical code and data sharing offers opportunities to other researchers to replicate statistical methods and results,45,46 it can also facilitate the reporting of errors,16,21 aggregate findings,47,48 and combine data from different sources to answer research questions that can’t be answered using single datasets.16,21 Unavailability of code and data hinders the sports medicine community’s ability to confirm results and combine data, to improve cumulative science. 47 It should be noted that while a number of studies reported their data is available upon request statement, which was technically meeting specific open science criteria, this statement is woefully inadequate, and has not resulted in increased access to data within the greater scientific literature.49 Thus, the overall prevalence of open data is likely lower than the reported results.

Randomized controlled trials demonstrated better adoption of open science practices compared to other study designs. Randomized controlled trials are required to register protocols before study recruitment prior at registries such as clinicaltrials.gov. Further, many journals, require RCTs to submit Consolidated Standards of Reporting Trials (CONSORT)35 checklists at the time of manuscript submission. The stricter study registration and methodological reporting of RCTs is due to the inherent risk, and thus patient protection required. Other methodological designs used in sport medicine, most notably observational studies, should require the same registration and methodological rigor, as these studies also inform evidence-based practice.50

As demonstrated within this meta-research systematic review, the reported open science practices are limited for studies published across the top five ranked journals within sports on medicine and science in the Clarivate journal citation rankings. While this is only a 6-month sample of selected sports medicine and science journals, it is possible that open science practice in other sports medicine journals may be even more limited, due to the smaller scientific barriers attributed to lower ranking journals.51

Based on the findings of this review, we strongly argue that the sports medicine community and journal editorial boards should make open science practices a priority before publication. Mandating study registration, availability of protocols, analytical plans, data, open access code, and requiring reporting author conflicts of interest, funding, and guideline checklists at submission are low hanging fruit, which can be easily implemented across all journals. The practices should also be viewed as just doing good science.52,53 Reporting patient public involvement, also known as citizen science, is an easy accessible open science practice that can and should be mandated across all journals. While there may be special concerns about sharing sports medicine data,16,21 many of these barriers can be circumvented, as already shown through other biomedical scientific fields.2,5,18 Potential solutions include creating synthetic (i.e., simulated) data that mirrors the characteristics of the actual data, creating a gatekeeper warehouse for data access, and using federated access (i.e., data is housed and analyzed only within the data owner’s servers). Nevertheless, there is no current consensus on the barriers and facilitators or legal ramifications of open access data within sport, and there is a need and opportunity to engage all relevant parties in this discussion.

There were limitations to this study. This study only included the top five sports medicine journals, as ranked by Clarivate. Other sports medicine journals may demonstrate different open science practices. A 6-month sample was taken from these journals, which may decrease the overall precision in these findings, however this was a contemporary refection of current open science practice. Scoping reviews are broad in nature, which decreases the precision of specific scientific questions.

Conclusions

Less than 20% of recommended open science practices were currently met by studies published in the top five sports medicine journals. Sharing code, data, and availability of analysis plans were the least followed open science practices. Randomized controlled trials revealed better adherence to open science practices compared to observational studies. The sports medicine research community, including journals, researchers, and relevant parties (i.e., patient public involvement) should prioritize open science practices before publication. Without implementing these practices, trust in methods and results will be compromised, thereby negatively impacting how the literature influences evidence-based practice. Understanding the barriers, particularly those associated legality of data-sharing, is likely essential to advancing this progress.

Data Availability

Data is available at https://osf.io/4amek/ Code is available at https://osf.io/4amek/

https://osf.io/4amek/

Contributions

GSB and GSC conceived the study idea. GB and GC designed the study. GB and GC wrote the initial draft. GB, PW, FI, SF, TH, CH, BW, KD, KH, EB, KH, AR, CG, GF, JW, KN, TS, RZ, PD, RR, GC critically revised the manuscript. GB, PW, FI, SF, TH, CH, BW, KD, KH, EB, KH, AR, CG, GF, JW, KN, TS, RZ, PD, RR, GC approved the manuscript.

Funding

This study received no funding.

Conflicts of Interest

All authors declare no conflicts of interest.

Role of Funder

This study received no funding.

Data Availability

Data is available at https://osf.io/4amek/

Code Availability

Code is available at https://osf.io/4amek/

References

  1. 1.↵
    Wuchty S, Jones BF, Uzzi B. The increasing dominance of teams in production of knowledge. Science 2007;316(5827):1036–39.
    OpenUrlAbstract/FREE Full Text
  2. 2.↵
    Bertagnolli MM, Sartor O, Chabner BA, et al. Advantages of a truly open-access data-sharing model. New Eng J Med 2017;376(12):1178–81.
    OpenUrlPubMed
  3. 3.
    Kingwell BA, Anderson GP, Morris ME, et al. Evaluation of NHMRC funded research completed in 1992, 1997 and 2003: gains in knowledge, health and wealth. Med J Aus 2006;184(6):282–86.
    OpenUrl
  4. 4.↵
    Bornmann L, Mutz R. Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references. J Assoc Inform Sci Tech 2015;66(11):2215–22.
    OpenUrl
  5. 5.↵
    Rockhold F, Nisen P, Freeman A. Data sharing at a crossroads. New Eng J Med 2016;375(12):1115-17.
    OpenUrlCrossRefPubMed
  6. 6.
    Maneiro R, Losada JL, Casal CA, et al. Advances in Sport Science: Latest Findings and New Scientific Proposals. Front Psych 2022:2019.
  7. 7.↵
    Asif IM. Celebrating the advancement of science in sports medicine: a 10-year collaboration between sports health and the AMSSM: Sage Publications Sage CA: Los Angeles, CA, 2018:110–12.
  8. 8.↵
    Altman DG. The scandal of poor medical research BMJ 1994:283–84.
  9. 9.↵
    Ioannidis JP. Why most published research findings are false. PLoS medicine 2005;2(8):e124.
    OpenUrl
  10. 10.↵
    Caldwell AR, Vigotsky AD, Tenan MS, et al. Moving sport and exercise science forward: A call for the adoption of more transparent research practices. Sport Med 2020;50(3):449–59.
    OpenUrl
  11. 11.↵
    Andrade C. HARKing, cherry-picking, p-hacking, fishing expeditions, and data dredging and mining as questionable research practices. J Clin Psych 2021;82(1):25941.
    OpenUrl
  12. 12.↵
    Riley RD, Ensor J, Snell KI, et al. External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. Bmj 2016;353
  13. 13.↵
    Riley RD, Lambert PC, Abo-Zaid G. Meta-analysis of individual participant data: rationale, conduct, and reporting. Bmj 2010;340
  14. 14.↵
    Hughes T, Riley RD, Callaghan MJ, et al. The value of preseason screening for injury prediction: the development and internal validation of a multivariable prognostic model to predict indirect muscle injury risk in elite football (soccer) players. Sports Med Open 2020;6(1):1–13.
    OpenUrl
  15. 15.↵
    Ford RA. Trade secrets and information security in the age of sports analytics. The Oxford Handbook of American Sports Law 2018
  16. 16.↵
    Bullock GS, Ward P, Peters S, et al. Call for open science in sports medicine. Br J Sport Med 2022;1;56(20):1143–4.
    OpenUrlFREE Full Text
  17. 17.↵
    Woelfle M, Olliaro P, Todd MH. Open science is a research accelerator. Nature Chem 2011;3(10):745–48.
    OpenUrl
  18. 18.↵
    Kadakia KT, Beckman AL, Ross JS, et al. Leveraging open science to accelerate research. New Eng J Med 2021;384(17):e61.
    OpenUrl
  19. 19.↵
    Wilkinson MD, Dumontier M, Aalbersberg IJ, et al. The FAIR Guiding Principles for scientific data management and stewardship. Scientific data 2016;3(1):1–9.
    OpenUrl
  20. 20.↵
    Bullock GS, Hughes T, Arundale AH, et al. Black Box Prediction Methods in Sports Medicine Deserve a Red Card for Reckless Practice: A Change of Tactics is Needed to Advance Athlete Care. Sports Med 2022:1–7.
  21. 21.↵
    Bullock GS, Ward P, Impellizzeri FM, et al. The trade secret taboo: open science methods are required to improve prediction models in sports medicine and performance. J Orthop Sport Phys Ther 2023.
  22. 22.↵
    Hardwicke TE, Wallach JD, Kidwell MC, et al. An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017). Royal Society Open Science 2020;7(2):190806.
    OpenUrlPubMed
  23. 23.↵
    Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. BMJ evidence-based medicine 2017;22(4):139–42.
    OpenUrl
  24. 24.↵
    Moher D, Shamseer L, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic reviews 2015;4(1):1–9.
    OpenUrlCrossRef
  25. 25.↵
    Mayo-Wilson E, Grant S, Supplee L, et al. Evaluating implementation of the Transparency and Openness Promotion (TOP) guidelines: the TRUST process for rating journal policies, procedures, and practices. Res Integ Peer Review 2021;6(1):1–11.
    OpenUrl
  26. 26.↵
    Tennant JP, Waldner F, Jacques DC, et al. The academic, economic and societal impacts of Open Access: an evidence-based review. F1000Research 2016;5
  27. 27.↵
    Nisonger TE. A methodological issue concerning the use of Social Sciences Citation Index Journal Citation Reports impact factor data for journal ranking. Library Acquisitions: Practice & Theory 1994;18(4):447–58.
    OpenUrlCrossRef
  28. 28.↵
    Gruber T. Academic sell-out: how an obsession with metrics and rankings is damaging academia. J Market Higher Ed 2014;24(2):165–77.
    OpenUrl
  29. 29.↵
    Murray AD, Daines L, Archibald D, et al. The relationships between golf and health: a scoping review. Br J Sport Med 2017;51(1):12–19.
    OpenUrlAbstract/FREE Full Text
  30. 30.
    Bullock GS, Perera N, Murray A, et al. The Relationship Between Cricket Participation, Health, and Wellbeing: A Systematic Scoping Review. Int J Sport Med 2021
  31. 31.↵
    Bullock GS, Uhan J, Harriss EK, et al. The relationship between baseball participation and health: a systematic scoping review. J Orthop Sport Phys Ther 2020;50(2):55–66.
    OpenUrl
  32. 32.
    Munn Z, Tufanaru C, Aromataris E. JBI’s systematic reviews: data extraction and synthesis. AJN The Am J Nurs 2014;114(7):49–54.
    OpenUrl
  33. 33.
    Conaghan PG, Dickson J, Grant RL. Care and management of osteoarthritis in adults: summary of NICE guidance. Bmj 2008;336(7642):502–03.
    OpenUrlFREE Full Text
  34. 34.↵
    Puza B, O’neill T. Generalised Clopper–Pearson confidence intervals for the binomial proportion. Journal of Statistical Computation and Simulation 2006;76(6):489–508.
    OpenUrl
  35. 35.↵
    Altman DG, Schulz KF, Moher D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Int Med 2001;134(8):663–94.
    OpenUrlCrossRefPubMedWeb of Science
  36. 36.↵
    Vandenbroucke JP, Elm Ev, Altman DG, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. Ann Int Med 2007;147(8):W-163-W-94.
    OpenUrlCrossRefPubMed
  37. 37.↵
    Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. Ann Int Med 2015;162(1):55–63.
    OpenUrlCrossRefPubMed
  38. 38.↵
    Eysenbach G. Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES): Gunther Eysenbach Centre for Global eHealth Innovation, Toronto, Canada, 2004:e34.
  39. 39.↵
    Twomey R, Yingling V, Warne J, et al. The Nature of Our Literature: A Registered Report on the Positive Result Rate and Reporting Practices in Kinesiology. Comm Kines 2021;1(3)
  40. 40.↵
    Büttner F, Toomey E, McClean S, et al. Are questionable research practices facilitating new discoveries in sport and exercise medicine? The proportion of supported hypotheses is implausibly high. Br J Sport Med 2020;54(22):1365–71.
    OpenUrlAbstract/FREE Full Text
  41. 41.↵
    Fraley RC, Vazire S. The N-pact factor: Evaluating the quality of empirical journals with respect to sample size and statistical power. PloS one 2014;9(10):e109019.
    OpenUrlCrossRefPubMed
  42. 42.↵
    Stanley TD, Carter EC, Doucouliagos H. What meta-analyses reveal about the replicability of psychological research. Psychological bulletin 2018;144(12):1325.
    OpenUrlPubMed
  43. 43.↵
    Mesquida C, Murphy J, Lakens D, et al. Replication concerns in sports and exercise science: a narrative review of selected methodological issues in the field. Royal Society Open Science 2022;9(12):220946.
    OpenUrl
  44. 44.↵
    Collaboration OS. Estimating the reproducibility of psychological science. Science 2015;349(6251):aac4716.
    OpenUrlAbstract/FREE Full Text
  45. 45.↵
    Errington TM, Denis A, Perfito N, et al. Reproducibility in cancer biology: challenges for assessing replicability in preclinical cancer biology. Elife 2021;10:e67995.
    OpenUrl
  46. 46.↵
    Camerer CF, Dreber A, Holzmeister F, et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour 2018;2(9):637–44.
    OpenUrl
  47. 47.↵
    Lakens D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Front Psych 2013;4:863.
    OpenUrl
  48. 48.↵
    Riley RD, Lambert PC, Staessen JA, et al. Meta-analysis of continuous outcomes combining individual patient data and aggregate data. Stat Med 2008;27(11):1870–93.
    OpenUrlCrossRefPubMedWeb of Science
  49. 49.↵
    Langille MG, Ravel J, Fricke WF. “Available upon request”: not good enough for microbiome data!: Springer, 2018:1–2.
  50. 50.↵
    Ligthelm RJ, Borzí V, Gumprecht J, et al. Importance of observational studies in clinical practice. Clin Therapeutics 2007;29(6):1284–92.
    OpenUrlCrossRefPubMedWeb of Science
  51. 51.↵
    Sims JL, McGhee CN. Citation analysis and journal impact factors in ophthalmology and vision science journals. Clin Exper Ophthalmal 2003;31(1):14–22.
    OpenUrl
  52. 52.↵
    Moher D, Bouter L, Kleinert S, et al. The Hong Kong Principles for assessing researchers: Fostering research integrity. PLoS Biology 2020;18(7):e3000737.
    OpenUrlCrossRefPubMed
  53. 53.↵
    Cobey KD, Haustein S, Brehaut J, et al. Community consensus on core open science practices to monitor in biomedicine. Plos Biology 2023;21(1):e3001949.
    OpenUrl
Back to top
Posted March 30, 2023.
Download PDF

Supplementary Material

Data/Code
Email
Up front and open, shrouded in secrecy, or somewhere in between? A Meta Research Systematic Review of Open Science Practices in Sport Medicine Research
Garrett S. Bullock, Patrick Ward, Franco M. Impellizzeri, Stefan Kluzek, Tom Hughes, Charles Hillman, Brian R. Waterman, Kerry Danelson, Kaitlin Henry, Emily Barr, Kelsey Healey, Anu M. Räisänen, Christina Gomez, Garrett Fernandez, Jakob Wolf, Kristen F. Nicholson, Tim Sell, Ryan Zerega, Paula Dhiman, Richard D. Riley, Gary S Collins
medRxiv 2023.03.30.23287959; doi: https://doi.org/10.1101/2023.03.30.23287959
Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools

Subject Area