Investigation of Research Quality and Transparency in Neurosurgery Through the Utilization of Open Science Practices

INTRODUCTION

Open science practices refer to scientific research’s reproducibility and transparency, including open data, methods, results, and accessibility. These have increasingly become a part of research in the medical field, and efforts have been made to encourage open practices in all scientific research. In 2023, the Biden-Harris administration, through the White House Office of Science and Technology, launched the Year of Open Science, which included actions “to advance national open science policy, provide access to the results of the nation’s taxpayer-supported research, accelerate discovery and innovation, promote public trust, and drive more equitable outcomes.” ¹ Lack of transparency in research has various implications, such as giving providers and patients incomplete information about the risks and benefits of treatments. Furthermore, it can lead to unnecessary spending of vital health system funds on ineffective therapies or guidelines.^2,3 Open science practices enable evidence-based and transparent decisions in medicine, which bolsters trust among patients and researchers.^1,3,4

Inconsistency in research practices has also resulted in the reproducibility crisis, where published research findings often cannot be replicated. Moreover, research waste, including biases and methodological errors, has been estimated to account for 85% of all medical research funding.^4,5 Lievore et. al. found an inverse relationship between the percentage of publications retracted and the Journal Citation Report (JCR).⁶ This indicated that errors in publications and lack of reproducibility led to fewer citations and use of those publications, hindering scientific progress and contributing to overall mismanagement. Between 1992 and 2012, papers retracted due to misconduct alone accounted for about $58 million of NIH research funding.⁷ As a result, there is a critical need for improved regulation of research quality, which open science standards aim to address. However, it is essential to evaluate whether these standards are being properly implemented and working, particularly across various types of journals. Research practices have been investigated diffusely throughout medicine, but there is no current literature on the use of open science practices in the neurosurgical literature. Thus, the purpose of this study was to explore the use of these open-science practices in papers published in top neurosurgical journals.

METHODS

RESULTS

One journal required (20%), and three journals (60%) recommended utilizing EQUATOR guidelines. Three journals (60%) allowed preprints, and all five journals (100%) recommended or required preregistration of clinical trials, but only two (40%) recommended preregistration for systematic reviews. All five journals (100%) recommended or required methods to be publicly available, but none (0%) published peer-review comments (Figure 1).

• Download figure
• Open in new tab

Figure 1.

Percentage of neurosurgical journals (n=5) in recommending or requiring each open science practice within their journal guidelines.

Across all 150 studies, 109 (73%) utilized at least one or more open science practices. Ninety-four studies (63%) from our sample properly reported EQUATOR guidelines. Twenty-four articles (16%) provided open access to data, while no studies published preprints (0%) or peer review comments (0%). Preregistration rates varied by study type, with all RCTs (100%) preregistered. OSs (6%) and SRs (17%) provided less adherence to preregistration (Figure 2). Throughout these 150 articles, only 9 (6%) studies required reconciliation by the third reviewer.

• Download figure
• Open in new tab

Figure 2.

Percentage of open science utilization among all 150 articles.

Neurosurgical Review utilized the most open science practices, with a mean utilization of 1.4 open science practices per publication versus 0.9 across the other four journals (p < 0.001). Moreover, Neurosurgical Review significantly utilized more open science practices versus the Journal of Neurosurgery (p < 0.050) and World Neurosurgery (p < 0.050). There was no relationship between journal rank and the use of open science practices (R² = 0.32)

RCTs and SRs utilized the most open science practices per publication, with a mean utilization of 1.2 and 1.3 open science practices, respectively. In contrast, OSs utilized the least open science practices with a mean utilization of 0.3 practices per publication. Both RCTs (p < 0.001) and SRs (p < 0.001) significantly utilized more open science practices compared to OSs. There was no significant difference in open science practice utilization between RCTs and SRs (p = 0.916).

DISCUSSION

This study investigated the utilization of open science practices in five leading neurosurgical journals. Findings revealed that there is significant variability in their application of open science usage in both the journal and article levels. One journal required and three journals recommended the use of EQUATOR guidelines. Notably, all five journals recommended or required methods to be publicly available, but none published peer-review comments. Neurosurgical Review demonstrated the highest mean utilization of open science practices per publication. When analyzing the utilization of open science practices in specific types of studies, RCTs and SRs applied the most open science practices per publication.

Despite the efforts of journals to implement open science practices, adherence to these practices remains generally poor. The use of open science practices in imaging journals found that 65% had policies on protocol registration and reporting guidelines.⁸ However, similar to our results in neurosurgical journals, adherence at the article level was significantly lower and variable, potentially highlighting a disconnect between editorial policy and open science practice implementation. Nonetheless, journal policies could benefit from further refinement. For example, policies regarding data availability in general and internal medicine journals are often inconsistent, and this inconsistency could potentially lead to confusion among researchers seeking to adhere to these guidelines.⁹ Additionally, Grant et. al analyzed the implementation of ten Transparency and Openness Promotion (TOP) standards within a broad range of journals (339) and highlighted that while some journals implement various TOP standards, most lack a comprehensive adherence to open science practices or consistent enforcement at the article level.¹⁰ Moreover, multiple smaller studies have demonstrated that various types of journals exhibit low compliance with transparency guidelines.^11–15 Specifically, general surgery journals were poorly compliant with specific open science guidelines. Of the 240 articles they surveyed, 34% used one or more open science practices, 26% complied with EQUATOR guidelines, 5% pre-registered protocols, and 9% fully disclosed methods to the public.¹⁶ The low use of open science standards in neurosurgical journals, as found in our study, aligns with several of these aforementioned findings, demonstrating that publicly available practices and guidelines may be insufficient to achieve a truly transparent research community. Hence, there is a necessary call to action for several journal types, including neurosurgical journals, to better promote open science usage and to prevent unintended consequences. For example, non-adherence to open science practices can delay advances in treatment and management guidelines, as illustrated by previous research on post-heart-transplant antibody-mediated-rejection (AMR). Scientific research in the 1990s found evidence that AMR most often presented in the first three months post-transplant and would often recur within the first year post-transplant.¹⁷ Further, patients with more than 3 AMR episodes were likely to die of cardiovascular-related causes. However, it took 24 years since the first description of post-heart transplant AMR for diagnostic criteria to be published in official guidelines. Despite having the knowledge, the development of guidelines was delayed by affinity bias and publication bias.¹⁷ Open science guidelines aim to address these concerns by collaborating knowledge and thereby accelerating the translation of research findings into clinical applications.

The quality and reproducibility of studies published in neurosurgical journals have previously been investigated, revealing potential gaps and areas for improvement. Among 633 review articles across ten leading specialty neurosurgical journals, only 45.97% of search strategies were reproducible.¹⁸ Rothoerl et al. explored the level of evidence (LOE) and citation index across neurosurgical journals. 55% of 982 articles investigated were rankable by LOE, and only 0.4% and 3.0% of these studies met the criteria for LOE Ia. and LOE Ib, respectively. The mean citation index across all 982 articles was suboptimal at 4.80, although for literature reviews, clinical studies, and experimental studies, the citation indices were 10.66, 6.78, and 5.03, respectively.¹⁹ Therefore, it is crucial to advocate for increased adoption of open science practices, as these measures can significantly enhance neurosurgical research. However, there may be several specific challenges to applying open science practices in surgical journals diffusely.

Research transparency promotion, such as through data-sharing policies, is quite limited in surgical journals.^20–22 Therefore, many surgical studies, specifically concerning RCTs, have not been reproducible.²³ These results could allude to the fact that implementing open science practices in surgical research could be quite difficult due to unique challenges. These challenges may stem from the complex nature of surgical research, which involves balancing patient safety, privacy, and intellectual property with the need for transparency and reproducibility. However, these challenges should be addressed, as incomplete transparency in methodology can significantly impact research conclusions. In a recent study examining outcomes for head and neck reconstruction surgery using the National Surgical Quality Improvement Program (NSQIP) database, different approaches to case selection and statistical analysis yielded significantly varying results, thereby highlighting the need for transparency when reporting methods.²⁴ This allows other researchers to reproduce the work published in surgical journals while also aiding publishing journals in evaluating methodologies and improving surgical literature aimed at maximizing surgical outcomes.

Additionally, other barriers to promoting open science usage include a lack of institutional funding and incentives, the presence of fees in adhering to open science practice, and the lack of mandates. Thus, incentives and “badges” can be implemented to recognize publications and journals that adopt open science practices, which would highlight their commitment to research publication quality control and encourage others to integrate these practices in their work. The Development Psychology Department of Brigham Young University introduced an incentivized quality measurement system for evaluating faculty research. The measurement system grades research quality based on criteria such as the use of preregistration and adherence to EQUATOR guidelines. The purpose of this system was to encourage better science practice and reduce pressure on faculty to accumulate publications and novel findings, and instead focus on valuable but underutilized research initiatives that implement longitudinal studies, larger sample sizes, and other time-consuming criteria.^25,26 The hope is that this system motivates other faculties and institutions to adopt similar incentivized initiatives and address challenges seen between the competitive nature of academia and open sharing.²⁷ Moreover, many have expressed uncertainty and ambiguity about the exact requirements needed for them for open science usage.^27–29 The numerous guidelines and expectations can cause confusion among researchers and lead to improper usage. In a survey of 198 cardiology researchers, most participants indicated that they received no formal training with open science practices and utilized these strategies at their discretion.³⁰ To enhance journal communication with researchers, it is essential to include specific and standardized instructions that detail how to improve transparency in their submissions. Journal requirements have been shown to influence researcher’s ensuing practices.³¹ This approach would ensure clarity and consistency in expectations regarding transparency practices.

Despite these barriers, there is growing sentiment and motivation for the scientific community to expand open science usage for the advancement of scientific knowledge.^32–34 Researchers were surveyed on outstanding motivators for open science practices. For personal motivation, open science was mentioned to potentially improve researchers’ professional standing. Open access would encourage unanticipated collaboration between groups, creating a more diverse and visible community and augmenting publication productivity. Furthermore, due to the competitive nature of scientific research, open science could be used strategically to maintain strong research metrics. For example, data sharing with other groups could prove useful if proper attribution is included, such as through citation, acknowledgment, or co-authorship. From a broader perspective, interviewees highlighted ethical motivation as open sharing would help reduce delays in the publication of publicly funded research and prevent “bottlenecking” of data prior to publication. Broadly sharing data samples would foster greater access to research output and maximize research efficiency and discovery.³⁴

CONCLUSION

This study aims to pave the way for the analysis of the adherence of journals to open science protocols. There is ample room for further development in this subject. For example, similar studies should be conducted in other surgical subspecialties. Other studies can continue to investigate the validity and potential of open science practices.³⁵ By promoting the incorporation of these open science protocols, a high standard of quality throughout the research community can be maintained. Improved research quality can directly lead to better patient outcomes, enhanced patient safety measures, and stronger trust between the provider and patient with reliable research results and conclusions.

Data availability Statement

The datasets generated and/or analyzed during the current study are available from the corresponding author upon reasonable request.