Investigating the Nature of Open Science Practices Across Complementary, Alternative, and Integrative Medicine Journals: An Audit

Approach and Open Science Statement

CAIM journals were identified and items from the TOP guidelines were used to extract data from each journal [34]. These items were assessed using the TOP rubric [35] to find the TOP Factor and determine which open science practices are implemented, encouraged, and required for their publications. The protocol, study material, and blank extraction form, were made available on Open Science Framework (OSF) [36].

Journal Selection

A list of CAIM journals was obtained from Table 2 of Ng’s (2021) bibliometric analysis of CAIM journals [37]. Ng conducted a search on Scopus for CAIM journals. Compared to other databases such as Web of Science, Scopus includes more CAIM-categorized journals in its database [38]. The search was based on journals belonging to the category of “Complementary and Alternative Medicine” (Code 2707), of the All-Science Journal Classification (ASJC) [37].

Eligibility Criteria

The list obtained from Table 2 of Ng’s bibliometric analysis was modified for this study to include only journals with the exact words “complementary”, “alternative”, and/or “integrative” in the journal title [37]. The modified list was further narrowed to exclude journals that were discontinued, renamed, inaccessible via their website, incorporated into other journals already found in the list, or only published in print. In other words, the revised list only includes journals that are currently active and accessible at the time of data collection. Table 1 contains the comprehensive list of journals along with the exclusion criteria that were applied.

TOP Factor

The TOP guidelines are a set of eight standards (Table 2) that journals are encouraged to adopt to enhance open science practices [7]. The TOP Factor is a metric that rates how well journals implement each of the TOP guidelines [9]. The TOP Factor is scored using the TOP rubric, a publicly available rubric created by the COS to guide journal assessment [35]. The TOP rubric includes four possible levels where journals can be ranked, from 0 to 3 [35]. Each TOP guideline has individual requirements for each score, with a higher score indicating stronger adherence to each guideline. For example, for data sharing, a score of 0 indicates that the journal only encourages or does not mention data sharing. A score of 1 indicates that the journal requires the author to disclose where data are publicly accessible. A score of 2 indicates that the journal mandates that authors share their data except for certain circumstances such as sensitive health data and proprietary data. A score of 3 implies that the journal not only requires data sharing, but also includes a verification process to ensure that the data is consistent with the reported findings in the published article [35]. The highest possible TOP Factor a journal can receive is 24.

Data Extraction and Assessment of Journal Practices

The first draft of the data extraction form was developed by BL and TP, which was reviewed by JYN with careful detail. The revised data extraction form was then circulated to HC and DM for their feedback and amended before beginning data extraction. The following items were extracted from the journals being evaluated in this study: website Uniform Resource Locator (URL), International Standard Serial Number (ISSN), Journal Citation Reports (JCR) impact factor, publisher, whether the journal is available in print or online, first year of print publication (if applicable), first year of online publication, and the specific text associated with the qualitative features as outlined by the items in the TOP guidelines.

The degree to which open science practices were required by journals was assessed using the TOP Factor [39]. Journals can be submitted for evaluation by the COS or journals can be self-evaluated using the TOP rubric and submitted to the COS for verification [34]. We self-assessed journals and submitted our evaluations to the COS through a Google form posted on their website [40]. Journals that were found to have already been assessed by the COS were used to pilot test the data extraction procedure [35]. A pilot data extraction procedure was performed by BL and TP on 5 journals in the list, of which 2 were previously assessed by COS. The purpose of the pilot extraction was to allow an opportunity for standardization among scorers before proceeding to the independent extractions. Most of the journals selected for the pilot had different publishers to account for anticipated differences in the requirement of open science practices. The extracted data was compared between BL and TP, then carefully reviewed by JYN. The remaining conflicting TOP ratings were resolved with consultation from HC and DM.

Data Analysis and Presentation

The CAIM journals’ open science practices were evaluated using the TOP rubric to identify the individual scores for each journal. CAIM journals were then ranked by their overall TOP Factor scores. To evaluate which TOP guidelines were most engaged with by the journals, scores for each TOP guideline were summed across all journals to find the TOP Factor.

Description of Journals

A total of 19 CAIM journals were included in this audit, two of which were previously assessed by the COS. A flow chart of the journal exclusion process can be found in Figure 1. Table 3 includes each journal’s URL and general information about their impact factors, whether they are available in print or online, and their first year of publication. In summary, the journals’ impact factors ranged from 1.449 to 4.473. Seventeen journals (89%) are available in print and online and two (11%) are only available online. The first year of online publication ranges from 1995 to 2022.

Overall TOP Scores

Table 4 shows the TOP guideline scores and TOP Factor calculated for all 19 journals. The mean TOP Factor across all journals was 2.95, with a median score of 2 and standard deviation of 2.63. The journals with the highest overall TOP Factors were Evidence-based Complementary and Alternative Medicine (8), Complementary Medicine Research (7), Integrative Cancer Therapies (7), and Journal of Evidence-Based Integrative Medicine (7). Four journals (21%) had a TOP Factor of 0, including African Journal of Traditional, Complementary and Alternative Medicines, Chinese Journal of Integrative Medicine, Integrative Medicine, and Journal of Complementary and Integrative Medicine.

Data Citation

Information about the specific text associated with each TOP guideline for each journal can be found in Supplementary File 1. Six journals (32%) provided no information regarding data citation practices and received a score of 0 in this category. Ten journals (53%) encouraged citation of datasets, though the practice was not required nor mandated for publication, receiving a score of 1. As noted in the TOP rubric, the word “should” was presumed as encouragement rather than a requirement [35]. Three journals (16%) required appropriate citation for all data obtained elsewhere, resulting in a score of 2. No journals requested enforced data citation as a condition for publication.

Data, Analytical Code, and Research Materials Transparency

The majority of journals (68%) received scores of 0 for the data transparency, analytical code transparency, and research materials transparency categories. Of these, 8 journals (42%) had no mention of data sharing, while 5 (26%) only encouraged it.

Two journals, BMC Complementary Medicine and Therapies and Complementary Medicine Research (11%) required a Data Availability Statement outlining whether and where research, including code data, can be found. Another journal, Evidence-based Complementary and Alternative Medicine, required that “research published in the journal must be: as reproducible as possible – sharing underlying data, code, and supporting materials wherever able.” [41] All three of these journals (16%) were ranked a score of 1 in both the data transparency and analytical code transparency categories. However, BMC Complementary Medicine and Therapies only encouraged rather than required sharing research materials upon request, which resulted in a score of 0 for research materials transparency, unlike the other two journals which received a score of 1.

One journal, Integrative Medicine Research, received a score of 2 in the data transparency category for mandating a Data Availability Statement and, where applicable, noting reasons why data may not be shared. This journal, however, only encouraged code and material sharing, receiving a score of 0 for both the analytical code transparency and research materials transparency categories.

Two journals, Integrative Cancer Therapies and Journal of Evidence-Based Integrative Medicine, (11%) required publicly available data as a condition of publication, receiving a full score of 3 for data transparency. Both of these journals did not include any specific information about code or material sharing, receiving a score of 0 for these latter categories.

Design & Analysis Transparency

Seven journals (37%) received a score of 0 in this category as four journals (21%) did not mention reporting guidelines at all and three journals (16%) only mentioned CONSORT, which does not encompass the majority of articles published within journals [35]. Seven journals (37%) received a score of 1 for encouraging use of reporting guidelines across several study types. As noted in the TOP rubric, the word “expect” was presumed as a requirement [35] Five journals (26%) required adherence to appropriate reporting guidelines for publication and received a score of 2. None of the journals enforced this adherence and hence were not allotted a full score of 3.

Study and Analysis Plan Preregistration

Sixteen journals (84%) received a score of 0 for study preregistration. Preregistration, as defined by OSF, refers to the creation and storage of the research plan in a public repository at the start of the study [42]. Of these journals, five (26%) did not mention preregistration at all and 11 (58%) only mentioned information about preregistration for clinical trials. Three journals, Evidence-based Complementary and Alternative Medicine, European Journal of Integrative Medicine, and Complementary Therapies in Clinical Practice (16%), received a score of 1 for including a statement about article preregistration in their authorship guidelines.

Eighteen journals (95%) received a score of 0 for analysis plan preregistration due to no explicit mention of preregistration with an analysis plan. One journal, Complementary Medicine Research, (5%) received a score of 1 for including a statement about analysis plan preregistration in its authorship guidelines.

Replication

Most journals (95%) provided no information regarding submission of replication studies. One journal, Evidence-based Complementary and Alternative Medicine, (5%) encouraged replication studies, resulting in a score of 1 for this category.

Summary of Results

The purpose of this study was to conduct an audit which investigates the nature of open science practices across CAIM journals. The mean TOP Factor across all journals was found to be 2.95, with a median score of 2 and standard deviation of 2.63. The range of TOP scores reflects the variability in open science practices within CAIM journals. Four journals (21%) had a final TOP score of 0, indicating no adherence to open science practices. The highest TOP factor was a 8, out of a maximum possible of 24. Across the audit, journals most frequently adhered to some requirements of data citation and design and analysis transparency, as evidenced by the fewest scores of zero in these categories. Journals had the lowest scores in the analysis plan preregistration and replication categories. A common trend across several categories was encouragement rather than requirement or enforcement of open science practices. TOP scores of 0 and 1, corresponding to no mention or encouragement, were more common in all categories than scores of 2 and 3, corresponding to a mandate or condition of publication. Overall, our findings suggest that CAIM journals provide minimal guidelines to encourage or require authors to adhere to open science practices, as reflected by the average TOP Factor of 2.95.

This low usage of open science practices across journals is similar to findings in other disciplines and suggests that journals have an opportunity to improve research practices through regulating the use of open science practices. For example, research shows open science practices are not highly encouraged nor required by many communication sciences and disorders journals, with a mean TOP Factor of 1.4 [17]. A similar finding was seen in an audit of open science practices across health and medical science journals, where transparency and open science practices in these journals were not frequently encouraged or mandated [18]. However, the mean TOP Factor was found to be 7, which is higher than the mean for the CAIM journals included in the present study. Similar audits were conducted on pain journals and sleep and chronobiology journals, which found that there was relatively low journal engagement with open science standards in these fields with a median TOP Factors of 3.5 and 3, respectively [43,44]. Furthermore, an audit done on clinical psychology journals found that many journal recommendations, such as use of reporting guidelines, are not frequently enforced in journals [16]. In the field of CAIM, open science practices have the potential to increase the quality reproducibility of research, as better data transparency can promote replicability and peer review to ensure reliability and credibility of findings. Open science practices in CAIM may also expedite the research process through emphasis on preregistration, for example. With improved article quality, stronger experimental evidence, and greater dissemination of CAIM research, negative perceptions of the field may be addressed. Increased public and provider trust in the field may then allow more funding and resources to be allocated for CAIM research, allowing continual improvement of research quality in the field.

With a growing interest in CAIM from patients and practitioners,, evident by the numerous patient trials each year, it is becoming increasingly important to ensure that the research quality within this field of medicine is improved and standardized [45]. As many patients use research to make decisions about their healthcare, it is critical for journals to hold submitted articles to a greater standard of open science practices to avoid patient misinformation and improve provider attitudes [46,47]. For example, a study found that many patients resorted to and trusted information on CAIM therapy from medical journals, in the form of clinical trials, compared to social networks or other online media [46]. Another study also noted patients used medical and lay publications in addition to the Internet for better understanding their CAIM options [48].

Given the potential benefits of open science practices for CAIM research, it is important to consider barriers to its usage and ways to overcome them. A potential reason for low TOP scores includes a low incentive for journals to implement the usage of open science practices due to the added time and resources to mandate the change [50]. Moreover, support from editors is crucial for promoting open science practices reforms within journals. Despite overall positive attitudes towards open science, a survey conducted by Naaman et al. identified that editors see the time and effort required to implement open science practices as a major barrier for open science promotion at the journal level [50]. For example, if data citation was mandated as a condition of publication, peer reviewers may not want to take additional time to ensure articles are published in accordance with the journal’s open science practices. Publishers can potentially alleviate this barrier by changing the default settings in popular manuscript submission systems to allow for more uniform data input in line with TOP (e.g., a mandatory field requiring input of a data availability statement) [51]. Such changes have been attempted in a study performed by Giofrè et al. that suggests that journal-specific submission guidelines could promote positive alterations in the practices of authors [15]. This adjustment would require authors to include TOP items in the manuscript submission and thus reduce the time and effort required from peer reviewers to check for usage of open science practices. Based on this discussion, a next step is to investigate how well CAIM authors adhere to open science practices encouraged or required by these CAIM journals. This type of study can allow comparison of standards set by journals with the actual implementation of open science practices by authors in the field.

Another barrier to the implementation of open science practices at a journal level stems from the often poor reporting of CAIM research [52]. When research is poorly reported, there is a decreased ability to draw conclusive results and make comparison with similar research studies, reducing research efficiency, validity, and replicability [52,53]. The multimodal treatment delivery of some types of CAIM are not well substantiated by research, often limiting its focus to one or two treatment interventions [53]. For example, while acupuncture needling is researched as the primary technique employed by licensed acupuncturists, their use of other interventions (e.g., cupping and massage) that are also employed in their multimodal treatment plans are not well researched [53]. Adapting research methods commonly used in Western biomedicine, such as clinical trials, for use in CAIM research in addition to using other research methods such as qualitative studies can improve its research basis [52,54]. Components of open science practice can support this research through the requirement of data sharing and a methodologically sound study design. Furthermore, there is low support for CAIM research from institutions, leading to its inadequate research infrastructure development and funding for research training in the field [54,55].By training CAIM researchers in open science practices, which emphasize open data sharing and preregistration, its encouraged use can improve the rigor of CAIM study reporting and methodology [53]. Consequently, the number of inadequately designed studies can be reduced, thereby making research more efficient [53]. With improved research and open science training, CAIM researchers may be more inclined to use open science practices in research development, which also makes journal level open science practice changes more likely to be made. For example, in the field of economics, researcher attitudes are readily accepting of strong transparency standards, which has led many high-impact economics journals to also adopt strong open science requirements [7].

Furthermore, increased funding needs to be directed towards CAIM research to train and incentivize these practitioners to perform research in this field. While many research disciplines face challenges with limited funding, the allocation of funding for CAIM research is notably lower compared to other healthcare sectors [56]. The safety and efficacy of CAIM therapies have been long debated and uncertainty with findings in the field significantly contribute to the lack of financing allocated to CAIM research [22]. Failure to obtain CAIM research funding in the past has often been attributed to skepticism and low perceived priority of CAIM research by grant reviewers, as well as low institutional support of CAIM research [57]. Funding applications for CAIM research have also been seen to fall short in comparison to conventional medicine due to insufficient infrastructure and fewer university level researchers showing interest in the field [22,55]. As noted by Veziari et al., as compared to advocating for linear research funding models, creating multi-faceted funding programs, though previously limited to other disciplines, may be favorable for CAIM as well [58] For example, funding research programs that require interaction between sponsors, CAIM researchers, and users, which include practitioners and patients, allows the opportunity to identify and address misconceptions regarding quality of findings within the field [52,59]. Investing in CAIM research training can also promote research quality that meets a more comparable standard to conventional medical research, and, in turn, makes for more competitive funding applications [55]. The adoption of open science practices, itself, provides an opportunity to present transparent, reproducible, and evidence-based CAIM research, which can also incentivize funding agencies to invest in CAIM research [33].

Several fields of research have seen improvements in the quality of their research articles and other metrics after implementing open science practices. Studies in the field of psychology have focused on registered reports, which is a type of registration that requires a two-stage peer review process [60]. A protocol’s introduction, materials, and methods undergo a first stage of peer review prior to the execution of the methodology, and is peer reviewed a second time once the manuscript is completed, checking for adherence to the original methodology [60]. Soderberg et al. found that psychology research articles published with registered reports have significantly improved research thoroughness and reputability, as well as include a more rigorous methodology and analysis [61]. Additionally, Obels et al. found that compared to other articles, registered reports have higher rates of data and code sharing [62]. However, studies have also found that even when data sharing is required by journals, data reporting is often inadequate for reproducibility which shows that open data alone is unable to achieve the proposed benefits of data sharing [63,64]. To draw conclusions about the impact of open science in CAIM, more research is warranted.

Strengths and Limitations

Our study has notable strengths. We sourced our subset of CAIM journals from a bibliometric analysis conducted by Ng, which was sourced from the Scopus Source List, a list of journals with established ASJC categories [37]. Furthermore, screening, data extraction, and TOP rubric assessment of the CAIM journals was conducted independently and in duplicate. All collected information was reviewed by all authors, and any discrepancies was resolved via consensus. Furthermore, comparing definitions against the TOP rubric provided a standardized reference for comparing data across different journals. The detailed and specific methodology also allows for the replicability of the study’s findings.

Regarding limitations, this study only includes CAIM journals that published online, which may not reflect the open science practices of CAIM journals that only publish in print format. Additionally, only journals that publish in the English language were included, thus our findings may not be representative of CAIM journals published in other languages. Furthermore, the audit relies on the information made publicly available by the included journals. If relevant data on open science practices was not readily accessible or inconsistently reported, the study’s findings may be incomplete or inaccurate. We also acknowledge that the audit may not capture the reasons behind the observed open science practices. Without additional qualitative data or information from journal editors, authors, or reviewers, it can be challenging to fully understand the motivations or barriers influencing open science practices in CAIM journals. Further, this audit represents a snapshot of open science practices at a specific point in time, and it can be expected that this information may change over time. Lastly, the TOP guidelines also do not address open science practices in regard to open access publishing models, which is a notable aspect of open science.

Ethics Approval and Consent to Participate

Ethics approval was not sought as this study only involved the use of publicly available data.