Characteristics and Impact of Librarian Co-authored Systematic Reviews: A Bibliometric Analysis

INTRODUCTION

With their expertise in searching and information management, librarians are well positioned to support the production of systematic reviews (SRs). This expertise is recognized by external stakeholders as is evident by the recommendation to include librarians and information specialists in the SR process, as outlined in the Institute of Medicine (IOM) standards on conducting SRs, Finding What Works in Health Care, released in 2011 [1]. Many librarians across different professional settings are engaged in SRs, and supporting SRs has become increasingly standard in the scope of work performed by health sciences librarians. The growing role of librarians in SRs is reflected in the extensive programming on this topic at Medical Library Association annual meetings and continuing education efforts and is the subject of a recent scoping review [2].

SRs are labor- and time-intensive. When SR support is just one of many other job responsibilities, as is the case in most academic and hospital settings, librarians and libraries can become overwhelmed with the demand for SR services, and struggle to find the best way to balance workloads. The increasing demand in requests for SR support has led some library administrators to question the value of providing such a service [3], and some libraries have even implemented a pay-per-service model [4, 5].

Another source of SR-related stress is the fact that in some academic libraries with tenure tracks, the contributions of librarians who successfully collaborate on SRs as co-authors, including coordinating large sections of the SR process, writing portions of the manuscript, and editing manuscripts, may not be considered to meet scholarship requirements because they do not directly focus on a topic within the field of library and information science (LIS). Thus, further research exploring librarians’ substantive role in SRs would give the information professional community a better understanding of the scholarly contributions and impact of this type of work.

Previous research in the LIS literature examined the added value of librarians’ contributions to the quality of SRs. Meert et al. identified SRs in which librarians contributed as co-authors or team members and scored those SRs using a 15-item reporting methodology checklist. They found that librarian involvement correlated with a higher quality score for the literature search component of SRs [6]. Rethlefsen et al. found that librarian/information specialist co-authored SRs reported higher-quality search strategies [7]. Also, Koffel found a positive correlation between librarian involvement in SRs and the use of SR guideline-recommended search methods [8]. These studies provide evidence of the value of librarian contributions to performing and reporting SR searches. However, there is limited published research on other characteristics and the overall impact of librarian co-authored SRs.

Bibliometrics is “the application of mathematics and statistical methods to books and other media of communication” as coined by Pritchard in 1969 [9]. Data regarding research output, impact, and collaborations are some examples of bibliometric measures that can be valuable in strategic decision-making and research assessment. While citation analysis alone provides insight into research impact, combining citation analysis with qualitative methods and analyses can provide greater context and improve its usefulness. Research assessment models such as the Becker Model [10] were developed to provide a framework for highlighting the different ways by which research impact can be captured.

Citation analysis as a method to assess impact has been used in previous LIS literature investigating the value of medical library services on healthcare literature. For example, Sherwill-Navarro et al. conducted a citation analysis and examination of the dissemination of four selected library services research studies to demonstrate impact [11]. However, this study did not look at SRs specifically nor aimed to achieve evaluation of the scope of studies analyzed that our study aimed to do. Regarding bibliometrics in the healthcare literature, Royle et al. conducted a bibliometric analysis of SRs and further analysis to determine potential predictors of the citation impact; and Rosas et al. used bibliometric analysis and network visualization to evaluate the research and impact of National Institute of Allergy and Infectious Diseases (NIAID) publications [12, 13]. Neither of these studies looked at librarian co-authored SRs, specifically. Furthermore, Rosas et al. examined the uptake of the NIAID publications in research reviews and syntheses [13]. In our research, we also went beyond citation analysis counts to examine what types of publications the SR research we studied were cited in and similarly we conducted visual mapping analysis as part of the bibliometric methods. Given the limited published research on the scholarly contributions and impact of librarians’ work related to their collaborative role in the production of SRs, we sought to demonstrate the impact of librarian co-authored SRs and to identify characteristics of librarian co-authored SRs. In this study, we went beyond conducting citation analysis to examine where research was cited to shed light on the diffusion of SR research. This type of content analysis along with citation analysis provides us with a more informed view about the impact of librarian co-authored SRs.

METHODS

RESULTS

The WOS search for librarian co-authored SRs yielded an initial 1,817 records (Figure 1). Of these, 1,711 records were selected as SRs co-authored by librarians according to our inclusion and exclusion criteria.

• Download figure
• Open in new tab

Figure 1

Flow chart of the retrieval and selection of records.

Characteristics of librarian co-authored SRs

In VOS Viewer, of the 33,100 terms identified in the 1,711 SRs, 350 met the frequency threshold, and 210 were sufficiently relevant to be selected for visualization. The density visualization map in Figure 2 depicts the topic areas of the SRs. The most common types of SRs appeared to be for interventions and diagnostic test accuracy, and the two most represented disease categories were “cancer” and “cardiovascular disease”. The overlay visualization map in Figure 3 depicts the normalized citation impact of the SRs. Normalized citation impact corrects for the age of a document defined by VOSviewer as follows: the “normalized number of citations of a document equals the number of citations of the document divided by the average number of citations of all documents published in the same year and included in the data that is provided to VOSviewer” [20]. Our overlay visualization map illustrating citation impact shows that SR topics such as cardiovascular disease, obesity, diabetes, and training are particularly impactful.

• Download figure
• Open in new tab

Figure 2

Density visualization map of librarian co-authored SRs. Term coloring is proportional to the number of term occurrences, with yellow representing the highest frequency of occurrences and blue representing lower frequencies. Font size is also proportional to the number of term occurrences, with larger sizes indicating higher frequencies.

• Download figure
• Open in new tab

Figure 3

Overlay visualization map of librarian co-authored SRs. Circle size reflects the number of term occurrences across records. Lines between circles indicate term co-occurrence across records. Circle color reflects the average normalized citation impact of all records in which the term occurs in the title or abstract. Blue indicates lower citation impact, and yellow indicates higher citation impact.

The number of times librarian co-authored SRs were cited ranged from 0 to 1,012, with an average of 26 and a median of 10. Of the 1,711 SRs, 187 (10.9%) were uncited, which comprised of 65 (3.8%) of the WOS document types “article” or “review”; 120 (7.0%) of the WOS document type “meeting abstract”; and 2 (0.1%) of the WOS document type “proceedings paper”. Also, 41.7% of the uncited SRs were published in 2017. The number of authors on the SRs ranged from 2 to 39, with an average of 7 authors. Additional characteristics of the SRs are outlined in Table 1.

View this table:

• View inline
• View popup
• Download powerpoint

Table 1

Characteristics of the librarian co-authored SRs.

The number of SRs published over time from 1996-2017 is shown in Figure 4. The temporal trend demonstrates the start of an increasing number of librarian co-authored SRs in the year 2006 with a steeper increase beginning in the year 2011.

• Download figure
• Open in new tab

Figure 4

Number of librarian co-authored SRs published over time.

DISCUSSION

The rise in the number of SRs co-authored by librarians over time is consistent with professional practice and the evolving nature of services provided by health sciences libraries over the last decade [21]. Our findings shed light on the extent and nature of the scholarly contributions of librarian collaborators on SRs and constitute a benchmark for future research.

Our analysis revealed that the rate of librarian co-authored SRs appeared to accelerate around 2011. This coincides with the year the IOM SR standards were released, which includes a standard to work with a librarian (Standard 3.1) [1]. The publication of these standards may have positively impacted the number of SR service requests received by librarians, leading to an increased publication rate of librarian co-authored SRs. A related reason for this increased publication rate could be that librarians began advocating more aggressively for authorship. Supporting resources such as the IOM standards, studies showing that librarian involvement improves the quality of SRs [6-8], and librarian authored publications in leading health sciences journals advocating for librarians as members of SR teams [22] may have all played a role in facilitating librarian co-authorship. In tandem with librarian advocacy, increased researcher adherence to authorship guidelines [23] may also have contributed to the increased number of librarian co-authored SRs.

Co-authorship demonstrates value in expertise and the resulting collaboration, and partnerships with research groups. It is encouraging to see that librarians are increasingly being offered co-authorship for their contributions. This trend is particularly encouraging in light of other studies showing that librarians involved in SRs are often not included as authors, or only included in article acknowledgments, despite being deeply involved and making significant contributions to the SRs [8].

Our density visualization map shows that many librarian co-authored SRs addressed diagnostic test accuracy, as well as interventions, which are traditionally the most common clinical foci for SRs. This is consistent with the history of the Cochrane Collaboration, which originally developed only SRs on interventions but which began to develop SRs of diagnostic test accuracy in 2007 [24]. In addition, this map indicates that cancer and cardiovascular disease are the two disease categories most highly represented by librarian co-authored SRs. This is consistent with data from National Vital Statistics Reports and Statistics Canada indicating that cardiovascular-and cancer-related diseases are the top two leading causes of death in the United States and Canada [25, 26].

We found that librarian co-authored SRs were cited an average of 26.4 times, consistent with previous studies showing high citation rates for SRs and meta-analyses [27, 28]. This is also consistent with the mean citation rate of 26.5 over 4 years reported by Royle et al. in their bibliometric analysis of SRs [12]. We found that 10.9% of librarian co-authored SRs were uncited, which included the WOS document types “article,” “review,” “meeting abstract,” and “proceedings paper.” This percentage is higher than those reported by Royle et al. (1.6%) [12] and Van Noorden (4%) [29] and is likely a result of including meeting abstracts in our study. In our sample, the uncited rate for articles and reviews, excluding meeting abstracts and proceedings papers, was 3.8%, which is closer to the aforementioned studies’ uncited literature rates [12, 29]. There are many reasons for articles not being cited, not all of them negative [29]. These include the use of a single source (e.g., Web of Science Core Collection) for citation data and insufficient time for the dissemination of research. Of the 10.9% of uncited SRs, 41.7% were from 2017, indicating that insufficient time may have passed for these papers to receive citations.

Librarian co-authored SRs had an average of 7 authors, similar to that reported by Royle et al. for their top 50 most cited papers (6 authors) [12]. The top countries in which librarian co-authored SRs were most often published were the United States, Canada, Netherlands, England, China, and Australia. These results match the leading countries publishing SRs identified by Fontelo et al. in their PubMed analysis of evidence-based publication trends across countries which were, in rank order, the United States, United Kingdom, China, Australia, Canada, and Netherlands [30].

Not surprisingly, librarian co-authored SRs were most often published in the Cochrane Database of Systematic Reviews. The Cochrane Collaboration is a SR-producing entity, and Trial Search Coordinators/Information Specialists are routinely included as SR team members. One specialty journal, the Journal of Clinical Endocrinology & Metabolism, was among the top five sources of librarian co-authored SRs, which aligns with our density visualization map showing diabetes as a prominent research area. In our study, the leading journals publishing the top 10% most cited SRs included: BMJ, Cochrane Database of Systematic Reviews, JAMA, and the Journal of Clinical Endocrinology & Metabolism. The presence of the Journal of Clinical Endocrinology & Metabolism in this list is consistent with the Overlay Visualization Map (Figure 3), which shows “diabetes” as one of the higher citation impact areas among the research areas represented on the map.

In addition to analyzing librarian co-authored SRs themselves, we also conducted content analysis of a large number (>28k) of citing articles to those reviews. Our aim was to provide a more narrative view of impact by demonstrating the nature of librarian co-authored SRs’ diffusion within the scholarly literature. In the literature, The Becker Model provides a framework for research impact assessment that goes beyond traditional citation analysis to quantitatively demonstrate impact as a complement to citation impact [10]. Elements of the Becker model for demonstrating impact include, for example, evidence of knowledge transfer by way of a research study being cited in a meta-analysis, consensus development conference, patent, etc. and evidence of clinical implementation (defined as “the application or adoption of research outputs in clinical practice” [10]) such as a research study cited in guidelines, continuing education materials, or private and public health care benefit plans, etc.

In our study, 412 citing articles were characterized as Practice Guidelines and Guidelines. This demonstrates that librarian co-authored SRs were disseminated into guidelines with the potential for clinical implementation. While we did not examine the nature of how the librarian co-authored SRs were used in these citing sources, it is possible they were cited as part of the body of evidence to support a recommendation in the guidelines thus serving as surrogate for clinical implementation, as defined by the Becker Model. The diffusion of the SRs into the various study types suggests that the librarian co-authored SRs in this study were used to influence future research studies. Our study found that citing sources to librarian co-authored SRs were characterized as 2,285 Randomized Controlled Trails, followed by 1,955 Meta-analyses, 1,830 Comparative Studies, and 1,339 Multicenter Studies, to name a few of the top “Study Characteristics” Publication Types. These examples of our results along with the detailed list in Table 2, additionally demonstrate the impact of librarian co-authored SRs in a more descriptive manner which complements the citation analysis we conducted.

Librarian SR services provide opportunity for collaboration, deeper engagement, and education. Despite many challenges related to librarians providing SR support, various strategies can be employed by library administrators to incorporate SR services and allay concerns. The results of the present study could be useful for health sciences library departments and for institutions to demonstrate the value of librarian services for SRs by highlighting librarian contributions to research and practice in other disciplines. Librarians on tenure/tenure-equivalent tracks may face challenges in convincing their tenure and promotion committees that non-LIS SR scholarship is equivalent to LIS scholarship. However, our study provides evidence of the impact of librarian scholarship on the knowledgebase of non-LIS disciplines, which requires expertise that is uniquely within the skill set of librarians. Thus, our findings may be useful in expanding the definition of scholarship for academic librarians.

CONCLUSION

Our characterization of the scholarly outputs and impact of librarian co-authored SRs emphasizes the value of the unique SR expertise of librarians. SRs are research that can have significant impact in many areas such as policy, patient care, and further research [31]. That librarians are increasingly providing SR services, despite the challenges involved, strengthens the vital role that librarians can play in research overall and could help expand the definition of scholarship within the LIS profession. This study may serve as a benchmark for administrative decision making about library SR services and for future research in this area.

DECLARATIONS