An analysis of COVID-19 article dissemination by Twitter compared to citation rates

Introduction

In December 2019, a collection of patients in Wuhan, China presented to hospital with respiratory symptoms and pneumonia. These patients were diagnosed with a novel coronavirus of the Orthocoronavirinae subfamily (SARS-CoV-2) [1,2]. SARS-CoV-2 is readily capable of transmission between human hosts, and is associated with a range of clinical severity from mild respiratory symptoms to severe hypoxemic respiratory failure and death [3,4]. Although originating in China, SARS-CoV-2 has resulted in over 1,000,000 cases of infection in 181 countries worldwide, and has claimed the lives of over 60,000 (as of April 5th) [5]. Whereas SARS-CoV-2 refers to the virus itself, its associated syndrome is referred to as coronavirus disease (COVID-19). COVID-19 has drastically impacted the lives of millions worldwide [5].

The global impact of COVID-19 has led to a surge of related research with over 1328 related articles indexed on PubMed as of March 20^th, 2020 [6,7]. With the amount of research being performed, streamlined and timely methods for disseminating information to frontline clinicians has become central to providing evidence informed care. To facilitate this, clinicians have increasingly turned to social media platforms including Twitter, to share, discuss, and respond to peer reviewed studies, sometimes even before the formal peer review process [8].

The use of social media to disseminate knowledge among the medical community is not new, with Twitter serving as an important social media platform for scholarly communication [9,10]. Whereas traditional citation-based metrics can track the amount of times a publication is cited by other researchers, they do not consider the dissemination of the publication online across social media platforms. The online database Altmetric is a tool that tracks social media sharing, and then collects and displays metrics from sources including Twitter and Facebook [11,12]. The Altmetric database has previously been interrogated to explore how variables such as study discipline, number of authors, and country of origin influence the number of tweets pertaining to a given study [13,14] and as a tool to examine research dissemination. As there is a lack of correlation between altmetrics and citation rates, it has been suggested that twitter-based metrics reflect different, yet complementary information to traditionally used citation rates [15,16]. Therefore, altmetrics may serve as another tool to capture the wider impact of a given study [17].

Determining the impact and dissemination of research is especially important for the COVID-19 pandemic, as healthcare workers rely on current and impactful research to inform practice. Traditional citation metrics may fail to portray dissemination accurately, as there is a significant time-lag due to the publication process. Altmetrics do not have the same limitation, as they do not rely on the publication process, suggesting that they may provide an effective marker of research dissemination during the COVID-19 pandemic.

The objective of this study is to assess the impact of COVID-19 research articles as measured by Twitter dissemination, compared to a traditional citation-based metric. This study will also seek to determine which characteristics are associated with an article’s Twitter dissemination. This research serves as an important first step in understanding how best to rapidly disseminate new and important research during the COVID-19 pandemic.

Methods

Research ethics board approval for this type of research is waived at our institution. The pre-specified study protocol is available on the Open Science Framework [18].

Results

Discussion

The objective of this study was to compare tweet rates to citation rates for articles about COVID-19 that were extracted from the LitCovid database, and to identify article characteristics that correlate with the number of tweets an article receives. The tweet rates were found to be significantly higher that the citations rates. The number of tweets an article received was positively correlated with 2018 journal impact factor and topic (Diagnosis, Transmission, Treatment, and Epidemic Forecasting relative to Case Report) and was negatively correlated with open access journals (relative to subscription-based journals) and the continent of the corresponding author (Africa, Australia, and South America, relative to Asia).

A significant difference was found between the tweet and citation rate. This result is in accordance with previously described literature from Haustein et. al. which found a lack of association between tweets [15] and citation rates for journal articles in the biomedical sciences and from Costas et. al. showing a weak correlation between tweets and citation rates from different disciplines indexed from Web of Science [16]. Thus, altmetrics provide different and supplementary information to previously used citation based metrics for article dissemination analysis. Further, the lack of association demonstrates that during the COVID-19 pandemic, citation rates are not optimal to quantify article dissemination due to their associated time-lag. As altmetrics do not suffer from this limitation, they are better equipped to quantify dissemination of articles and provide an objective metric for healthcare workers or researchers to find impactful research, as evidenced by our finding that tweet rates were significantly higher than citation.

A journal’s 2018 impact factor was found to be positively associated with tweet rate. This is fitting as journals with higher impact factors are often exposed to larger audiences, increasing the likelihood that articles published in these journals would be tweeted about.

The continent of the corresponding author was found to impact the number of tweets with South America having the largest negative association, followed by Africa, then Australia, relative to articles published from Asia. An analysis of worldwide Twitter usage using tweet geolocation, found that Twitter has a strong prevalence in North America and Europe compared to other continents with African and South American countries having the lowest involvement [23]. This could explain the lower number of tweets associated with Africa and South America. There was also a lower research output, as reflected by the relatively low number of articles from Africa, South America and Australia which could account for the lower number of tweets received. Similarly, Asia had the highest number of articles, followed by Europe and North America which could account for their higher number of tweets. The number of tweets also followed the progression of COVID-19 across continents, with early cases surging in Asia, followed by Europe, then North America. Therefore, when looking at articles on social media platforms it is important to acknowledge that the articles presented may be biased towards countries with higher social media use or in general more research on the topic.

Epidemic forecasting, Transmission, and Diagnosis had significantly higher rates of tweets (compared to General Info, Mechanism, and Prevention), relative to Case Reports. This result could convey an increased interest in these topics for the COVID-19 pandemic, resulting in increased tweet rates. Interestingly, these topics had a relatively low amount of articles, relative to the other topics, suggesting that general perceived interest in a topic and not absolute number of publications could influence online dissemination.

Surprisingly, journals that were open access had a negative association of tweets compared to subscription-based journals. This finding was supported by the statistically significant interaction between 2018 impact factor and open access status in the regression model. Since journals with higher impact factor were more likely to be subscription-based, articles published in subscription-based journals had an increased likelihood of Twitter dissemination. It is important to note that many prominent subscription-based journals have made COVID-19 related research open access in an effort to improve dissemination to readers. This could have influenced the number of tweets made about articles from subscription-based journals.

There are a few limitations to this study. There may be residual confounding variables that were not measured in this analysis. The authors acknowledge that self-promotion (eg: authors tweeting out their own article) may influence the number of tweets an article receives; however, the extent is difficult to quantify. Further, this study did not quantify the sentiment of the tweets and citations, which could provide useful information. However, it has been found that 94.8% of tweets linking to scientific papers and 75.85% of citations in clinical trial papers were neutral [24,25]. Thus, this suggests that the majority of citations and tweets within this study were neutral in sentiment. Further research should be conducted in this area to clarify this. Also, there is a loss in precision due to the use of categorical variables such as the use of continents instead of countries and use of impact factor median split instead of impact factor. This was used in order to get an appropriate fit of the model, but loss of precision is a consequence. Similarly, the relatively high proportion of missing values (20%) could have impacted the robustness of the model fit to the data, despite the imputation performed to address this concern.

Research surrounding the COVID-19 pandemic is being performed at an ever increasing rate. This study represents a snapshot of the early research into the pandemic, repeating this analysis as more research becomes available will be important to confirm findings. In addition, sentiment analysis (categorizing the content of the tweets as positive, negative, or neutral) may be valuable to gain further insight into the content of tweets shared online for a given article. A tool known as SentiStrength has shown potential in this area by categorizing sentiment strengths of social media text, and may prove useful in this type of analysis [26].

LitCovid provides a centralized curated hub for readers to find research related to COVID-19. However, not all periods of rapidly expanding research has this same easily, accessible resource. Thus, the results of this study suggest that the amount of tweets, or online dissemination in general by means of altmetrics, can be used to locate highly disseminated articles in other areas of rapidly evolving research, such as the vaping-associated lung disease which sporadically appeared in the summer of 2019 [27], where citation based metrics may fail to capture article dissemination.

Conclusion

This study collected data from LitCovid on articles related to COVID-19 and showed that these articles had significantly higher tweets rates compared to citation rates. This study also identified that the number of tweets an article received was positively correlated with 2018 journal impact factor and topic (Diagnosis, Transmission, Treatment, and Epidemic Forecasting relative to Case Report) and was negatively correlated with open access journals (relative to subscription-based journals) and the continent of the corresponding author (Africa, Australia, and South America, relative to Asia). This analysis expands the current literature on the use of altmetrics in categorizing research dissemination and its practical use in periods of rapidly expanding research, such as the COVID-19 pandemic, to localize highly disseminated articles.