#ChronicPain: Automatic establishment of a chronic pain cohort from social media using machine learning for studying opioid-alternative therapies

Data collection

We collected data through the Twitter academic streaming application programming interface (API).²⁵ The API enables the collection of a sample of data in real-time based on keywords. We used the hashtag ‘#chronicpain’ and the phrase ‘chronic pain’. For the phrases, we first collected tweets using the terms individually, and then applied a regular expression pattern to detect the exact phrase. We excluded retweets and tweets that did not meet the pattern.

Annotation

We randomly selected 5000 posts from our entire collected set for manual annotation. The objective of the annotation was to determine if a post represented a personal experience or not.

Thus, the annotation was a binary labeling task indicating if a tweet represented a self-report (label: Y) or something other than a self-report (label: N). Label N represented tweets that were general information about chronic pain or those that could be about raising awareness or promotion. We conducted the annotation in three batches. In the first batch, we annotated a small sample of tweets (n=200) without an explicit guideline and we discussed the disagreements from this round of annotation to create a simple annotation guideline. We annotated a larger sample (n=1300; batch 2) following this guideline, and we further discussed disagreements and resolved them. We made minor updates to the annotation guideline based on these discussions. Since the disagreements were relatively low after the second round of annotations, we annotated the remaining tweets (batch 3).

Randomly selected subsets of the tweets were annotated by all three annotators. We used these overlapping annotations to compute inter-annotator agreements (IAA). Since the interpretation of the tweets can be subjective, it was important to capture the extent of human agreement as it represents a potential ceiling for any machine learning algorithm used to automate the process. We used the Cohen’s kappa measure to measure IAA. All disagreements were resolved by the last/senior author of the article (AS). The annotation guideline is available as supplementary material. Table 1 presents examples of tweets and their labels.

Supervised classification

We divided the annotated data into 3 sets—60% for training, 20% for validation, and 20% for testing/evaluation. Transformer-based approaches that use large pre-trained language models such as BERT are currently state-of-the-art for text classification, both in and outside of the medical domain. We, therefore, fine-tuned and evaluated several transformer-based classifiers. The following is an outline of the classifiers we used:

1. RoBERTa: Transformer-based model popular for its training on big batches and long sequences.²⁶

2. SciBERT: Model is trained on a large corpus of scientific data and with the same model structure as BERT.²⁷

3. BioClinicalBERT: This was modelled from BioBERT (BioBERT-Base v1.0 + PubMed 200K + PMC 270K) and was trained on MIMIC III notes.²⁸

4. BERTweet: Large scale pre-trained model for English tweets.²⁹

5. BioBERT: This is specifically trained for bio-medical text and is widely used for biomedical text mining.³⁰

We compared the performances of the classifiers based on the F₁ score for the Y class. The F₁ score is the harmonic mean of precision and recall. We focused our evaluation on the Y class since that is our class of interest. We also report overall classifier accuracy, but this metric is primarily driven by the majority (N) class. We computed the 95% confidence intervals for the Y class F₁ scores using the bootstrap resampling technique.³¹ In this method, the F₁ score is calculated 1000 times for randomly selected (with replacement) training sets. Out of these, 25^th and 975^th values are considered as lower bound and upper bound values at 95% confidence interval.

Post-classification analyses

We used the best-performing classifier from the previous subsection to classify unlabeled posts collected from January 2021 to March 2022. Then, for a sample of subscribers whose posts were classified to be self-reports of chronic pain (Y), we collected all their past posts available via the API. The Twitter API allows the collection of approximately 3200 past public posts by each subscriber. Thus, for many subscribers—members of our chronic pain cohort—this enabled us to obtain multiple years of posts. We were only able to collect this for a subset of the subscribers due to the API limitation of 10 million post collection per month. Finally, we semi-automatically analyzed samples of the cohort posts to assess the presence and type of chronic pain-related information. The semi-automatic analysis involved using a lexicon-based fuzzy matching approach³² to detect potential treatments mentioned, including pharmaceuticals such as opioid pain relievers (e.g., oxycodone, hydrocodone) and non-standard ones such as behavioral therapies. We performed a number of analyses involving the posts that mentioned specific therapies and also the past posts collected from the subscribers. We outline these below.

Data and annotation

Of 5000 posts selected for annotation, two were excluded due to issues with text encoding, leading to the annotation of 4998 posts in total. 550 posts were at least double annotated. Pair-wise inter-annotator agreement (IAA) among 3 annotators was k=0.82 (Cohen’s kappa³³), which can be considered to be almost-perfect agreement.³⁴ 719 (14.4%) were self-reports of chronic pain while the rest (85.6%) were not. The class distribution was thus slightly imbalanced. This was unsurprising since our past studies have found similar imbalances.^35,36

Classification performance

Table 2 presents the results of our automatic classification experiments. The table shows the overall accuracy of the classifiers, the F1 scores for the positive class and the 95% confidence intervals. The RoBERTa classifier achieved the highest F1 score among all the classifiers. For the best-performing classifier, we conducted ablation experiments using 10% subsets of the original training data. Figure 2 shows, and as is typically seen from such experiments, that the performance of the classifier increases approximately logarithmically as more training data is added. The best-fit logarithmic trendline suggests that F1 scores of 0.87 would require us to increase the annotated dataset by three times. Accuracy scores are relatively unchanged with training data size since this value is primarily driven by the majority negative class.

• Download figure
• Open in new tab

Figure 2:

Classifier F₁ scores or the positive (Y) class at different training set sizes and projected scores based on a logarithmic trendline for the RoBERTa model.

Post classification analyses

The application of our classifier on the unlabeled posts resulted in the identification of 41,262 self-reports of chronic pain from 22,795 subscribers. Collecting their past data resulted in 3,461,619 posts. We performed the post classification analyses on these posts, and we described the findings in the following subsections.

Sentiment Analysis

Sentiment Analysis is performed to estimate the polarity distribution of feelings associated with the chronic pain experienced by the cohort members who also mention specific treatment keywords. 600 posts were manually annotated in total with an IAA of 0.88 (Cohen’s kappa). Figure 3 presents the sentiment distributions for different types of therapies. The figure shows that most of the users who mentioned “meditation”, “Guided imagery”, and “Tai Chi” in their posts, also showed more positive emotions, while users who mentioned “hydrocodone” and “tramadol” show negative emotion compared to others.

• Download figure
• Open in new tab

Figure 3:

Distribution of manually annotated sentiments (positive, negative and neutral) for different chronic pain therapies, as discussed by the cohort members.

Related work

Several past studies have demonstrated the utility of social media data for chronic pain-related research. Many early studies focused on studying the effects of social media for chronic disease management, rather than the use of it to study chronic pain specifically.^37,38 Through a global online survey conducted partially via social media, Merolli et al.³⁸ concluded that areas of research that warranted attention included the ability to (i) filter information and guide people to pertinent information, (ii) connect sufferers of chronic pain (i.e., cohort members), and (iii) explore relationships between the therapeutic affordances of social media and health outcomes. In a separate, questionnaire-based study, Ressler et al., found that posting about chronic pain online may decrease the sense of isolation and increase a sense of purpose.³⁹ Similar findings were reported by Tsai et al. in a more recent study.⁴⁰ Gonzalez-Polledo et al. studied social media (Tumblr and Flickr) and chronic pain from the perspective of digital anthropology, characterized chronic pain narratives over these platforms, and presented a typology of chronic pain expressions.⁴⁰

Works most closely related to ours are those by Sendra and Farre and Mullins et al. ^39,41. In the former study, the authors analyzed a small sample of data (n=350) and concluded that social media is changing the way patients live with their chronic pain and care providers could benefit from paying attention to self-reported information by these individuals. Mullins et al. collected a small sample of data from Twitter and performed NLP-driven analysis of discussion topics, sentiments and advice provided. The study concluded that the pain-related discussions on the platform can enrich our understanding of the chronic pain experience. Our study builds on these and proposes a sustainable infrastructure, driven by NLP and machine learning, that can automatically curate knowledge from a chronic pain cohort over time. No prior work has proposed as thorough an approach. Crucially, our study goes beyond relying on post-level information by focusing on building a chronic pain cohort and collecting data solely from this cohort. This ensures that the texts included in the analyses are more likely to be related to personal experiences of chronic pain rather than general discussions about the topic. Our past efforts to build targeted cohorts have led to the creation of massive, multi-year datasets involving hundreds of thousands of cohort members (e.g., people who use substances⁴² and pregnant people²³).

Utility of social media

Our findings illustrate that social media is a rich source of information for studying chronic pain. Chronic pain-related discussions are common on Twitter, but this resource has thus far been underutilized in both epidemiological and interventional research. A prime reason for this is the difficulty of handling such massive data and processing complex language. Our proposed machine learning and NLP methods have the potential of overcoming the barriers leading to the underutilization of such an effective source of information. Successful deployment of this pipeline is likely to increase the utility of social media in chronic pain research. Our study also found that the Twitter chronic pain cohort discusses a variety of pain-related topics including but not limited to therapies, addiction, mental health, and support. Conducting more detailed analyses of texts associated with these topics may lead to important breakthroughs in chronic pain research. For example, analyses of the opioid alternatives discussed by subscribers can generate hypotheses about their efficacies, which can be tested in future research.

Studying a social media-based cohort may also reveal information about factors impacting the sufferers of chronic pain that may not be available from any other source. These may include, for example, their social support (e.g., number of followers, number of interactions with a given post) and its influence (if any) on the quality of life of chronic pain patients. There is also the potential of going beyond epidemiological studies over social media and conducting interventional studies.

Error analysis

We conducted manual analysis on the misclassified tweets to identify patterns of errors. We focused on both false positives (classified erroneously as self-report) false negatives (classified as not self-report). Under the former category, we found that most posts were misclassified when users intended to share the experiences of the people around them, not their personal experiences. One such post is, “today is mother’s day and it’s not an easy day. although my mom is still biologically alive, it feels like i’ve lost her to her chronic pain years ago. they say to treasure every moment but it’s not always easy”. Under the latter category where model fails to classify as self-reports, we found that one common reason for misclassification was due to the expressions being implicit, making it difficult for the machine learning algorithms to capture their true meanings. One such example is, “music therapy for <hashtag> chronicpain. thinking of ways to make movement enjoyable might involve <hashtag> music <hashtag> dance. sometimes I’ll just chuck on some of my favorites and dance, mindfully. i always feel a million bucks after as well as more connected with my body.” Another reason for misclassification in this category was determined to be incomplete information/partial information. For instance, “<hashtag> finally starting to ease, been trying to focus my mind on other things to get my brain to turn down the chronic <hashtag> pain it’s at <number> out of <number>”. While for a human reviewer the post has enough context, it could not be picked up by the machine learning classifier.