Physical activity, sedentary behaviour, and sleep on Twitter: A labelled dataset for public health research

Collection and Preparation of the Dataset

We collected the data of this study from Twitter using the Twitter livestream application programming interface (API) between 28^th November 2018 to 19^th June 2020. The entire dataset (i.e. 1,902,980,841 tweets) was filtered for Canadian tweets potentially relevant to PASS. 103,911 tweets were selected from 22,729,110 collected Canadian tweets using keywords and regular expressions related to PASS categories. Each of these 103,911 tweets was labelled by three AMT workers, from which 98,722 tweets received three valid labels (i.e. multi/empty-labels are invalid and were rejected), with almost half of them related to physical activity. For the Canadian dataset, 610 unique workers participated in our data labelling tasks and completed 103,911 HITs, from which 5,189 HITs were removed as they did not receive three valid answers. The majority of these workers (87%, 530) completed less than 100 HITs each, among which 164 completed only one HIT each. Among the workers who completed more than 5,000 HITs, one worker completed 21,801 HITs, and three workers completed between 5,000 and 10,000 HITs.

In addition to the Canadian tweets, we filtered a random subset of the dataset for tweets that originated in the UK, USA, and Australia. This dataset spans the same data collection period as the Canadian dataset and contains 70,239 labels collected for 23,413 tweets (i.e. three labels per tweet). Adding the data from these countries will provide an important epidemiological diversity that can be utilized for implementing comparative studies and evaluating the generalizability of the PASS surveillance models trained on the Canadian dataset.

Labelling Process

We implemented a pipeline to create the crowdsourcing tasks, referred to as HITs by AMT, post them on AMT, collect the labels through a quality check process, approve/reject the HITs, and store the results. To minimize noisy and low-quality data, we added a qualification requirement to our tasks and granted the labelling access to workers who had demonstrated a high degree of success in performing a wide range of HITs across AMT (i.e. master qualification). In addition to this, we added a simple qualification question to each HIT to detect spammers or irresponsible workers. Each HIT contained four questions, including the qualification question and was assigned to three workers. Through different iterations of data labelling, workers were paid from US$0.03 to US$0.05 after completing each HIT. We collected the labels for the 122,135 tweets used in this study through different iterations, from April 2019 to November 2020. We regularly checked the quality of submitted tasks to detect low-quality workers during each iteration and revoke their access to our tasks. Before the formal initiation of the process, we pilot tested the design, response time, and complexity of the HITs in two iterations and revised the workflow accordingly. To label the dataset provided in this paper, we utilized AMT as a crowdsourcing service and did not collect any personally identifiable information from the workers (participants) during the data labelling task. The experiments were carried out in accordance with relevant guidelines and the University of Calgary Conjoint Faculties Research Ethics Board’s (CFREB) regulations. We implemented the entire workflow in python and used Boto3 python SDK to connect to and work with AMT.

Time Adjustment

Twitter API returns the date and time that a tweet is published in UTC. To adjust this timezone base on each tweet’s location, we utilized the bounding box of coordinates, which enabled spatial mapping to tweets’ respective city locations and used timezonefinder in python. Given that daytime, month, and weekday can be influential factors in twitting about each of the PASS categories, and to better utilize the datetime data (%a %b %d %H:%M:%S %Y), we extracted a: weekday, b: month, and H: hour fields and stored them as separate features. Figure 1a shows the temporal distribution of tweets for each of the PASS categories in the Canadian dataset. Moreover, the stacked area charts presented in Figures 1b-1d detail the frequency of tweets for each of PASS categories for the top ten Canadian cities.

• Download figure
• Open in new tab

Figure 1.

The temporal distribution of the tweets for the Canadian dataset. To make fair comparisons, we used the data from December 1^st 2018 to May 31^st 2020 for these visualizations and removed the data collected during the last two days of November 2018 and the first two weeks of June 2020.

Location Inference

The geospatial metadata provided by the Twitter API is derived from three main sources, including the (1) geotagged location, (2) profile location, and (3) mentioned location in the tweet text. Geotagged location can be defined by exact location (i.e. device location) at the time of tweeting, by assigned twitter place (i.e. at the neighbourhood level), or both. While the exact location field provides the highest level of precision, it is not a default setting, and only a small portion of users share their exact latitude and longitude (e.g. only 1-2% of tweets are geo-tagged²⁵). Thus, to infer the location of each tweet in LPHEADA, we proposed and developed a five-step process that utilizes tweet-neighbourhood location (i.e. place.name and place.full_name), profile information (i.e. profile description and location), and tweet text to map twitter’s geospatial metadata for each tweet t_i to physical locations in the form of ⟨c_i, p_i | s_i⟩, where c denotes the city of a tweet and p | s represent its corresponding province or state, respectively (Figure 2). To demo the proposed process, we used the Canadian geographical names dataset (i.e. location dictionary), provided by the Geographical Names Board of Canada. Each geographical name provided by this dataset is mapped to a province and is classified to a geographic area such as city, town, village, lake, administrative sectors, or recreational centers. As illustrated in Figure 2, for each t_i, we first use a simple search function to map the ‘place.name’ field associated with each tweet to its corresponding c_i in the location dictionary (LD). If found, the corresponding province field p_i was defined using equation 1:

• Download figure
• Open in new tab

Figure 2.

Five-step location inference process. The location dictionary is an external regional geographical metadata used to extract the exact locations of the tweets. The area field in this process refers to regions at different scales such as city, regional, municipality, town, township municipality, municipal district, dispersed rural community, village, or country.

Where place.full_name_ci [1] denotes the the second component of the field when the first component is c_i (e.g. Ontario in the illustrative example of Figure 2). This will detect geographical areas with the same names but in different provinces (e.g. Leduc is a town/city in both Alberta and Quebec).

Steps 3-5 of the process deal with unstructured text objects that can come from all three sources of geospatial information. To extract the location information from these fields, we developed a string matching function to detect the longest common substring between the unstructured text of the dataset and the area field of the location dictionary (e.g. first time boating in Lake Louise #AB is mapped to ⟨Lake Louise, Alberta⟩ instead of to ⟨Louise, Quebec⟩). To manage the complexity of information extraction from the unstructured text, we only used a subset of areas listed in LD, with high population density (e.g. city, municipality, town, village, and country). Thus, we excluded areas classified as lake, mountain, river, bridge, or river.

Demographic attribute inference

The demographics variable of age and gender and the information about the source of each tweet (e.g. organization vs. real users) were not available within the dataset collected from Twitter. We estimated these variables for each tweet using the M3inference package in python²⁶, which uses a multimodal deep neural architecture for joint classification of age (binned into four groups: ≤18, 19-29, 30-39, ≥40), gender, and information-source of social media data. This approach uses four sources of information, including username, screen name, biography, and profile image of public profiles to develop two separate pipelines for processing a profile image and each of the three text sources of information. The models provided in this package are trained on 14.53M, 2.61M, and 23.86M profiles for each of the gender, age, and is-organisation categories, respectively.

Labels

Let L denote the set of j unique class label, t present the tweet text, and w_k present the k^th worker who labeled the tweet, where k ∈ {1, 2, 3 }. Each l _j∈ L is defined based on two conditions: whether or not the tweet is self-reported (c₁ ∈ {0, 1 }); and whether or not the tweet reports a recent PASS experience (c₂∈ {0, 1 }). For each PASS category across the four countries, the dataset contains the following two subsets of labels for each tweet:

User’s demographics data

In the demographic dataset, each tweet t is mapped to quadruple 𝒟 = ⟨tweetID, a, g, o ⟩, where g ∈ {male, female} presents the gender of the user who posted the tweet, a ∈ {≤ 18, 19 − 29, 30 − 39, 40≤} represents their age range, and o ∈ {0,1} shows the source of the tweet (i.e. o = 1 if the tweet posted by an organization, o = 0 otherwise). Figure 4 shows the demographic distribution of the Canadian dataset based on gender and age-range of the unique users associated with tweet IDs. We can see that at least 75% of female users in this dataset are inferred to be younger than 40, while this number for the male users is 60%. Also, the most populated age category for female users across all PASS categories is 19-29, while this range for male users is ≥40. Excluding the (female, sleep quality) category, the age range ≤18, regardless of the user’s sex, is the least populated category across all PASS categories.

• Download figure
• Open in new tab

Figure 4. Demographic information of 42,603 unique users associated with the tweets originated in Canada.

Inferred location data

Each row of the location dataset is presented in the form of 𝒜 = ⟨tweetID, c, p ⟩, where c and p denote the city and province/state associated with each tweet, respectively. Using the TweetID parameter, the location data can be mapped to other datasets, including labels, user’s demographics, time, and twitter metadata. Each of the c and p variables are inferred based on the raw variables in Twitter metadata, including text, place objects, and user’s profile description (Figures 2 and 3). For example, Figure 5 shows the distribution of labelled tweets for each PASS category across Canadian provinces (i.e. p). For the top five provinces, the overall size of the dataset is directly proportional to the population size of each province. However, as only English tweets from Twitter users are included in the dataset, LPHEADA represents only English-speaking Quebecor’s and Francophone Quebecor’s tweets in English, placing the province in fourth place. Moreover, with a lower population than BC, Alberta contains more SB and SQ tweets and places in second place (Figure 5).

• Download figure
• Open in new tab

Figure 5. Geo-spatial details of the Canadian dataset

Tweet object

This object comprises of hashtags mentioned in each tweet, emojis, user mentions, urls, and media (e.g. images and videos). For example, Figure 6 illustrates the distribution of the top 10 hashtags per label for each of the PASS categories in the entire dataset. Hashtags are basically keywords or word strings prefixed with the symbol ‘#’ that are used for categorizing and communicating tweets pertaining to the same topics. While the high level of intersection between the hashtags of positive and negative classes in our dataset makes this feature a less discriminating feature for the development of machine learning models (e.g. annotated hashtags in Figure 6), this field can still be utilized by PASS-related advocacy campaigns on Twitter to brand their movement and open up their campaigns to users who need more information about the context²⁷. As tagged tweets are easily archived and accessible, the hashtag field can be effectively leveraged to improve the public’s engagement in digital PHS discussions.

• Download figure
• Open in new tab

Figure 6.

The distribution of top 10 hashtags per label for each of the physical activity, sedentary behaviour, and sleep quality categories. The number at the end of each bar presents the frequency of its corresponding hashtag. The intersections between two class of labels for each PASS category are annotated using filled circles (•). This Figure is based on all data collected from Canada, US, UK, and Australia.

Methods of label agreement

To measure the consistency of labels generated by AMT workers, we calculated label consistency (ℒ C) as the average entropy of the collected labels for each PASS category²⁸. For each tweet t_i ∈ 𝒯 _s, where 𝒯 _s denotes the set of all tweets related to surveillance category s ∈ {physical activity, sleep quality, sedentary behaviour}, n_{i j} defines the number of answers given to the j^th choice (j ∈ {1, 2, 3, 4, 5}), as we have five choices for each tweet). We calculate ℒC_s as: |s| denotes the size of the surveillance category s and as we collect three labels for each tweet, the denominators in the entropy formula receive the constant value of three. ℒC ranges from 0 to 1, and the values close to zero show less consistency between the workers’ input. After calculating the label consistency (ℒC) for each PASS category, we had ℒC > 0.52 for the multi-class labelling, and ℒC > 0.73 for the binary labelling tasks.

To consolidate the collected labels for each tweet, we used the majority voting (MV) technique (Table 1). Defining the estimated label as , and the submitted label by worker w as l_w, the MV approach, for a binary labeling task, assigns 1 to if , and assigns 0, otherwise. The discrepancy between the number of binary and multi-class labels presented in Table 1 caused by the way that MV approach calculates the truth label for each of these categories. In addition to MV, there are models, such as David and Skene (DS)²⁹, Raykar (RY)³⁰, and generative model of labels, abilities, and difficulties (GLAD)³¹, that incorporate the error rate of annotators (workers), task complexity, and context-sensitive features into the inference process and can be utilized to predict truth labels from crowd-labelled data.

Semantic consistency

To validate the semantic consistency of the datasets collected from different countries, we transformed the dataset of each PASS category into a semantic space of low dimensionality using latent semantic analysis (LSA). For the vector presentation of each dataset, to capture high-level semantics of the text, we ran the PASS category of each country through a pre-trained word2vec embedding model. This model contains 300-dimensional vectors of three million words and phrases trained on 100 billion words from a Google News dataset. Then, the resulting 300-dimensional vectors were averaged for each tweet. For each tweet

T composed of words ⟨w₁, w₂, …, w_n⟩, with . defining the embedding of w_i, the embedding of tweet T can be calculated as: We then applied truncated SVD on the new vectorized dataset and only kept the top two dimensions of the dataset containing the most variance (e.g. those directions in vector space of the dataset that contain more information). The scatter plots presented in Figure 7 illustrate our datasets in a two-dimensional latent semantic space. The high level of overlap between the datasets of each PASS category indicates that the data from different countries cover similar semantic space, but the space is scaled differently based on the size of datasets.

• Download figure
• Open in new tab

Figure 7.

Scatter plots of the first two dimensions of latent semantic analysis (LSA) performed on PASS categories and classified based on the geographic source of the data.

Moreover, to further investigate the internal consistency of the datasets presented in this paper, we trained three convolutional neural network (CNN) multiclass classifiers (i.e. one for each PASS category) to classify the tweets into one of Canada, USA, Australia, or UK classes. Given the highly imbalanced distribution of the classes in our dataset due to the highly unequal number of samples from each country, we used average precision metric (AP) to measure the discrimination ability of our predictive models. The poor performance of these classifiers (AP_PA: 37%, AP_SB: 32%, AP_SQ: 31%) in detecting each tweet’s country implies a high level of semantic and syntactic cohesion among the 4 countries in our dataset.

Classification of PASS categories

For the PASS classification experiment, we utilized a standard CNN classifier with one layer of convolution with global max-pooling on top of a word2vec embedding trained on 100 billion words of Google News. The vectors have a dimensionality of 300 and were trained using the continuous bag-of-words architecture³². We used the binary labels of the dataset to train and evaluate the model on each of 12 datasets provided in LPHEADA. Due to the imbalanced distribution of binary labels across all of these datasets (Table 1), in addition to precision, recall, F1, and AUC scores, we used average precision (AP) to measure the weighted mean of precision at different thresholds to make the score robust to heterogeneous and imbalanced class distributions. Like AUC score, AP is a model-wide and threshold-free evaluation metric. However, for imbalance class distributions with the negatives outnumbering the positives, AP is more informative than AUC, as it mainly evaluates the fraction of true positive samples among positive predictions and is more robust to the relationship between false-positive (FP) and false-negative (FN) rates³³. As shown in Table 2, for each of the Canada, USA, and UK datasets, we find a steady increase in the overall performance of the classifier as the size of the dataset increases (i.e. | PA |>| SQ |>| SB |). Interestingly, the UK dataset achieves the highest performance for the PA category than all the other countries.

Linguistic Properties

To understand and validate the linguistic properties of each dataset, we measured and visually compared the following metrics for each PASS category grouped by countries: (i) sentence count, (2) grammar score (i.e. number of grammar errors), (3) the average number of syllables per word and the average sentence length (i.e. Flesch-Kincaid Grade Level index³⁴), (4) the average number of words per sentence and the percentage of words with three or more syllables (i.e. Gunning Fog index³⁵), (5) a combination of average sentence length and percentage of difficult words (i.e. Dale–Chall readability³⁶), (6) Sentence length and number of polysyllables (i.e. Linsear Write readability³⁶), (7) number of characters (i.e. Coleman-Liau Index^36,37), (8) the average number of characters per word and number of words per sentence (i.e. automated readability index³⁸), and (9) the text standard score based on number of sentences, words, syllables, and characters in each tweet (i.e. text readability consensus). Figure 8 illustrates these comparisons based on the minimum (red), average (pink), and maximum (light pink) value of each feature. While all datasets have similar behaviour in terms of each feature’s minimum value, the Canadian dataset has a lower score for the average number of syllables per word and the average sentence length for all PASS categories. Interestingly, the sleep quality dataset, compared to other PASS categories, has a higher value for the maximum number of ‘grammar errors’ and ‘sentence count’ metrics, while all datasets show the same behaviour for the minimum and the average values of these metrics. These location-specific linguistic characteristics should be considered when utilizing these datasets to train and evaluate PASS surveillance ML models. For example, a model trained on the Canadian dataset may not present some linguistic features of a dataset that originated in Australia and vice versa.

• Download figure
• Open in new tab

Figure 8.

sc: sentence count, g: Gunning Fog index, ar: automated readability index, f :Flesch-Kincaid grade level, c: Coleman-Liau Index, l: Linsear Write readability, d: Dale–Chall readability, gs: grammar score, and t: text readability consensus.

Data access

LPHEADA is publicly available online. Retrieving the data using TweetIDs provided in the dataset requires Twitter’s approval. Data access also requires a data use agreement between the data user and Twitter to govern the access and use of the licensed material returned by Twitter API. Once approved, Twitter metadata can be downloaded as a JSON file to be mapped to other subsets of data (e.g. location, time, and demographics) using the TweetID field.