Twitter and Census Data Analytics to Explore Socioeconomic Factors for Post-COVID-19 Reopening Sentiment

2.1. Twitter data analytics

Twitter data has been used for a wide range of analyses, including but not limited to healthcare, retail marketing, stock trading, education and politics [17–24]. Twitter data offers a wide range of variables depending on the download programming interface or mechanism used. The use of rsenti package in R allows for the download of 90 variables(including variables such as type of device used, stated location, hast tags, display text width, reply to user ID, quote, retweet, favorite count, retweet count, URLs used, followers count, and date and time, to name a few) providing a rich array of variables associated with each post, which can be used to better understand the sentiment associated with the Tweet [25]. In addition to the rich diversity of Twitter variables that lend themselves to analysis, Twitter posts or “Tweets” contain textual which are not easily manipulated, and therefore requires specialized analysis. Additional elaboration on the specific steps used is provided under the methods section (Section 3.2).

2.2. Human behavior and sentiment

The present study utilizes public sentiment derived from social media posts as a key variable, and this is supported by extant research which has used sentiment analysis for diverse research purposes such as decision support, education, politics, opinion mining, data visualization, healthcare and hate crimes, and the importance of education, gender sensitivity and motivation [7, 22, 24, 26–32]. These studies have used a wide range of methods, tools and languages such as Python and R, and their associated libraries, to estimate sentiment from social media posts. Sentiment estimation can be broadly classified into two buckets, the first is the assignment of a score which ends to be continuous within a given range of an approximate minimum negative value (such as −2) to an approximate maximum positive value (such as +2), and the second consists of binary classification mechanisms (usually into positive and negative sentiment classes) or categorical classifications of data into sentiment classes such as fear, trust and sadness. For the purposes of this study, we use the R statistical modeling language from CRAN, and its associated sentiment analysis packages called sentimentr and syuzhet [33–35]. Past studies have also developed customized mechanisms to study human characteristic traits such as dominance, with the potential for corresponding emotions of anger and elation expressed through textual communications, and identified via manual or automated textual analytics [36, 37].

2.3. US Census data and socioeconomic analysis

To find out the socioeconomic factor associations of reopening sentiments, this study also collected socioeconomic and demographic (e.g., income, education, age, family type and size, race, housing type etc.) information of the people from the American Community Survey (ACS) [14]. Moreover, data on the factor of the built environment (e.g., population density) was collected from ACS to assess the impacts of urban form on the Coronavirus fears and reopening sentiments of the people. ACS is conducted by the US Census Bureau each year to collect vital information about the citizens. The data is free, publicly available, and considered as an important source of information for researchers from different disciplines. Many previous studies collected information from ACS and leveraged with Twitter data to analyze sentiment of the people in the arena of public health [28], urban spaces [38], politics [39, 40], disasters management [41], racial conflicts [30, 42], and gender disparity [43]. Thus, linking Twitter data with Coronavirus data is a common practice among the researchers to evaluate the impacts of socioeconomic and demographic characteristics on the sentiments of the people towards a subject of interest. Moreover, the name of the four regions from where the tweets were generated was collected based on the US Census to evaluate regional impacts on reopening sentiments [44]. Dummies (0 or 1) of the regions were created to include them in the model. This study also collected information on the number of cases and deaths from Worldometer [45] to understand how the severity of Coronavirus influence the sentiment of the people about reopening the economy. Considering the unavailability of the exact location of Twitter users, state averaged Census and Coronavirus data were collected and integrated with Twitter data to perform the analysis. The process of Twitter, Census and Coronavirus data integration has been illustrated in Section 3. Table 1 represents the descriptive statistics of the dependent and independent variables used in the binary model.

2.4. Reopening risk assessments

Restarting the US businesses would cause a rise in the mortality [46]. A study using agent-based Susceptible, Exposed, Infected, and Recovered (SEIR) model [47] on the population-specific data (such as contact patterns, household structure, age distribution, and comorbidity rates) evaluated an alternative lockdown and reopening scenarios for three specific states in the US (Florida, Georgia, and Mississippi). The model assessed that imposing lockdown one week earlier in all the three states could have saved hundreds of lives from COVID-19. However, to reopen the economy even with a limited capacity it is required to reduce the population contact down to 20-25% and implement strict social distancing measures along with the use of personal protective equipment [47]. Besides, a robust testing capacity would help the policymakers to estimate more precise reopening dates and it would be beneficial to detect and isolate asymptomatic carriers in a quicker manner after reopening. Various empirical evidence, trials, and observations suggested that the proper use of medical masks, combined with other non-pharmaceutical interventions (NPIs) such as thorough handwashing and strict social distancing, testing, contact tracing, and quarantine, is an effective way to reduce or interrupt the transmission of respiratory infections of COVID-19. Therefore, the deployment of masks in the public zones combined with other measures may eventually help in reopening the economy and transitioning into the post COVID-19 world [48, 49].

3.1. Data

This study uses Twitter data collected between April 30, 2020, and May 08, 2020, to understand the sentiment of the people towards the reopening of the US economy [8]. A total of 293,597 tweets with 90 variables were downloaded using the keyword “reopen”. The detailed methodology of data acquisition, saving, cleaning, and filtering for obtaining a subset of them that were generated from different states of the US has been described in Section 3.2. After systematic cleaning and filtering, a final dataset consisting of 2507 tweets and twenty-nine variables were used for sentiment analysis and exploring the socioeconomic factor associations that influence sentiments of the people towards reopening the economy. Fig. 2a shows total number of tweets collected from each states in the US. The figure demonstrates that a large number of tweets were generated from the Western and Northeastern regions of the US. Most of the tweets were collected from California, Texas, New work, Florida, Pennsylvania, Illinois, Ohio, North Carolina, Virginia, New Jersey, Arizona, and Nevada. In contrast, no tweets were collected from North Dakota. Fig. 1 displays the Census regions of the US with the states in each region.

• Download figure
• Open in new tab

Figure 2:

Number of tweets and Twitter-driven sentiment score.

Sentiment analysis is a popular topic of research among the researchers after collecting data from Twitter, webpage, product reviews, newspaper etc. [26, 50–52]. The primary objective of the sentiment analysis is to investigate opinions, attitudes, and emotions of the users (i.e., positive or negative feeling) towards a subject matter of interest (e.g., entity, person, issue, event, and topic). Sentiment analysis is beneficial for both customers and service providers to improve the mutual relationships [26]. In this study, Twitter data was analyzed to understand the sentiments of the Americans towards the new normal reopening amidst COVID-19 outbreak. Using R packages Syuzhet and sentimentr, sentiments were classified and assigned a score based on matching keywords, word sequences, and prewritten lexicons. Sentiment score was assigned within a range of −2 to +2. The maximum negative value indicates negative sentiment, whereas the maximum positive value indicates positive sentiment, with a score 0 indicates neutral sentiment. Results from a preliminary analysis showed that about 48.27% of the users expressed positive sentiment. In contrast, about 36.82% and 14.92% of the users expressed negative and neutral sentiments, respectively. Fig. 2b shows spatial distribution of the Twitter-driven sentiment score towards reopening the economy. It indicates that most of the states showed positive sentiment towards reopening of the economy. Calculating an average score value for the US, we found that mean value of the sentiment score is 0.0271 considering positive, negative and neutral tweets. Thus, most of the Twitter users posted positive information about the reopening. States with highest positive sentiments include South Dakota, Wisconsin, Oklahoma, Montana, Tennessee, Minnesota, and Missouri. On the other hand, Vermont, Nebraska, Utah, Maine, Illinois, New Hampshire, and Rhode Island showed the highest negative sentiment towards the reopening.

3.2. Study methods

A detailed description of the systematic methodologies adopted in this study, starting from data collection to results discussion and reporting, is shown in Fig. 3. We collected data from Twitter to understand feelings of the people by using the rTweet package in R and associated Twitter API. However, this method is generally applicable for collecting data from any social media platforms (e.g., Facebook, LinkedIn, Instagram, and news agency) regarding any real-world social events (e.g., manmade and natural disasters, political affairs, religious and racial conflicts). After filtering the tweet’s information, the data is saved in CSV format for subsequent prepossessing. Data preprocessing (e.g., cleaning, removing noises) is an important step in text analysis and classification. In the data preprocessing steps, unnecessary words, such as stop words (i.e., pronouns, articles, prepositions such as ‘the’, ‘a’, ‘about’, ‘we’, ‘our’, etc. that do not have any significant contribution to text classification), noises (i.e., punctuation, special characters), and abusive words (i.e., slang words) were removed from the text to improve the efficiency of the system [53], [54].

• Download figure
• Open in new tab

Figure 3:

Study design flowchart.

Moreover, tokenization and normalization techniques are applied to process the text. Tokenization is the process of dividing texts into words, phrases, and symbols which are known as tokens [55]. The main goal of the tokenization is to find out the words or group of words in a sentence, which is the foundation of text analysis. Tokenization is very important for text analysis because a meaningful computation primarily depends on the components (tokens) of the text, not on the full text. For example, if there is a sentence like ‘we have collected data from twitter’. After tokenization, the set of tokens will be:{‘we’, ‘have’, ‘collected’, ‘data’, ‘from’, ‘twitter’}. Normalization transforms words into a common form that allows the computer to identify the same words with similar meaning and remove one of them [54], consequently significantly reduces data size and increases computational efficiency. Lowercasing, stemming and lemmatization are three important and useful methods of text normalization. Lowercasing converts every letter to lowercase and make similar words consistent to each other [53]. For example, lowercasing converts each letter of ‘An’ and ‘an’ to lowercase and helps the computer to identify the two words are identical. Stemming converts words to their base form (stem) [56]. The same word appears in different forms in the texts that are consolidated into the same features [57]. For example, the semantic meaning of ‘read’ and ‘reading’ is the same, thus they are combined as ‘read’ to avoid any confusion. Lemmatization is the most advanced method of text normalization, which replaces words with their same morphological root using a dictionary [54]. Lemmatization is very similar to stemming. However, the basic difference between them is that lemmatization does not produce a stem but replace the suffix of the word to normalize it with usually a different word suffix [58]. Normalized words with both stemming and lemmatization could be the same. For example, normalized word of ‘reads’, ‘reads’, and ‘reading’ is the same in both methods. However, normalization of the same words could be different in stemming and lemmatization. For instance, the normalized word of computes, computing, computed is ‘comput’ in stemming and compute in lemmatization. After processing the data, different data exploration techniques were used to extract insights from the Twitter data as stated below:

1. Bag-of-words (BoW) count different word frequencies in the text to determine the focal point of the text analysis, ignoring the order of the words [53]. For example, the sentence “Although the order of the words is ignored, multiplicity is counted and used to determine the focal point of the text analysis” can be tokenized as {‘Although’, ‘the’, ‘order’, ‘of’, ‘the’, ‘words’, ‘is’, ‘ignored’, ‘multiplicity’, ‘is’, ‘counted’, ‘and’, ‘used’, ‘to’, ‘determine’, ‘the’, ‘focal’, ‘point’, ‘of’, ‘the’, ‘text’, ‘analysis’}. Thus, the corresponding BoW is {1, 4, 1, 2, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}.

2. Document term matrix (DTM) is used to represent text corpus (i.e., collection of texts) in a bag-of-words [54]. DTM is a matrix where rows contain documents, columns contain terms, and cells contain the number of each term that occurred in each text document. TDM allows the researchers to analyze data with vector and matrix algebra, which effectively convert text to numbers. Moreover, text data stored in the DTM format improve memory efficiency and optimize the operation of the data analysis.

3. Parts-of-speech (POS) tagging is a basic part of the syntactic analysis, which has numerous application in NLP [59]. In POS tagging, worlds in the text such as nouns, verbs, articles, and adjectives are identified to understand the context of the text [54]. For example, tagging nouns and proper names researchers identify similar events in news items. It is also a good approach to remove articles and pronouns from the text, which has no meaningful role in the text analysis.

4. Dependency parsing (DP) illustrates the syntactic relationship between different tokens [60]. For example: ‘John is an assistant professor at UNCC and he is very popular among the students’. In the example, it is understood that ‘John’ is a nominal subject and ‘professor’ is an adjective, which indicates that John is a professor who teaches students.

5. The N-gram technique is used to understand the association between the words. N-gram is not a representation of text, rather presents a set of n-words in the text with their order [53]. N-gram techniques tokenize texts into single words (unigrams), sequences of two words (bigrams), three words (trigrams), and so on to maintain the order of words and syntactical properties [54].

The findings of the exploratory analysis using steps and processes described above helped to gain a clearer understanding of public perspectives on reopening [8]. After data exploration, sentiment score was generated for each tweet by using the R package sentimentr and sentiments were classified into positive, negative, and neutral based on matching keywords, word sequences, and prewritten lexicons. Twitter data was then integrated with state-wide averaged socioeconomic data collected from the Census to conduct the analysis. A binary logit model was used to evaluate the factors that influence people’s sentiment towards a new normal reopening. To perform the analysis, the categorical variable of tweet sentiments was converted to dummy variables where positive sentiment was assigned a value of ‘1’ and negative and neutral sentiments were assigned a value of ‘0’. The category of the tweets was used as the dependent variable in the binary model. On the other hand, different socioeconomic and demographic variables, regional dummies, and Coronavirus cases and deaths have been used as the independent variables in the binary logit model. Finaly, the model results are discussed and reported to get some insights about the reopening sentiment of the people. The following subsection discusses about the theoretical framework of binary logit model.

3.3. Binary logit model

In the linear regression model, the response variable Y is quantitative. However, in many situations, the response variable is rather qualitative or categorical, for instance, the sentiment of a tweet could be categorized into positive or negative. The logistic regression model also known as logit model classifies the sentiment based on the probability. Assume X is the set of features, [x₁, x₂, …, x_n], where n is the total number of features in reopening sentiment analysis. pr(Y = 1|x) denotes the probability of positive sentiment about reopening given the feature x. Conversely, pr(Y = 0|x) denotes the probability of negative reopening sentiment given the feature x. Using the linear regression model, pr(Y = 1|X) = P(X) can be computed as, where β₀ is the intercept and β₁ is the co-efficient of X. However, Eq. (1) may predict P(X) < 0 for X close to 0 and P(X) > 1 for large value of X. To bound P(X) in the range 0 and 1, we use the following logistic function [61],

Consider the boundary value is 0.5. We get the following estimated probability from Eq. (2),

After some mathematical manipulation from Eq. (2), we can get,

After taking log on both side of Eq. (4) we get,

The left-hand side of Eq. (5) is called log-odds or logit, which shows that logistic regression model has a logit which is linear in X. The coefficients β₀ and β₁ are estimated using the maximum likelihood method. The idea is to estimate the value of β₀ and β₁ for each feature (x_i) so that it minimizes the difference between the predicted probability and observed probability P(x_i). The likelihood function can be expressed as follows [61],

To increase the computation speed, we take log on Eq. (6) and get the following log likelihood function,

In other words, estimate the values of β₀ and β₁ so that Eq. (7) yields the maximum value.

4.1. Calibrated model

The binary logit model is calibrated and estimated using STATA 15 software [62]. The log-likelihood method is used to calculate the coefficients. The final results of the model indicating the impacts of independent variables on the dependent variables are presented in Table 2. The table also reports the standard error, z-value, and probability level (P-value) of the estimates. Many of the coefficients are statistically significant at 0.001, 0.05, and 0.10 levels. However, some of the coefficients with a P-value greater than 0.10 are retained in the model to portray the relationships between dependent and some statistically insignificant independent variables, yet important factors that can influence sentiment of the persons.

4.2. Discussion of results

Results presented in Table 2 show that longer tweets (0.003) are more likely to indicate positive sentiment about the reopening. The influence of tweet width on positive sentiment is statistically significant at P-value of 0.05. Thus, people are more likely to post a longer statement on Twitter to express their positive feelings about the reopening of the economy with useful information. A study analyzed Twitter data to understand sentiments of the citizens for allocating resources during Hurricane Irma in 2017 in Florida [41]. Upon analyzing data the study found that longer tweets are more likely to have useful information with sentiment contents. It also revealed that popular tweeters were more likely to have positive sentiments and less likely to have useful information about the disaster. Thus, longer tweets can provide insightful information on the public sentiments and can be used for crises management during any man-made and natural pandemics.

4.2.4. Fit

The overall fit of the calibrated model is evaluated based on several key goodness-of-fit statistics. The likelihood ratio chi-square (LR chi2) statistic of the estimated model is 63.39 (Table 2). A lower value of the Chi-square indicates a better fit of the model. P-value (<0.000) of the Chi-square statistic indicates that overall the model is statistically significant and better than a model with no predictors. Thus, the model can significantly fit the observed data because the P-value is less than 0.000.

Other fit statistics also confirm the overall fit of the estimated model (Table 3a). We calculated Pearson chi-square fit statistics to evaluate the overall fit of the model. This is the formal test of the null hypothesis to assess whether the fitted model is correct. The P-value of this hypothesis testing ranges between 0 and 1. P-value specified α level (i.e., 0.05) indicates that the model is not statistically significant and acceptable. A higher P-value indicates a better fit of the model. Pearson chi-square test statistics (Prob >chi2 = 0.1929) presented in Table 3a indicates that we cannot reject the null hypothesis and hence, the model is overall fit. In a nutshell, the above discussed fit statistics indicate that the model can adequately fit the observed data.

View this table:

• View inline
• View popup
• Download powerpoint

Table 3:

Fit statistics and classification summary.

We also reported classification statistics to evaluate the accuracy and efficiency of the model (Table 3b). Overall, the model can correctly classify 56.18% of the sentiments with 62.32% of specificity (i.e., correctly classify positive sentiment) and 49.59% sensitivity (i.e., correctly classify negative sentiment). Thus, the classification statistics indicate a good prediction accuracy of the model.

Normal distribution of the residuals is shown in Fig. 4. The Normal QQ plot and normal probability plot show that the residuals fall perfectly along a linear line at 45°angle. Thus, the residuals are normally distributed. Normal distribution of the residuals indicates that the amount of error in the model is consistent across the observed dataset. Therefore, the predictive capability of the explanatory variables is same for the full range of dependent variables. Moreover, the model can explain all the variations in the dataset sufficiently.

• Download figure
• Open in new tab

Figure 4:

Normal distribution of the residuals.

4.3. Marginal effects

The results of the binary logit model presented in Table 2 provide an indication of the effects (i.e., positive or negative) of independent variables on the dependent variable. However, it is difficult to interpret the magnitude of marginal effects. Hence, we calculated the marginal effects of explanatory variables using the ‘margins’ command in Stata. Margins make result interpretation easier and report elasticities (i.e., percentage change in the likelihood of positive sentiment for a unit change in an explanatory variable). The results of the marginal effects of explanatory variables have been presented in Table 4. The results in Table 4 indicate that a 1% increase in tweet generation from the west region of the US increases the probability of positive sentiment for the reopening by 0.175%. The results also indicate that employment status has positive impacts on the reopening. People with low household income are associated with the positive reopening sentiment. A 1% increase in low-income people increases the probability of positive sentiment for reopening by 0.051%. Similarly, high house rent motivates people toward the positive sentiment of reopening the US economy. The result indicates that a 1% increase in house rent leads to the probability of a 0.001% increase in positive sentiment. Interestingly, increasing Coronavirus cases have a limited affect on reopening sentiment - A 1% increase in Coronavirus cases reduces positive sentiment by only 0.000001%.

5.1. Implications

Several policy recommendations can be drawn from the analysis. By investigating the socioeconomic factor associations for reopening sentiments, this study implicitly pointed out which strata of the people, and which regions bear positive and negative sentiments for reopening the economy. Thus, the federal and state governments and agencies can take necessary decisions based on the study findings where to intervene to secure people and economy of the country. The findings can also be used to improve information dissemination, as information categories can impact performance, and increase effectiveness of other preventive measures such as antiseptic and disinfectant protocol (e.g., hand washing, body and nasal spray) are testified to reduce infection of the people [64, 65]. Real-time COVID-19 incidence and socioeconomic characteristics of the people provide essential directions to the policymakers and health professionals to allocate resources for developing vaccines and therapeutics to protect people [66]. Thus, adequate protective actions need to be undertaken for anticipated risks and pave the way to a new normal reopening.

5.2. Limitations

Despite timely contribution to the literature, this study has some limitations. First, using tweeter data does not represent the complete section of the population. Very few people use Tweeter [67–69]. Thus, the study using Twitter data unable to provide a general idea about the subject matter. Second, sentiment analysis is unable to pick up nuanced or ambiguous meanings (e.g., slang, misspellings, nuanced or ambiguous meanings, Twitter lexicon, inside references, current events, intention, mood) of the tweets which give misleading information [67, 69]. Third, it is very difficult to analyze and identify valuable information from a very large quantity of unstructured and heterogeneous data from social media to acquire useful information for decision making [70]. Fourth, socioeconomic and household information is averaged at the state level which provides a little variation. Thus, a study with fine geographic resolutions (e.g., county, zip code etc.) might provide more insights.

5.3. Future research and conclusion

This research uses a novel methodological variation by combining sentiment analytics from Twitter data, with a custom selection of socioeconomic variables from Census data, to create insights that can contribute to developing a clearer understanding of the factors driving post-COVID-19 reopening sentiment. Sentiment and human behavior can be affected by a wide range of factors, including the information propagation formats, and future research could therefore include relevant time-matched news articles and responses to the Tweets data for sentiment analysis [16, 51, 71]. This study opens up a valuable stream of research in identifying factors contributing to post-crisis public sentiment using sentiment analysis and can influence future research in policy formation, public mental health, information systems and applications of sentiment analytics. In summary. this study provides interesting insights to researchers and policymakers, and makes two key contributions: a) the study makes a novel methodological contribution by combining sentiment analytics using Twitter data with Census data, for socioeconomic analysis which can be used for further research, and b) the study provides post-COVID-19 reopening insights into positive sentiment and negative sentiment population segmentations, which can be useful for focused and effective communication and policy management.