The role and influence of perceived experts in an anti-vaccine misinformation community

3.1 Perceived experts are present throughout the coengagement network, including in the anti-vaccine community

The coengagemement network consisted of 7,720 accounts (nodes) linked by 72,034 edges (Figure 1). 5,171 of those accounts had English language profiles and appeared to correspond to individual users. 24 of the individual users with English profiles added or removed expertise cues in their profiles over the course of the study and were thus excluded from the analysis. Of the remaining 5,147 individual users, 13.1% (678 users) were perceived experts. Perceived experts rarely provided cues of expertise in their name alone. Instead, almost all users indicated expertise in their description, with approximately half of users including expertise cues in both their name and description (Supplemental Figure 7). There was substantial agreement between coders on whether a user was in an excluded category, a perceived non-expert, or a perceived expert (κ = 0.715).

The two largest communities contained 79.6% of total accounts and 66.1% of English-language individual accounts in the network. Stance toward COVID-19 vaccines by users with the greatest degree centrality was relatively consistent, meaning that very few users posted a combination of tweets that were positive and negative in stance, although most users posted some neutral tweets as well (Supplemental Figure 8). Further, stance was generally shared within communities; popular tweets by central users in the anti- and pro-vaccine communities almost exclusively expressed negative and positive stances, respectively, or neutral stances, with a few exceptions. Although there was moderate inter-rater reliability (κ = 0.567) between coders on whether individual tweets were negative, positive or neutral about vaccines, there was high agreement on the overall stance of each user (κ = 0.805 for whether a given user posted more anti-vaccine tweets, pro-vaccine tweets, or equal numbers of either).

The pro-vaccine community was larger than the anti-vaccine community (3,443 and 2,704 users respectively). The anti-vaccine community was made up of a large, densely connected component split into two communities and linked to two additional medium-sized subcommunities (Figure 1A), which were dominated by users tweeting in languages other than English (Italian and French) (see Supplemental Figure 10 and Supplemental Table 4 for visualization and further discussion of subcommunities). The pro-vaccine community similarly consisted of one large component that was connected to the next largest community, which was focused on vaccination in India, and two medium-sized sub-communities (Figure 1C). The largest component of the pro-vaccine community was divided into two subcommunities, one of which was dominated by non-individual accounts. Perceived experts were present in both communities, but constituted a larger share of individual users in the pro-vaccine community (17.2 % or 386 accounts) compared to the anti-vaccine community (9.8 % or 185 accounts) (Figure 3). The two communities were connected by a few bridging nodes, including some perceived experts (Figure 1B).

3.2 Perceived experts in the anti-vaccine community share both low quality and academic sources

We found marked differences in sharing of low quality and academic sources depending on community and perceived expertise Figure 2. For both types of sources, we calculated a user-level metric (the proportion of users who shared at least one link of a given type; Figure 2B, D) and a link-level metric (the proportion of all links shared during the study period that were of a given type; Figure 2A, C). Perceived experts posted more frequently, and included links in a greater proportion of their posts (Supplemental Figure 9)

• Download figure
• Open in new tab

Figure 2: Users in the anti-vaccine community share low quality sources and perceived experts share academic research.

Each panel compares a different metric of link-sharing by perceived experts and perceived non-experts in the anti- (pink) and pro- (green) vaccine communities. The metrics are: (A) proportion of checked links that were from low quality sources, (B) proportion of users that shared at least one low quality source, (C) proportion of checked links that were from academic research sources, and (D) proportion of users that shared at least one academic research source. 95% binomial proportion confidence intervals are indicated by black error bars.

Compared to the pro-vaccine community, perceived experts and perceived non-experts in the anti-vaccine community shared low quality sources at significantly greater rates (p < 0.001 for proportion of links and proportion of users). Low quality sources were almost exclusively shared in the anti-vaccine community, although low quality sources generally comprised a relatively small proportion of assessed links (Figure 2A, B). Many users in the anti-vaccine community shared at least one low quality link (30% of perceived experts and 25% of perceived non-experts) compared to fewer than 0.5% of users in the pro-vaccine community (two perceived experts and seven perceived non-experts) (Figure 2B). Compared to perceived experts, perceived non-experts in the anti-vaccine community shared low quality sources as a significantly greater proportion of all of their links (p = 0.015, Figure 2A), but there was no statistically significant difference in the proportion of users who shared at least one low quality source (p = 0.191, Figure 2B).

In both communities, perceived experts shared academic research links at a significantly greater rate compared to other individual users (Figure 2C, D) (p < 0.001 for proportion of links and proportion of users sharing academic sources). There was a slightly greater proportion of academic link-sharing for perceived experts in the pro-vaccine community compared to the anti-vaccine community, which was statistically significant at the link level (p = 0.028, Figure 2C) but not at the user level (p = 0.074, Figure 2D).

3.3 Perceived experts are overrepresented as central and bridging users

Although perceived experts represented a relatively small share of the individual users in the coengagement network, they disproportionately occupied important positions in the network as central and bridging nodes (Figure 3, Figure 4). Perceived experts were overrepresented among users with the greatest betweenness, degree, and eigenvector centrality in both communities, although this effect was not always statistically significant for the pro-vaccine community (Figure 3).

• Download figure
• Open in new tab

Figure 3: Perceived experts are overrepresented as the most central users in the pro-vaccine (green) and anti-vaccine (pink) communities.

Plots are arranged in a grid where each row corresponds to users in one of the two largest communities: the anti- and pro-vaccine communities (top and bottom respectively). Each column corresponds to a different centrality metric: betweenness centrality (left), degree centrality (middle), and eigenvector centrality (right). For each plot, we subset the network to the n nodes with the greatest values for a given centrality metric, where n is full population size (all individual users in the community), 500, or 50 (x-axis). Bar height indicates the proportion of users in each subset that are perceived experts and error bars give 95% binomial proportion confidence intervals. Stars above the error bars indicate whether perceived experts are significantly overrepresented within a given sample of central users (one star indicates p < 0.05, two stars indicate p < 0.01, three stars indicate p < 0.001).

• Download figure
• Open in new tab

Figure 4: Perceived experts disproportionately act as key bridges between the pro-vaccine and anti-vaccine communities.

Bar height indicates the proportion of users in each population sample that are perceived experts and error bars give 95% confidence intervals for the proportions. The x-axis indicates the size of each subset (n), corresponding to the full population of 5,147 (all individual users as a basis of comparison), and the 500, 50, or 10 users with the greatest community bridging score (x-axis). Stars above the error bars indicate whether perceived experts are significantly overrepresented within a given sample of bridging users (one star indicates p < 0.05, two stars indicate p < 0.01, three stars indicate p < 0.001).

Perceived experts in both communities were highly overrepresented among the five hundred users with the greatest betweenness centrality (p < 0.001 for both communities) and overrepresented among the fifty users with the greatest betweenness centrality by a factor of about two (p = 0.014 and p < 0.001 for the anti- and pro-vaccine communities respectively) (Figure 3).

Ranking on degree centrality and eigenvector centrality, perceived experts were more strongly overrepresented in the anti-vaccine community compared to the pro-vaccine community (Figure 3). Perceived experts in the anti-vaccine community were about two times more prevalent in the group of central users (20% of the fifty top users ranked on both metrics) compared to their share of the population, while perceived experts in the pro-vaccine community were overrepresented by a factor of approximately 1.6. By both metrics, perceived experts in the anti-vaccine community were significantly overrepresented in the 500 most central users (p = 0.001 and p < 0.001 for degree and eigenvector centrality respectively) and the fifty most central users (p = 0.014 for both degree and eigenvector centrality). In the pro-vaccine community, perceived experts were significantly overrepresented in the 50 most central users (p = 0.032 for both degree and eigenvector centrality) but not in the 500 most central users (p = 0.1038 and p = 0.082). It is unsurprising that the results for eigenvector and degree centrality were similar given that the PageRank algorithm used to compute eigenvector centrality is proportional to degree centrality on an undirected graph [Grolmusz, 2015].

Perceived experts were significantly overrepresented in the group of the 500 and ten most important bridges between the pro- and anti-vaccine communities (p < 0.001, p = 0.001 respectively), but not significantly overrepresented in the fifty most important bridges (p = 0.09). Figure 4. About 20% of the top 500 and top fifty users ranked by community bridging scores were perceived experts, and five of the 10 users with the greatest community bridging scores were perceived experts (Figure 4, Figure 1B). There was little variation in which users were highly ranked between different network metrics (Supplemental Figure 12, Supplemental Figure 13, Supplemental Figure 14, Supplemental Figure 15).

3.4 Perceived experts are more influential than other individuals in the anti- and pro-vaccine communities

Using propensity score matching, we achieved excellent balance across matching covariates within the anti-vaccine community (Supplemental Figure 16) and good matching within the pro-vaccine community for all variables except for the percent of posts with links (Supplemental Figure 17 and Supplemental Figure 18). There were minimal differences in the results of propensity score matching with and without matching on subcommunities (compare Figure 5 and Supplemental Figure 19; Supplemental Table 6, Supplemental Table 7, and Supplemental Table 8). Using these propensity-matched pairs for comparison, on average, perceived experts received more engagements (likes and retweets) on their posts, but were not more central than other individual users (Figure 5, Supplemental Figure 19, Supplemental Table 6, Supplemental Table 7, Supplemental Table 8).

• Download figure
• Open in new tab

Figure 5: In the anti-vaccine and pro-vaccine communities, perceived experts receive greater engagements compared to other users.

For each influence metric (y-axis, Supplemental Table 3), we plot the standardized average treatment effect on the treated (ATT) as a point and corresponding 95% confidence interval. Values are shown for the matching analysis directly comparing the anti-vaccine (pink) and pro-vaccine (green) communities (corresponding to Supplemental Table 8). Positive values (to the right of the vertical line) indicate an influence boost for perceived experts. Instances where the effects were significantly greater than zero (p < 0.05) are indicated with an additional circle around the point estimate.

• Download figure
• Open in new tab

Figure 6: The coengagement network of users tweeting about COVID-19 vaccines is divided into two large communities.

Users are represented as circles and scaled by degree centrality. Edges connect users that were retweeted at least twice by at least ten of the same users. Nodes in the two largest communities detected using the Infomap algorithm are colored in pink (anti-vaccine) and green (pro-vaccine). Shades indicate account type: non-individual and non-English accounts excluded from analyses (light); individual perceived non-expert (medium); and perceived experts (dark). Nodes outside of the two largest communities are gray. Each edge is colored based on the color of one of the two nodes it connects, randomly selected. Note that this figure is identical to Figure 1 without annotations and in higher resolution.

• Download figure
• Open in new tab

Figure 7: Signals of expertise may appear in different parts of the profile.

Panel A (top) gives examples of different ways that expertise signals may be presented. Starting in the bottom right and moving counterclockwise: signal in description only (red), signal in name and description (purple), signal in name but description clarifies user is not an expert (green), or signal in name only no information about perceived expertise in description (blue). The gray profile has no signals of expertise and is therefore not a perceived expert. Panel B (bottom) is the proportion of perceived experts that provide expertise signals in each way across different group (x-axis): the full coengagement network, the pro and anti-vaccine communities combined, the pro-vaccine community alone, and the anti-vaccine community alone.

Perceived experts in the anti-vaccine community received 20% (95% CI: 3 - 43%) more retweets and 37% (95% CI: 9 - 63%) more likes on their median post than would be expected if they were not perceived experts (Figure 5, Supplemental Figure 19, Supplemental Table 6). We also found a significant effect of perceived expertise when using an alternative metric, the h-index, to assess engagements across tweets posted during the time period (ATT for natural logged retweets: 1.90 (95% CI: 0.36-3.44); ATT for natural logged likes: 3.43 (95% CI: 1.11-5.76)) (Figure 5, Supplemental Figure 19, Supplemental Table 6). In contrast to these measures of engagement, perceived expertise did not significantly affect centrality (betweenness, degree, and eigenvector) within the anti-vaccine community on average (Figure 5, Supplemental Table 8, Supplemental Table 6), although we found in the previous section that perceived experts were overrepresented in the tail of the distribution as highly central users compared to the full (unmatched) set of perceived non-experts (see subsection 7.8 for further discussion reconciling these two results).

Perceived experts in the pro-vaccine community also had a positive ATT across engagement metrics and additionally had a significantly positive ATT for betweenness centrality (Figure 5, Supplemental Figure 19, Supplemental Table 7). Although perceived experts in the pro-vaccine community tended to have a greater ATT across all influence metrics than those in the anti-vaccine community, the differences in ATT between groupswere not statistically significant for any influence metric (Supplemental Figure 20, Figure 5, Supplemental Table 9). Matching results were robust to matching parameters and exclusion of different matching covariates (Supplemental Figure 22, Supplemental Figure 23, and Supplemental Figure 24).

Limitations and Extensions

This study relies on proxies for tweet content and user activity that may miss important variation and nuance in stance. Breaking the network into pro- and anti-vaccine communities is common across studies of social media networks [Hagen et al., 2022, Sharma et al., 2021, ?, Jamison et al., 2020], and stance was generally consistent across popular tweets in either community (Figure 8), but there were several notable exceptions. Positive tweets from users in the anti-vaccine community tended to cite vaccine efficacy as an argument against mandating vaccines, while negative tweets from users in the pro-vaccine community (particularly those by perceived experts) praised vaccine regulators for responding to safety signals. Further, sharing a particular misinformation or academic link may not constitute endorsement, particularly in fact-checking contexts. Although academic sources were shared in the anti-vaccine community, academic sources may be misrepresented by these users [?Beers et al., 2022, Kata, 2010] or include articles that have been retracted [Van Schalkwyk, 2019, Abhari et al., 2023]. Future work may examine which academic links were shared within the anti-vaccine community and how these sources were interpreted. More detailed content analysis may reveal important differences between the communities beyond attitudes toward vaccine (e.g. attitudes toward non-pharmaceutical interventions and perceptions of severity of COVID-19 infection), heterogeneity in vaccine opinion within groups, and specific rhetorical strategies utilized by perceived experts in either group [Bradshaw et al., 2020, Maddox, 2022].

• Download figure
• Open in new tab

Figure 8: User stance is largely consistent within the two largest communities.

Plots display tweet stance across the ten perceived non-experts (left column) and ten perceived experts with the greatest degree centrality (right column) in the anti- and pro-vaccine communities (top and bottom row respectively). Users are ranked and arranged by degree centrality (x-axis). Three coders assessed the stance of a sample of highly retweeted tweets from each user, and results from all coders are displayed side-by-side for each user. Bars are shaded by the number of tweets assessed as positive (green), neutral (gray), or negative (pink).

• Download figure
• Open in new tab

Figure 9: Link-sharing activity across communities and perceived expertise.

Each panel compares a different metric of link-sharing by perceived experts and perceived non-experts in the anti- (pink) and pro- (green) vaccine communities. The metrics are: (A) average number of posts per user (B) proportion of all tweets with links (C) proportion of links that were checked (did not link back to Twitter or timed out during link expansion) (D) the total number of checked links in a given category. Error bars give 95% binomial proportion confidence intervals for proportions in panel B and C.

An important limitation of our study is that we were unable to assess how many users are exposed to a given tweet. By relying on engagements as an indicator of tweet popularity, we may underestimate the true reach of content. We also could not ascertain how exposure to vaccine-related information in this study influenced health decision-making and behavior, questions that could be directly evaluated in an experimental setting [Jiménez et al., 2018]. Further work may additionally examine the relative contribution of prestige bias to vaccine decision-making compared to other types of biases and how the effects of perceived expertise are moderated by other source characteristics (e.g., trust-worthiness and competence) [Pornpitakpan, 2004] and other forms of bias (e.g., confirmation bias, repetition bias, or success bias) [Acerbi, 2016, Jiménez et al., 2018, Brand et al., 2020, Berl et al., 2021, Nadarevic et al., 2020, Lachapelle et al., 2014, Lin et al., 2016].

This analysis is limited to a single month, based on events in the United States and English keywords, constrained to Twitter, and focused on conversations about COVID-19 vaccination. Initial vaccination capacity, trust in experts, and vaccine uptake vary considerably between countries [Wilson and Wiysonge, 2020] and our own analysis found considerable geographically clustering across the coengagement network (Figure 10). Further work may examine how the role of perceived experts in conversations about vaccination differed between regions and across different time periods (including prior to the COVID-19 pandemic and after the emergence of variants of concern with high breakthrough infection rates). Although we excluded the relatively small set of individuals who added or removed signals of expertise from their profile during the month of the study, extending the study period to expand this set of users could enable further examination of how user behavior and influence changes depending on perceived expertise [Hasan et al., 2022]. Twitter users are not representative of the general population [Wojcik and Hughes, 2019], and patterns in vaccine conversations on social media do not necessarily reflect actual vaccination trends [Cascini et al., 2022, Tavoschi et al., 2020]. The generalizability of these findings should be assessed on different social media platforms [Jones et al., 2012, Cascini et al., 2022, Tangcharoensathien et al., 2020]. The role of experts, particularly those who take counter-consensus positions, is relevant across other scientific topics including climate change, tobacco, and AIDS etiology and treatment [Oreskes and Conway, 2022, Epstein, 1996]. These methods could be applied to compare the role of perceived experts in conversations about different scientific and non-scientific topics (e.g., politics and entertainment) to test the extent to which domain specific credentials are necessary to be perceived as an experts and compare the effects of perceived expertise relevant to different conversations (e.g., whether politicians speaking on political matters receive an influence boost comparable to that of medical professionals and scientists discussing COVID-19 vaccines).

• Download figure
• Open in new tab

Figure 10: The main subcommunities of the coengagement network of users tweeting about COVID-19 vaccines.

Users are represented as circles and scaled by degree centrality. Edges connect users that were retweeted at least twice by at least ten of the same users. All edges are gray. Nodes in the ten largest subcommunities detected using the Infomap algorithm are colored in different shades of pink and red if they are in the anti-vaccine community (top to bottom: Italian, French, Main Anti (B), Main Anti (A)), green and blue if they are in the pro-vaccine community (top to bottom: Australia, Media, Main Pro, and Canada), and shades of orange if they are in the community focused on vaccination in India (top to bottom: India (A), India (B)). Nodes outside of the ten largest subcommunities are gray.

• Download figure
• Open in new tab

Figure 11: The main communities of the coengagement network of users tweeting about COVID-19 vaccines detected using the Louvain algorithm roughly correspond to subcommunities detected using the Infomap algorithm (compare to Supplemental Figure 10).

Users are represented as circles and scaled by degree centrality. Edges connect users that were retweeted at least twice by at least ten of the same users. All edges are gray. Nodes in the nine largest communities detected using the Louvain algorithm are colored in different shades of pink if they are in the anti-vaccine community (top to bottom: Italian, French, Main Anti (B), Main Anti (A)), green if they are in the pro-vaccine community (top to bottom: Australia, Main Pro, and Canada), and shades of orange if they are in the community focused on vaccination in India (top to bottom: India (A), India (B)). Nodes outside of the nine largest subcommunities are gray.

• Download figure
• Open in new tab

Figure 12:

Node size corresponds to community bridging score. As in Figure 1, users are represented as circles. Edges connect users that were retweeted at least twice by at least ten of the same users. All edges are gray. Nodes in the two largest communities detected using the Infomap algorithm are colored in pink (anti-vaccine) and green (pro-vaccine). Shades indicate account type: non-individual and non-English accounts excluded from analyses (light); individual perceived non-expert (medium); and perceived experts (dark). Nodes outside of the two largest communities are gray. Users that rank in the top 500, top 50, and top ten users by community bridging across the whole coengagement network are labeled with one, two, and three stars respectively.

• Download figure
• Open in new tab

Figure 13:

Node size corresponds to degree centrality. As in Figure 1, users are represented as circles. Edges connect users that were retweeted at least twice by at least ten of the same users. All edges are gray. Nodes in the two largest communities detected using the Infomap algorithm are colored in pink (anti-vaccine) and green (pro-vaccine). Shades indicate account type: non-individual and non-English accounts excluded from analyses (light); individual perceived non-expert (medium); and perceived experts (dark). Nodes outside of the two largest communities are gray. Users that rank in the top 500 and top 50 users by degree centrality within the anti-vaccine or pro-vaccine community are labeled with one and and two stars.

• Download figure
• Open in new tab

Figure 14:

Node size corresponds to betweenness centrality. As in Figure 1, users are represented as circles. Edges connect users that were retweeted at least twice by at least ten of the same users. All edges are gray. Nodes in the two largest communities detected using the Infomap algorithm are colored in pink (anti-vaccine) and green (pro-vaccine). Shades indicate account type: non-individual and non-English accounts excluded from analyses (light); individual perceived non-expert (medium); and perceived experts (dark). Nodes outside of the two largest communities are gray. Users that rank in the top 500 and top 50 users by betweenness centrality within the anti-vaccine or pro-vaccine community are labeled with one and and two stars.

• Download figure
• Open in new tab

Figure 15:

Node size corresponds to eigenvector centrality. As in Figure 1, users are represented as circles. Edges connect users that were retweeted at least twice by at least ten of the same users. All edges are gray. Nodes in the two largest communities detected using the Infomap algorithm are colored in pink (anti-vaccine) and green (pro-vaccine). Shades indicate account type: non-individual and non-English accounts excluded from analyses (light); individual perceived non-expert (medium); and perceived experts (dark). Nodes outside of the two largest communities are gray. Users that rank in the top 500 and top 50 users by eigenvector centrality within the anti-vaccine or pro-vaccine community are labeled with one and and two stars.

• Download figure
• Open in new tab

Figure 16: Love plot demonstrating the balance across matching covariates for individuals in the anti-vaccine community before (orange) and after (blue) propensity score matching was performed to test H1.

Each row corresponds to a different matching covariate (described in Supplement Table 2. The x-axis is absolute standardized mean difference, where values closer to zero correspond to better balance. The horizontal line indicates 0.1, the threshold below which balance is generally considered good.

• Download figure
• Open in new tab

Figure 17: Love plot demonstrating the balance across matching covariates for individuals in the pro-vaccine community before (orange) and after (blue) propensity score matching was performed to test H2.

Each row corresponds to a different matching covariate (described in Supplement Table 2. The x-axis is absolute standardized mean difference, where values closer to zero correspond to better balance. The horizontal line indicates 0.1, the threshold below which balance is generally considered good.

• Download figure
• Open in new tab

Figure 18: Love plot demonstrating the balance across matching covariates for individuals in the pro- and anti-vaccine communities before (orange) and after (blue) propensity score matching was performed to test H3.

Each row corresponds to a different matching covariate (described in Supplement Table 2. The x-axis is absolute standardized mean difference, where values closer to zero correspond to better balance. The horizontal line indicates 0.1, the threshold below which balance is generally considered good.

• Download figure
• Open in new tab

Figure 19: In the anti-vaccine and pro-vaccine communities, perceived experts receive greater engagements compared to other users.

For each influence metric (y-axis, Supplemental Table 3, we plot the standardized average treatment effect on the treated (ATT) as a point and corresponding 95% confidence interval. Values are shown for the matching analyses comparing perceived experts and perceived non-experts within the anti-vaccine (pink, left) and pro-vaccine (green, right) communities (corresponding to Supplemental Table 6 and Supplemental Table 7 respectively) with subcommunity matching. Positive values (to the right of the vertical line) indicate an influence boost for perceived experts. Instances where the effects were significantly greater than zero (p < 0.05) are indicated with an additional circle around the point estimate.

• Download figure
• Open in new tab

Figure 20: There is no significant difference in the influence boost for perceived experts between the anti- and pro-vaccine communities.

For each influence metric (y-axis, Supplemental Table 3, we plot the difference in the standardized average treatment effect on the treated (ATT) between the pro- and anti-vaccine communities as a point and corresponding 95% confidence interval (Supplemental Table 9). Positive values (to the right of the vertical line) indicate a greater influence boost for perceived experts in the anti-vaccine commmunity compared to the pro-vaccine community. None of the differences were significant at the p < 0.05 level.

• Download figure
• Open in new tab

Figure 21: Matching reduces the proportion of perceived non-experts with low centrality.

Each plot shows the frequency distribution of users for a given logged centrality metric. Values are log-transformed because distributions are highly skewed. Lines are colored by the subset of users: perceived experts (red), perceived non-experts in the full community prior to matching (green), and perceived non-experts in the community subset for matching (blue). Values to the right of the vertical lines indicate users in the top 500 (gray) or 50 (black) users ranked by the given centr₅a₄lity metric. Each row corresponds to a different centrality metric: betweeenness (top), degree (middle), and eigenvector (bottom). Each columns corresponds to a different community: anti-vaccine (left) or pro-vaccine (right).

• Download figure
• Open in new tab

Figure 22: Matching balance and results are robust to k, the number of nearest neighbors to which each perceived expert is matched.

Each row corresponds to a different value of k: 1 (top), 2 (middle), or 3 (bottom). The left column contains Love plots demonstrating the balance across matching covariates for individuals in the pro- and anti-vaccine communities before (orange) and after (blue) propensity score matching was performed. Within a plot, each row corresponds to a different matching covariate (described in Supplement Table 2. The x-axis is absolute standardized mean difference, where values closer to zero correspond to better balance. The horizontal line indicates 0.1, the threshold below which balance is generally considered good. The right column shows the standardized average treatment effect on the treated (point) and corresponding 95 % confidence interval for each influence metric (y-axis, Supplemental Table 3). Values are shown for the third matching analysis directly comparing the anti-vaccine (pink) and pro-vaccine (green) communities. Positive values (to the right of the vertical line) indicate an influence boost for perceived experts. Instances where the average treatment effect on the treated was significantly greater than zero (p < 0.05) are indicated with an additional circle around the point estimate.

• Download figure
• Open in new tab

Figure 23: Matching balance is robust to matching covariates.

Matching covariates were dropped one at a time to detect whether inclusion of a single covariate affected balance across other matching covariates. Each plot corresponds to a different dropped covariate indicated in the panel title (described in Supplement Table 2). Note that the variable posting time encompasses the three binary variables: morning poster, evening poster, and night poster. Each Love plots demonstrates the balance across matching covariates for individuals in the pro- and anti-vaccine communities before (orange) and after (blue) propensity score matching was performed. Each row within a figure corresponds to a different matching covariate. The x-axis is absolute standardized mean difference, where values closer to zero correspond to better balance. The horizontal line indicates 0.1, the threshold below which balance is generally considered good.

• Download figure
• Open in new tab

Figure 24: Average treatment effect on the treated (ATT) is robust to matching covariates.

Matching covariates were dropped one at a time to detect whether inclusion of a single covariate affected ATT. Each plot corresponds to a different influence metric indicated in the panel title (described in Supplemental Table 3). Plots show the standardized average treatment effect on the treated (point) and corresponding 95 % confidence interval depending on which matching covariate was dropped (Supplement Table 2). Note that the variable posting timeencompasses the three binary variables: morning poster, evening poster, and night poster. Values are shown for the third matching analysis directly comparing the anti-vaccine (pink) and pro-vaccine (green) communities. Positive values (to the right of the vertical line) indicate an influence boost for perceived experts compared to other individual users. Instances where the average treatment effect on the treated was significantly greater than zero (p < 0.05) are indicated with an additional circle around the point estimate.

Conclusion

We found that the set of anti-vaccine perceived experts extends far beyond prominent examples noted by others previously [Smith, 2017, The Virality Project, 2022], suggesting that they should be addressed as a unique and sizable group that blends misinformation with arguments that appear scientific. We also found evidence that perceived experts are more influential than other individuals in the anti-vaccine community, as they disproportionately occupied central network positions where they could reach large audiences and received significantly more engagements (20% more likes and 37% more retweets) on their vaccine-related posts compared to perceived non-experts. Perceived experts are not only some of the most effective voices speaking out against vaccine misinformation; they may be some of its most persuasive sources.

5.1 Data collection

The Institutional Review Board of Washington University determined that this study (STUDY00017030) was exempt. Our analysis was conducted on a subset of collections of public tweets containing keywords related to vaccines and the COVID-19 pandemic, retrieved and stored in by the University of Washington Center for an Informed Public in real-time as they were posted. Tweets that were later deleted and public tweets from accounts that were suspended or became private are included in the dataset.

We constrained our search period to April of 2021, noting that all individuals sixteen and older were eligible for vaccination by April 19th, marking this time period as an especially critical window for vaccine decision-making [Roy, 2021]. Further, administration of the Johnson & Johnson (Janssen) vaccine was paused in the United States between April 13th and April 23rd while the Centers for Disease Control and Prevention (CDC) and Food and Drug Administration (FDA) investigated a safety signal involving six reported cases of severe blood clots [U.S. Food and Drug Administration, 2021]. Focusing on April of 2021 also allows us to examine how different communities reacted to credible news of a serious but rare vaccine safety signal. During the same time period, fact checkers and researchers responded to several false claims about vaccination, including rumors that vaccinated people were able to “shed” vaccine components that might infect and harm unvaccinated people [The Virality Project, 2021].

To focus our analysis on content related to COVID-19 vaccines, we selected tweets (i.e., posts) within the collections that mentioned keywords related to vaccines in general or the specific manufacturers of the three COVID-19 vaccines initially offered in the United States (Supplemental File 1). Although this protocol focuses on vaccine administration in the United States and English language, tweet selection was not constrained to a specific geographic region. In total, we retrieved 4,276,842 unique tweets including quote tweets (when another post is shared with commentary) and replies (a direct response to another user’s tweet) from April. 5,523,595 unique users participated in the Twitter conversation about COVID-19 vaccines during the study period by either posting original content or retweeting (i.e., sharing) another user’s tweet on the topic. We additionally retrieved retweets, quote tweets, and replies linked to tweets in the April collection that were posted within 28 days of the original tweet (extending the dataset to May 28th) to compare the number of likes and retweets each tweet in the April study period received across a window of the same length (four weeks).

We randomly generated a unique numeric identifier for each user to protect user privacy (particularly for users who are not public figures) while allowing our findings to be reproduced. Individual users are not named in the analyses, reinforcing that we aim to characterize the group of perceived experts within the anti-vaccine community instead of focusing on individual, high-profile examples. User profiles and numeric identifiers for accounts used by Twitter were stored separately from tweets, which were instead linked to the anonymous identifiers we assigned. We also stored a dictionary that allows for translation between these two distinct numeric identifiers and utilized the dictionary to connect statistics about tweets to properties of the network and users (e.g., to determine community stance after tagging the stance of a sample of popular tweets, to compare the types of links shared depending on perceived expertise and community, and to collect matching covariates based on Twitter activity and engagements). In tweets we coded for stance, we anonymized mentions of other usernames and removed links to other posts and images on Twitter to prevent user identification. Therefore, researchers were blinded to user identity when assessing tweet stance. We have made tweet and user information relevant to results in this study publicly available without the linking dictionary.

5.2 Coengagement network construction

We examined the activity of influential users with shared audiences by generating a coengagement network. First, we constructed a directed graph where each edge connects a user to another user whom they have retweeted at least twice. Next, we used a Docker container developed by Beers et al. [2023] to transform the network into an undirected graph. Edges link accounts that were retweeted at least twice by at least ten of the same users. The resulting graph therefore filters users to those that received some amount of repeated engagement from several accounts rather than those that produced a single viral tweet. We used the Infomap hierarchical clustering algorithm implemented at mapequation.org to detect communities within the coengagement network. Infomap balances the detection of potential substructures (i.e., subcommunities within larger communities) against concisely describing a random walker’s movements through the network to determine the total number of levels to use [Rosvall and Bergstrom, 2008, Holmgren et al., 2022]. The subcommunities detected using Infomap correspond well to communities detected using the Louvain method, an alternative community detection algorithm (Supplemental Figure 10, Supplemental Figure 11, Supplemental Table 4). Coengagement networks were visualized using the open-source software package Gephi with the ForceAtlas2 layout algorithm [Bastian et al., 2009, Jacomy et al., 2014].

5.3 Tagging profiles

Based on username, display name, and user description collected each time a user tweeted or was retweeted during the study period, we manually labeled each profile in this group as a perceived expert or not (Supplemental Figure 7). The lead author tagged all users in the coengagement network and two additional authors labeled a subset of 500 users to test robustness of tags based on interrater reliability. We first noted whether a profile indicated that the user primarily tweeted in a language other than English. Because we were unable to assess non-English expertise signals and tweets associated with these users, who may also have reached a substantially different audience compared to English-language accounts, we excluded non-English accounts from our analysis. We also noted accounts that appeared not to represent individuals (e.g., accounts for media groups, non-profit organizations, governmental agencies, and bot accounts). Non-English and non-individual accounts (2,549 accounts in total) remained in the network for visualizations and calculations related to network centrality but were excluded from analyses focused on comparing individuals with and without signals of expertise in their profiles.

For the remaining individual profiles, we noted whether there were signals of expertise in the account username or display name (which are listed with tweets in a user’s feed) or the account description (which is only visible if a user mouses over the author’s tweet or directly visits the account) (Supplemental Figure 7). Signals of expertise included academic prefixes (e.g., Dr. or Professor), suffixes (e.g., MD, MPH, RN, PhD), and professional information (e.g., scientist, retired nurse). We limited our definition of perceived expertise to include training or professional experience in a potentially relevant field but excluded individuals who expressed an interest in a related topic without providing qualifications (e.g., “virology is the coolest”). We included anonymous accounts and ones that may have been parodies (e.g., “Dr. Evil” and “The Mad Scientist”) since we expect that users evaluate profiles based on heuristics and without investigating the veracity of information provided [Sundar, 2007]. We assumed that medical and wellness professionals, including practitioners of alternative medicine, may broadly be perceived as experts regardless of specialty [Jiménez and Mesoudi, 2019]. To focus on biomedical expertise, we did not code users from other fields as perceived experts (e.g., science journalists, disability rights advocates, and governmental officials without biomedical backgrounds), acknowledging that these sources may provide trusted and knowledgeable perspectives relevant to health decision-making.

Users that indicated expertise in any part of their profile (names, description, or both) at any point in time were tagged as perceived experts based on experimental evidence that users with biomedical expertise signals in their profiles are perceived to have greater expertise on COVID-19 vaccines [Jalbert et al.]. Users that changed their profiles over the course of the study in a manner that changed whether they might be perceived as an expert (24 accounts) were excluded from the following analyses.

5.4 Determining community stance

To understand the stance of individuals in the two largest communities, we analyzed tweets that received retweets during April 2021. Of the 251,040 tweets satisfying those criteria, we tagged a subset posted by the ten perceived experts and perceived non-experts in both communities with the greatest degree centrality. We collected the ten most retweeted tweets by the forty selected users. A few of the users had fewer than ten tweets that were retweeted during the study period, so we instead retrieved all of their tweets that received retweets in April. In total, we reviewed 392 tweets.

For each tweet, three coders assessed stance toward COVID-19 vaccines as positive, negative, or neutral. Stance was evaluated as positive if the tweet supported vaccination, including by providing evidence that COVID-19 vaccines are safe and effective. Tweets expressing the opposite view were coded as negative. Stance was often implicit. For example, calls for deployment of vaccines to “hotspots” with elevated disease burden suggest that the author believes the vaccine can prevent disease, even if this rationale is not stated directly. Unless the author refuted or negatively reacted to a quote they provided, a tweet they quote tweeted, or an article they linked, the post was assumed to take the same stance as referenced sources. Tweets containing multiple contrasting viewpoints or providing information without a clearly implied position (e.g., sharing statistics about the pace of vaccination without any additional commentary) were coded as neutral. The neutral tag was also applied to tweets if the coder was uncertain about the argument being made or context was missing (e.g., criticism of an article that could not be retrieved). We noted many tweets during this time period that opposed vaccine mandates and coded these tweets as neutral unless the author justified their position using a specific argument regarding vaccine safety or efficacy. Tweets comparing multiple types of vaccines with negative stances toward some and positive stances toward others were tagged neutral. Tweets that took a uniform stance toward different types of COVID-19 vaccines while explaining their differences were coded based on the corresponding stance. Based on this analysis, we determined that negative stances toward COVID-19 vaccines were prevalent in tweets from one community, which we refer to as the anti-vaccine community in following analyses (Supplemental Figure 8). The other community, in which tweets mainly expressed positive stances, was the pro-vaccine community that served as a basis of comparison.

5.5 Academic and low-quality link-sharing

We assessed the types of sources users referenced in their tweets by comparing links shared to databases of: (1) low quality sources and (2) academic research journals and pre-print servers. We expanded shortened URLs using the RCurl package in R [Temple Lang, 2023]. In some cases, this process timed out or links connected to other content shared on Twitter, in which case links were excluded from the following analysis. Across community and perceived expertise, we compared the average number of tweets per user, proportion of tweets with links, and proportion of links that were checked, and number of links that were checked to ensure there was minimal risk of differences in link-sharing behavior leading to bias in the following analyses (Supplemental Figure 9). We checked the remaining 52,073 links, first determining whether the domain name was rated “low” or “very-low” quality by Media Bias/Fact Check according to the Iffy Index of Unreliable Sources [Golding, 2023, Van Zandt], a common proxy for misinformation sharing [Sharma et al., 2021, DeVerna et al., 2022]. To assess sharing of academic research, we also checked links to research publications [Newbold et al., 2022] and pre-print servers used in biomedical and medical sciences [Kirkham et al., 2020]. For perceived experts and perceived non-experts in each of the two large communities, we calculated: (1) the proportion of assessed links from low quality or academic research sources and (2) the proportion of total users that shared at least one link from a low quality or academic research source in their original posts during April 2021. For each metric, we calculated the binomial proportion 95% confidence interval as where p^ is the observed proportion and n is the total links or users in a given category. To compare proportions for different categories of users, we conducted two proportion one-tailed Z-tests with Yates’ continuity correction to account for the small number of links from low quality sources shared by the pro-vaccine community.

5.6 Network centrality and bridging metrics

We calculated degree, betweenness, and eigenvector centrality to describe network position using the igraph package in R [Csardi and Nepusz, 2006] (see Supplemental Table 1 for a description of centrality metrics and their interpretation). Specifically, we calculated eigenvector centrality using the PageRank algorithm. We additionally detected nodes with audiences spanning the anti- and pro-vaccine communities using a community bridging index calculated as the minimum number of edges that a node has linking it to either of the two communities.

We tested whether perceived experts were overrepresented within the group of highly ranked users by each metric by calculating the proportion of perceived experts in the top n users ranked by a given metric (where n varies between 500 and 50 for all metrics and ten for bridging score). Several perceived experts and perceived non-experts had the same bridging scores, leading to ties for ranking in the top 500 and 50 bridges. In these cases, we randomly drew 1,000 samples from the tied users without replacement to complete the set of top bridges and calculated the mean proportion of perceived experts across all samples. To calculate the 95% confidence interval in these cases, we added the margin of error to the .025th and 97.5th percentile proportion value across the 1,000 samples. For the remaining metrics and subsets, we calculated the binomial proportion 95% confidence interval as in the previous section except when examining the ten top bridges, in which case we calculated the Wilson score interval to account for the small sample size.

To test whether perceived experts were significantly overrepresented in each sample of highly ranked individuals, we conducted two proportion Z-tests with the alternative hypothesis that perceived experts were a significantly greater proportion of the top n individuals than individuals in the complement (i.e., individuals who were not among the top n individuals ranked by a given metric). When considering the top ten bridges, we instead used a Fischer’s exact t-test to account for the small sample size. We reported p-values for significance tests.

5.7 Matching to assess relative influence of perceived experts

This portion of the analysis was pre-registered on OSF Registries prior to hypothesis testing (https://osf.io/6u3rn). We assessed how perceived expertise affects influence using propensity score matching, a technique to select a sample of individual perceived non-experts with a similar covariate distribution to perceived exerts in the dataset [Rosenbaum and Rubin, 1983]. We can then compare the average influence between perceived experts and perceived non-experts while adjusting for confounding from covariates that may predict being a perceived expert without being a direct consequence of perceived expertise. Propensity score matching has been applied previously to compare the effectiveness of different users in the context of online advertising [Yan et al., 2018, Kuang et al., 2018] and to understand whether following prominent public health institutions on Twitter impacts vaccine sentiment [Rehman et al., 2016]. Propensity score matching was performed using the MatchIt package in R [Ho et al., 2011].

We matched on the following covariates: natural logged follower count at the beginning of April 2021, total on-topic posts during the study period (i.e., posts included in our data set because they contained a COVID-19 vaccine keyword), account creation date, whether the account is verified, percent of on-topic posts containing links, percent of on-topic tweets that were original posts versus retweets, posting time of day, and uniformity of posting date across the study period (Supplemental Table 2). For the first two analyses focused on relative influence within the anti-vaccine and pro-vaccine communities, we also matched on subcommunity assignment based on the Infomap community detection algorithm Figure 10. We used logistic regression to compute propensity score as the predicted probability that each user is a perceived expert given their covariates. Based on propensity scores, each perceived expert was matched with replacement to their three nearest neighbor perceived non-experts. To test the first two hypotheses about influence within the anti-vaccine and pro-vaccine communities respectively, we included only users in either community. To test the third hypothesis comparing the influence of perceived experts in the antiversus pro-vaccine community, we calculated the interaction of perceived expertise and community. Sample sizes for all analyses are provided in Supplemental Table 5.

The outcome variables for influence are: eigenvector centrality, degree centrality, betweenness centrality, natural logged median likes, natural logged median retweets, h-index for likes, and h-index for retweets (Supplemental Table 3). h-index is the maximum number h such that the user wrote at least h original tweets (including replies and quote tweets) that received at least h likes or retweets. We calculated engagements (likes and retweets) for each tweet as the maximum number observed for that tweet in our collection within twenty-eight days of when it was posted. Importantly, we only retrieved like counts for a given tweet when another user engaged with the tweet through a quote tweet or retweet.

We then estimated the effect of perceived expertise on influence by calculating the average treatment effect on the treated (ATT), which corresponds to the difference in expected influence ((Y (T)) with and without perceived expertise given that a user is a perceived expert (T = 1) with covariates drawn from the distributions for the matched users (X):

ATT was estimated using the marginal effects package [Arel-Bundock, 2023]. We tested the robustness of our results to different matching parameters and covariates, as displayed in Supplemental Figure 22, Supplemental Figure 23, and Supplemental Figure 24.