Effectiveness of digital health interventions against COVID-19 misinformation: a systematic realist review of intervention trials

Study characteristics

26 papers were found that met inclusion criteria, including 6 out of reviewing the bibliographies of the 20 studies found through the search strategy described above. 636 were initially resulting from the searches, with 230 duplicate results removed, 341 deselected by title, and 45 deselected by full-text review, resulting in 26 eligible papers (Figure 1). Papers were published between 2020 and 2023, with a variety of national, regional, and international participant groups and study origination countries. The papers reviewed utilised participant groups coming mainly from the USA through private research participant companies like MTurk, Lucid, Prolific, Pollfish, and YouGov but also targeted audiences within the US like essential workers [10] and ‘Latinx’ communities [11]. Beyond the US, participant groups from Germany, the UK, Hong Kong, China, Canada, the Netherlands, Brazil, Kyrgyzstan, India, and internationally were included in the reviewed studies. These studies split into the intervention framework developed in the data extraction process as follows: 6 studies using Accuracy Nudges; 6 using education; 3 using Prebunking; 3 using Games; 3 using message framing; 3 using Community Engagement; and 2 using Debunking (Table 1). Full details of the studies can be seen in the appendices.

All eligible studies underwent quality assessment using Kennedy et al.’s [12] risk of bias tool for assessing study rigor, results are shown in Appendices Table 1.1. Many studies lost several points due to lack of follow-up elements or not giving information on whether comparison groups were equivalent on demographics or baseline outcome measures. Iles et al. [13] and Maertens et al. [14] stand out as the only perfect scoring studies, with Veletsianos et al. [15] on the other side scoring only 1/8 as the lowest score of the assessed studies. When sorted into intervention-types, the average quality scores are relatively similar for each group, indicating a similar level of quality across the intervention-types.

Intervention characteristics

The studies in this review tested interventions with far greater heterogeneity than the dominant interventions proposed before the COVID-19 pandemic (accuracy nudges and fact-checking). As can be seen above in the study characteristics table, the studies were iteratively sorted into intervention-types as laid out in the methodology section. These intervention-types included: accuracy nudges, community engagement, debunking, prebunking, education, games, and message framing. This section will briefly introduce these intervention types and their defining characteristics.

Accuracy nudges in the reviewed studies consisted of mechanisms including: stimulating attention [16], source credibility labels [17], logo banners to help identify trustworthiness of sources [18], accuracy reminders [4], and tags that mark information as false [19]. These various intervention mechanisms fit under accuracy nudges due to their common characteristics as simple, fast, attention-grabbing labels or reminders that ‘nudge’ the participant to consider information veracity and bring that consideration into the forefront of their minds immediately before reading the information.

Community engagement is difficult to characterise by intervention mechanism because the defining aspect of community engagement occurs before intervention mechanism is determined in the research design. Instead of pre-determining intervention mechanisms and delivery methods, community engagement involves co-creation of the intervention alongside and in collaboration with the targeted community, to be bespoke to the unique context and circumstances of the community [10, 11, 20].

Debunking refers broadly to reactive interventions (e.g. fact-checking) that seek to ‘debunk’ existing misinformation and help people exposed to it rethink their belief and formulate new understandings of the relevant information [21, 22]. In contrast, ‘prebunking’ interventions seek to build resilience to misinformation in people preemptively before exposure has occurred, and potentially even before the piece of misinformation has been created/spread [23]. This typically takes the form of inoculation messages administered to participants before exposure to potential misinformation. In this way they are similar to accuracy nudges – the key difference being that prebunking is more extensive than accuracy nudges. The inoculation messages are more significant, take longer to process, and are intended to take the full attention of the participant for the duration of the message, whereas accuracy nudges are fast and often involve the periphery of a participant’s attention. In the reviewed studies characterised as prebunking, all three involve inoculation messages as their intervention mechanism [24, 25, 26].

Education is the most heterogeneous of the intervention-types and can be difficult to categorise as educating the participant is essential to all interventions working to address misinformation. In the reviewed studies, this intervention-type involved mechanisms such as: videos [27], comics [15], infographics [28, 29], and a multimodal intervention using authentic social media messaging [30]. The defining characteristic of the reviewed studies in this intervention-type is the primacy and exclusivity of education as the goal of the intervention. For instance, in Vandormael et al. [27], an educational video was released and distributed internationally with the goal of maximising viewership, but with no additional features of the intervention beyond watching the video.

Game intervention-types are characterised by the inclusion of a computer game for participants to play as the primary intervention-mechanism. This can be seen in all three of the included studies under this categorisation. These games inform players (participants) on the tactics and manipulation used to create and spread misinformation, with the goal of creating an inoculation effect and helping bolster veracity-judgment in participants. For example, Bad News is the name of the game used in Maertens et al. [14], a popular game used in many studies outside the purview of this review as well. In this game, players take on the role of an antagonist, creating misinformation and working to spread it through social media and the internet.

Message framing as an intervention-type is characterised by the use of psychological framing in the development of the language used in the intervention. Whether presenting information via video or written information, what distinguishes these studies as message framing is their strategic use of language to attempt to make their information transfer to participants as congruent with their extant worldview as possible. This then helps participants internalise that information effectively and can address intervention design concerns around potential backfire effects.

Intervention effectiveness

The two variables most central to answering which interventions work ‘best’ appear to be scalability and impact. If impact is too low, the intervention might not actually engender sufficient behavioural change in participants to combat the misinformation. Similarly, if an intervention cannot be upscaled, it has no capacity to address COVID-19 misinformation at a systemic level. The ‘ladder’ visuals below represent the intervention-types relative to one another across these two variables (Figures 2 & 3). Relative impact is determined by the measured impact on participants in each study. These measures are not consistent, yet with the authors’ interpretations, comparisons are possible. These relative measurements focus on the impact per participant, with no regard to number of participants or scalability. Inversely, the scalability ladder visual focuses on scalability with no regard to impact per participant. These visuals are meant to simplify and ease understanding of the results, and are purely relative.

• Download figure
• Open in new tab

Figure 2: Ladder of Impact on Participants.

• Download figure
• Open in new tab

Figure 3: Ladder of Scalability.

On the bottom rung of impact per participant is accuracy nudges, whose impact is heavily debated. Some authors in this review such as Pennycook et al. [4], champion this intervention type and claim significant impact in their results. Gavin et al. [8], who replicated Pennycook et al. [4], found mixed results that stood at odds with the original study. Amin et al. [16] found impact on decision behaviour and tendency to share misinformation with their study, but the rest of the studies in this intervention group found either minimal impact [19], only impact on certain groups [17], or no impact [18] who even noted potential counterproductivity.

Framing of public health messages is next along the impact ladder. Studies testing interventions using different framings of public health messaging found significant impact [13, 31, 32], although not as high as other intervention-types included in this review. This impact was largely reserved for those heaviest consumers of misinformation and those most vaccine-hesitant [31].

Debunking by its nature must occur retroactively, which limits impact as the initial exposure must be overcome. In this way, debunking has two independent goals: to both disprove internalised misinformation and convince the participant of the veracity of legitimate information. This is a barrier to impact, which is noted by both Vijaykumar et al. [21] and Yousuf et al. [22]. Vijaykumar et al. [21] found no impact on perception or willingness to share misinformation yet found enhanced credibility and readiness to share accurate information because of their intervention. However, Yousuf et al. [22] found that exposure to their intervention did result in enhanced trust in government and significantly stronger rejection of vaccination misconceptions.

Prebunking, as the preventative version of debunking, scores better on impact. Prevention is found to be more powerful in a variety of aspects than reactive debunking. All three included studies [24, 25, 26] found significant impact among participants, although in the case of Amazeen et al. [24] this significance was limited to those with preexisting ‘healthy’ attitudes. Impact on participants was found to include generating resilience against misinformation, less willingness to share misinformation, and greater willingness to receive a vaccine.

Education is the most varied type of intervention with a range of impact between the individual tested interventions (the relative score here is an aggregate). At best, educational interventions have the potential to be a form of systematic prebunking with great effectiveness. In the reviewed studies, they were found to improve knowledge and increase resilience to misinformation at significant levels, particularly among populations with low preexisting knowledge levels [27, 30]. However, Van Stekelenburg et al. [29] found no significant impact, highlighting the variability of this intervention-type.

Games were consistently found to be highly impactful across the various populations who played them, with high levels of durability and longevity compared to other intervention-types reviewed and significant impact levels for all kinds of preexisting attitudes towards vaccination and COVID-19 [14, 23, 33]. Every reviewed study found significant impact, which corresponds with a high relative impact score, although still below the bespoke and prolonged interventions within community engagement.

Community engagement is the single most impactful intervention-type, with sustained interaction and bespoke interventions to specifically targeted communities who then themselves are brought into the intervention process and invited to participate, make their voices heard, and have their concerns addressed in a bespoke, personal, and trusted manner. All reviewed studies found significant and extensive impact among their participants.

Community engagement is essentially impossible to scale upwards. It inherently requires small numbers of participants and high levels of resource and time investment by those implementing the intervention. The interventions themselves are then not even intended to be generalisable, but rather bespoke to and befitting the contextual needs of the community involved. Community engagement can only effectively be done at a small scale over long periods of time involving the building of trust with community, the proactive engagement and co-creation of interventions and implementation strategies with the community itself, and the implementation strategies themselves can take years to accomplish [10, 20].

Debunking scores quite low in scalability. Debunking at a large scale is extremely difficult as it is inherently based on preexisting misinformation and cannot effectively prevent additional misinformation. Further, it must always attempt to reach those specific populations initially exposed to the targeted misinformation to be ‘debunked’, which is difficult and resource-intensive.

Prebunking does not need to attempt to find and target those who already saw some misinformation as the intervention occurs before misinformation is seen. For this reason, prebunking is easier to scale upwards than debunking and is relatively lower in resource-cost. Implementation of prebunking involves the development of ‘inoculation messages’ [24, 25] as written messages or video content intended to raise resiliency of participants against misinformation.

Without being built into the public education system, educational interventions may struggle to scale upwards, relying on peer educational champions [34] or social media ‘virality’ to spread [27]. Adjusting anything within the public education system is highly resource-intensive, even though those changes are then highly impactful and wide-reaching. However, when performed in smaller scale as in the included interventions, educational interventions can be substantially reduced in resource intensity [27].

Although not as easily scalable as message framing or accuracy nudges, games are nonetheless relatively highly scalable when compared to the other intervention-types in this review. As the games are already developed, introducing them to new populations is then relatively simple, resource-inexpensive, and quick.

Message framing has high scalability with the simple addition of language strategising and purposeful narrative framings applied to extant and new public health messaging. Message framing is only slightly more resource intensive than accuracy nudges in that it must be bespoke to particular narratives, communities, and groups. However, in each bespoke circumstance, still the resource intensity would be low.

Accuracy nudges is undeniably the highest scalability intervention-type. The core reason why accuracy nudges are so scalable is the extremely low resource intensity needed to implement them. It requires the insertion of nudges in social media feeds and news articles. This would be easy and inexpensive for social media corporations and newspapers to implement, even when scaled into the extreme levels of interaction and users involved in contemporary social media.

Durability of effect

In the studies that did test for longevity/durability of impact in their tested intervention, consistent low levels were found, with findings indicating high reliance on intervention repetition and regular testing of misinformation resilience over a sustained (and potentially indefinite) period to reach functional durability of effect. The study that looked most closely at this was Maertens et al. [14] which performed one of the only longitudinal studies included in this review explicitly investigating longevity of impact using the ‘Bad News’ game as its chosen intervention. They found that their intervention resulted in a significant increase in ability to discern misinformation with lasting effects if regular misinformation resilience testing occurred over time. Without regular testing they found significant decay over a 2 month period ending in a loss of inoculation effect [14].

Special groups and circumstances

There appears to be a significant distinction in how these interventions work between those with preexisting ‘healthy’ understandings of public health information and those who are the heaviest consumers of misinformation. This was noted in several studies ([17, 18, 19, 23, 24, 28, 32], and in ways that do not initially appear congruent with one another. It is clear this subgroup of heaviest misinformation consumers is impacted differently by many of the interventions included in this review, but that change in impact is not a consistent factor - instead it is an ephemeral variable, difficult to spot and even harder to plan for in study design.

The table below lays out the contexts in which each relevant intervention type was found to be most effective in the groups tested, alongside the groups included within the included studies, relevant findings from the authors regarding context and their intervention, and an overall level of generalisability (Table 2).

Eight reviewed studies found insignificant trends in intervention impact between baseline participants and special groups, with several more looking for such trends and finding none. This indicates the specificity of these intervention-types, and that although context and social group could be determinants of intervention effectiveness, such effects are likely to be small. For example, Bender et al. [32] noted that their intervention framing worked best on those already strongly anti-vaccine. Conversely, Johnson et al. [30] found their intervention worked best on those with less vaccine hesitancy, and that those with higher social political conservatism performed worse on knowledge scores. The insignificant trends found in these studies were typically tied to either age, ethnic group, or political ideology as core identities tied to perceptions and experiences of COVID-19 and the public health responses thereto. Political (rightwing/conservative) ideology was noted in many studies as a subgroup of particular importance and was found to coincide with less accurate pre-intervention beliefs [23, 28].

Accuracy nudges were tested with participants from the USA, Kyrgyzstan, and India, with findings that suggest that their impact is difficult to predict and changes depending on the context [8]. Dias et al. [18] noted the potential for a ‘backfire’ effect among those people most bought-in to misinformation, whereas Kreps et al. [19] found no evidence of this effect. Aslett et al. [17] find that their intervention only worked on those who consume the highest levels of misinformation in their participant group and had minimal effect on anyone else. This conflicts with concerns about backfire effects.

Prebunking was tested with participants from online recruiters in the USA and Hong Kong undergraduates. Amazeen et al. [24] found that the intervention only worked on those with preexisting ‘healthy’ attitudes, meaning those whose beliefs already coincided most closely with legitimate public health messaging. Because this intervention-type is intended to inoculate the ‘average’ person against misinformation, it only working on those with preexisting ‘healthy’ attitudes does not reduce the usefulness of prebunking.

Games were tested in the USA and China with proportionally representative online groups. Basol et al. [23] as well as Ma et al. [33] respectively found that the interventions worked across both the political spectrum and the public in general. This indicates high generalisability, particularly with the proportionally representative and relatively large participant cohorts in these studies. However, by the nature of a digital intervention type like games, older people and those with low levels of digital literacy (who are among those most desirable to target for the intervention) may have less desire or ability to play the game.

Debunking was tested in the UK and Brazil among Whatsapp users, and in the Netherlands among the elderly. Interestingly, Vijaykumar et al. [21] found that their intervention was most effective on older people. This indicates that this type of intervention might be most useful among elderly populations and communities. Vijaykumar et al. [21] and Yousuf et al. [22] speculate that perhaps older people have higher baseline trust in governmental messaging and are therefore more open to changing their internalised beliefs based on new information from legitimate sources. By the nature of debunking, it can only be applied reactively to widely believed misinformation, which significantly limits its generalisability.

Education was tested in the USA, Canada, Hong Kong, and internationally through social media sharing. Johnson et al. [30] found their intervention worked best on elderly people and those with less hesitancy around COVID-19 vaccination. Similarly, Veletsianos et al. [15] found that their intervention caused a noteworthy ‘backfire’ effect among conservative US republicans (as the most vaccine-hesitant and ‘bought-in’ to misinformation already). Vandormael et al. [27] suggested educational interventions might be most effective among populations with a low baseline knowledge level, as their own participant group has relatively high levels of baseline knowledge (although nonetheless the intervention successfully boosted knowledge of COVID-19 prevention). When taken together, these findings indicate that the groups most ideal for this type of intervention are communities with low baseline knowledge of public health information or communities distrustful of government where peer and individual study might be able to penetrate that distrust.

Message framing was tested in Germany [32], the US [13], and the UK [31] all through online interventions testing framed messaging against traditional extant public health informative messaging. Bender et al. [32] found that extant framing (which typically focuses on collective benefits and informing about vaccination side-effects) worked best for those anxious about vaccination, whereas the intervention framing worked best for those strongly anti-vaccine. Similarly, Freeman et al. [31] found that emphasising personal benefit (the intervention framing) was more effective on the most vaccine-hesitant, whereas emphasising collective benefit (the control/extant framing) was far less effective and even resulted in ‘backfire’ effects. Together these findings make a strong case for message framing interventions to effectively target those communities most distrustful of government messaging, those most ‘bought in’ to conspiracy and misinformation already, and the most politically radicalised.

Community engagement was tested in the US among ‘Latinx’ communities [11], young Black adults [20], and ‘essential workers’ [10]. By its nature, community engagement is very low generalisability as it is more contextually specific, resource intensive, and time-consuming than any other intervention type. Degarmo et al. [11] found their intervention was successful at mitigating health disparities in the communities they engaged. This suggests community engagement would be most effectively utilised in deprived communities, vulnerable communities, and those areas most difficult to reach for any reason.

Politics and partisan bias

Both the theoretical and intervention literatures around COVID-19 misinformation hint at its politically polarising elements yet fail to address this influence head on. Dispersed throughout the findings and discussions of the included studies are the political elements of COVID-19 misinformation. It is consistently found that political conservatives, particularly in the US, are uniquely vulnerable and bought-in to misinformation and conspiracism [7, 35]. This group was found to have its own unique interactions with many of the tested interventions in this review. When this happened, the authors mention this difference and give some speculation as to why that might be the case, but do not investigate this finding further, or seek to use explanations in the wider literature to support their findings (see [29] for the most comprehensive discussion of this issue in the eligible studies). Additionally, there has been very little work to explicitly begin from this starting point and deep dive into why this might be the case and how interventions might most effectively impact this group. This presents a significant detriment to reaching the stated goal of these interventions - effectively combatting COVID-19 misinformation.

Pennycook et al. [4] is the most influential study included in this review in terms of citation count, references throughout the reviewed studies, and the extent to which their study has been replicated and critiqued within both the studies under review and the wider literature. Within that study they champion the theory that the systemic sharing behaviour of COVID-19 misinformation in our society is “because [people] simply fail to think sufficiently about whether or not the content is accurate when deciding what to share” [4, p. 770].

Pennycook et al. claim that their findings and this theory indicate that accuracy nudges are not only simple and effective, but the only intervention needed against COVID-19 misinformation. In doing so, they negate the claims of many of the other included studies in this review. This has brought significant criticism against this core idea of what is causing vulnerability to COVID-19 misinformation. If the only issue is a lack of thinking, then accuracy nudges are the obvious intervention. Yet although the findings of Pennycook et al. [4] do suggest the effectiveness of accuracy nudges and the need for interventions that make people think more about their sharing decisions, this ‘theory’ they promote is insufficiently supported when applied to negating the findings of other studies. Their findings suggest the effectiveness of accuracy nudges, but not the ineffectiveness of other interventions. The alternative proposed answer to what is causing vulnerability to misinformation is partisan bias. This explanation posits that it is not failing to think sufficiently or lower cognitive ability that leads to vulnerability to misinformation, but rather the inherent bias that arises from adherence to political ideology in the context of intense political division and polarisation as is affecting the contemporary United States very deeply but also affects many countries today [36]. This debate on partisan bias vs insufficient thinking punctuates the literature on misinformation, including many of the studies included in this review.

Limitations

A primary limitation in this review comes from the heterogeneity of the studies and interventions disallowing meta-analysis and other forms of traditional systematic review analysis that rely on similar outcome measures and methodologies within the eligible studies. This limitation is accentuated by the potential for interpretation bias. The interpretation of the data herein is biased by the perspective and worldview of the authors.

Additionally, there is limited consistency between realist reviews and limited standards and assessments available to apply to this review. This does not necessarily limit the rigor of the review but makes analysing that rigor and validity more difficult. The development of more and consistent direction and assessments for realist reviews would address this limitation currently present within the method. Lastly, the limited engagement in the intervention literature with theory limits the extent to which theoretical insights can be drawn from this study.

Future Research Directions

Although a variety of interventions tested in the studies herein found success in the short term, in the long-term it is impossible to avoid the urgent need for mass-scale education on digital literacy if the goal is to make a population as resilient as possible against misinformation. Future research in this direction is pivotal, with experiment-groups in classrooms a clear next step. Additionally, future research on how to address the political difficulties in implementing such a wide-scale intervention is required.

Out of all intervention-types reviewed, games appear to create the highest impact while still being highly scalable and resource-inexpensive, with the potential for longevity in the right conditions [14]. Relative to the other intervention-types, games scores maximally in terms of impact on participants, while still being relatively high on scalability. Future research in this direction is needed to refine and test these results. Longitudinal testing is an obvious follow-up to gain insight into durability of inoculation effect.

Additional areas for future research include: 1) theoretical research into how to build a resilient population and how to address vulnerability to misinformation systemically versus individually; 2) the role of politics and partisan bias in the functioning of these interventions; 3) where misinformation comes from and who gains from it; 4) the role of political polarisation and radicalisation in vulnerability to and the spread of misinformation.

Conclusions

This review included 26 studies of interventions combatting COVID-19 misinformation. The interventions reviewed varied widely in terms of scalability, resource intensity, impact on participants, the contexts within which each best works, the people onto whom the interventions will have greatest effect, and research quality. The tests performed in the included studies hold rich contributions toward better understanding how misinformation functions, how veracity judgement occurs in individuals and communities, and which interventions work best in which contexts and for whom. COVID-19 showed precisely how harmful and deadly misinformation can be, and what a public health threat it can represent. In this fight against systemic misinformation in our society, a final takeaway from this review is the need for acknowledgement of misinformation as a societal and systemic issue that requires significant investment and time to resolve, if resolution is possible.

Search strategy

This review included a systematic search of Web of Science, Scopus, ASSIA, Psycinfo, and Pubmed to identify English language articles written between January 1, 2020 and June 22, 2023 performed following a pre-registered protocol conforming to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement [37]. No secondary searches were performed. Search strategy followed the protocol using pre-determined search terms, with results imported into Excel sheets for ease of deselection. Duplicates were removed and then an initial title-based screening was performed. Screening then followed based on abstract and then full-text review. Additional searching among the references of the included studies followed. Duplicate screening was performed by a team member (K.G.) on ∼15% of studies through all screening stages, with any disagreement resolved via discussion. Inter-rater agreement was found to be very high (∼92%).

The full search-string chosen for this review, which was only applied to Titles and Abstracts, is as follows: (conspirac* OR anti-vax* OR anti-vaccine OR ‘anti vaccine’ OR misinform* OR fake OR fals*) AND (messag* OR rumor* OR argu* OR rhetoric OR spread*) AND (COVID OR COVID-19 OR coronavirus OR ‘corona virus’ OR pandemic*) AND interven*

Eligibility

Trials or experimental studies were eligible if they were focused on reducing the spread of and vulnerability to COVID-19 misinformation in their participants, and tested an intervention meant to combat COVID-19 misinformation. Studies were required to be in the English language and been published between 2020 and 2023 as searching before the COVID-19 pandemic began was unnecessary.

Quality assessment

The methodological quality of each study chosen for inclusion was assessed via Kennedy et al.’s [12] risk of bias tool for assessing study rigor. It includes eight items for appraisal: (1) cohort, (2) control or comparison group, (3) pre-post intervention data, (4) random assignment of participants to the intervention, (5) random selection of participants for assessment, (6) follow-up rate of 80% or more, (7) comparison groups equivalent on sociodemographics, and (8) comparison groups equivalent at baseline on outcome measures. This assessment tool was used for its flexibility regarding type of methods and interventions in the studies being assessed. Although this analysis was performed, no studies were excluded due to quality as realist reviews explicitly disagree with exclusion from quality concerns as explained below.

Data extraction and analyses

The following information from included studies was extracted into a table to highlight study characteristics as can be seen in the next section: Study, intervention-type, ‘working ingredient’, ‘delivery method’, country of origin, methodology, number of participants, and whether the intervention was found to be successful.

Additionally, a variety of other information was extracted to inform the other tables and charts found in the results section. All text from the eligible studies was imported into NVivo Pro 14 and the methods, results, and discussion sections underwent qualitative coding. Coding was done iteratively to categorise the findings. This iterative process evolved into a developed framework as the coding took place. For instance, if one intervention was identified from an article during coding, the coder attempted to assign it to a category within the emerging intervention framework. New subcategories were created if the current categories were insufficient, until all interventions were categorised. As coding progressed, the intervention framework came to be populated through the included studies. The heterogeneity of the included studies and their respective measures disallowed quantitative meta-analysis.

Regarding effectiveness, impact per participant and scalability were the primary variables analysed. Impact per participant refers to the level of individual behavioural change experienced by the participants of each intervention reviewed, as all were centred on individual behaviour. Scalability is a more complex variable consisting of several combined factors including generalisability (how effectively can results be replicated in other contexts and with other groups), resource-intensity (how expensive is the intervention in terms of time, money, and overall resource expenditure), and capacity for upscaling (how many people it could reach). With impact and scalability thus defined, effectiveness can be then analysed by how many people could be impacted and to what extent per person. A sub-analysis of context was undertaken by comparing context by intervention type, and laying out which participant groups were targeted by the interventions. Additional analysis was performed to investigate context beyond the community of participants within the intervention.

It is important to define ‘circumstances’ as used in RQ3. Here, circumstances refers to the context within which an intervention is taking place - such as geographic location, identities and wealth of the targeted community, and structural and institutional factors within which the community and intervention will take place. Additionally, circumstances refers to the experience, resources, and capacity of the research team or implementing body performing the intervention.