The utility of infectious disease modelling in informing policy for outbreak response: a scoping review

Context: Infectious disease modelling as a tool for outbreak response

Policymakers in public health face complex decisions during infectious disease outbreaks, such as AIDS, Ebola, Mpox and COVID-19. These decisions often involve balancing multiple competing priorities under significant uncertainty, including the timing and scale of interventions, resource allocation, and the social and economic impacts of control measures (e.g. (1)). This complexity underscores the critical need for effective decision-making tools. Mathematical modelling has emerged as an important tool in this context, often referred to as ‘infectious disease modelling’ (e.g. (2)). It supports outbreak response in multiple ways: Modellers can for example estimate key epidemiological parameters, forecast outbreak size and assess the potential impact of different control measures (e.g. (3, 4)). Thereby, modelling translates complex epidemiological data into actionable insights for decision-makers as highlighted by its use during outbreaks of e.g. A/H1N1 influenza, Ebola and COVID-19 (e.g. (3, 4)).

Context: Challenges in the use of infectious disease modelling for outbreak response

However, there is extensive debate about whether these models could be trusted and used effectively to guide policy choices (e.g. (1)). Especially when there are conflicting results, models face criticism from politicians, the press, experts from other disciplines and the public (e.g. (3)). These tensions sometimes cause potentially avoidable delays in implementing measures and makes modellers apprehensive to further contributing to outbreak response (e.g. (5)). Hence, it can be challenging to utilise modelling successfully for outbreak response.

Context: Need for a better understanding of the use of infectious disease modelling for outbreak response

These challenges highlight the need for continued efforts in enhancing how the results of infectious disease modelling for outbreak response are used, as recognised by the ‘Lancet Commission on Strengthening the Use of Epidemiological Modeling of Emerging and Pandemic Infectious Diseases’(6).

Rationale: Need for a scoping review

Previously, a scoping review on evidence use for decision-making during infectious disease outbreaks was conducted that tangentially considers the role of modelling evidence (7). However, as it lacks a detailed perspective on modelling, focuses on Europe and considers records only from before the COVID- 19 pandemic, an updated more comprehensive overview is needed.

To improve the utility of modelling for policy, it is necessary to identify typical characteristics, facilitators and challenges of the modelling to policy pathway. Such mapping is particularly timely given the ongoing need to strengthen the science-policy interface in outbreak response, such as for the ongoing Mpox epidemic.

At the time of writing, there are two other projects similar to ours: First, there is a pre-registered scoping review focused on the question ‘What defined policy-relevant advanced analytics over time during the COVID-19 pandemic in Europe?’ (8). Second, there is a study that was published after we had already conducted and pre-registered our search and screening; it also contains a scoping review focusing on the effective translation of ‘modelled evidence into policy decisions’ in Lower- and Middle-Income Countries in the context of the COVID-19 pandemic (9). We complement this work by conducting a scoping review that is not limited geographically and also considers diseases beyond COVID-19.

Questions and objectives: What has been the utility of infectious disease modelling in informing policies for outbreak response?

This review examines how mathematical modelling has informed policy decisions during infectious disease outbreaks and epidemics. We intended to investigate what types of infectious disease models have been used and how they have been utilised in policy decision-making, with particular focus on evaluations of how useful models were in informing disease outbreak control or mitigation efforts.

Our analysis addresses several interconnected questions about modelling-to-policy translation: What characterizes modelling-to-policy processes? What are the facilitators and barriers for successful integration of modelling evidence into policy? We examine publications that reflect on or evaluate modelling engagement with policies for outbreak response, including who has studied these questions, in what contexts, and what recommendations emerge for improving modelling-to-policy translation.

Eligibility criteria

The eligibility criteria are based on the PCC framework (population, concept and context), as suggested by the Joanna Briggs Institute (12). The detailed inclusion and exclusion criteria by categories as well as types of evidence sources are presented in Table 1.

Search strategy

To define the search components, we adapted the CoCoPop scheme (Condition, context, population) to CoCoPop +E to include the evaluation component (13). We searched the database Ovid (Medline, Embase) on June 5 2024 using the search strategy depicted in Supplementary Figure 1. The rationales for the details of the search are listed in Supplementary Table 1.

Following the search, all identified citations were collected and uploaded into EndNote X7 (Clarivate Analytics, PA, USA) and duplicates were removed. Titles and abstracts were screened by the research team for assessment against the eligibility criteria for the review. Additional literature was added through expert judgement in two ways: (a) The senior authors of this review suggested relevant additional publications and (b) the selected literature of a similar scoping review (9) was considered. These additional publications were also checked against the eligibility criteria and included/excluded accordingly.

Article selection

The full text of selected citations was assessed in detail against the eligibility criteria by the research team. Two reviewers, AT and DR, independently screened all articles for inclusion. Disagreements were resolved by additional reviewers.

Data extraction

A data extraction table (via Google Sheets) was developed and updated in an iterative way, resulting in five broad and 21 sub categories including their definitions (Table 2). The table updates comprised clarifications of definitions as well as re-conceptionalisations and re-clustering of categories to achieve better coherence.

Data from each included publication was extracted independently by two reviewers, respectively. The involved reviewers were AT, DR, and ENI. In case of immediate agreement, the wording of the first reviewer was chosen. Complementary data was added, where necessary. Any arising disagreements were discussed and resolved between the two reviewers. Where the data extraction for certain publications was deemed difficult by the reviewers, help was sought from additional co-authors.

Synthesis of results

The purpose of this scoping review is to map the research done on the utility of infectious disease modelling in informing policy for outbreak response, as well as to identify any existing gaps in knowledge on this topic. As we did not find objective assessments of utility to include into our review but subjective reflections by practitioners, we instead summarised aspects of the modelling to policy process that are claimed to be essential for the utility by the authors of the included publication. We performed a descriptive qualitative content analysis analysis and synthesis of the data extraction results along the topics that we chose in our data extraction table.

Article selection

Out of the 4007 articles identified through database search via 3 databases 3965 were screened for inclusion after removal of duplicates (Fig 1). Out of the title and abstract screened articles, the full texts of 42 articles were screened against the eligibility criteria and 26 of them were included in the review. Full texts of additional 12 articles suggested based on expert judgement were also screened against the criteria and 7 of them were included in the review. Data was extracted from all 33 included articles (Supplementary Material).

• Download figure
• Open in new tab

Fig 1:

Flow diagram of the article selection process.

Summary of extracted data

The full extracted data are presented in the data extraction matrix (Supplementary Material).

No study provided objective assessments of utility. 31 of the 33 included articles are descriptions and reflections by modellers about their own modelling efforts where they highlight aspects they deemed relevant for the utility of their work in informing policy for outbreak response. Only in two articles, the perspectives of decision makers are portrayed (14, 15).

The included literature represents all continents (Fig 2). However, most of the studies come from High Income Countries, with multiple articles on modelling efforts in the USA (13 articles) and UK (5 articles). Most Low-and-Middle Income Countries (LMICs) that are highlighted in the graph are considered through the included articles related to the same modelling consortium (‘CoMo Consortium’) (16, 17). These articles were included based on the study by Owek et al. (9).

• Download figure
• Open in new tab

Fig 2: Many countries are represented.

World map highlighting the countries that were considered in the included articles. Although all continents are represented, the majority of articles consider the Global North. Note that some articles consider multiple countries; two of these articles account for most of the countries of the Global South in this figure (16, 17).

Out of the 33 articles included, 27 had models on COVID-19, one combined modelling of COVID-19 and Influenza (18) with two models on Influenza alone (19, 20), one on Diphtheria (21), one on HIV (22) and one article considered models on various infectious diseases including Influenza, Pertussis, Tuberculosis and others (23). Our data extraction yielded no key differences between modelling efforts about COVID- 19 and other diseases.

Visualisations of the considered diseases, publication years and considered population groups can be found in the Supplementary Information (Supplementary Figures 2, 3 and 4).

Authors’ affiliations and modelling initiatives

Out of the 33 included articles, 17 articles were by authors with solely academic affiliations while the remaining 16 articles were authored by both those with academic and policy affiliations.

Infectious disease modelling efforts were initiated at various levels (Table 3). While most modelling efforts were conceptualised and started either by scientists working in academia or by officials at decision- making authorities, some initiatives were also seen as a collaborative work between academia and local, regional or national decision-making authorities. All modelling initiatives were found to be in line with the modelling infrastructures defined in a previous global analysis on the use of epidemiological modelling in health crises, i.e. “one modelling team; multiple small teams functioning as one; a consortium consisting of multiple teams and multiple models led by a modelling committee; and finally, teams working in isolation feeding independently into government” (2).

Policy objectives and role of modelling

Infectious disease modelling was done to cater various policy objectives. Most modelling efforts were done to guide operational planning and decision-making in the context of high uncertainty. Mathematical models were used in understanding transmission dynamics, and to forecast the scale of disease outbreak by providing both short- and long-term projections of confirmed cases, hospitalizations, and deaths, while aiding in healthcare resource management and allocation such as hospital bed occupancy and staffing needs (18, 20, 24, 26, 30–32) (15).

Many modelling approaches were also undertaken to assess the effects and impacts of various mitigation strategies and answer policy questions under different disease outbreak scenarios. For instance, during the COVID-19 pandemic, modelling determined the timeframe of the lifting of the state of emergency (SOE) in Tokyo while controlling infection and minimizing the economic loss (33); while the CoMo Consortium’s model was used to determine lockdown and school opening and closing strategies (16). In New York, the modelling aimed to meet the benchmark of the ‘HIV Ending the Epidemic Initiative’ under the most realistic conditions, with a focus on prioritizing men who have sex with men (22).

Modelling approach

In 15 of the modelling efforts, practitioners developed and used compartmental models, i.e. models based on differential equations. 14 of the articles used Agent Based Models (e.g. (27)) or stochastic/statistical models (e.g. (34)). 10 studies mentioned more statistical approaches, such as using neural networks or Generalized Additive Models (GAM) (e.g. (19, 29, 32, 35)). 4 studies utilized ensemble approaches, synthesizing results from multiple models that included Agent based, Stochastic and compartmental models among others - often by different modelling teams (e.g. (20)). Note many articles considered multiple models and model types.

Decision maker

Infectious disease modelling was used to inform decision makers across multiple governance levels. (Table 4). While some articles considered decision makers on one governance level, some considered multiple levels, e.g. in the Philippines (36) and South Africa (26).

Approaches to policy interaction

We identified various communication formats used by modellers to convey their findings to policymakers, other modellers, and other relevant professionals and organizations.

These communication approaches ranged from formalized, structured processes to more ad-hoc arrangements. Formal structures included established reporting templates (27), regular meeting schedules (e.g. (22)), and dedicated communication channels (e.g. (17)). Diverse communication methods were employed for these interactions, from daily reports (e.g. (28)) to weekly communications (e.g. (38)) and monthly meetings (e.g. (14)). The formats included written reports (e.g. (18)), face-to-face or virtual meetings (e.g. (39)), and formal presentations (e.g. (16)). Several initiatives established dedicated coordination teams or workstreams to manage the interface between modellers and decision-makers (e.g. (40)).

Hierarchical paths were frequently observed, where results were first presented to technical groups, then to advisory bodies, and finally to government through chief scientific advisors (e.g. (41)).

A notable theme across multiple studies was the emphasis on participatory and collaborative approaches to ensure effective knowledge translation. This was highlighted in The CoMo Consortium’s stepwise method to establish channels of communication between local policymakers and modellers, efforts by Médecins Sans Frontières and Canada’s bi-directional communication between modellers and decision- makers via multiple workshops to identify translational barriers (17, 21, 23). The adaptation of communication methods to different audiences was also common, with both technical reports and simplified briefings for broader audiences. Nigeria’s approach with regular evening calls between technical teams and policy task forces enabled rapid response to emerging questions (14). Public communication with several studies incorporating press releases, media engagement, and public-facing websites ensured broader dissemination of modelling insights (e.g. (33)).

Stakeholder input

Stakeholder input refers to the contributions, feedback, and interactions of individuals or groups with an interest in the modelling process or those who may be impacted by its outcomes. The input was incorporated in various aspects of the project such as during the selection of variables, setting assumptions, and identifying critical questions to address in the models (16, 39, 42), with most studies aiming for collaborative improvement of the modelling process via expert input (e.g. (28, 32)).

15 out of 33 studies benefited from stakeholder input, where stakeholders were often professionals, including policy makers, cabinet officers, public health professional, clinicians, socioeconomic experts and modellers with little to no community level engagement. This input was considered in most studies after the modelling process to improve models but collaborative approaches were also seen with input being incorporated throughout the project via iterative processes and sector-specific expertise to refine models. Such efforts are exemplified in New Zealand’s two-round modelling approaches that allowed for clarification between teams and ’scenario sandpit’ discussions (40), where policymakers requested specific simulations, and the formation of specialized working groups in Nigeria (14).

Stakeholder engagement occurred through various structured feedback mechanisms like surveys and Plan- Do-Study-Act cycles implemented among clinicians, hospital staff, commissioners, and public health consultants by National Health Service teams (37, 38).

Direct communication channels with advisory groups and the media were established in multiple projects, including those in Canada with Indigenous communities through ’Community of Practice’ models (23). Additionally, international collaborations like the CoMo Consortium emphasized the role of local experts in guiding policy decisions and ensuring models reflected local contexts (17).

Validity assessments of model results

Modellers tried to ensure the validity of their models and results through diverse quantitative and qualitative methods across studies. Quantitative model validation approaches included comparing forecasts against actual data (e.g. (19)), calibrating models on actual data (e.g. (36)) and conducting sensitivity analyses (e.g. (33)). Cross-validation between different modelling approaches was also observed (e.g. (28)) and some studies utilized ensemble approaches, benefiting from the fact that combined forecasts often outperformed individual models (e.g. (20)). Decision makers themselves (in North Carolina, USA) also appreciated comparisons between different models and teams, gaining confidence in the results if they confirmed each other and uncertainty if they diverged (15).

Validation frequently involved qualitative iterative processes, such as real-time assessment of forecast performance (33), allowing for continuous refinement of models. Several initiatives also implemented peer review mechanisms, exemplified by New Zealand’s rapid multi-disciplinary review process (40) and the use of multiple independent modelling groups (e.g. (42)).

Facilitators and Barriers

The included publications reflected on various facilitators and barriers to informing policies and decisions. They can be roughly clustered around the themes ‘data infrastructure’, ‘institutional and collaborative structures’, ‘expertise and capacity’, and ‘communication and trust’ (Table 5).

The modelling process was strongly facilitated by well-established institutional and collaborative frameworks. Multi-stakeholder partnerships involving experts from government, decision-making bodies, and academia led to improved predictive accuracy and efficient process execution according to e.g. the US FluSight Network organized by the CDC (18, 20). This pre-existing Influenza modelling network also laid the groundwork for the COVID-19 and Influenza Scenario Modelling Hub, which offered long-term operational support to inform outbreak response policy in the US (18, 32). Substantial funding through research grants from the CDC further facilitated these modelling initiatives.

In contrast, inadequate institutional and collaborative structures were identified as significant barriers to the modelling process and the translation of evidence into policy. For example in Nigeria, several systematic challenges were observed, including fragmented decision-making structures, and limited availability of public health funding to support modelling efforts (14).

13 of the included studies highlighted barriers in data infrastructure. These challenges encompassed issues such as data unavailability (e.g. (33)), occasional unreliability of data (e.g. (44)), reporting delays (e.g. (21)), the presence of data from multiple sources leading to inconsistencies and integration difficulties (e.g. (24)), and restricted access to granular data, which sometimes lacked key epidemiological parameters (e.g. (19)).

6 of the reviewed articles highlighted uncertainties in the projections due to the novel nature of the coronavirus and the use of varying parameters and assumptions by different modelling teams. This led to a discordant set of results presented to policymakers, fostering confusion, mistrust, and diminishing interest in the models among decision-makers, as illustrated by a study in the Eastern Mediterranean region (16). Keeling et al. also noted the challenge of effectively communicating such uncertainties in long-term projections to policymakers and the public (42). The results were frequently misinterpreted as definitive predictions in the media and decision-making contexts, which, in some cases, led to the potential dismissal of the modelling outcomes (43).

Local decision makers were especially concerned about translating model results based on national data and perspectives into their local context and expressed scepticism towards the ability of models to appropriately incorporate human behaviour (15). In their case, having internal professionals to interpret and translate the results was reported as helpful.