Abstract
Background The COVID-19 Pandemic has popularized computer-based decision-support models as a tool for decision-makers to manage their organizations. It is unclear how decision-makers have considered these models to inform COVID-19-related decisions.
Methods We interviewed decision-makers from North Carolina across diverse organizational backgrounds to assess major decision-making processes during COVID-19, including the use of modeling as an input to inform decision-making.
Results Interviewees were aware of models during COVID-19, with some depending upon multiple models. Models were used to compare trends in disease spread across localities, allocate scarce resources, and track disease spread within small geographic areas. Decision-makers desired models to project disease spread within subpopulations and estimate where local outbreaks could occur as well as estimate the outcomes of social distancing policies, including consequences beyond typical health-related outcomes. Challenges to the use of modeling included doubts that models could reflect nuances of human behavior, concerns about the quality of data used in models, and the limited amount of modeling at the local level.
Conclusions Throughout COVID-19, decision-makers perceived modeling as valuable for understanding disease spread within their communities and to inform organization decisions, yet there were variations in organizations’ ability and willingness to use models for these purposes.
Introduction
The COVID-19 Pandemic has challenged decision-makers to manage their organizations through a public health crisis.1–3 Many organizations have shifted the type of services offered, establish structures for emergency decision-making, and develop the capacity to track and act upon continuously updated COVID-19-related information.4,5 In parallel, the pandemic has accelerated health sciences research and popularized many tools of health science researchers,6–9 including computer-based decision-support models designed to simulate the future course of disease spread under different epidemiologic and policy assumptions.10,11 Throughout the pandemic, it has been unclear how decision-makers across a range of organizational backgrounds (especially those outside Public Health or Healthcare) have used these models. In this qualitative study, we aimed to better understand the use of COVID-19 models for organization-level decision-making.
Methods
From October 2020 to January 2021, we interviewed decision-makers (n = 44) from North Carolina across diverse organizational backgrounds, including three or more interviewees from each of the following eight sectors: education (primary/secondary and higher), public health, public safety, government, healthcare, business, faith-based, and public transportation. Through semi-structured interviews, we assessed their organization’s major decision-making processes throughout the pandemic. A codebook was developed using an inductive approach to label responses across several broad categories, including decision inputs for COVID-19-related decisions. All interviews were double coded independently by two members of the research team. In this analysis, we focused on coded segments discussing the use of modeling as an input to inform decision-making, and identify key themes to help modelers appreciate current perceptions about modeling and opportunities to improve the use of modeling in decision-making. As asked in the interview, different types of simulation-based models were not distinguished (e.g. curve-fitting models, agent-based simulations, compartmental models, etc.), but were treated as a general category.
This study was exempted from IRB review by the [blinded-for-review] Office of Human Research Ethics.
Results
Among the 44 decision-makers interviewed, twenty substantively discussed modeling as a decision input (Table 1); others were either not aware of modeling or had not considered it as an input to decision-making. Comments among this subset were categorized into instances of actual use, recommended applications, and resistance toward model-informed decision-making during COVID-19. Representative quotes across themes are in Table 2.
Use of Modeling during COVID-19
Decision-makers consistently demonstrated awareness of COVID-19 decision-support models, with several following one or more local, state, or national-level models. Model sources varied among decision-makers. Whereas some organizations built their own internal models, most relied on local, external modeling groups. In the absence of local modeling resources, several decision-makers relied on national models, especially those from the Institute for Health Metrics and Evaluation (University of Washington), to inform initial decision-making around shut-downs and safety protocols. One decision-maker from the healthcare sector commented on the use of multiple models, with recognition that one model could not provide all the answers (Healthcare, E3). In other instances, decision-makers used external models for comparison with internal models, so as to check that decision-making at the local level made sense in the context of state-or national-level trends (Public Safety, H3). This aligns with a general wariness, as expressed across most interviewees, toward making a decision that would draw negative attention to their locality or organization if their local model was wrong.
Three healthcare organizations utilized models for allocating scarce resources and critical care employees. Among healthcare system decision-makers and others, hospital surge capacity models were the most commonly used models for decision-making (Public Safety, H5). In two other instances, decision makers used modeling to monitor disease spread in comparable localities to estimate what could be expected in their own jurisdiction (Public Safety, H3). Comparisons occurred not just within North Carolina, but also across other states, especially those that had been hardest hit by the Pandemic early on (Healthcare, E3). Three other respondents emphasized how valuable modeling was for tracking the spread of disease within small geographic areas, especially those that correspond with vulnerable communities that could be reached through targeted communication, testing, and contact tracing efforts. A local health department director, for instance, reflected on the importance of modeling infection rates by zip code (Public Health, D1).
There were two instances in which decision-makers emphasized the value of model visualizations and how they helped clarify the impact of policies on disease spread for multiple audiences. Visualizations helped to make the abstract complexity of models concrete (Education, C3). They were also educational. One Public Health official described the process of working with a government epidemiologist as they presented graphically how different policies may shift the “curve” of total infections over time in a way that was both intuitive and easily actionable (Public Health, D1).
Having someone in the organization who is trained to interpret models, point to key takeaways, and situate findings within the broader literature was a facilitator to using models to inform decision-making. In the absence of this internal knowledge, interviewees noted using models that had been approved by their local health director as a way to discern which models could be trusted (Government, A3).
Recommended Applications for COVID-19 Modeling
There were multiple decision-makers who, while they had not used modeling at the time of the interview, proposed applications for which modeling would be useful. For instance, two individuals emphasized a desire for models to show disease spread within subpopulations, including a Public Safety officer who desired models to both understand how subpopulations with a high burden of disease shifted throughout the pandemic and to identify potential outbreaks (Public Safety, H5). Additionally, a public health officer wanted modeling to help show the outcomes of social distancing policies, and specifically to determine what minimum number of restrictions need to be in place to ensure a safe re-opening. They also proposed modeling to inform communication campaigns for vaccination rollout, with particular interest in identifying target coverage levels (Public Health, D4). A university administration officer similarly desired modeling to inform testing policies if schools were to return in-person (Education, C1). Another interviewee expressed a desire for modeling the downstream consequences of the pandemic beyond typical health-related outcomes, particularly assessing developmental outcomes for kids who have been in school virtually (Education, C7). In general, decision-makers used visual language (e.g. “show”, “see”) when describing their desire for future modeling work, and avoiding more technical words like “predict” and “forecast.”
Resistance toward COVID-19 Modeling
Interviewees expressed resistance to modeling, including the recognition that models can be highly uncertain and sensitive to differing assumptions, which may lead to inconsistent results across different models (Education, C3; Education, C1; Public Health, D5). Points of resistance also included doubts that models could effectively incorporate the nuances of human behavior, particularly compliance with official mask usage guidelines (Public Health, D5; Education, C2; Education C5). Among such interviewees, there was an implicit assumption that models must rigidly mirror official guidelines (e.g. everyone wearing masks), without an appreciation for the flexibility models provide. Other decision-makers questioned the quality of data used in the models, acknowledging that “models are only as good as the comfort level you have with the data going into it and the assumptions being made” (Education, C3). This critique nevertheless acknowledged that models could become an effective tool for decision-making if quality data were used and transparently presented (Public Health, D5; Education, C1). Another concern raised was that models did not capture the nuances of local communities; decision-makers identified unique characteristics of their own communities which made them substantively different from the state at large. One local health director commented on the inability of state-level models to translate to individual counties (Public Health, D5).
Discussion
Modeling has become a well-recognized tool for COVID-19-related decision-making within private and public organizations, including local organizations. Among decision-makers interviewed for our study, modeling was seen as valuable for understanding the trajectory of the pandemic within their communities and to inform organization decisions, such as opening facilities or allocating resources. Visualizations were valuable for understanding model outcomes. Notable points of resistance toward the implementation of modeling results included the difficulty of creating models which can be useful to local decision-makers accounting for differences across areas. Accurate modeling at the local level and modeling the impact of policy among subpopulations (especially vulnerable communities) both demand access to granular data which was difficult to obtain during COVID-19. 12 Examples include mobility data or data on mask usage, neither of which are easily accessible by age and race or ethnicity.13 Our study results emphasize the need to modernize the data infrastructure in the United States to support these model development and calibration needs. In the absence of internal modeling capacity and data collection locally, local decision-makers may have to rely on externally built models that may not take into account important features of the local context.14 Given the resistances raised, additional guidance is needed to help local decision-makers become better consumers of modeling, including the capacity to understand decision objectives, potential model approaches and outcomes (e.g., predictions, forecasts), the uncertainty associated with model structure and estimates, and effective communication of model-informed decisions.15
Data Availability
Qualitative Data used for this analyses may be accessed upon request.
