Summary
Background In March and April 2020, public health authorities in the United States acted to mitigate transmission of and hospitalizations from the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), which causes coronavirus disease 2019 (COVID-19). These actions were not coordinated at the national level, which raises the question of what might have happened if they were. It also creates an opportunity to use spatial and temporal variation to measure their effect with greater accuracy.
Methods We combine publicly available data sources on the timing of stay-at-home orders and daily confirmed COVID-19 cases at the county level in the United States (N = 132,048). We then derive from the classic SIR model a two-way fixed-effects model and apply it to the data with controls for unmeasured differences between counties and over time. This enables us to estimate the effect of stay-at-home orders while accounting for local variation in factors like health systems and demographics, and temporal variation in national mitigation actions, access to tests, or exposure to media reports that could influence the course of the disease.
Findings Mean county-level daily growth in COVID-19 infections peaked at 17.2% just before stay-at-home orders were issued. Two way fixed-effects regression estimates suggest that orders were associated with a 3.8 percentage point (95% CI 0.7 to 8.6) reduction in the growth rate after one week and an 8.6 percentage point (3.0 to 14.1) reduction after two weeks. By day 22 the reduction (18.2 percentage points, 12.3 to 24.0) had surpassed the growth at the peak, indicating that growth had turned negative and the number of new daily infections was beginning to decline. A hypothetical national stay-at-home order issued on March 13, 2020 when a national emergency was declared might have reduced cumulative county infections by 62.3%, and might have helped to reverse exponential growth in the disease by April 5.
Interpretation Although stay-at-home orders impose great costs to society, delayed responses and piecemeal application of these orders generate similar costs without obtaining the full potential benefits suggested by this analysis. The results here suggest that a coordinated nationwide stay-at-home order may have reduced by hundreds of thousands the current number of infections and by thousands the total number of deaths from COVID-19. Future efforts in the United States and elsewhere to control pandemics should coordinate stay-at-home orders at the national level, especially for diseases for which local spread has already occurred and testing availability is delayed. Since stay-at-home orders reduce infection growth rates, early implementation when infection counts are still low would be most beneficial.
Funding None.
Introduction
Coronavirus disease 2019 (COVID-19) first appeared as a cluster of pneumonia cases in Wuhan, China on December 31, 20191 and was declared a global pandemic by the World Health Organization (WHO) on March 11, 2020.2 As of April 12, 2020, the European Centers for Disease Control reports that worldwide there have been 1,734,913 confirmed cases of coronavirus disease 2019 (COVID-19), resulting in 108,192 deaths.3
The United States recently became the country with both the highest number of cases (529,951)3 and deaths (20,608) due to the disease. As a result, the U.S. government has been widely criticized for inaction in the early stages of the pandemic.2 Although the first confirmed case of COVID-19 was reported to the Centers for Disease Control on January 21, 2020 and documented transmission commenced immediately4, a national state of emergency was not declared until nearly two months later on March 13. At that time, no mandatory actions were ordered at the national level other than international travel restrictions.5
While the national government has the authority to act, the United States is a federal political system where public health is normally the purview of the fifty states. Furthermore, each state often delegates health authority to cities and/or counties, geographic political units nested within states. As a result, responses to COVID-19 varied across states and counties and led to spatial and temporal variation in implementation of mitigation procedures. This variation in policy responses has likely contributed to significant variation in the incidence and growth of infections across jurisdictions in the United States.6
A variety of government policies have been proposed and used to mitigate the spread and consequence of pandemic diseases like COVID-19, ranging from investments in medical testing, contact tracing, and clinical management, to school closures, banning of mass gatherings, quarantines, and population stay-at-home orders7. China’s extensive interventions appear to have been successful at limiting the outbreak.8,9 These include quarantines both for those diagnosed and those undiagnosed but who had been in Hubei province during the outbreak10, and restrictions on travel to and from affected areas.11 In contrast, school closures across East Asia were estimated to be much less effective.12
With estimates that nearly half of transmissions occur from pre-symptomatic and asymptomatic individuals, epidemiological simulations suggest that quarantines of symptomatic individuals alone will be insufficient to halt the pandemic.13 This has led to widespread adoption of population-wide policies to dramatically reduce social contact.
Here, we study the role of stay-at-home orders, perhaps the most common policy intervention in the United States and Europe. Stay-at-home orders require citizens to shelter in their residence with very few exceptions, and they have typically been implemented along with school closures, bans on mass gatherings, and closure of non-essential businesses. These policies are associated with a significant reduction in observed mobility,14 and initial evidence from New York City suggests that they can be effective in reducing case growth in the United States.15 Yet, because each locality in the U.S. has many factors that contribute to differential rates of transmission, statistical efforts to control for potential confounds and to identify the precise effects of stay-at-home orders are critical to understanding whether -- and to what degree -- such policies are working.
Methods
Data
The time and date of county-level “stay-at-home” or “shelter-in-place” orders for each state and locality were aggregated and reported on a web page maintained by the New York Times starting on March 24, 2020.16 As new orders went into effect, this page was updated. We checked it once daily to update the data through April 11, 2020. In some cases a statewide order was reported with reference to earlier city-level or county-level orders in the state without specifying where they occurred. In those cases, we searched local news outlets to find references to official city and county orders in the state that preceded the statewide order. For each county in each state we recorded the earliest time and date that a city, county, or statewide order came into effect. Figure 1a shows the distribution of order dates. As of April 11, 2020, 18 states (1,451 counties) exhibited county-level variation in order dates, 27 states and the District of Columbia had statewide orders with no local variation (1,307 counties) and 5 states (386 counties) had no order in place.
(a) Distribution of stay-at-home orders at the county level by date in the United States. (b) Log of cumulative confirmed COVID-19 active infections by county and date, gradient-colored by date of first case (blue = early to light blue = middle to orange = late).
County-level data on cumulative COVID-19 confirmed cases were also aggregated daily by the New York Times.17 We discarded all observations where cases were not assigned to a specific county (these account for 1.3% of total cases). We retained observations where cumulative cases declined from one day to the next due to official revisions to the counts (0.4% of cases). Figure 1b shows that the number of cases grew exponentially in each county over time from March 1, 2020 to April 11, 2020. It also suggests that efforts to “flatten the curve” initiated in mid-to-late March may have helped to reduce the rate of exponential growth. Below we describe how we use data on cumulative cases, which include both currently and previously infectious individuals, to estimate the growth rate in the total number of currently-active infections.
Availability of tests for COVID-19 in the United States has not been uniform over the date range of the study.18 To mitigate the effect of changes in rates of testing on our measure of confirmed cases, we also collected data on the number of tests administered each day. This information is not currently available for each county, but it is available for each state by date from the COVID Tracking Project19 for about 80% of our observations. We merged this data with information about stay-at-home orders and confirmed cases.
Estimation
The spread of disease in a population (N) is an exponential process, with each person among those currently infected (I) capable of infecting one or more other individuals who are susceptible (S). After a period of time, infected individuals recover from a disease (R) and are no longer infectious. The classic SIR model suggests that for a county c on date t the daily rate of new infection is
where β is the infectiousness of the disease (how many other people does an infected person infect per day) and γ is the rate of recovery (what fraction of infected individuals cease to be infectious).
In our data, we observe cumulative infections (yct) but these include both infected and recovered individuals yct = Ict + Rct. We do not observe the number of recovered individuals, but we assume it approximately equals the number of infected individuals d days prior to the current period (Rct = yct-d), where d represents the number of days individuals remain infected. Because a specific value for d is unknown for COVID-19, we estimate models with different assumed values. We also assume that the portion of the population that is susceptible Sct/Nct = 1, since the number of observed cumulative cases is typically 1% or less of the population in each county for this study, suggesting the rest remain susceptible (we elaborate on the limitations of this assumption in the discussion).
Under these assumptions, we can rewrite the above equation as
Notice that the left hand side is simply an expression for rate of growth in cumulative cases (%Δy) that adjusts for recovered individuals or, put differently, rate of growth in active infections. To ensure that this value was not undefined, we added 1 to the denominator for all observations.
If we assume that the left hand side of equation (2) is measured with error uct, and rates of increase in active infections (β −) are a linear function of fixed factors within each county (αc), factors that apply to all counties but vary over time (αt), and county-specific stay-at-home orders (xct), this equation can be rewritten as a two-way fixed-effects ordinary least squares regression model20
Because we do not know the temporal dynamics of stay-at-home orders, we measure the effect of stay-at-home orders non-parametrically as they unfold in the days following the order. We sum (Στ≠−1) over all possible observations of number of days τ prior to or after an order (excluding the reference day τ =− 1 immediately prior to the order). For each τ, we estimate an effect size δτ using a set of indicator variables xctτ that equal 1 if the number of days since a stay-at-home order in county c on date t equals τ, and otherwise equal 0. Negative values of τ allow us to measure effects on days prior to the order in order to evaluate whether or not differences in case growth rates might cause changes in the date an order is enacted, rather than the other way around.
A strength of this model is that county-level fixed effects αc control for all time-invariant features of each county that might drive rates of case growth in the epidemic.21 For example, each county has its own age profile, socioeconomic status, local health care system, base rate of population health, and date on which a first case of COVID-19 was observed. Additionally, time fixed effects αt control for factors that vary over time.21 For example, case rates could be affected by changes in the availability of testing nationally, in social behaviors influenced by daily events reported in the media, and national-level policies that vary from one day to the next. Finally, we cluster standard errors uct at the state level. This adjusts the estimated standard errors for unobservable factors correlated between counties within the same state.
We identify the effects of stay-at-home orders δτ using variation in the timing of implementation by counties and municipalities. Our two-way fixed effects regression is equivalent to a difference-in-difference model with variation in treatment timing. Models with variation in treatment timing are known to exhibit bias if the treatment effect is heterogeneous over time.22 We test for temporal treatment heterogeneity bias by performing a Bacon decomposition.23 Results from this analysis indicate that this source of bias is moderate (weight of Later Treatment vs. Earlier Control = 0.29) and that bias attenuates our results towards 0, in which case our estimates are a lower bound for the effect of stay-at-home orders on case growth.
Our estimator captures the causal effect of stay-at-home orders on case growth if counties that implement these orders on a specific date would have had similar changes in case growth to counties that had not yet implemented these orders had the implementing counties not implemented the order on that date. This is the standard parallel trends assumption of difference-in-difference models.24
Role of the funding source
There was no funding source for this study. The authors had full access to all the data in the study and had final responsibility for the decision to submit for publication.
Results
Figure 2a shows how the mean county-level daily growth rate in COVID-19 infections has changed over time. After peaking on March 25, 2020 at 18.0%, it declined quickly to 3.6% as of April 11. Figure 2b restricts observations to those that had implemented a stay-at-home order and shows that peak growth in these counties occurred three days before the order went into effect (17.2%) and declined to 2.7% three weeks afterward.
Mean U.S. daily growth in total active COVID-19 confirmed infections (a) by date, and (b) by the number of days before or after the stay-at-home order. Growth in active infections declines after stay-at-home orders towards 0, where the number of new active infections is approximately equal to the number of already-infected who recover. When growth goes below 0, the number of daily infections begins to decline.
Growth rates begin to decline following the orders. However, a number of factors may confound this association in the raw data. For example, stay-at-home orders may closely follow earlier targeted mitigation measures at the national level (such as travel restrictions issued by the State Department or recommendations by the CDC on mass gatherings). There may also exist spurious correlation between local factors (such as susceptibility to the disease or the capacity of the health system) and the timing of stay-at-home orders. To control for these factors, we apply the fixed-effects model in equation (3) to the data.
Table 1 shows four versions of the model. The main parameter that creates some uncertainty is d, the number of days an infected person remains contagious. Recent research suggests this period is about two weeks, so that is our assumption in Model 1. In Model 2 we show results if we set the value of d to 7 and in Model 3 we set d to 21. There is also some concern that measures of growth in cases of COVID-19 may be affected by the rate in growth in the availability of tests for the disease, so in Model 4 we include that variable as a control. Parameter estimates for all models are similar, so we will focus on Model 1 for the remainder of this analysis.
Coefficient estimates from regressions of daily growth rate of COVID-19 infections on variables indicating the number of days before and since a stay-at-home order went into effect. The variable d for each model indicates the assumption made about the number of days cases remain infectious. All models include fixed effects for county and date and standard errors clustered on state. Model 4 includes as a control a daily state-level measure of the growth rate in the number of tests administered for COVID-19. All models include coefficients for days prior to the order that suggest differences in the case growth rate do not predict the timing of stay-in-place orders (Models 1-3 shown in Figure 3).
The numbers in Table 1 can be interpreted as percentage point changes in the rate of growth of COVID-19 infections associated with the number of days since a stay-at-home order has gone into effect. Negative numbers indicate a slowing of rate of growth though it is important to note that a smaller rate of growth still means an increasing number of total cases. For example, by Day 2, counties with stay-at-home orders achieve their first statistically meaningful reduction in the case growth rate (2.8 percentage points, 95% CI 0.3 to 5.3). After a week the reduction in the rate is 3.8 percentage points (0.7 to 6.9). At two weeks the reduction in the rate is 8.6 percentage points (3.1 to 14.0). And by Day 22, the expected reduction in the infection growth rate (18.2 percentage points, CI 12.3 to 24.0) has surpassed the average magnitude of growth rate at its peak (17.2%). When the growth rate turns negative, the number of new daily infections will start to decline and the epidemic will eventually come to a halt.
Figure 3 shows these estimates along with the estimates for each day prior to the day a stay-at-home order goes into effect. Each panel of the figure displays the results given different assumptions about the number of days cases remain infectious. Unlike the raw data shown in Figure 2b, the estimates here are adjusted for unobserved factors that vary over time and between counties that can influence the course of the disease. Notice that the estimates in Figure 3 before the order goes into effect stay very close to zero. This suggests that differences in case growth are not influencing the timing of stay-at-home orders, helping to rule out the possibility that the later associations we see are driven by reverse-causality or differential trends.
Estimated total effect of stay-at-home orders on the daily growth rate of COVID-19 confirmed infections in U.S. counties that implemented these orders, by the number of full days since the orders were issued. Estimates are derived from a fixed-effects regression of the daily growth rate of cases on days since initiation of the stay-at-home order as a categorical variable, where the reference category is the day prior to an order (Equation 3). Each panel of the figure displays a different model from Table 1, representing a different assumption about the number of days d cases remain active (a) d = 7 (Model 2); (b) d = 14 (Model 1); (c) d = 21 (Model 3). The models control for all county level and date fixed effects and for correlated observations with cluster-robust standard errors at the state level.
To better understand the scale of these estimates, consider that these growth rates are cumulative. Lower growth in infections today means fewer infections tomorrow, so the full effect on future infections is the product of these improvements. The coefficients suggest that counties with stay-at-home orders have 20.8% fewer infections by day 7 and 48.6% fewer by day 14.
It is a complex task to model what might have happened if the United States Federal government had coordinated a nationwide stay-in-place order when it declared a national state of emergency on March 13, 2020. We have already seen county-level results that suggest the rate of growth in infections turns negative by day 22, suggesting a nationwide policy might have done the same by April 5.
Moreover, consider this: the median date a stay-in-place order was issued for counties in the United States was a full 17 days later on March 30. Our coefficient estimates suggest that acting 17 days earlier in each county would have reduced new infections by 62.3%. It is thus highly likely that a nationwide order would have reduced the number of infections by hundreds of thousands. And with case fatality rates currently ranging from 1% to 4% in the United States, it is possible that such an order could have also prevented thousands of deaths.
Discussion
The results here suggest that the United States federal government may have erred in not acting to coordinate stay-at-home orders at an early stage in the outbreak of COVID-19. We find that these stay-at-home orders appear to be effective at the county level in limiting spread, and provide some hope that the physical distancing measures now widely implemented are working to flatten the curve.
With that said, we note numerous limitations in our analysis. Stay-at-home policies are ultimately assigned endogenously so we cannot say for certain that the associations we have measured are the result of a causal effect. Our tests of reverse causality suggest that stay-at-home orders influence case growth and not the other way around, but there is no way around the fact that these are observational data from which causal estimates are notoriously difficult to obtain. Our dependent variable, growth in infections, is based on incomplete data. It is well-known that rates of testing in the United States were extremely low in the early part of the pandemic,2,18 so measures of cumulative cases over time probably increased faster than the disease itself due to previously undetected infections. We attempt to control for this issue with county and time fixed effects and a measure of the growth in testing at the state level, but we are unlikely to have entirely adjusted for daily local variation in access to tests.
Our model of the disease is derived from the classic SIR model to allow for an empirical strategy that estimates causal effects of stay-at-home interventions. Using this model, however, requires us to make strong assumptions. For example, we must assume the number of days that infected individuals are contagious even though the scientific community is currently unsure about the precise distribution of this parameter. We account for this uncertainty by ensuring our results are robust to different assumed values, but it is still possible that the true value falls outside the range we show here. We also assume that the proportion of the population susceptible to the disease is constant, at 1, over space and time. If this assumption is strongly violated it might influence our estimates since they are linearly related to a parameter that is multiplied by that value. At this stage in the disease we believe high susceptibility is a reasonable assumption, but low rates of testing coupled with pre-symptomatic and asymptomatic transmission suggest it is possible that the proportion susceptible is lower than is currently indicated by the data.
Our independent variable, stay-at-home order status, measures a policy intervention that was often implemented simultaneously or within days of several other local interventions, such as bans on mass gatherings and closures of schools, non-essential businesses, and/or public areas. Given the uncertainty about how many days infected individuals are contagious both before and after the onset of symptoms, efforts to generate a sharp estimate of the effects of policies that were implemented within days of each other are difficult. Moreover, our analysis suggests these other local interventions may also have an effect on infection growth. On average, the peak of infections happens three days prior to the stay-at-home order. In addition, we see significant reductions in the growth rate of infections at just two days after the order. This is in spite of the fact that case identification during the early part of our observations was based on tests that often took a week to be resolved.2,18
With our current empirical approach we cannot perfectly separate the effects of these other local interventions from that of stay-at-home orders. This means that our estimates should properly be interpreted as the effect of stay-at-home orders bundled with the effects of these other local interventions. As such, our model compares a “do everything” approach to a counterfactual mix of “do something” and “do nothing” approaches at the local level, which is the status quo that prevailed in the United States until mid-March. An interesting question which we leave for future work is which local interventions in the policy mix helped the most.
One final limitation is that we assume the effects of stay-at-home orders between localities are independent, but it is likely that significant spillovers exist. Consider the effect of the epidemic in New York City on neighboring counties in New Jersey, Connecticut, and as far away as Rhode Island. Or the effect of Mardi Gras in Louisiana or Spring Break in Florida on a variety of locales throughout the United States. To the extent these spillovers are positive -- as seems reasonable to assume -- we have likely underestimated the effect of a hypothetical coordinated effort at the national level.
Would a hypothetical reduction in infections from an early nationwide stay-at-home order have limited fatalities? We do not know. Perhaps such an order would simply have delayed the timing of the pandemic. We do know, however, that slowing the initial rate of growth in infections helps hospital systems to better figure out how to provide supportive care for COVID-19, enables the implementation of better testing and tracing procedures, and provides time for clinical trials to produce results regarding immunization and treatment. Perhaps most importantly, spreading out infections over more time can help prevent the number of total cases requiring hospitalization from spiking above existing hospital capacity, which is relatively fixed in the short-run. Thus, it seems reasonable to assume that death rates seen in early cases might be higher than if earlier stay-at-home orders had moved these infections later in time. If this is the case, then even a moderate delay in infections could produce a significant decrease in total fatalities from the disease.
It is important to note that although we are currently observing decreases in the rate of growth of daily COVID-19 infections in the United States, it remains positive so that total cases continue to increase exponentially. Only when the rate of growth turns negative will we know whether or not we slowed the disease in time to keep it from overrunning our health system capacity. There is much still to be done, and we are hopeful that the work here will help our fellow scientists, policymakers, and the public-at-large to plan for the next steps in managing this disease.
Data Availability
All data used in this paper are publicly available on internet sites.
Contributors
All authors contributed to collection of data, design and execution of analysis, and drafting, review, revision, and approval of the final manuscript.
Declaration of Interests
We declare no competing interests.
Research in Context
Evidence before this study
Past research has shown that stay-in-place orders can be effective for mitigating influenza pandemics and recent research suggests stay-in-place orders have helped halt the spread of COVID-19 in China.
Added value of this study
To our knowledge, this is the first nationwide study of the effectiveness of COVID-19 mitigation efforts in the United States. The uncoordinated nature of the response to the disease has created variation between localities and over time that we can exploit to more precisely estimate the effect these policies have on growth in infections. And we can use these estimates to quantify the effect that early coordinated action by the United States federal government might have had. The results show that the suite of physical distancing interventions that were implemented with stay-in-place orders probably greatly reduced the rate of growth in COVID-19 infections.
Implications of all the available evidence
Even as we submit this paper for publication, policymakers are debating whether to keep stay-in-place orders intact, and this study contributes to the conversation with a scientific estimate of their effect so far specifically for the COVID-19 epidemic in the United States. The evidence here suggests that the effectiveness of stay-in-place orders shown in studies of pandemic flu apply more broadly to other diseases as well.
Acknowledgments
We thank Robert Bond, Chistopher Dawes, Micah Gell-Redman, Lauren Gilbert, Tim Johnson, Arman Khachiyan, Sam Krumholz, Brad Leveck, Peter Loewen, Lucas de Abreu Maia, Robyn Migliorini, Niccolò Pescetelli, Daniel Rubenson, Wayne Sandholtz, and Clara Suong for helpful comments.
