ABSTRACT
In the absence of a vaccine and effective antiviral medications, most of the non-pharmaceutical interventions focus on reducing social contact rates through different social distancing policies. However, the effectiveness of different policies and their relative impact vis-a-vis that of mechanisms driven by public awareness and voluntary actions have not been studied. This is crucial since in most places we observe significant reductions in social interaction before any policy was implemented. Variations in types and effective dates of different social distancing policies across different states in the US create a natural experiment to study the causal impact of each policy during the early stage of the outbreak. Using these policy variations and the aggregate human mobility and location trends published by Google for the month of March 2020, we employ a quasi-experimental approach to measure the impact of six common policies on people’s presence at home and their mobility in different types of public places. Our results rank six common social distancing policies based on the magnitude and significance of their impact, beyond what has already been achieved through voluntary actions. They show that while strong policies such as statewide stay home mandate and non-essential business closure have strong causal impact on reducing social interactions, most of the expected impact of more lenient policies (such as large gathering ban and school closure mandates) are already reaped from non-policy mechanisms such as voluntary actions and public awareness.
Introduction
In the absence of antiviral drugs and vaccines in response to the current COVID-19 pandemic, social distancing has been the major mechanism adopted by various impacted countries.1,2. These attempts are made, largely to keep the peak level below the resource capacity of healthcare systems and buy time for possible drug and vaccine development.
Reduction in social contact rate during pandemic outbreaks is driven by a combination of awareness-driven voluntary actions by individuals and businesses, and an array of non-pharmaceutical interventions (NPI) implemented at the national, state, or local level. Research on the 1918 influenza pandemic in different cities in the US point to the role of both these mechanisms in lowering the mortality rate3.
There is strong evidence that social distancing has played a significant role in containing the first wave of COVID-19 outbreak in China4–6 and the latest evidence indicates their effectiveness in Italy and Spain at the end of the first week of April 2020. However, the relative impact of social-awareness versus policy interventions is yet to be determined for the existing outbreak. In addition, several complementary policies were adopted to increase social distancing, some of which might have unintended consequences. Therefore, identifying effective policies could help policymakers respond efficiently to the outbreak. In most cases, decoupling these factors is challenging, since, in most countries, the timing and strength of NPI are highly correlated with public awareness. Moreover, it is crucial to determine which interventions have a significant impact on lowering the contact rate beyond what can be achieved via awareness mechanisms. Furthermore, the evaluation of these policies could provide valuable lessons, especially for the states that have not yet adopted these policies.
While most countries follow a central policy scheme in the current pandemic, the federal nature of the United States creates a suitable natural experiment setting to tackle these questions. In the United States and as of this writing, the federal government has left NPI decisions to individual states, creating a high level of variation in the type and timing of such policies7 (See Figure 1 in the Data Section). While there are strong evidences for reduced social contact in the US, not all these reductions can be attributed to NPIs. In fact, mobility data show that people in most states had already started to reduce the time they spend outside of home before any NPI was implemented (See Figure AP.1 in the appendix).In fact, for some states such as Idaho, Missouri, Wyoming and the District of Columbia, people’s presence at home had already increased close to a level of saturation before any social distancing policy went into effect. These pieces of evidence suggest that attributing current reductions in social interaction to policy measures can be misleading and further underscore the need for a formal study to disentangle the direct impact of NPIs from other factors such as awareness and spillover effects.
United States COVID-19 policy adoption timeline for six common social distancing policies until March 30 2020
This study utilizes the daily state-level variations in the adoption of six different intervention policies – statewide stay-home order, more limited stay-home orders, non-essential business closure, large gathering ban, school closure mandate, and restaurant and bar limits – to investigate their causal effect on different indicators of social distancing. We use Google-released daily human mobility indicators in different categories of places such as residential areas, groceries and pharmacies, parks, retails and recreations, transit stations, and workplaces. The studied intervention policies are each employed in a subgroup of the US states and at different times during March 2020, making them suitable for quasi-experimental methods. We employ a difference-in-differences approach, which is commonly used to evaluate the effect of policies. Interestingly, the difference-in-difference method was first developed in a simple form by John Snow in 18498 to study the cause of the Cholera outbreak in London and resulted in policy adoptions that effectively ended the outbreak.
Our results give a clear picture of the causal impact and effectiveness of different NPIs on location-specific human mobility at the early stage of the pandemic. At this stage, where changes in behavior are driven by a combination of policy forces and voluntary actions, our results help us rank NPIs based on the strength and significance of their impact, beyond what has already been achieved through voluntary actions. Specifically, our results show that the tendency to remain at home is driven strongly by the statewide stay-home order and moderately by non-essential business closure and policies related to restaurant and bar limits. Other policies such as school closure mandate, large gatherings bans, and more limited stay-home orders do not show any significant impact on keeping people at home. Furthermore, statewide stay-home orders have a strong and significant impact on reducing mobility in all outside home place categories. The effectiveness of non-essential business closure and restaurant/bar limits are also significant on most (but not all) place categories. Limited stay-home orders and large gathering ban do not show any impact on increasing presence at home or reducing mobility in other place categories. If anything, in the absence of other policies, a large gathering ban results in a moderate increase in mobility in transit stations. School closure mandate gives us a mixed picture, with no significant impact on presence at home and moderate, but significant impact on mobility in retail locations and transit stations. Finally, our Event study analysis further makes a stronger case for the causal nature of these findings.
These results show that at the early stages of the pandemic much of the expected impact of some more lenient NPIs are reaped from other factors such as voluntary actions and awareness driven mechanisms. Thus, in order to achieve social distancing beyond what is gained through those mechanisms, states need to adopt strong interventions such as statewide stay-home order.
Data
Mobility Trend Data
We use Google-released aggregated, anonymized daily location data on movement trends over time by geography, across different categories of places for the month of March 20209. The data are gathered by Google from users who have enabled the Location History setting on their accounts and are the same data used by Google Maps to track human traffic at various restaurants and other locations. The data include mobility trends for six location categories including retail and recreation, groceries and pharmacies, parks, transit stations, workplaces, and residential. These data were publicly released on April 2, 2020, in a form of charts that plot country-specific mobility trends and the percentage of changes with respect to a baseline in the period of Feb 16 to March 29. The US data also include mobility trends at the level of states and counties. Given that most intervention policies are implemented state-wide, we base our analysis at the state level. Each movement trend includes multiple data points per day. We aggregated these points to get a single mobility index per day for each trend chart. We should note that these data do not include people without phones, people not carrying their phones to places, etc. However, it is unlikely that the COVID-19 policies affect changes in recorded behavior. Overall, in most cases we have data on movements for 50 states and the District of Columbia for 29 days, providing us with 1479 observations. However, for some measures, we were not able to restore up to two observations for a couple of states.
The success of social distancing policies can be evaluated on two fronts. A primary goal of these policies is to decrease the time people spend outside of their homes. In this study, we use the time people spend at residential locations (referred here as presence at home as a proxy to measure the success level of this goal1. Besides encouraging people to stay at home, social distancing policies also intend on keeping people away from crowded locations and large gatherings, even when they go out. We use the impact of policies on the changes in the remaining five location categories (i.e. retail, transit stations, parks, and work places). Given that these categories do not cover all possible gathering places (e.g. places of worship), we base our findings mainly on the impact of policies on presence at home, but discuss all other categories, acknowledging this limitation.
State Policy Data
We collected all COVID-19 related policies, their issue and effective dates for all 50 states and the District of Columbia since the report of the first positive case in the United States. Since there are some discrepancies in policy start dates among datasets available on third party sources, we used the original documents issued by the state governments, collected by the Kaiser Family Foundation10 to determine the type and date of each state policy. We considered the effective date as the first day in which the policy in question has been in full effect, acknowledging that this decision creates potential biases since some states had policies that went into effect immediately, resulting in fractions of a day of policy that are missed in our data. This, however, does not substantially change the magnitude and significance of our results.
We performed this study on those policies that aim at social distancing, which we divide into six categories: statewide stay-home order, other stay-home orders, non-essential business closure, large gatherings ban, school closure mandate, and restaurant and bar limits. Other stay-home orders incorporates stay-at-home orders for the senior population as well as those targeting specific cities or counties within a given state. Besides these policies, some states have implemented other COVID-19 policies such as cost-sharing waivers for testing or treatment or mandatory quarantine for travelers that we do not consider in this study. Figure 1 summarizes the policy adoption timeline for each policy by showing the number of states who have each policy in effect on any given day during March 2020, suggesting a wide heterogeneity in both the type and the adoption date of each policy during this period.
Temperature Data
To control for the impact of temperature variation on human mobility, we construct daily temperature for each state by taking web scraping daily temperature data for the top 5 biggest cities in each state from the Weather Underground, a commercial weather service that provides real-time weather information. We calculated an aggregate daily temperature for each state by taking the average of daily temperatures of top 5 cities weighted by their populations. Table 1 shows the summary statistics for these three categories of data.
Summary statistics for the data used in the study
Method and Results
We employ difference-in-differences, a quasi-experimental approach commonly used in social sciences to evaluate the effect of policies. In particular, we compare the daily changes in visits from various locations in states that adopt various COVID 19-related policies with those that have not (yet) done so, before and after these policies take effect (See Appendix B for the regression equation and more details). For each policy, we define a binary variable, set to one if a given state adopts that policy after a certain day during the sample period, and otherwise zero. Since changes in visits are serially correlated within the same state over time, we cluster standard errors at the state level11. Note that the validity of this approach hinges on the assumption of parallel trends in changes in visits absent the policies, an assumption which we empirically test using an event study approach. In the event study specification, we replace each policy indicator variable (one at a time) with 15 binary variables, which estimate the effect of that particular policy seven or more days, six days, five days, four days, three days, two days, and one day before and one day, two-to-six days (four binary variables), and seven days and more after the implementation of the policy (See Appendix B for the regression equation and more details). These variables are all zero for states without those policies. We normalize the coefficient for the day before the implementation to zero.
Table 2 reports the results for the effect of policies introduced above on changes in daily visits from various places. As explained in the Data section, our main focus is on presence at home for which we report the results in column (1). Results indicate that statewide stay-home orders significantly increase the measure associated with presence at home by about six fold (relative to states without such a policy) while more limited stay-home orders have a small and statistically insignificant effect. Non-essential business closure and restaurant and bar limits are other policies that have a positive and statistically significant impact on presence at home, although their effect sizes are around half of what is observed for statewide stay-home orders.
Effect of COVID-19 policies on community mobility
The rest of the table presents results on changes in visits from out-of-home places. Interestingly, they suggest a decline in all of these measures which provide evidence that the results obtained for presence at home are not spurious. Among all of the statistically significant estimated coefficients in Table 2, those related to statewide stay-home orders are the most pronounced ones, which strongly suggests that this particular policy is likely the most effective policy in promoting social distancing. On the other hand, policies such as large gatherings ban seems to have a limited and statistically insignificant effect on keeping people at home. Table AP.1 provides estimates by adding state-specific day-of-week to each model. Overall, results are very similar and suggest that state-specific day-of-week variations in outcomes are are not driving our results.
Next, we provide evidence on the dynamic effects of the policies of interests on presence at home in Figure 2. Overall, except for “other stay-home orders” and to some extent, non-essential business closure, there are no differences in presence at home trends between the states with and without those policies. This is evident through the flat trends before the policy took effect on day zero. Consistent with results in Table 2, we observe the largest effect on presence at home through statewide stay-home order, although the magnitude of the effect declines after a week. The effects of other stay-home orders and non-essential business closure are noisy. large gatherings ban illustrates flat trends before and after the implementation suggesting that this policy is ineffective in changing individuals’ behavior towards staying at home. Finally, both school closure and restaurant and bar limits seem to positively affect presence at home. However, their effects are either weak and marginally significant or there are some evidence of upward trends before the policy implementation.
Event study of policies of interest on presence at home. Gray area highlights the 95% confidence intervals.
The preexisting trends in the outcome in states that adopt other stay-home orders and non-essential business closure provide further evidence supporting the role of public awareness in changing home-stay behavior. It is worth noting that we considered the start date of each policy as the first day in which the policy was in effect for 24 hours. This assumption can impact what we see on the last day before the policy in the event study graphs. Therefore, changes immediately before the policy date must be interpreted keeping this in mind.
Given that the COVID-19 spread in the US started from the State of Washington, California, and New York and these states experienced a higher volume of positive cases and deaths, there is always a concern that the estimated policy effects are driven by these states. We provide a version of a permutation test in which we drop each state from the sample one at a time and estimate the effect of policies. The estimated coefficients were consistent when dropping each state, suggesting the effects are not driven by a particular group of states. Results presented in Appendix D support this hypothesis.
Discussion and Conclusion
Our findings show the effectiveness of different social distancing policies on reducing out of home social interaction during the early stage of the COVID-19 outbreak. We show that reductions in out-of-home social interactions are driven by a combination of policy and voluntary measures and point to the strong causal impact of state-wide stay at home and more moderate impact of non-essential business closures and bar/restaurant limits. At this stage of the outbreak and for the US, other policy measures such as school closure mandates or large gathering bans seem to have had no significant causal impact on keeping people at home. We need to be cautious when generalizing the results of this early stage of the pandemic to the later stages and to the possible future waves of the outbreak. Specifically, we need to emphasize that our results do not claim that more lenient social distancing policies such as school closure or large gathering bans are always causally inefficient in reducing social interaction. While it is evident that most of the social distancing capacity of such measures are already absorbed in non-policy driven changes –possibly caused by social awareness –, it is expected that as the pandemic lasts longer, voluntary social distancing measures start to wane, making such policies and a combination of them more effective in later stages of the pandemic12.
Our study has a number of limitations. First, The Google database is not based on the universe of all cellphone users and it only includes those individuals who have enabled the Location History setting on their account. However, given that around 90 percent of users keep their location services on13 our estimates should not be largely affected. Besides, the data are imperfect since they don’t include people without smartphones and those not carrying their phones to places. However, this should not affect changes in recorded behavior and is expected to have little impact on our results.
Finally, this study provides a partial picture of the effectiveness of social distancing policies, by focusing on the first stage, i.e. success of such mechanisms in achieving a lower level of out-of-home social interaction. The more crucial question is how and to what extent such reductions in different forms of social interaction impact the infection rate and disease mortality. This is especially important since the relationship between social interaction and the infection rate is not fully understood and is expected to be highly nonlinear with strong dependencies on the location in which such interactions take place (e.g. parks vs. retail stores). Tackling these questions based on positive cases require more reliable data on the number, location, and policies for infection testing, something that is not widely available in most US states at the time of this writing. This problem can be in part mitigated by analyzing the policy effect on the death cases. Given that the median time from infection to death is reported to be close to 17 days, and since many states have issued their stronger policies in the last week of March, this study needs to wait for now until enough reliable data are collected.
Data Availability
This article uses publicly available data with sources listed under the references.
Acknowledgement
The authors would like to thank two Northeastern University PhD students Soumyakant Padhee and Amir Abodollahi for their valuable assistance with data collection in this study. We also thank David Powell from the RAND Corporation and Dan O’Brien from Northeastern University for their comments and suggestions.
Appendix
A. Evidence for the start of decline in out-of-home social interaction before the implementation of the first social-distancing policy
Trends in presence at home and the start date of the first social distancing policy implemented in each state.
B. Difference-in-differences Estimation
To study the effect of COVID-19 policies, we estimate the following regression equation:
where Y is the changes in visiting various places such as home, grocery and pharmacy, park, retail and recreation, transit station, and workplace. X is the matrix for COVID-19 policies introduced before. temp represents state-level mean daily temperature five days ago. δ and τ are sets of state and day-of-the-month fixed effects, respectively. For the event study, we estimate the following regression equation:
Note that the coefficient corresponding to τ = −1 is normalized to zero and event study for each policy is conducted while including binary variables on other policies in the regression model.
C. Results with state-specific day-of-week fixed effects
Effect of COVID-19 policies on community mobility, results with state-specific day-of-week fixed effects
D. Sensitivity of estimates to dropping states one at a time
Note: each dot represents the point estimate from the regression model in which the corresponding state was dropped from the sample. Gray area highlights the 95% confidence intervals.
C. Sensitivity of estimates to dropping states one at a time (cont.)
Note: each dot represents the point estimate from the regression model in which the corresponding state was dropped from the sample. Gray area highlights the 95% confidence intervals.
Footnotes
