On the effectiveness of COVID-19 restrictions and lockdowns: Pan metron ariston

Tradeoffs in Covid-19 research

Tradeoffs also exist in the choice of pandemic research methodologies, both in terms of the type of model and the data it is estimated on. For example, models may be estimated on data at the national or subnational level. Working at the level of the latter has the benefit of better controlling for country infection dynamics and avoiding possible issues arising from differences in pandemic accounting standards across countries (May 2020). Since data at the national level is broader in terms of country coverage than sub-national data, analyses at the level of the former may carry more external validity and will be less likely to fall prey to overfitting due to small sample noise. Finally, the tradeoff between aggregated (assuming homogeneity or pooled) or disaggregated (individual) modeling depends strongly on the amount and quality of the data.

The impact of policy interventions can be examined under the prism of three types of different models with their own set of tradeoffs: a) detailed models of epidemiological processes such as SIR-based models (Visscher 2020; Dehning et al. 2020; Davies et al. 2020), b) agent based modeling of transmission in a population (e.g., Aleta et al. 2020) and c) reduced form models that abstract away from these epidemiological processes and agent interactions at the micro-level, by using simple econometric models based on empirical data, rather than inference of complex epidemiological parameters. Epidemiological models, while desirable as they directly model the underlying mechanisms, are prone to overfitting in the presence of scant or low-quality, noisy data, leading to non-robustness (see Chin et al. 2020; Soltesz et al. 2020) since they require estimates of key parameters that are highly uncertain, and whose impact may reverberate significantly in highly non-linear, exponential growth models. No single approach dominates the rest, especially at the current stage of our understanding of the pandemic. Different approaches must be simultaneously pursued whilst acknowledging the limitations and advantages of each in an attempt to consolidate the findings.

As the literature on Covid-19 is burgeoning, I briefly survey the most similar studies, primarily those employing reduced-form equations. Early discourse was heavily influenced by the epidemiological modeling of the Imperial College COVID-19 Response Team (Flaxman et al. 2020), which concluded that lockdowns were a crucial and necessary strategy for the subduement of the pandemic. Sub-national data from six countries revealed that while not all specific NPIs have a significant impact, overall the implementation of NPIs significantly reduced infections (Hsiang et al. 2020), without however, explicitly accounting for how restrictive individual NPIs were. Dichotomizing the range of NPIs into less-restrictive (e.g., social distancing, discouragement of international and domestic travel, and large gatherings bans) and more-restrictive (e.g., stay-at-home and business closure orders) reveals no evidence of additional gains associated with the latter in ten countries at the subnational level (Bendavid et al. 2021). Similarly, stringent lockdowns in 24 European countries over the first half of 2020 did not significantly improve pandemic dynamics over less stringent lockdowns (Bjørnskov 2021). This was confirmed using data from 108 countries until May, 2020, (Bonardi et al. 2020) with the implication that NPIs may have a signalling effect leading to voluntary behavioral changes by citizens. Less stringent NPIs already provide a strong enough signal, so that there is nothing to gain from more stringent NPIs. The importance of voluntary behavioral changes is exemplified by the significant changes in mobility found to occur earlier than mandatory restrictions on movement (Abouk and Heydari 2021; Farboodi, Jarosch and Shimer 2020), and by the fact that mobility did not return to previous levels after the easing of lockdown restrictions (Gapen et al. 2020; Allcott et al. 2020).

I seek to combine the advantages of reduced-form approaches, whilst alleviating some of the drawbacks in their implementations to date. Specifically, earlier studies: a) eschewed behavioral components and agents’ incentives, b) estimated a single regression disavowing variable endogeneity, and c) were estimated on datasets from the first few months of the pandemic without the benefit of the currently accumulated data covering other developments such as newer variants.

The importance of behavioral models

Models can be classified according to whether they are behavioral or not, i.e., are the behavioral responses of citizens and governments allowed to vary endogenously or are they assumed to be fixed? Agent-based models are by construct behavioral, however standard SIR models are not, and reduced-form work typically estimates single-equation regressions of effects of various NPI variables on either the confirmed case (and or death) growth rate or mobility data, thereby implicitly assuming exogeneity of these variables. However, it reasonable to believe that mobility is dependent on the severity of the NPIs in place, i.e., it may mediate the impact of NPIs on Covid-19 dynamics. Furthermore, NPI stringency may also be endogenous if governments base their policy decisions on recent epidemiological data and trends, e.g., infection growth. Consequently beyond modeling only SARS-CoV-2 dynamics, agents’ adaptive reactions to the situation merit attention: those of citizens (Cowling et al. 2020) (through mobility, precautionary and voluntary measures and other behavioral responses) and those of governments through the choice and timing of policies.

Beyond the foundational use of behavior in agent-based models, behavioral components can be introduced both to epidemiological and reduced form models. The former is significantly more common than the latter, which I will pursue in this study. The incorporation of behavioral components in SIR models leads to very different long-run predictions than models based on standard non-behavioral SIRs (Kermack and McKendrick 1927). A common finding is that the system tends to an equilibrium reproduction rate of 1 (Gans 2020), sometimes with oscillations if behavioral components operate with lags (Cochrane 2020; Atkeson 2021). These studies highlight the importance of modeling the evolution of behavioral responses, as they can lead to important qualitative, not just quantitative, changes in pandemic dynamics. The canonical behavioral response depends on the perceived risk of infection and severity; e.g., consumers’ behavioral changes in shopping habits were predominantly of their own volition rather than through the imposition of legal restrictions (Goolsbee and Syverson 2021). Behavioral incentives can have a stabilizing effect on the infection growth rate, as the higher it is, the more likely citizens are to react by adjusting their behavior to decrease the chances of infection. However, it is important to also bear in mind that individual behavior may become more reckless as risks are mitigated—see the risk-compensation literature (Peltzman 1975). For example, as the pandemic starts to wane, citizens may adopt more risky practices, thereby slowing down the decrease in the infection rate. Similarly, extensive testing and vaccinations may elicit adverse behavioral responses if citizens believe they are less likely to contract Covid-19 and infect others.

The importance of beliefs and expectations

Importantly, there may be latent (unobservable) variables, such as culture (Laliotis and Minos 2020), at the country-level that jointly influence key variables that are typically assumed to be exogenously determined instead. For example, low social capital and trust in government may affect case growth rates directly and indirectly, through other explanatory variables. Suppose that low-trust countries also tend to have inadequate public health systems. This will have a direct impact on case and mortality growth due to inadequate health care, but also an indirect effect. The government, knowing that the health system is weak (e.g., scarce intensive care units), may impose more stringent NPIs compared to countries with better health systems, in a attempt to prevent them from filling to capacity. Expectations and beliefs of citizens and governments can further introduce endogeneity. Low trust between citizens and government institutions may lead citizens to underestimate the severity of the situation as presented by governments, and the government, expecting this, may act to impose stricter restrictions in anticipation of the weaker behavioral response by its citizens.

Simultaneously modeling endogeneity and unobservable variables, behavioral incentives and beliefs

Acknowledging that confirmed case and death growth, mobility and government policies are endogenous requires a system of four equations to fully model these interactions and avoid biases resulting from simpler regressions that implicitly impose exogeneity. I complement the literature by employing econometric procedures, namely structural multi-equation modeling including unobservable or latent variables and their effects, to improve upon the external validity of prior analyses. Such a system of equations can be viewed through the lens of game theory, that is, the modeling of different agents, who react to each others strategies and expectations thereof. I propose, an admittedly rudimentary, model of the adaptive interactions of these three agents.¹ My approach complements existing studies by investigating a large number of countries, with the associated benefits of pooling estimates for robustness, but other disadvantages such as the assumption of homogeneity across countries and the need to focus on aggregated measures of key endogenous variables–rather than their individual components (Haug et al. 2020)–to avoid a computationally-intractable and unidentifiable system of equations. A central question is whether there are decreasing returns to the effectiveness of NPIs with severity, and if so, to estimate the approximately optimal severity of government policies. This remains a critical question not only for future pandemics, but also because NPIs may be valuable even during the vaccination phase (Rella et al. 2021).

Highly correlated latent (unobservable variables) influence both government policy and confirmed case/death growth rates

The estimated covariances (and associated correlation) between the country-level latent variables in equations 1, 2 and 4 are all positive and significantly different from zero at the 0.001% level: ρ_λ,12=0.98 [95% CI: 0.95,1.00], ρ_λ,14=0.66 [0.54,0.79], ρ_λ,24=0.68 [0.56,0.80]. This validates our hypothesis that there exist significant unobservable variables that may simultaneously influence case and death rates, but also government responses to the pandemic in terms of the severity of restrictions. Ignoring these relationships by estimating a single regression of case or death growth rates, as is commonly done in the literature, would have led to biased estimates.

Adaptive expectations of the risk of infection impact non-residential mobility

Beyond the effect of NPIs, citizens react to increases in the 7-day lag in the growth rates of confirmed cases by reducing mobility. This is consistent with a theory of citizens forming adaptive expectations about the severity of the pandemic at any point in time and the risk of contraction based on the data. The impact of the daily growth in confirmed cases is [-0.255,-0.174]. Putting the effect size into perspective, an increase in the case rate of 1% point, would result on average in a relatively weak 0.21% point reduction in mobility compared to the baseline.

The indirect links between NPIs and confirmed cases/deaths

The indirect links consist of two components in each case, (SI → M) and (M → Ċ), (SI → M) and . The following results will establish that both indirect components are significantly different from zero. Consequently, non-residential mobility acts as the mediator in this relationship between NPI stringency and case/death growth rates. First I report the evidence regarding the SI → M component that is common to both instances and then the second component.

Higher NPI severity restricts non-residential mobility

Non-residential mobility is clearly impacted by the SI (see the estimates in Table 2 and Figure 3 below), as the null hypothesis that all SI dummy variables are equal to zero is rejected (χ² (10) = 428.38, p < 0.0001). Furthermore, it is monotonically decreasing in the ten SI ranges.

• Download figure
• Open in new tab

Figure 3:

Impact of policy Stringency index on the percentage change in non-residential mobility (γ_1,l)

Lower non-residential mobility increases case and death growth rates

The theoretical motivation behind restricting mobility through lockdown measures is that its reduction will lead to a fall in the number of social interactions, with the hope that this will reduce transmission and infection. However, a reduction in non-residential mobility must, by definition, be mirrored by an increase in the time spent within residences—indeed, they are highly negatively correlated in the Google mobility data; the median correlation within panels is -0.89. Consequently, reductions in non-residential mobility may have a detrimental effect on case/death growth as although transmission outside the home may be reduced, within-household transmission may be enhanced as people spend increasingly more time within the confines of a closed space with others (Sun et al. 2021). Since the two effects work in opposite directions, whether restricting non-residential mobility reduces case/death growth or not must be resolved empirically. Our finding that the coefficient of M_i,t−7 is significantly negative for cases, and similarly M_i,t−21 for deaths , supports the hypothesis that the benefits of reduced non-residential mobility are more than outweighed by the detrimental effects of increased within-household transmission (conditioning on the stringency of government policies). The effect size is moderate, as a 10% point decrease in mobility leads to an increase in 0.4% points of the case growth rate.

The direct link between SI and case/death growth rates

Turning to the direct effect of the stringency index of restrictions on case growth rates, the hypothesis that the set of estimates are all equal to zero is rejected (χ² (10) = 137.28, p < 0.0001). Furthermore, there is evidence of a nonlinear relationship, with strongly decreasing returns to the effectiveness of restrictions with increasing severity (see Table 2, and note that the conclusions are similar for death growth rates).

The total effect of NPI stringency on case/death growths and the optimal level of SI

The total effect of NPI stringency on case growth can be computed by adding the direct β_2,l and indirect paths γ_1,l × β₁ for each stringency level of l. This combined effect of SI on Ċ_i,t and is documented in Table 4 and presented graphically below in Figure 4.

• Download figure
• Open in new tab

Figure 4:

Total effect of the level of the Stringency Index

The maximum impact of SI on case growth is observed for the SI range of 61–70. We test the difference in effectiveness of all other levels against the most effective range 61–70, correcting for multiple comparisons using the Sidak correction.² There is no difference in effectiveness for the ranges 51–60 and 71–80 (see Table 5 for the test statistics); the ranges (81–90, 91–100) are significantly less effective than the range of 61–70. Consequently, there are no further gains to be achieved beyond the SI range of 51–60. The socially optimal SI range, however, must account not only for the positive effects of NPIs, but also for the significant economic and other health-related costs that result from restrictions (for example, see Miles, Stedman and Heald 2020; Susskind and Vines 2020; Altman 2020). While this would require a full cost-benefit analysis (Rowthorn and Maciejowski 2020;Appleby 2020) that is beyond the scope of this paper, we can derive the approximate upper bound of the socially optimal level of the SI with a single assumption about the cost profile of different SI levels. Namely, I only assume that the costs are monotonically increasing in the SI level. Consequently, without the need to quantify costs, I conclude that the upper bound of the socially optimal SI level, SI* is 51–60, i.e., the minimum SI range that is not significantly different from the maximum effect at 61–70.

While the quantification of the exact costs is an important endeavour, it is fraught with many difficulties, such as converting non-economic outcomes into monetary terms and would require many assumptions about highly uncertain possible effects, many in the distant future. By contrast, this upper-bound on SI* is extremely robust, albeit less informative. The above derivation of SI* is based on statistical significance, however it is possible that lower SIs may be statistically significantly different from SI*, but that the difference in the effect size is practically of little importance. Indeed, the ratio of the effect sizes of an SI range of 31–40 relative to the average of the 51–80 range, is 91% [85%, 97%]. The relative effectiveness of the even laxer 21–30 range is 72% [57%, 86%]. If the costs of NPIs increase quickly with NPI stringency, it is conceivable that moderately severe policy responses in the 30–40 SI range (and possibly, even 21–30 SI) may in fact be close to the socially optimal SI* (arising from a full cost-benefit analysis), as it already achieves 91% effectiveness without accounting for costs.

Similarly, the maximum effectiveness on death growth rates is also observed for an SI range of 61–70. We test the difference in effectiveness of all other levels against this range. There is no significant increase in effectiveness for levels beyond 41–50 (see Table 7). In terms of the effect size relative to the average of the 41–90 range, an SI range of 31–40 achieves 93% [87%, 98%] of the effectiveness of the former and the laxer 21–30 range achieves 73% [58%, 88%].

Finally, while the SI index aggregates various individual NPI policies, examining the median values of the individual policies in the dataset for each SI range can be informative—see Table 1. Note, that for the 31–40 SI range, which achieves at least 90% of the maximum impact, the median policies do not include any level of restriction on transport and internal movement, and no stay-home restrictions. Furthermore, only recommendations for closing schools and workplaces and cancellation of public events were issued, i.e., these were not mandatory. The only stringent individual policies typically arising in the 31–40 SI range were quarantining high-risk cases from international travel and restrictions on gatherings of 100-1000 people; however, these two policies are not reflective of citizens’ everyday behavior. Consequently, voluntary–rather than mandatory–behavioral changes are more important drivers of the impact of NPIs on case (and death growth).³ This is consistent with other studies that also conclude that the flattening of NPI effectiveness with increasing stringency reflects a relatively stronger voluntary rather than mandatory component to behavioral changes (e.g., Allcott et al. 2020; Bjørnskov 2021; Bonardi et al. 2020; Bendavid et al. 2021).

Whence arises this voluntary behavioral change? As we showed above, part of it is from citizens’ expectations of the risk of infection and severity as captured by in equation 3, whose effect size, however, was found to be relatively small. The rest, and majority of the voluntary behavioral change, is likely due to the signalling value of policy decisions, i.e., citizens can also use information about the stringency of the government measures to infer the severity of the pandemic. This implies that recommendations by governments, for SI ranges of up to 40, were heeded by citizens, who significantly changed their behavior in ways beyond those captured by mobility in eq. 3, e.g., preventative measures such as diligent hand washing and mask-wearing, effective self-isolation when infected et cetera.

Extensive public testing significantly reduces case and death growth, contact tracing does not

Figure 5 presents the estimated coefficients and associated 95% confidence intervals—see Table 2 for detailed regression results. All three levels of the testing regime are jointly significantly different from zero (χ² (3) = 66.03, p < 0.0001), leading to progressively greater declines in case growth as testing becomes more extensive (robust to multiple comparison Sidak corrections).⁴ Note, that the most extensive testing policy has an impact of -7.39 [-9.926,-4.854] which is 51% [27%,76%] of that of the most impactful SI range (61–70).

• Download figure
• Open in new tab

Figure 5:

Estimated coefficients across levels of testing policy and contact tracing

Similarly, testing policy significantly reduces the death growth rate (χ² (3) = 42.81, p < 0.0001). Apart from the monetary costs associated with implementing a testing regime it does not impose other negative social externalities, in contrast to lockdowns. Coupled with the significant impact on Covid-19 dynamics, this renders extensive testing a desirable tool.

Contact tracing of any level does not have a significant impact on confirmed case growth (χ² (2) = 1.93, p = 0.38) or death growth (χ² (2) = 1.31, p = 0.52). One should keep in mind that contact tracing may still be effective if the number of daily new cases is small, when efficient tracing is more manageable. However, there remain important challenges to scaling contact tracing (Chowdhury et al. 2020; Quilty et al. 2021; Kretzschmar et al. 2020) that could hinder its effectiveness during significant outbreaks.

The proportion of the population tested daily does not significantly affect cases and deaths

While both estimates are negative, as expected, increasing the proportion of daily tests does not significantly reduce the case growth, , nor the growth rate in deaths, -0.126 [−0.315, 0.063, p = 0.191]. Note, that an earlier analysis with datapoints till the end of January 2021 only, revealed a statistically significant impact. The inclusion of different levels of the ordinal testing policy variable may partially capture this effect, as they will be correlated to some degree with the proportion tested, i.e., extensive testing as coded in the ordinal variable would likely imply more testing. Finally, this may be the result of implicitly assuming exogeneity, however testing may also be endogenously determined as governments are likely to step up testing during phases with higher transmission.

Vaccination reduces case and death growth rates

Despite few datapoints where vaccination was already well underway, each 1% point increase in the cumulative vaccination % results in a reduction of the case growth of -0.017% points [−0.032, −0.002, p = 0.022] and in the death growth rate -0.0187% points [−0.035, −0.002, p = 0.026]. Note that the median (non-zero) cumulative vaccination % across countries is only 2.6% and the 10th and 90th percentiles are 0.07% and 19.8%, respectively. Consequently, these relatively low estimates should not be assumed to extrapolate for higher vaccination levels, especially since this is likely to be a nonlinear relationship in reality, i.e., the effect of vaccinations will increase at an accelerating rate as it approaches the herd immunity level.

Government policy is endogenous and exhibits hysteresis

Government policy is strongly endogenous, in contrast to the common implicit assumption of exogeneity. The seven-day lagged confirmed growth rate coefficient is positively related to NPI severity. Furthermore, I find significant hysteresis in the de-escalation of NPIs, that is for the same case growth, NPIs are significantly more stringent if the case growth has been falling recently, than the opposite: .

Interpretation and implications for policy

What implications does this have for policy? I report in Table 1 the median values of the individual NPI constituents of the SI index for these upper bounds on the socially optimal level of NPI stringency for both confirmed cases (51–60) and deaths (41–50), and the range of 31–40, which is near-optimal. Note, that there is significant heterogeneity across individual NPIs in terms of their severity. The most severe implemented restrictions are those on gatherings and international movement. At the other extreme, regarding transport and internal movements, no restrictions or recommendations were made in these three SI ranges. Stay-home restrictions and recommendations are also absent with the exception of a recommendation to stay home associated with the 51–60 range. Moderately severe restrictions are typically implemented for schools and workplaces, that is, the optimal upper bound for confirmed cases enforces partial closing (targeted, not across the board) whereas that for death rates and the near-optimal SI range (31–40) call only for recommendations to work from home.

These findings are generally aligned with studies finding that more severe restrictions were not significantly more effective than less restrictive policies (Bendavid et al. 2021; Haug et al. 2020; Bjørnskov 2021; Bonardi et al. 2020). However, they stand in contrast to others that concluded that strict lockdowns were effective (Flaxman et al. 2020; Islam et al. 2020), whilst differing in some policy interventions such as school and workplace closures, but agreeing on others such as mass gathering restrictions (Islam et al. 2020). Differences with these studies may arise due to their more limited time-series and/or the lack of explicit behavioral modeling of citizens’ reactions to the pandemic. The majority of the effect of NPIs on case and death growth rates could be attributed to voluntary behavioral changes rather than mandatory, government-imposed imperatives. Note also, that the 31–40 range typically includes a coordinated public informational campaign aimed at influencing voluntary behavior. This highlights the importance of modeling the behavioral incentives of both governments and citizens in conjunction with the pandemic dynamics.

Importantly, the study’s conclusions were shown to be robust to the rectification of econometric issues such as endogeneity and correlated unobservable latent variables, and accounting for citizens’ and governments’ voluntary behavior.

Strengths and limitations of this study

The simultaneous modeling of pandemic dynamics with behavioral models of citizens’ behavioral adaptation to the pandemic, along with a model of governments’ decision-making processes regarding policy implementation is the primary strength of this study with respect to earlier work. Identification of this more sophisticated model with behavioral components was made possible by the larger amount of accumulated data including testing and vaccination rates. Nonetheless, certain simplifications were still necessary to ensure identification and to rein in the computational complexity of the estimation processes. These simplifications included examining the effects of nations’ stringency index compiled from individual NPIs, rather than examining each NPI separately. Similarly, an average measure of the change in mobility was used instead of disaggregated sub-measures of the type of mobility. Furthermore, while random-effects allowed for variation across countries in unobservable variables, estimates of the variables of interest (NPIs and other interventions) were pooled across countries.

Unanswered questions and further research

As more data becomes available over time, future research should be directed towards relaxing some of the acknowledged limitations of the current modeling. For example, allowing for heterogeneity in the variable estimates across countries would be desirable rather than pooling estimates across countries, as would the use of sub-national data. The analysis of vaccinations should extended once data is available for higher levels of vaccination to properly estimate the likely nonlinear effect beyond the currently low levels reported in this dataset.

Finally, the conclusions reached herein must be placed within the context of and validated by other methodological approaches, such as SIR and agent-based models. However, I have presented significant evidence that strict NPIs provide no further benefits over less stringent ones, and that the latter function primarily as signals for significant voluntary changes in behavior rather than mandatory changes.