Ranking the effectiveness of worldwide COVID-19 government interventions

Data

Case-control analysis

We perform a case-control regression analysis to quantify the impact of implementing an NPI measure on the effective reproduction number, R_t. The central idea is to compare all countries that have implemented a certain measure with all countries that have not implemented this measure at the same stage of the epidemic while adjusting for several country-specific covariates through regression analysis. These covariates include timing (time-span between the day on which more than 30 cases were confirmed and the day the measure was implemented), baseline R_t (reproduction number before the measure was implemented), size and population density, as well as covariates for how many other measures have already been implemented. It is assumed that implementing a NPI on day t will have measurable impacts on R_t at day t + τ. We also study how various country-specific indicators for economic and human development as well as governance impact on these results.

Exposure variable

We consider each NPI implemented in more than ten countries. We include measures published by ²⁵ on two different resolution levels (L2 and L3) separately. Let T_M be the day on which a country implemented measure M. The covariates and outcome variables are measured relative to this point in time. As an exposure variable, we use a dummy variable, X, encoding whether a country has implemented the measure during the observation window or not.

• Download figure
• Open in new tab

Figure S1:

Comparison of effectiveness rankings on the coarsest hierarchical level for the case-control analysis (CC), LASSO regression (LASSO), random forest regression (RF), and transformer analysis (TF). To obtain a ranking of the eight different themes (L1) of NPIs, we sum the impacts of the 3 highest ranked categories of each theme and then rank the themes according to this cumulative impact. All methods indicate that NPIs belonging to the themes of social distancing, travel restrictions as well as healthcare and public health capacity lead to the most significant reductions of R_t. Environmental measures (e.g., cleansing public places) are ranked least effective in each approach.

• Download figure
• Open in new tab

Figure S2:

Normalised scores of the NPI categories in CCCSL, averaged over the four different approaches.

• Download figure
• Open in new tab

Figure S3:

Normalised scores of the NPI categories in CoronaNet, averaged over the four different approaches.

• Download figure
• Open in new tab

Figure S4:

Normalised scores of the NPI categories in WHOPHSM, averaged over the four different approaches.

Covariates

We include the following covariates in all analyses. First, a country’s epidemic age when measure M was implemented, A(M), defined as the number of days between the implementation of the measure, T_M, and the first day with more than 30 confirmed cases, denoted by T₀, giving A(M) = T_M − T₀. Second, the baseline effective reproduction number, , is taken at day T_M + τ. Third and fourth, we include for each country the logarithms of its total population P and population density D ⁵⁷. Furthermore, we include the number N^All(T_M) of all implemented L2 measures and the number N^L2(T_M) of all measures (L2 or L3) from the same categories as M that have been implemented before T_M. These covariates are supposed to capture whether a country introduces the intervention under consideration early or late in relation to its epidemic age and the number of measures already taken.

Outcome variable

As a dependent variable in the regression, we consider the change in effective reproduction number after implementation of the measure, , where is the value of R_t taken on day T_M + τ. We report the average of ΔR_t(τ) taken over time lags 0 < τ ≤ 28.

Matching

As cases we include all countries implementing a given measure at a time A(M^c) > τ. As controls we consider all other countries that have not implemented this measure within the observation window but that implemented any other intervention not more than one day sooner or later than its matched case. These matching criteria ensure that we do not introduce a bias by comparing countries that implemented any response with countries that did nothing at all and maybe had the epidemics already under control.

Additional country variables

We consider variables that capture economic and human development, as well as different dimensions of governance. Economic development is measured by the country’s per capita GDP adjusted for power purchasing parity (PPP) ⁵⁸. Human development is quantified by the human development index (HDI), an indicator provided by the United Nations Development Programme taking life expectancy, education, and per capita income into account ⁵⁹. Finally, we consider six different dimensions of governance as quantified by the World Governance Indicators (WGI) ⁶⁰. These indicators include Voice & Accountability (free media, the extent to which citizens participate in the government), Political Stability (including the absence of violence), Government Effectiveness (quality of public services), Regulatory Quality (the ability of government to implement sound policies), Rule of Law (the extent to which citizens abide by the rules of society) and Control of Corruption.

Statistical analysis and multiple testing

A linear regression model of the form, is evaluated for all NPIs meeting the inclusion criteria. We evaluate the above model for all measures and permissible delays τ. We adjust for multiple testing by controlling the false discovery rate at level 0.0001 via the Benjamini-Hochberg procedure. We investigate the impact of the additional country variables by evaluating models as in Eq. (S1) with an additional term for the considered indicator.

Lasso regression

We assume that without any implemented NPIs, the R(t) is constant, and deviations are caused by a time-delayed onset of the effects of each NPI on L2. This hypothesis is an oversimplification of the reality, but it allows to estimate this time-delay itself and to quantify magnitudes of impacts for each NPI. A similar approach has been reported, for instance, in ¹⁹, although the authors use a smaller list of NPIs.

Formally, this model for a single territory c is captured in the expression Here, bold symbols indicate vectors containing discrete time-series, starting at 30 confirmed cases for each country. The vector Θ_t models a discrete version of the Heaviside function with Θ_t = 0, …, 0, 1, …, 1, with the step from 0 to 1 at position t. In Eq. (S2), A(M, c) indicates the day when a NPI M is implemented in territory c, and τ is the offset in days to account for the time-delay until the NPIs affect the case numbers. The first term in Eq. (S2) is a constant composed of the average effects β_avg and potential territory-specific effects β_c. For inference, we concatenate all territories into one large vector. Thus, in total we need to estimate 126/326/265 coefficients (1 constant, 79/218/224 territories and 46/107/40 NPIs for CCCSL/CORONANET/WHOPHSM) in this model.

Regression is carried out by minimizing the target function to obtain values for β. The first term is the Residual Sum of Squares (RSS) of the model with respect to actual observation data. The second term penalises too large values of the coefficients β, with α a penalty parameter that indicates the weight of this penalization in the optimization procedure. Here, ‖·‖₂ and ‖·‖₁ denote the 2-norm (euclidean distance) and 1-norm (sum of absolute values), respectively. The 1-norm for the penalizing term is characteristic for lasso regression ⁶¹, which acts as feature selection by estimating multiple coefficient values β as zero. Formally, this additional penalty term can be shown to be equivalent to assuming a Laplace-distribution prior on all values in β. Thus, the tuning parameter α effectively balances the trade-off between two objectives: the goodness of fit to the data (RSS) with the complexity of the model (second term)^{61, 62}.

Cross-validation and meta-parameter search

This model includes two meta-parameters (τ, α), which are estimated by cross-validation to obtain a minimal RSS: At first, countries are randomly assigned to one of 10 groups. Then, each of these country groups is dropped from the vectors R, and coefficients β are estimated via the minimization in Eq. (S3). These coefficients β from the training group are used to compute an value of the model on the dropped countries to test how good the model can predict previously unseen observations. As the different country groups can contain a different number of observations, we compute the overall for a given set of (τ, α) as which weighs the individual coefficients of determination r² for each test with the variance in the reproduction number R_g within the test group. This whole procedure is carried out on a grid of possible (τ, α) values, to find a set of meta-parameters, where the model Eq. (S2) can best describe the data. Fig. S5 depicts the resulting values of for a sufficient large range of meta-parameters. As the overall curve for is relatively flat, we repeat the cross-validation 30 times with different assignments to the 10 country groups, which reduces the overall noise. The final is then computed using the same weighting procedure in Eq. (S4) for the set of all 20 repetitions and each of the 10 territory groups within a repetition. We find that a time-offset of τ = 11/11/15 days and a penalty parameter α = 0.005623/0.00100/0.004217 are the best parameters to describe the observations with this model for the CCCSL/CORONANET/WHOPHSM, see Fig. S5.

Final coefficient estimation

We estimate the ranges of each NPI effectiveness shown in Figs. S12, S16 and S20, using a territory-wise cross-validation, i.e., by reducing the sizes of test groups to one. For each estimation leaving out one territory, we compute the coefficients β_M for each NPI M and β_c for all territories c. We identify these coefficients β_M and β_c as the change ΔR_t. Shown intervals contain 95% of all observed values, with the center indicating the median. From the overall 126/355/305 coefficients β, the feature selection aspect of lasso regression finds 23/19/23 relevant NPI coefficients and 1/1/23 relevant territory coefficients, while other coefficients are estimated as zero (for CCCSL/CORONANET/WHOPHSM) Note that this does not indicate that these NPIs are useless to reduce the spread of the virus, but rather that the algorithm can explain all observations with a smaller number of coefficients. The number of selected coefficients is highly sensitive to the value of the penalty parameter, however, the more impactful NPIs are consistently present with similar coefficients (see Fig S5CFI).

• Download figure
• Open in new tab

Figure S5:

Cross-validation results for LASSO regression. (A,B,D,E,G,H) r² values for the test and training data, averaged over 30 repetitions of assigning countries into 10 groups for each of the three data sets (CCCSL/CORONANET/WHOPHSM). With smaller penalty parameters α, increases as the algorithm uses more NPIs to explain the training data. This overfitting, however, reduces the ability to fit the previously unseen test data. In contrast, for large penalty parameters α, not enough NPIs are used to explain the data, and the method cannot explain either training or test data. The optimal meta parameters are found as (τ, α)_CCCSL = (11, 0.005623), (τ, α)_CORONANET = (15, 0.004217) and (τ, α)_WHOPHSM = (11, 0.00100) (C,F,I) Effects of all NPIs and territories vary when changing the penalty parameter α. However, most of the effects of the significant NPIs (see Figs. S12, S16 and S20) stay roughly constant, only the number of significant NPIs increases.

Random forest regression

We use random forest regression ⁶³ as a third method to assess the impact of the implemented measures on the spreading of COVID-19, measured in terms of the effective reproduction number R_t.

We represent for each country and each day of the observation period the NPIs which have been implemented in that country until that day in the form of a binary vector. The analysis is performed on the NPI categories (L2). The binary data on implemented NPIs in a given country on day t is regressed on the value of R_t in that country τ days later, R_t+τ. The time shift τ accounts for the time delay between infection and case confirmation. We vary τ between τ = 0 and τ = 20.

Each random forest consists of 500 decision trees with maximum depth d. At each split of a node in one of the trees in the random forest, m randomly selected NPI categories out of the total number of 46 categories are considered. We employ bootstrapping, meaning that each decision tree is fitted on a different random subset containing 75% of the rows of the predictor matrix.

To implement the random forest regression, we use the RandomForestRegressor class from the python library scikit-learn ⁶⁴.

Cross validation

For each 0 ≤ τ ≤ 20, we use 10-fold country-wise cross validation to determine the optimal values of the maximum tree depth d and percentage of considered features m. We vary the parameters d and m in the range 1 ≤ d ≤ 15 and 1 ≤ m ≤ 100. For each combination of d and m, we randomly split the set of all territories into a training set and a test set. The random forest is trained on the training set data; subsequently, we measure the performance of this random forest in predicting the time series of R_t+τ for the countries in the test set. As a performance metric, we quantify the difference between the predicted time series of R_t+τ and the observed one by using the coefficient of determination r². We repeat the same procedure 10 times with different random splits of the set of territories into training and test set. Then we take the mean of the coefficient of determination over the 10 splits to obtain the average out-of-sample performance of the random forest for this combination of d and m. The heatmap in Fig. S6 shows the dependence of the performance of the random forest on unseen data depending on the parameters d and m for fixed time shift τ = 10.

Feature importance

To quantify the importance of a NPI, we measure the loss of predictive performance of the random forest if the information carried by the given NPI is replaced by noise. This measure of feature importance is also known as permutation importance. If the loss in performance is high for a given NPI, then we conclude that knowledge of the implementation status of this NPI is important for predicting R_t+τ.

Specifically, for each time shift 0 ≤ τ ≤ 20 and each NPI M, we measure the reduction in performance of the random forest in predicting R_t+τ for unseen data when the values in the column corresponding to M are randomly shuffled. The maximum tree depth d and the percentage m of features considered are set to their optimal values as determined in the cross-validation step. To obtain sharp estimates for the permutation importance of the different NPIs we use repeated 10-fold country-wise cross validation with 100 repetitions. Different NPIs attain their maximum value for different values of τ.

• Download figure
• Open in new tab

Figure S6:

Heatmap of the predictive performance of the random forest depending on the maximum tree depth d and the percentage of features m considered at each split for time shift τ = 10. The predictive performance is measured in terms of the coefficient of determination (r²) on unseen data, averaged over 10 random splits of the territories into training and test sets.

Transformers modeling

The temporal and dynamic nature of epidemic propagation finds a suitable tool to model such a temporal evolution in Transformers ⁵⁴. This neural network can recall past events presented at the input by leveraging its innate ability to take into account the previous information. The intrinsic Transformer architecture ensures this ability for all the temporal data is represented and considered at once without recurrent connections as in Recurrent Neural Networks. At a generic time t, to approximate the value of the effective reproduction number at time t + 1, R_t+1, we use a Transformer whose input is the daily representation of the adopted measures in a given country in binary form, similar to the encoding performed in Random Forest, along with the value of R_t measured in the same day. The best performing network has been identified as having four hidden layers of 128 neurons, an embedding size of 128, 8 heads, one output described by a linear output layer, and 54 inputs (measures ⊕ R_t, where ⊕ is the concatenation operator). NPIs are represented through a theta function over time, whose step position corresponds to the day when the specific measure has been adopted in the selected country. For the neural network training, 10 out of 78 territories were chosen as the validation set, and the best neural network was found when the mean square error between the values of the validation set and the corresponding network outputs reached the minimum amount. Different Transformers training can lead to slightly different predictions. Although the main trends of forecast maintain a substantial similarity across the various training, the corresponding time evolution may change when some containment measures are perturbed. This issue is possibly due to a large number of local minima affecting the neural network’s energetic panorama. To address this problem and to provide an adequate assessment of the importance of the measures, we built an ensemble of trained Transformers with a comparable level of training precision.

Transformer ranking assessment

This approach has been chosen to be consistent with the other methods implemented in this paper, in which the impact of the NPIs is related to the comparison between countries having or not having implemented a measure. Therefore the comparison is performed between the normal prediction of the Transformer after the application of a given NPI and the corresponding prediction when the same measure is removed. The difference between the predictions quantifies the impact of that NPI on the overall behaviour of R_t of the selected country. The final relative importance of the containment measure meas_i is then given by , where mean_{T F} (·) is the mean operator over the RNN ensemble, R^{T F} (t) is the reproduction number predicted by the TF th network, (t) is the corresponding temporal prediction when the measure meas_i is removed, and T is the period over which the assessment is performed.

Transformer model at country level

Implementing a Transformer as a general model for describing the dependence on the measures of R_t allows, at the same time, to evaluate the impact of the measures on a single country. Each country is identified by a specific temporal sequence of applied measures and a unique R_t evolution over time. Then, we can interrogate the Transformer to assess the importance of every single measure for that particular context by removing the measure under investigation from the specific sequence as in the general knockout case and building an associated ranking of NPIs for each country. For example, Fig. S7 shows the impact on R_t of the NPI “Small gathering cancellation” in Italy when such a measure is removed from the temporal sequence of adopted NPIs. This approach allows us to assess the country-specific NPI ranking. By comparing these ranking across all countries, we measure the specificity of the impact of its measures.

What-If experiments with Transformers

As mentioned above, Transformers offer the possibility to perform what-if experiments, i.e., exploring different scenarios corresponding to time-series of events different from the actual ones. This approach opens the possibility to examine scenarios corresponding to different combinations of NPIs. In the specific case of the Non-Pharmaceutical Interventions to mitigate the spread of COVID-19, starting from the actual sequence of NPIs adopted by one particular country, we tested the efficacy of artificial sequences by removing specific NPIs or shifting their adoption to other days. In this framework, one can compare the actual evolution obtained through the synthetic sequence of actions to suitable reference sequences. More specifically, we adopted the Transformers to assess what would have happened if a given NPI had been adopted on a different day with respect to the actual day of adoption. To this end, we adopted the following procedure:

• for each country and each NPI, we first compute a knockout evolution of the system by letting the Transformer simulate the evolution of the system once the specific NPI has been removed.

• we create a synthetic sequence by keeping the sequence of NPIs of the country untouched except for the specific NPI that is positioned in a generic day t_i. For that sequence, we compute the evolution of the system as given by the Transformer, i.e., we calculate the time evolution of R_t for the specific synthetic sequence.

• We repeat the above operation for a generic choice of the day of adoption, t_i, of the specific NPI.

• For each country and each NPI, we compute the average difference between the synthetic evolution of R_t with the specific NPI positioned at t_i and the knockout evolution over the 30 days following t_i. The average here is performed over several realisations of the Transformers.

• The outcome is a series of curves that, for each NPI, report the variation of R_t, ΔR_t, averaged over all the countries that adopted that NPI and over several realisations of the Transformers.

• Download figure
• Open in new tab

Figure S7:

Example of the country-specific impact of an NPI. The plot shows the R_t increase when the NPI “Small gathering cancellation” is removed from the temporal measures sequence in Italy. Shaded areas are the standard deviations of the Transformer prediction averaged on the networks ensemble.

Fig. S8 shows the effect of the selected NPI “Small gathering cancellation” in Italy if it had been adopted on different days, compared to the case where the same NPI is absent (knockout evolution). The reduction of R_t is visible, while the overall impact decreases as one shifts the day of adoption ahead. The Transformer evolution, when the selected NPI is adopted on any other day, takes into account the effect of all the other NPIs adopted in Italy and kept fixed throughout the simulation.

Fig. S9 reports the evolution of ΔR_t for a selection of NPIs that display a “the earlier, the better” behaviour, that is, whose ability to reduce R_t tends to decrease with the epidemic age of their adoption.

• Download figure
• Open in new tab

Figure S8:

Example illustrating the what-if experiment described in the text. Here we consider the NPI “Small gathering cancellation” in Italy and we simulate what would happen if it had been taken at epidemic ages 0, 10, 20, 30, 40, 50, 60. We then compare this evolution with an evolution obtained through a knowknout of the same NPI (knockout evolution). The other concurrent NPIs and their days of adoption are kept fixed throughout the simulation (green rectangles at the bottom).

• Download figure
• Open in new tab

Figure S9:

Outcome of the what-if experiment performed with the Transformers. We report, for several NPIs, the behaviour of the averaged variation of R_t, ΔR_t (see text for the definition), when the NPI is adopted at a generic epidemic age compared with an evolution in which that NPI has been knocked-out. Negative (Positive) values of ΔR_t indicate a decrease (increase) of R_t.

Results from the Case-control Analysis on country covariate effects

The impact of different country variables, including demographic and governance indicators, as well as measures for human and economic development, are shown in Fig. S10. This heatmap presents the average standardised coefficient values from all regression models for a given NPI category (L2) that yield a significant impact on R(t) for various delays (the average is taken over the delays). The standardised coefficient values are the coefficient values (estimated change in R_t) divided by its standard error (SE); the t-Statistic. The significant NPIs show effect sizes in reducing R_t of about 5 standard errors. Note that no statements concerning causal relationships can be made based on these findings; we report correlations from a cross-sectional analysis.

A number of NPIs appear to be less effective in countries with a high population density and low GDP ⁶⁵. Human development, as quantified by the Human Development Indicator, has no or only very mild impacts on measure effectiveness. The governance indicators Political Stability and Voice & Accountability are negatively correlated with the effectiveness of most measures. Most of the remaining governance indicators show either no or moderately negative correlations. There are, however, exceptions for some interventions that show consistently positive correlations with government effectiveness, regulatory quality, rule of law and control of corruption, in particular measures to increase the availability of medical supplies and PPE. It is unlikely that these results are confounded by poor reporting in less developed countries, as such biases would affect R_t values from before and after measure implementation similarly and hence cancel each other out. Although there is no clear pattern concerning the influence of the total number of NPIs already implemented, there seems to be a trend regarding the number of NPIs of the same category. As a general tendency, having already implemented measures from a given category makes additional NPIs from the same category more effective, hinting at a relationship where one measure (e.g., closure of restaurants) amplifies the effectiveness of other related measures (e.g., home office).

Results from the Random forest regression

Figs. S13, S17 and S21 give the rankings of the different NPIs in descending order according to their feature importance for the three analyzed datasets.

Results from Transformers Analysis

We trained 20 different Transformers after selecting the best out of 100 training procedures to build a representative ensemble of networks to address multiple local minima issues. All the Transformers reached similar values of test loss with a relative deviation of less than 5%. The NPIs impact evaluation is performed by averaging the difference between the prediction and the prediction when a given measure is removed. The area under such a difference normalized by the temporal amplitude represents the mean variation of ΔR(t).

Social distancing

Bans of small gatherings (gatherings of 50 persons or less) and the closure of educational institutions have a more substantial effect on R_t (but are also more intrusive to our daily lives) than the prohibition of mass gatherings, measures targeting special populations (e.g., elderly, vulnerable populations, hospitalised patients, prisoners or more exposed non-healthcare professionals) or adaptive measures for certain establishments (e.g., places of worship, administrative institutions, entertainment venues, nursing homes). While in earlier studies based on smaller numbers of countries, school closures had been attributed only a little effect on the spread of COVID-19 ^{19, 20}, more recent evidence has been in favour of the effectiveness of this NPI ^{28, 29}. This is also in line with a contact tracing study from South Korea which identified adolescents aged 10–19 as the biggest spreaders in household settings ³⁰. Social distancing measures are less effective in countries with a high population density and a high degree of citizen participation in the government, as well as freedom of expression or free media (WGI Voice & Accountability). The country-level analysis confirms that these NPIs have a particularly high entropy, meaning that their effectiveness varies indeed substantially across countries. An exception to that are the measures for public transport and work safety protocols, where the latter mostly refers to mandatory guidelines for, e.g., physical barriers or fever checks at workplaces. These two social distancing measures have a low effectiveness rank (little significance across the methods) and low entropy, meaning that they had no impact on R_t consistently across most countries.

• Download figure
• Open in new tab

Figure S23:

Observed values for R_t (solid lines), together with the range of values for R_t as predicted from the different methods (shaded regions).

• Download figure
• Open in new tab

Figure S24:

Analogue to Fig. 1 of the main text if the analysis is done on all countries except those in Europe and Africa.

• Download figure
• Open in new tab

Figure S25:

Analogue to Fig. 1 of the main text if the analysis is done on all countries except those in Asia and Oceania.

• Download figure
• Open in new tab

Figure S26:

Analogue to Fig. 1 of the main text if the analysis is done on all countries except those in North, Central or South America.

• Download figure
• Open in new tab

Figure S27:

Analogue to Fig. 1 of the main text if the analysis is done on the WHOPHSM data set.

• Download figure
• Open in new tab

Figure S28:

Analogue to Fig. 1 of the main text if the analysis is done on the CORONANET data set (continued on the next page).

• Download figure
• Open in new tab

Figure S29:

Analogue to Fig. 1 of the main text if the analysis is done on the CORONANET data set (continued).

Healthcare and public health capacity

An increase in the availability of personal protective equipment (PPE) to the healthcare workforce, together with measures aiming to reduce the number of non-COVID-19 or non-critical COVID-19 patients in medical centres and hospitals (by promoting self-isolation of mildly symptomatic patients, setting up health hotlines, etc.) are also essential building blocks of successful containment strategies. All of these measures combine high effectiveness of early implementation and low entropy, meaning that they are similarly effective in most countries. Actions aiming to enhance the health system are critical. They are the primary response to patients and have no (or few) negative repercussions on individual rights of liberty (exception on the travel restriction for healthcare workers imposed by several countries). Our results demonstrate that government support to the health system needs to be a priority during a health crisis in order to reduce mortality ⁶⁶. In line with our result “the earlier, the better”, we argue that those actions must be taken early enough to prepare for a surge in healthcare demand. Compared to other interventions, increased medical supplies and availability of PPE show substantially stronger positive correlations with several governance indicators including government effectiveness and control of corruption. Indeed, there are increased news reports currently on scandals related to government procurement of PPE ^67–69.

Travel restrictions

Different types of travel restrictions also show significant effects, in particular border restrictions (e.g., border closure, border controls), individual movement restrictions (e.g., curfews, the prohibition of non-essential activities) and cordons sanitaires (containment zones). The high effectiveness of border restrictions is driven by European countries (its impact on R_t turns insignificant in two of our methods after removing all European countries); most likely for geographic reasons. This finding is in line with a high entropy score of border, airport, port and ship as well as individual movement restrictions.

Effectiveness of ultimate measures such as stay-at-home orders or lockdowns is still controversial. Recent studies suggest that a national lockdown reduces R_t by an average of 5% ¹⁹ to 80% ²⁰, whereas other interventions seem to reduce the virus spread by 5% ²⁰ to 30% ¹⁹. In some countries or territories, the effect of a lockdown decided in the late stage of the epidemic may not be more effective than previously implemented bans on gatherings ^{19, 20, 70}. Our analysis highlights the importance of early national lockdowns by showing how the relative effectiveness of that measure correlates with the epidemic age of its adoption. However, the reduced effectiveness of lockdowns at higher epidemic age, as observed in Fig. 4, does not necessarily imply that taking this NPI late is useless.

Risk communication

In terms of risk communication, we find that pro-active communication with stakeholders from the private sector (e.g., business owners or chief executive officers) to promote voluntary safety protocols in enterprises, businesses, event organization, government administrations, etc., shows a significant effect in each of the four analyses, mainly when implemented early. Three out of four approaches also indicate a substantial impact of public health communication strategies (i.e., non-binding NPIs) encouraging citizen engagement and empowering them with information.

Resource allocation

Measures for resource allocation show limited impacts on R_t in our analysis (e.g., police and army interventions being insignificant in all studies) with relatively high entropy, meaning that country-level effects are important. Surprisingly, the implementation of crisis management plans turns out to be highly effective, except for the Americas. After removing countries from North and South America from the analyses, all four of our methods agree on significant effects of crisis management plans with an ΔR_t of down to −0.3, suggesting a lack of effective crisis plans in American territories. For instance, US states had to focus on providing health insurance and economic stimulus as well as facilitating administrative procedures, while European countries could develop their plans on top of a stronger socio-economic basis ^{71, 72}. Crisis management plans are also more effective in countries with a non-participatory government, meaning that countries with increasingly authoritarian practices might be at an advantage at implementing such policies, as can be seen in the swift response of Singapore ⁷³.

Case identification, contact tracing and related measures

NPIs related to case identification and contact tracing show some of the lowest effectiveness ranks and in some cases even increase R_t, consistently across most countries (NPIs with the five lowest entropy scores all belong to this theme). This result is to be expected, as, e.g., increased testing and faster contact tracing will on the short-run increase the numbers of found cases in return for reduced numbers in the long run. We do not assess such long-term effects (over timespans of more than a month) in the current work. Furthermore, note that our analysis considers mostly data from March and April 2020 where many countries experienced surges of case numbers that most likely hindered effective contact tracing and other case identification measures. This also applies to the relative ineffectiveness of quarantining people who either are infected or were exposed to infected persons, while the promotion of self-isolation of people with symptoms was one of the most effective NPIs. This result confirms a tendency in our results where voluntary measures are more effective than similar mandatory ones.