The revolution will be hard to evaluate: How simultaneous change in multiple policies affects policy-based health research

Abstract

Extensive empirical health research leverages variation in the timing and location of policy changes as quasi-experiments. Multiple social policies may be adopted simultaneously in the same locations, creating clustering which must be addressed analytically for valid inferences. The pervasiveness and consequences of policy clustering have received limited attention. We analyzed a systematic sample of 13 social policy databases covering diverse domains including poverty, paid family leave, and tobacco. We quantified policy clustering in each database as the fraction of variation in each policy measure across jurisdictions and times that could be explained by co-variation with other policies (R²). We used simulations to estimate the ratio of the variance of effect estimates under the observed policy clustering to variance if policies were independent. Policy clustering ranged from very high for state-level cannabis policies to low for country-level sexual minority rights policies. For 65% of policies, greater than 90% of the place-time variation was explained by other policies. Policy clustering increased the variance of effect estimates by a median of 57-fold. Policy clustering poses a major methodological challenge to rigorously evaluating health effects of individual social policies. Tools to enhance validity and precision for evaluating clustered policies are needed.