Using synthetic controls to estimate the population-level effects of Ontario’s recently implemented overdose prevention sites and consumption and treatment services

Study Design

We conducted a retrospective, ecological study that uses variations in the provision of OPS/CTS across Ontario’s public health units (PHU) to examine population-level health outcomes.

Setting

Ontario is Canada’s most populous province with approximately 14.75 million residents (Government of Ontario, 2020), 97% of whom are covered by the provincial health insurance program (Izenberg, Iroanyah, & Thompson, 2018). Within the province, health promotion and disease prevention are administered by PHUs, which have mutually exclusive and exhaustive health boundaries. The 34 PHUs range from 33,166 residents in Timiskaming, the province’s most sparsely populated health unit, to 3,094,237 for its densest, most populated urban unit, Toronto (Government of Ontario, 2020).

Beginning in 2016, several provincial and federal harm reduction interventions were implemented to stem the rising rate of opioid-related overdose deaths. In June 2016, the Ontario Naloxone Program for Pharmacies (ONPP) began offering naloxone injection kits at no charge to individuals at risk of opioid-related overdose or persons in a position to assist someone at risk of an overdose, while select community-based organizations continued to dispense naloxone to at-risk clients, friends and family through a program that first began in 2013 (Canadian Centre on Substance Abuse & Canadian Community Epidemiology Network on Drug Use, 2016; Canadian Pharmacists Association, 2017). Shortly thereafter, naloxone kits transitioned from an injectable formulation to nasal spray (January 2017), the federal government’s Good Samaritan Drug Overdose Act became law (May 2017) (Government of Canada, 2018), and PHUs began sub-distributing naloxone kits to community-based organizations much more widely through the Ontario Naloxone Program (ONP, September 2017) (Gogolishvili & Wasdell, 2020). Although some cities began operating unsanctioned sites earlier in the year, Health Canada issued an exemption to Ontario to establish legally sanctioned temporary OPSs on December 7, 2017 (Ontario Ministry of Health & Ontario Ministry of Long-Term Care, 2018). Beginning October 2018, the government of Ontario enforced a list of required services and metrics that needed to be met for continued approval and financial support of CTSs (Ministry of Health and Long-Term Care, 2018a) and applied a cap to the number of sites allowed, ostensibly restricting provision of sites to larger cities (Russell, Imtiaz, Ali, Elton-Marshall, & Rehm, 2020).

Between June 2017 and December 2020, nine of the 34 provincial PHUs began operating at least one OPS/CTSs. These were in Toronto (n=9, 2 additional shelter-hotel sites), Ottawa (n=5), and one each in Hamilton, Kingston, London, Niagara, Thunder Bay, Guelph, and Waterloo (Pivot Legal Society, 2020) (Supplementary Table 1 for implementation dates, hours of operation, and capacity).

We excluded two PHUs that merged during the study period (Southwestern and Huron-Perth) with a combined population of approximately 336,000 (2.3%) from subsequent analysis owing to the incomplete demographic data available and differences between public health units’ and Statistics Canada’s jurisdictional boundaries.

Data collection and measures

We used monthly counts of all opioid-related ED visits, acute care hospitalizations, and deaths between January 2014 and March 2021, reported at the PHU-level in Ontario’s publicly accessible Interactive Opioid Tool (Ontario Agency for Health Protection and Promotion (Public Health Ontario), 2020). Across the 32 PHUs included in the study, there were 53,806 ED visits, 13,734 hospitalizations and 9,377 mortalities for opioid-related overdose events during this time.

Ethics approval

This study was exempt from ethics review by McGill University’s Institutional Review Board.

Statistical analysis

We used controlled interrupted time series with segmented regression to test the effects of OPS/CTS booth-hours on opioid-related ED visits, hospitalizations, or deaths (Bernal JL, Cummins S, & Gasparrini A, 2017). Interrupted time series analysis is a quasi-experimental study design that estimates population-level effects of health service or policy interventions before and after implementation in contexts where randomized controlled trials are not feasible or ethical (Cook TD & DT, 1979; Zhang F, Wagner AK, & Ross-Degnan D, 2011). The advantage of including a well-chosen control as the counterfactual is that selection bias and within-group characteristics that change slowly over time, secular changes, random fluctuations from one time point to the next and regression to the mean are also controlled (Lopez Bernal, Cummins, & Gasparrini, 2018).

Treated groups included all PHUs that implemented at least one OPS/CTS during the study period. Synthetic controls were derived for each treated PHU per outcome from a donor pool of exclusively never-treated PHUs (Baker, Larcker, & Wang, 2021) using Abadie et al.’s data-driven method and the Synth package with its extensions in R (Abadie, Diamond, & Hainmueller, 2011; Castanho Silva & DeWitt, 2020; Degli Esposti et al., 2020). The pre-intervention period was restricted to two years to optimize prediction of the treatment group by the comparison group (Bilinski, 2021).

We fitted the following linear regression model for intervention status j (pre-intervention=0, post-intervention=1), for group k (control group=0, treated group=1), at time t (pre-intervention implementation <0, month of intervention implementation=0, post-intervention implementation>0): Where β₀ is the outcome rate intercept for the control; β₁ represents the pre-intervention temporal trend for the control’s outcome rate; β₂ indicates the pre-intervention difference between the treated group’s outcome rate intercept and the controls (level difference); β₃ captures the pre-intervention difference between treated and control groups’ temporal trends; β₄ represents the difference in the control outcome rate immediately post-intervention compared with the control outcome rate at the beginning of the pre-intervention observation period (level difference); β₅ indicates the change in temporal trend for the control outcome rate post-intervention compared with pre-intervention; β₆ is the effect of the booth-hours on the treated group’s outcome rate; and β₇ indicates the difference between treated and control groups’ temporal trends post-implementation (i.e., the treated groups change in trend relative to the control group’s change in trend, β₅). We also adjusted for other relevant time-varying confounders following the disjunctive cause criterion for confounder selection such that all covariates are known or potential causes of the exposure (i.e., OPS/CTS implementation) and/or the outcome (i.e., opioid-related ED visits, hospitalizations or death) (VanderWeele & Shpitser, 2011). Finally, we corrected for seasonality using harmonic terms, and used Newey-West standard errors to account for autocorrelation identified using plots of residuals.

Since the initial OPS/CTSs was implemented at different times across the nine PHUs, we applied a multiple baseline approach (i.e. using intervention time instead of calendar time)(Hawkins, Sanson-Fisher, Shakeshaft, D’Este, & Green, 2007) to measure the effect of booth-hours on population-level ED visits, hospitalizations, and deaths. We aggregated the data across treated PHUs with well-fitted synthetic controls and report the results of these aggregate analyses and their sensitivity analyses in the main text. We also report the disaggregated results per PHU treated/synthetic control pair in the supplementary material.

For sensitivity analysis, we tested the effects of the Good Samaritan Drug Overdose Act(Government of Canada, 2018) across all treated/synthetic control pairs (not shown here), aggregated all treated PHU/synthetic control pairs irrespective of fit, and terminated the observation period to before March 2020 and the onset of COVID-19 related service restrictions and border closures.

All analyses were conducted using R version 4.0.5(R Core Team, 2014). All data used for this study are publicly available through the Ontario Opioid Tool, Ontario Drug Policy Research Network, news articles and OPS/CTS listed hours of operation (see Supplement). All results are reported using 95% CI and accompanying p-values. Because we created a unique synthetic control per treated PHU per outcome, we did not use a p-value correction for interpreting our results. However, a simple Bonferroni correction per treated unit assuming we were using the same synthetic control across all three outcomes would change the accepted p-value from 0.05 to 0.017.

ED visits

Per booth-hour, ED visits increased in Guelph (0.02, 95% CI: 0.01, 0.03; p-value=0.0059) but declined in Ottawa (−0.0023, 95% CI: -0.0037, -0.0008; p-value=0.0021) and London (−0.02, 95% CI: -0.03, 0.00; p-value=0.0177). Meanwhile, prescription opioids for pain management increased ED visits in London (0.01, 95% CI: 0.00, 0.01; p-value=0.0266), Kingston (0.01, 95% CI: 0.01, 0.01; p-value=0.0001), Thunder Bay (0.0044, 95% CI: 0.0003, 0.0084; p-value=0.0380) and Niagara (0.01, 95% CI: 0.00, 0.01; p-value=0.0463).

Our sensitivity analyses found no effects of booth-hours on ED visits but prescription opioids for pain management decreased ED visits in Toronto (−0.0027, 95% CI: -0.0052, -0.0001; p-value=0.04555), and Ottawa (−0.0039, 95% CI: -0.0059, -0.0019; p-value=0.0003).

Hospitalizations

There was no statistically significant effect of booth-hours on hospitalizations. However, prescription opioids for pain management increased hospitalizations in Kingston (0.0017, 95% CI: 0.0001, 0.0032; p-value=0.0350) but decreased hospitalizations in Waterloo (−0.0022, 95% CI: -0.0032, -0.0011; p-value=0.0002). Results from the sensitivity analyses found no effect of opioids for pain management on hospitalizations in the period before COVID-19 public health measures were implemented.

Deaths

Per booth-hour, deaths declined in London (−0.0039, 95% CI: -0.0062, -0.0015; p-value=0.0020), and Thunder Bay (−0.0038, 95% CI: -0.0074, -0.0002; p-value=0.0396). Prescription opioids for pain management increased deaths in London (0.0010, 95% CI: 0.0001, 0.0019; p-value=0.0349). Paradoxically, OAT reduced deaths in London (−0.02, 95% CI: -0.03, -0.01; p-value<0.0001) but increased deaths in Hamilton (0.01, 95% CI: 0.00, 0.03; p-value=0.0169). Naloxone kits reduced deaths in Toronto (−0.0043, 95% CI: -0.01, 0.00) and London (−0.0027, 95% CI: -0.0042, -0.0012; p-value=0.0004).

Our sensitivity analyses found the effects of prescription opioids for pain management (0.0013, 95% CI: 0.0003, 0.0022; p-value=0.0113) and OAT (−0.02, 95% CI: -0.03, -0.01; p-value<0.0001) persisted in London; while OAT (0.01, 95% CI: 0.01, 0.02; p-value=0.0005) and naloxone (0.0077, 95% CI: 0.0022, 0.0131; p-value=0.0074) increased deaths in Kingston. DISCUSSION

Our aggregate analyses found no effects of booth-hours on opioid-related ED visits, hospitalizations, or deaths; but OAT reduced ED visits and deaths. Small, but significant reductions in ED visits (Ottawa, London) and deaths (London, Thunder Bay) were observed locally; while there was an increase in ED visits for Guelph. In other words, 5 of the 36 coefficients for the local effects of booth-hours appeared to be significant at the p=0.05 level.

Our models estimated 18.76 (95% CI: -34.61, -7.50) and 16.47 (95% CI: -28.67, -7.17) ED visits averted per month in Ottawa and London, respectively; and an additional 10.34 (95% CI: 5.17, 15.51) ED visits per month for Guelph. Assuming each visit cost the health care system approximately $304 (Canadian Institute for Health Information, 2020), this equates to $5703 and $5007 saved, and $3134 incurred per month, respectively. Further, we found London’s and Thunder Bay’s CTSs averted 3.21 (95% CI: -5.11, -1.24) and 2.90 (95% CI: -5.63, -0.15) deaths per month, respectively. Using a very modest value of life of $1,000,000 (Griffiths & Vadlamudi, 2016) this equates to $3,210,000 and $2,900,000 saved monthly. However, we do not encourage limiting cost-effectiveness or cost-benefit exercises to overdose events averted since costs of operating OPSs, for example, while minimal (e.g. $20,367/month each, 40 hours/week) still include other program, administrative, phone, data management and IT expenses that are not specific to these outcomes (Ministry of Health and Long-Term Care, 2018a, 2018b). To better determine if OPS/CTSs are a cost-effective or even cost-saving intervention, we recommend incorporating costs averted from reduced transmission of blood borne infections and other drug use injuries (i.e. skin, soft tissue, and vascular infections and their sequelae) (Panagiotoglou et al., 2021).

Extensive research on the effects of supervised consumption sites (and their variants) has yet to show consistent population-level benefits. Until recently, all the evidence on the effectiveness of supervised consumption sites in Canada was based on Vancouver’s Insite. First implemented in September 2003, Insite is in the Downtown Eastside – a unique neighborhood with one of the highest concentrations of people who use illicit drugs globally (Hayashi et al., 2019). Numerous studies have explored Insite’s effectiveness on drug related harms and acceptability in the community. Among the most rigorous of these, a study led by Marshall et al. demonstrated Insite decreased the local fatal overdose rate by 35% from 253.1 to 165.1 deaths per 100,000 person-years (Marshall BDL, Milloy MJ, Wood E, Montaner JS, & Kerr T, 2011). Differences in magnitude between Marshall et al.’s findings and ours may be a consequence of several important methodological differences. Their analysis compared overdose mortality rates for the population within 500m of Insite to the rest of the city’s population pre-(1 January 2001 to 20 September 2003) and post-implementation (21 September 2005 to 31 December 2005). While innovative at the time, this seminal paper did not account for underlying epidemiological trends or other co-occurring and time-varying interventions, nor restricted comparison to neighborhoods comparable to the Downtown Eastside, potentially overestimating the effect of Insite on fatal overdose rates.

More recently, a mathematical model of the individual and synergistic effects of British Columbia’s harm reduction interventions (OAT, take home naloxone, and OPS) found all three play a significant role in stemming overdose deaths, with OPS averting 290 deaths (credible interval (CrI): 160 – 350) over 21 months (Michael A. Irvine et al., 2019). Differences in the effect sizes between their findings and ours may be, in part, owing to a key assumption underlying their model: in the absence of medical or peer intervention each overdose results in death (M. A. Irvine et al., 2018; Michael A. Irvine et al., 2019). While overdoses do cause life-threatening bradypnea (also referred to as opioid induced respiratory depression, classified as < 12 breaths per minute), respiratory distress, particularly when mild, can spontaneously resolve (Ballantyne, 2007; Toronto Public Health, 2022). In fact, in Irvine et al.’s 2019 paper, although the baseline death rate was set to 1.0 with large credible intervals, posterior estimates showed this to be much smaller (0.0682, 95% CrI: 0.0663 – 0.0701) (Michael A. Irvine et al., 2019). While the authors do not speak to this discrepancy directly in the 2019 paper, they do briefly acknowledge a large difference between the prior and posterior estimates in a prior paper examining the effects of BC’s take home naloxone program (M. A. Irvine et al., 2018). This uncertainty around the baseline death rate per overdose event coupled with differences in population substance use preferences, patterns and geographic diffusion; extent of fentanyl contamination of the illicit drug supply; observation period (i.e. the BC paper examined the effects of these interventions when most services and overdose events remained concentrated within Vancouver’s Downtown Eastside); and delivery models of OPS/CTSs may further explain why Irvine et al. observed substantially larger effect sizes for OPSs in British Columbia than we do here.

Our study is not the first to find a lack of effect of supervised consumption sites on mortality, but it is the largest. An evaluation of Sydney, Australia’s supervised injection centre (NCHECR, 2007) and another of OPSs in mid-sized communities on Vancouver Island (The Evaluation of Overdose Prevention Sites Working Group & Lori Wagar, 2018) both noted a decrease in local paramedic attendance, but no effect on overdose related mortality. The authors of these reports considered the rarity of fatal events and subsequent challenges with study power as possible explanations for the absence of a measurable effect; and specific to Sydney, a co-occurring decline in the local heroin supply.

Our results also agree with simpler analysis we conducted to evaluate British Columbia’s recently implemented OPSs (Panagiotoglou, 2022). In both studies, we used controlled interrupted time series and reported aggregate, population-level results. However, in the BC study we were not able to control for other time-varying interventions nor the intensity of the intervention itself. Despite these differences, both studies found a lack of effect on hospitalization and mortality rates. The BC analysis found OPSs reduced ED visits whereas our Ontario analysis did not. This may be due to differences in provincial OPS guidelines for medical care follow-up after on-site overdose reversal (BC Centre on Substance Use, 2017). Finally, both studies found a decline in the death rate trend in their respective aggregate control populations which was mirrored in the treated units (Panagiotoglou, 2022). While the cause of this decline in underlying epidemiological trend is not clear, it may be a combination of gradual changes in opioid drug use behaviour (e.g. prioritizing using substances with a peer, testing substances for contaminants), the public’s perception of substance use disorder, and law enforcement activity. The effects of this cultural shift may be imperceptibly small at the local treated/control unit level, but discernable in the aggregate analyses, and worth additional exploration.

This common change in trend for the treated/control unit dyads lends additional credence to the comparability of the groups. Alternatively, a change in the control’s trend that was not also observed in the treated unit would indicate substantial and problematic differences between the populations.

The lack of observed effect of OPS/CTSs on our outcomes may be a consequence of access barriers first described elsewhere. Work examining successful implementation and uptake of OPSs in British Columbia found persisting stigma and police presence reduced their social acceptability and use by at-risk populations (Collins et al., 2019). Meanwhile, hours of operation, facility capacity, and absence of safe inhalation rooms limited their effectiveness (The Evaluation of Overdose Prevention Sites Working Group & Lori Wagar, 2018). Similar barriers have been described in qualitative work on Ontario’s OPSs (Foreman-Mackey, Bayoumi, Miskovic, Kolla, & Strike, 2019). This may also explain why OAT reduces deaths, since patients receiving OAT have already overcome challenges to access.

Despite concerns that harm reduction interventions may encourage risky behaviour (Hedlund, 2000) and therefore explain the lack of observed effects in our study, a team examining the prevalence of unintentional overdose in a longitudinal cohort of people who inject drugs in Toronto has found no evidence of risk compensation (Scheim et al., 2021). Finally, while drug use and fentanyl consumption, in particular, have increased since March 2020 (Statistics Canada, 2021), these consumption changes do not explain the lack of observed effects in our sensitivity analyses. Together, our results suggest the effectiveness of OPS/CTSs is more modest than described elsewhere, and/or context specific.