Author Response
Reviewer #1 (Public Review):
The authors have provided an impressive analysis of the effects of reporting of WGS results on IPC practices in 14 hospitals in the UK during the COVID-19 pandemic. After a median of 4 weeks, hospitals adopted a practice of "rapid" or "longer" turnaround phases for WGS reporting. After a median of 8 weeks, 8 of 9 "rapid" hospitals adopted the "longer" practice for a median of 4 weeks. After a median of 4 weeks, all 5 "longer" hospitals adopted the "rapid" practice for a median 8 weeks. Hence, there were twice as many weeks with the rapid, compared to the longer reporting practice.
The targeted turnaround times for reporting were 48 hours for the rapid and 5-10 days for the longer phase.
The primary outcomes of the study were: (1) incidence of IPC-defined SARS-CoV-2 HAIs per week per 100 currently admitted non-COVID-19 inpatients, and (2) for each HOCI, identification of linkage to individuals within an outbreak of SARS-CoV-2 nosocomial transmission using sequencing data as interpreted through the SRT that was not identified by pre-sequencing IPC evaluation during intervention phases.
Secondary outcomes were: (1) incidence of IPC-defined SARS-CoV-2 hospital outbreaks per week per 100 non-COVID-19 inpatients, (2) for each HOCI, any change to IPC actions following receipt of SRT report during intervention phases, (3) any recommended change to IPC actions (regardless of whether changes were implemented). The proportion of HOCI cases for which IPC reported the SRT report to be 'useful' was added as a further outcome.
A total of 2170 HOCIs were recorded for the study between 15 October 2020 and 26 April 2021.
The authors conclude that "While we did not demonstrate a direct impact of sequencing on the incidence of nosocomial transmission, our results suggest that sequencing can inform IPC response to HOCIs, particularly when returned within 5 days."
The research question is very relevant and, as said, the amount of data collected is impressive. Yet, interpretation of the data, obtained in a real-life setting with all hurdles and complexities created by the pandemic situation, is challenging. I have several questions related to data interpretation and difficulties in accepting the overall positive interpretation of the findings when it comes to feasibility and potential impact. Especially, as I consider the real-time availability of WGS results in the participating hospitals to be (much) higher than it will be in hospitals in most other countries.
We thank the reviewer for their summary and commentary on our work.
Specific questions
Not clear why sites started either with rapid or longer phase. Was there a randomization process? Please clarify.
There was no randomisation process as the ordering of intervention phases was largely driven by logistical concerns that needed to be overcome in order to be able to run the intervention during a very unpredictable period of the pandemic. We have added the following text to the second paragraph of the Methods to clarify this:
“The order of the intervention phases was pragmatically determined in some sites by the need to first run the ‘longer-turnaround’ phase to develop sample transport and sequencing procedures before attempting the ‘rapid’ sequencing phase, and the ordering was decided in the remaining sites to ensure a mixture of intervention phases over calendar time – there was no randomisation process in deciding the order of study phases.”
From Figures S2 it is clear that the pandemic peaked, after which the curve declined when vaccination had started, and these curves seem to resemble the incidence rates of HOCI in the hospitals.
Yes, there was also a full national lockdown in the UK in late 2020 and early 2021, which sharply reduced community incidence rates.
I had difficulties in interpreting S3 and S4 where I think the authors incorporated these disease dynamics occurring outside the hospital setting on the HOCI incidences. I would be helped by a better explanation of what actually was done.
Yes, this is correct. We have now added a more detailed model specification to the Appendix (also in response to the comments of Reviewer #3). We have also added some further explanation to the Figure captions: ‘The spline curves shown are estimated simultaneously within the final analysis model, and show how these factors have independent contributions to the prediction of the incidence rate for HAIs. The associations for each covariable indicated by model parameter point estimates are shown as solid lines, with 95%CIs shown as dashed lines.’
It is not clear why the difference between the groups in the intervention (providing rapid or not so rapid WGS reports) was too small to have an impact, compared to the baseline period without WGS reporting. Surprisingly sites F and G appear to do significantly worse during the "rapid" phase, according to Fig1. Please clarify.
With regards the issue of why it might be that we did not detect an effect of the intervention on the incidence rate of hospital-acquired infections, the Methods contains a relevant brief summary of our qualitative analysis within this study:
“The SRT did provide new and valued insights into transmission events, outbreaks and wider hospital functioning but mainly acted to offer confirmation and reassurance to IPC teams. Critically, the capacity to generate and respond to these insights effectively on a case-by-case basis was breached in most sites by the volume of HOCIs, and the limits of finite human and physical resource (e.g. hospital layout). “
We have now also added a sentence to flag this in the context of study limitations in the Discussion “Our qualitative analyses also found that the capacity of sites to react to information generated by the sequencing intervention was breached by the volume of HOCI and admitted COVID-19 cases in combination with the finite personnel resources and limited physical space for isolation that was available”.
We have noted in the Results section that “Our analysis models reveal important findings beyond the effect of the intervention. The analysis model for the incidence of HAIs identified independent positive associations with the proportion of current SARS-CoV-2 positive inpatients, the local community incidence of new SARS-CoV-2 cases … and calendar time …”. The ‘rapid’ phases for sites F and G were conducted during periods of high community incidence of SARS-CoV-2, with a high proportion of current inpatients SARS-CoV-2 positive and at a relatively early stage in the national vaccination roll-out. These factors clearly outweighed any potential reduction in the incidence of HAIs associated with the sequencing intervention.
We have added some further clarifications regarding the structure and interpretation of the analysis models for the outcome of the incidence of HAIs in response to comments from Reviewer #3.
The 'health economic findings' miss the health component. The costs of the intervention are described in detail, but not the benefits of the intervention. Is it possible to calculate the costs required to prevent a single case of HOCI?
Thank you for your comment. The scope of the economic evaluation was to evaluate the economic effects of SARS-CoV-2 genome sequencing in supporting infection control teams and not the health benefits of the intervention. Therefore, the outcome of the intervention would be rather the benefit of rapid/slow return of the sequencing report expressed as potential reduction in resource utilisation and costs. A paper presenting the methodology and findings is under preparation. Here we presented only the cost of the intervention. We will remove the “Health economic findings” heading and re-name it “Cost of SARS-CoV-2 genome sequencing”.
Reviewer #2 (Public Review):
This study evaluated the impact of rapid turnaround whole genome sequencing to discover unsuspected hospital-acquired SARS-CoV-2 on the incidence of hospital-acquired SARS-CoV-2 cases. Strengths of the study include the important question, the technical and logistic feat of making whole genome sequencing widely available, and the large number of participating sites.
We thank the reviewer for their summary and commentary on our work.
Major limitations of the study include the fact that only half of sequencing reports were returned to infection prevention programs and then only a small minority within the targeted reporting time-frame (5% for rapid phase, 21% for longer phase; median turnarounds were 5 days and 13 days respectively). This fundamentally undermines the premise of the study, namely to see if rapid turnaround of sequencing can impact infection control.
We have now explicitly stated in the Results that the median turnaround times achieved were substantially longer than the target values, and have added that ‘…and more timely reporting of results might be associated with greater impact on IPC actions’ in the Discussion.
More broadly, it does not appear that there was a standardized protocol on how hospitals were expected to respond to reports of clusters.
Review of Table S2 suggests that many of the potential actions were things that in retrospect probably don't have too much impact on transmission (e.g. checking soap stocks, signage assessments). The kinds of things that I think might decrease nosocomial transmission include minimizing use of shared rooms, improving ventilation, increased use of N95/FFP2 respirators for source control, more frequent surveillance testing cadences, etc. These were not options on the response lists perhaps explaining the lack of impact on transmission.
We have added the following paragraph to the Discussion: “Planning this study and developing the data collection forms during the early stages of a novel viral pandemic was challenging, as in the summer of 2020 there were still ongoing debates around the primary mode of viral transmission and optimal IPC practice, and global supply chains for personal protective equipment were strained. In the planning of an equivalent study now, there would be a greater focus on adjustments to ventilation, air filtration and respirator usage. It would also be possible to be more prescriptive and standardised regarding the recommended changes to IPC practice in response to sequencing findings.”
Reviewer #3 (Public Review):
This study, conducted in 14 acute hospital trusts in the United Kingdom, compared SARS-CoV-2 hospital infection outcomes in a four week baseline period with outcomes in periods with 'rapid' (<48h) and 'longer-turnaround' (5-10 day) sequencing with results fed-back to infection prevention and control teams using a bespoke sequencing reporting tool. The question of whether rapid sequencing of hospital-onset SARS-CoV-2 infection can, by informing infection prevention and control (IPC) actions, reduce nosocomial transmission is interesting and potentially important. To our knowledge, this study represents the first large-scale formal evaluation of such technology. While the results are, on the face of it, disappointing in that hospitals were largely unable to meet target turnaround times for sequencing and results provide no evidence of benefit of the intervention in reducing hospital-acquired infection (and in some cases, such as for the "hospital outbreaks" outcome, the confidence intervals are so wide as to be unable to rule out substantial benefits or harms of the intervention) the are a number of important strengths of the study. These include the relatively strong quasi-experimental design (a type of non-randomised cluster crossover), the pre-defined analysis plan, and adequate power for the primary outcomes.
Limitations of the study include the practical difficulties that participating hospitals had in reporting sequencing results to the IPC teams in a timely manner that could be acted on and lack of sufficient consideration for the ways in which sequencing information could have directly informed IPC activities in ways that would have been likely to substantially reduce the spread of infection (for example, Table S2 reports changes to IPC as a result of sequencing reports which include generic activities such as "Assessment of alcogel stocks" or "IPC signage assessment", which seem like things which should be done anyway, and don't obviously depend on information from pathogen sequencing).
We thank the reviewer for their summary and commentary on our work.
There are also some aspects of transparency that need to be addressed: the analytical methods are not reported in sufficient detail to enable the work to be repeated, and the results are not reported with sufficient detail to an enable an assessment of the appropriateness or otherwise of the statistical models used in the analysis. Additionally, while the study protocol specified six secondary outcomes, not all of these are reported even where it appears that some (partial) information is available for unreported outcomes.
