Abstract
Background Short-term forecasts of infectious disease can aid situational awareness and planning for outbreak response. Here, we report on multi-model forecasts of Covid-19 in the UK that were generated at regular intervals starting at the end of March 2020, in order to monitor expected healthcare utilisation and population impacts in real time.
Methods We evaluated the performance of individual model forecasts generated between 24 March and 14 July 2020, using a variety of metrics including the weighted interval score as well as metrics that assess the calibration, sharpness, bias and absolute error of forecasts separately. We further combined the predictions from individual models into ensemble forecasts using a simple mean as well as a quantile regression average that aimed to maximise performance. We compared model performance to a null model of no change.
Results In most cases, individual models performed better than the null model, and ensembles models were well calibrated and performed comparatively to the best individual models. The quantile regression average did not noticeably outperform the mean ensemble.
Conclusions Ensembles of multi-model forecasts can inform the policy response to the Covid-19 pandemic by assessing future resource needs and expected population impact of morbidity and mortality.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
The following funding sources are acknowledged as providing funding for the named authors: Wellcome Trust (210758/Z/18/Z, 206250/Z/17/Z); Health Data Research UK (grant: MR/S003975/1), the Medical Research Council (MC_PC 19065, MR/V009761/1); the National Institute for Health Research (NIHR) Health Protection Research Unite (HPRU) in Modelling and Health Economics, a partnership between PHE, Imperial College London and the London School of Hygiene & Tropical Medicine (grant code NIHR200908); the HPRU in the Health Protection Research Unit in Genomics and Enabling Data; the NIHR (16/137/109) using UK aid from the UK Government to support global health research; the European Union's Horizon 2020 research and innovation programme - project EpiPose (No 101003688); the Bill & Melinda Gates Foundation (INV-003174). This research was partly funded by the Global Challenges Research Fund (GCRF) project 'RECAP' managed through RCUK and ESRC (ES/P010873/1); the Wellcome Trust and the Royal Society (202562/Z/16/Z, 107652/Z/15/Z); the Department of Mathematics at the University of Manchester for PhD funding; the Canadian Institute of Health Research (CIHR) 2019 Novel Coronavirus (COVID-19) rapid research program. The views expressed are those of the authors and not necessarily those of the United Kingdom (UK) Department of Health and Social Care, the National Health Service, the National Institute for Health Research (NIHR), or Public Health England (PHE). The following funding sources are acknowledged as providing funding for the CMMID COVID-19 Working Group authors: Alan Turing Institute; BBSRC LIDP: BB/M009513/1; B&MGF: INV-003174, INV-001754, OPP1157270, OPP1180644, OPP1183986, NTD Modelling Consortium OPP1184344, OPP1191821; DHSC/UK Aid/NIHR: ITCRZ 03010; Elrha R2HC/UK DFID/Wellcome Trust/NIHR, DFID/Wellcome Trust: Epidemic Preparedness Coronavirus research programme 221303/Z/20/Z; ERC Starting Grant: #757699; European Commission: 101003688; Global Challenges Research Fund: ES/P010873/1; MRC: MR/N013638/1, MC_PC_19065, MR/P014658/1; Nakajima Foundation; NIHR: 16/136/46, 16/137/109, NIHR200929, PR-OD-1017-20002; Royal Society: Dorothy Hodgkin Fellowship, RP\EA\180004; Wellcome Trust: 206250/Z/17/Z, 206471/Z/17/Z, 208812/Z/17/Z, 210758/Z/18/Z; UK MRC: LID DTP MR/N013638/1.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
London School of Hygiene & Tropical Medicine Ethics Committee (Ref. 22828).
All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Footnotes
Figure 1 and 3 clarified with what data was available at the time. Reference to the quantgen R package added.
Data Availability
All publicly available data used in the manuscript is available at https://github.com/epiforecasts/covid19.forecasts.uk Some of the data used for forecasts, and to generate the results in this manuscript, were shared with modellers in confidence and cannot be made available by the authors.
