Abstract
During the COVID-19 pandemic, epidemiological modelling has played a key role in public debate and policy making for anticipating the epidemic trajectory, as well as proposing and evaluating non-pharmaceuticals interventions. Despite its importance, evaluations of models’ ability to accurately represent the evolution of the disease remain scarce. Robust and systematic evaluation is needed to assess models. We investigate the following research question : were the COVID-19 scenarios proposed by modellers during the pandemic to policy-makers relevant for decision making ? To answer this, we conduct a retrospective assessment of modelling reports which guided policy response in France in 2020-2022. After systematically verifying the scenarios hypotheses (e.g., exclusion of no-lockdown scenarios when a lockdown was effectively in place), we find that out of 10 reports, reality was below the best-case scenario in 6 reports; within the best-case / worst case scenarios range in 3 reports; above the worst-case scenario in 1 report. Best-case scenarios were the closest to reality, but often came from report with a large span between best-case and worst-case scenarios beyond 2 weeks, precluding certainty about future outcomes at the time of publishing. Our results hint a systematic overestimation bias for these particular models used to anticipate epidemic evolution, which can be of importance if such models are used to contractually estimate the effectiveness of non pharmaceutical interventions. To our knowledge, this is the only national systematic retrospective assessment of COVID-19 pandemic scenarios assessing hospital burden; such an approach should be reproduced in other countries whenever possible.
Graphical Abstract Reality (black line) compared to prospective scenarios (colored lines) which informed policy during the COVID-19 pandemic in France for Intensive Care Units (top) and New Hospital Admissions (bottom). Colors indicate the error between reality and scenarios, expressed as a percentage of the 1st wave peak (horizontal dashed line).
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This study did not receive any funding.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Footnotes
Update of the abstract ; upload of list of prospective scenarios
Data Availability
All data produced are available online at https://github.com/maxime-langevin/retrospective_analysis
Abbreviations
- ICU
- Intensive Care Units
- COVID
- Coronavirus Disease
- MAE
- Mean Absolute Error
- MAPE
- Mean Absolute Percentage Error
- ME
- Mean Error
