Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations
============================================================================================

* K. Sherratt
* H. Gruson
* R. Grah
* H. Johnson
* R. Niehus
* B. Prasse
* F. Sandman
* J. Deuschel
* D. Wolffram
* S. Abbott
* A. Ullrich
* G. Gibson
* EL. Ray
* NG. Reich
* D. Sheldon
* Y. Wang
* N. Wattanachit
* L. Wang
* J. Trnka
* G. Obozinski
* T. Sun
* D. Thanou
* L. Pottier
* E. Krymova
* MV. Barbarossa
* N. Leithäuser
* J. Mohring
* J. Schneider
* J. Wlazlo
* J. Fuhrmann
* B. Lange
* I. Rodiah
* P. Baccam
* H. Gurung
* S. Stage
* B. Suchoski
* J. Budzinski
* R. Walraven
* I. Villanueva
* V. Tucek
* M. Šmíd
* M. Zajícek
* C. Pérez Álvarez
* B. Reina
* NI. Bosse
* S. Meakin
* P. Alaimo Di Loro
* A. Maruotti
* V. Eclerová
* A. Kraus
* D. Kraus
* L. Pribylova
* B. Dimitris
* ML. Li
* S. Saksham
* J. Dehning
* S. Mohr
* V. Priesemann
* G. Redlarski
* B. Bejar
* G. Ardenghi
* N. Parolini
* G. Ziarelli
* W. Bock
* S. Heyder
* T. Hotz
* D. E. Singh
* M. Guzman-Merino
* JL. Aznarte
* D. Moriña
* S. Alonso
* E. Álvarez
* D. López
* C. Prats
* JP. Burgard
* A. Rodloff
* T. Zimmermann
* A. Kuhlmann
* J. Zibert
* F. Pennoni
* F. Divino
* M. Català
* G. Lovison
* P. Giudici
* B. Tarantino
* F. Bartolucci
* G. Jona Lasinio
* M. Mingione
* A. Farcomeni
* A. Srivastava
* P. Montero-Manso
* A. Adiga
* B. Hurt
* B. Lewis
* M. Marathe
* P. Porebski
* S. Venkatramanan
* R. Bartczuk
* F. Dreger
* A. Gambin
* K. Gogolewski
* M. Gruziel-Slomka
* B. Krupa
* A. Moszynski
* K. Niedzielewski
* J. Nowosielski
* M. Radwan
* F. Rakowski
* M. Semeniuk
* E. Szczurek
* J. Zielinski
* J. Kisielewski
* B. Pabjan
* K. Holger
* Y. Kheifetz
* M. Scholz
* M. Bodych
* M. Filinski
* R. Idzikowski
* T. Krueger
* T. Ozanski
* J. Bracher
* S. Funk

## Abstract

**Background** Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022.

**Methods** We used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported from a standardised source over the next one to four weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models’ predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models’ forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models’ past predictive performance.

**Results** Over 52 weeks we collected and combined up to 28 forecast models for 32 countries. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 84% of participating models’ forecasts of incident cases (with a total N=862), and 92% of participating models’ forecasts of deaths (N=746). Across a one to four week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over four weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models.

**Conclusions** Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than two weeks.

**Code and data availability** All data and code are publicly available on Github: covid19-forecast-hub-europe/euro-hub-ensemble.

## Background

Epidemiological forecasts make quantitative statements about a disease outcome in the near future. Forecasting targets can include measures of prevalent or incident disease and its severity, for some population over a specified time horizon. Researchers, policy makers, and the general public have used such forecasts to understand and respond to the global outbreaks of COVID-19 [1]–[3].

Forecasters use a variety of methods and models for creating and publishing forecasts, varying in both defining the forecast outcome and in reporting the probability distribution of outcomes [4], [5]. Such variation makes it difficult to compare predictive performance between forecast models, and from there to derive objective arguments for using one forecast over another. This confounds the selection of a single representative forecast and reduces the reliability of the evidence base for decisions based on forecasts.

A “forecast hub” is a centralised effort to improve the transparency and usefulness of forecasts, by standardising and collating the work of many independent teams producing forecasts [6]. A hub sets a commonly agreed-upon structure for forecast targets, such as type of disease event, spatio-temporal units, or the set of quantiles of the probability distribution to include from probabilistic forecasts. For instance, a hub may collect predictions of the total number of cases reported in a given country for each day in the next two weeks. Forecasters can adopt this format and contribute forecasts for centralised storage in the public domain. This shared infrastructure allows forecasts produced from diverse teams and methods to be visualised and quantitatively compared on a like-for-like basis, which can strengthen public and policy use of disease forecasts.

The underlying approach to creating a forecast hub was pioneered in climate modelling and adapted for collaborative epidemiological forecasts of dengue [7] and influenza in the USA [6], [8]. This infrastructure was adapted for forecasts of short-term COVID-19 cases and deaths in the US [9], [10], prompting similar efforts in some European countries [11]–[13].

Standardising forecasts allows for combining multiple forecasts into a single ensemble with the potential for an improved predictive performance. Evidence from previous efforts in multi-model infectious disease forecasting suggests that forecasts from an ensemble of models can be consistently high performing compared to any one of the component models [7], [8], [14]. Elsewhere, weather forecasting has a long-standing use of building ensembles of models using diverse methods with standardised data and formatting in order to improve performance [15], [16].

The European COVID-19 Forecast Hub [17] is a project to collate short term forecasts of COVID-19 across 32 countries in the European region. The Hub is funded and supported by the European Centre for Disease Prevention and Control (ECDC), with the primary aim to provide reliable information about the near-term epidemiology of the COVID-19 pandemic to the research and policy communities and the general public [3]. Second, the Hub aims to create infrastructure for storing and analysing epidemiological forecasts made in real time by diverse research teams and methods across Europe. Third, the Hub aims to maintain a community of infectious disease modellers underpinned by open science principles.

We started formally collating and combining contributions to the European Forecast Hub in March 2021. Here, we investigate the predictive performance of an ensemble of all forecasts contributed to the Hub in real time each week, as well as the performance of variations of ensemble methods created retrospectively.

## Methods

We developed infrastructure to host and analyse prospective forecasts of COVID-19 cases and deaths. The infrastructure is compatible with equivalent research software from the US [18], [19] and German and Polish COVID-19 [20] Forecast Hubs, and easy to replicate for new forecasting collaborations.

### Forecast targets and models

We sought forecasts for the incidence of COVID-19 as the total reported number of cases and deaths per week. We considered forecasts for 32 countries in Europe, including all countries of the European Union, European Free Trade Area, and the United Kingdom. We compared forecasts against observed data reported for each country by Johns Hopkins University (JHU, [21]). JHU data sources included a mix of national and aggregated subnational data. We aggregated incidence over the Morbidity and Mortality Weekly Report (MMWR) epidemiological week definition of Sunday through Saturday.

Teams could express their uncertainty around any single forecast target by submitting predictions for up to 23 quantiles (from 0.01 to 0.99) of the predictive probability distribution. Teams could also submit a single point forecast. At the first submission we asked teams to add a pre-specified set of metadata briefly describing the forecasting team and methods (see supplementary information (SI)). No restrictions were placed on who could submit forecasts. To increase participation we actively contacted known forecasting teams across Europe and the US and advertised among the ECDC network. Teams submitted a broad spectrum of model types, ranging from mechanistic to empirical models, agent-based and statistical models, and ensembles of multiple quantitative or qualitative models (described at [22]). We maintain a full project specification with a detailed submissions protocol [23].

We collected forecasts submitted weekly in real time over the 52 week period from 08 March 2021 to 07 March 2022. Teams submitted at latest two days after the complete dataset for the latest forecasting week became available each Sunday. We implemented an automated validation programme to check that each new forecast conformed to standardised formatting. Forecast validation ensured a monotonic increase of predictions with each increasing quantile, integer-valued non-negative counts of predicted cases, as well as consistent date and location definitions.

Each week we used all available valid forecasts to create a weekly real-time ensemble model (referred to as “the ensemble” from here on), for each of the 256 possible forecast targets: incident cases and deaths in 32 locations over the following one through four weeks. The ensemble method was an unweighted average of all models’ forecast values, at each predictive quantile for a given location, target, and horizon. From 08 March 2021, we used the arithmetic mean. However we noticed that including highly anomalous forecasts in a mean ensemble produced extremely wide uncertainty. To mitigate this, from 26th July 2021 onwards the ensemble instead used a median of all predictive quantiles.

We created an open and publicly accessible interface to the forecasts and ensemble, including an online visualisation tool allowing viewers to see past data and interact with one or multiple forecasts for each country and target for up to four weeks’ horizon [24]. All forecast and meta data are freely available and held on Github [17] and Zoltar, a platform for hosting epidemiological forecasts [25], [26]. In the codebase for this study [27] we provide a simple method and instructions for downloading and preparing these data for analysis using R. We encourage other researchers to freely use and adapt this to support their own analyses.

### Forecast evaluation

In this study we focus only on the comparative performance of forecasting models relative to each other. For each model, we evaluated performance in terms of both accuracy (coverage) and overall predictive performance (weighted interval score). We evaluated all previous forecasts against actual observed values for each model, stratified by the forecast horizon, location, and target. We calculated scores using the *scoringutils* R package [28]. We removed any forecast surrounding (both the week of, and the first week after) a strongly anomalous data point. We defined anomalous as where any subsequent data release revised that data point by over 5%.

We established the accuracy of each model’s prediction boundaries as the coverage of the predictive intervals. We calculated coverage at a given interval level k, where *k* ∈ [0,1], as the proportion *p* of observations that fell within the corresponding central predictive intervals across locations and forecast dates. A perfectly calibrated model would have *p* = *k* at all 11 levels (corresponding to 22 quantiles excluding the median). An underconfident model at level *k* would have *p > k*, i.e. more observations fall within a given interval than expected. In contrast, an overconfident model at level *k* would have *p < k*, i.e. fewer observations fall within a given interval than expected. We here focus on coverage at the *k* = 0.5 and *k* = 0.95 levels.

We also assessed the overall predictive performance of weekly forecasts using the weighted interval score (WIS) across all available quantiles. The WIS represents a parsimonious approach to scoring forecasts based on uncertainty represented as forecast values across a set of quantiles [29], and is a strictly proper scoring rule, that is, it is optimal for predictions that come from the data-generating model. As a consequence, the WIS encourages forecasters to report predictions representing their true belief about the future [30]. Each forecast for a given location and date is scored based on an observed count of weekly incidence, the median of the predictive distribution and the predictive upper and lower quantiles corresponding to the central predictive interval level.

Not all models provided forecasts for all locations and dates, and we needed to compare predictive performance in the face of various levels of missingness across each forecast target. Therefore we calculated a relative WIS. This is a measure of forecast performance which takes into account that different teams may not cover the same set of forecast targets (i.e., weeks and locations). The relative WIS is computed using a *pairwise comparison tournament* where for each pair of models a mean score ratio is computed based on the set of shared targets. The relative WIS of a model with respect to another model is then the ratio of their respective geometric mean of the mean score ratios, such that smaller values indicate better performance.

We scaled the relative WIS of each model with the relative WIS of a baseline model, for each forecast target, location, date, and horizon. The baseline model assumes case or death counts stay the same as the latest data point over all future horizons, with expanding uncertainty, described previously in [31]. Here we report the relative WIS of each model with respect to the baseline model.

### Retrospective ensemble methods

We retrospectively explored alternative methods for combining forecasts for each target at each week. A natural way to combine probability distributions available in the quantile format used here is [32] ![Formula][1]</img>  

Where *F*1 … *F**n* are the cumulative distribution functions of the individual probability distributions (in our case, the predictive distributions of each forecast model *i* contributed to the hub), *w**i* are a set of weights in [0,1]; and *α* are the quantile levels such that ![Formula][2]</img>  

Different ensemble choices then mainly translate to the choice of weights *w**i*. An arithmetic mean ensemble uses weights at *w**i* = 1*/n*, where all weights are equal and sum up to 1.

Alternatively, we can choose a set of weights to apply to forecasts before they are combined. Numerous options exist for choosing these weights with the aim to maximise predictive performance, including choosing weights to reflect each forecast’s past performance (thereby moving from an untrained to a trained ensemble). A straightforward choice is so-called inverse score weighting. In this case, the weights are calculated as ![Formula][3]</img>  where *S**i* reflects the forecasting skill of forecaster *i*, normalised so that weights sum to 1. This method of weighting was found in the US to outperform unweighted scores during some time periods [33] but this was not confirmed in a similar study in Germany and Poland [11].

When constructing ensembles from quantile means, a single outlier can have an oversized effect on the ensemble forecast. Previous research has found that a median ensemble, replacing the arithmetic mean of each quantile with a median of the same values, yields competitive performance while maintaining robustness to outlying forecasts [34]. Building on this, we also created weighted median ensembles using the weights described above and a Harrel-Davis quantile estimator with a beta function to approximate the weighted percentiles [35]. We then compared the performance of unweighted and inverse relative WIS weighted mean and median ensembles.

## Results

An example of weekly forecasts from the ensemble model is shown in Figure 1.

![Figure 1:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/06/16/2022.06.16.22276024/F1.medium.gif)

[Figure 1:](http://medrxiv.org/content/early/2022/06/16/2022.06.16.22276024/F1)

Figure 1: 
Ensemble forecasts of weekly incident cases in Germany over periods of increasing SARS-CoV-2 variants Delta (B.1.617.2, left) and Omicron (B.1.1.529, right). Black indicates observed data. Coloured ribbons represent each weekly forecast of 1-4 weeks ahead (showing median, 50%, and 90% probability). For each variant, forecasts are shown over an x-axis bounded by the earliest dates at which 5% and 99% of sequenced cases were identified as the respective variant of concern, while vertical dotted lines indicate the approximate date that the variant reached dominance (>50% sequenced cases).

Over the whole study period, 26 independently participating forecasting teams contributed results from 28 unique forecasting models (see supplementary information (SI), Table 1). The number of models contributing to each ensemble forecast varied over time and by forecasting target (SI Figure 1). Not all modellers created forecasts for all locations, horizons, or variables. At most, 15 models contributed forecasts for cases in Germany at the 1 week horizon, with an accumulated 592 forecasts for that single target over the study period. In contrast, deaths in Finland at the 2 week horizon saw the smallest number of forecasts, with only 6 independent models contributing a total 24 forecasts. Similarly, not all teams forecast across all quantiles of the predictive distribution for each target, with only 23 models providing the full set of 23 quantiles. No ensemble forecast was composed of less than 3 independent models.

View this table:
[Table 1:](http://medrxiv.org/content/early/2022/06/16/2022.06.16.22276024/T1)

Table 1: 
Predictive performance of main ensembles, as measured by the scaled relative WIS.

Using all models and the ensemble, we created 2106 forecasting scores where each score summarises a unique combination of forecasting model, variable, country, and week ahead horizon (SI Figure 2). We visually compared the absolute performance of forecasts in predicting numbers of incident cases and deaths. We observed that forecasts predicted well in times of stable epidemic behaviour, while struggling to accurately predict at longer horizons around inflection points, for example during rapid changes in population-level behaviour or surveillance. Forecast models varied widely in their ability to predict and account for the introduction of new variants, giving the ensemble forecast over these periods a high level of uncertainty (Figure 1).

In relative terms, the ensemble of all models performed well compared to both its component models and the baseline. By relative WIS scaled against a baseline of 1 (where a score <1 indicates outperforming the baseline), the median score for participating models across all submitted forecasts was 1.04, while the median score of forecasts from the ensemble model was 0.71. Across all horizons and locations, the ensemble performed better on scaled relative WIS than 84% of participating model scores when forecasting cases (with a total N=862), and 92% of participating model scores for forecasts of incident deaths (N=746).

The performance of individual and ensemble forecasts varied by length of the forecast horizon (Figure 2). At each horizon, the typical performance of the ensemble outperformed both the baseline model and the aggregated scores of all its component models, although we saw wide variation between individual models in performance across horizons. Both individual models and the ensemble saw a trend of worsening performance at longer horizons when forecasting cases with the median scaled relative WIS of the ensemble across locations worsened from 0.62 for one-week ahead forecasts to 0.9 when forecasting four weeks ahead. Performance for forecasts of deaths was more stable over one through four weeks, with median ensemble performance moving from 0.69 to 0.76 across the four week horizons.

![Figure 2:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/06/16/2022.06.16.22276024/F2.medium.gif)

[Figure 2:](http://medrxiv.org/content/early/2022/06/16/2022.06.16.22276024/F2)

Figure 2: 
Performance of short-term forecasts aggregated across all individually submitted models and the Hub ensemble, by horizon, forecasting cases (left) and deaths (right). Performance measured by relative weighted interval score scaled against a baseline (dotted line, 1), and coverage of uncertainty at the 50% and 95% levels. Boxplot, with width proportional to number of observations, show interquartile ranges with outlying scores as faded points. The target range for each set of scores is shaded in yellow.

We observed similar trends in performance across horizon when considering how well the ensemble was calibrated with respect to the observed data. At one week ahead the case ensemble was well calibrated (ca. 50% and 95% nominal coverage at the 50% and 95% levels respectively). This did not hold at longer forecast horizons as the case forecasts became increasingly over-confident. Meanwhile, the ensemble of death forecasts was well calibrated at the 95% level across all horizons, and the calibration of death forecasts at the 50% level improved with lengthening horizons compared to being underconfident at shorter horizons.

The ensemble also performed consistently well in comparison to individual models when forecasting across countries (Figure 3). In total, across 32 countries forecasting for one through four weeks, when forecasting cases the ensemble outperformed 75% of component models in 21 countries, and outperformed all available models in 3 countries. When forecasting deaths, the ensemble outperformed 75% and 100% of models in 30 and 9 countries respectively. Considering only the the two-week horizon shown in Figure 3, the ensemble of case forecasts outperformed 75% models in 24 countries and all models in only 12 countries. At the two-week horizon for forecasts of deaths, the ensemble outperformed 75% and 100% of its component models in 30 and 26 countries respectively.

![Figure 3:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/06/16/2022.06.16.22276024/F3.medium.gif)

[Figure 3:](http://medrxiv.org/content/early/2022/06/16/2022.06.16.22276024/F3)

Figure 3: 
Performance of short-term forecasts across models and median ensemble (asterisk), by country, forecasting cases (top) and deaths (bottom) for two-week ahead forecasts, according to the relative weighted interval score. Boxplots show interquartile ranges, with outliers as faded points, and the ensemble model performance is marked by an asterisk. y-axis is cut-off to an upper bound of 4 for readability.

We considered alternative methods for creating ensembles from the participating forecasts, using either a mean or median to combine either weighted or unweighted forecasts (Table 1). Across locations we observed that the median outperformed the mean across all one through four week horizons and both cases and death targets, for all but cases at the 1 week horizon. This held regardless of whether the component forecasts were weighted or unweighted by their individual past performance. Between methods of combination, weighting made little difference to the performance of the median ensemble, but slightly improved performance of the mean ensemble.

## Discussion

We collated 12 months of forecasts of COVID-19 cases and deaths across 32 countries in Europe, collecting from multiple independent teams and using a principled approach to standardising both forecast targets and the predictive distribution of forecasts. We combined these into an ensemble forecast and compared the relative performance of forecasts between models, finding that the ensemble forecasts outperformed most individual models across all countries and horizons over time.

Across all models we observed that forecasting changes in trend in real time was particularly challenging. Our study period included multiple fundamental changes in viral-, individual-, and population-level factors driving the transmission of COVID-19 across Europe. In early 2021, the introduction of vaccination started to change population-level associations between infections, cases, and deaths [36], while the Delta variant emerged and became dominant [37]. Similarly from late 2021 we saw the interaction of individually waning immunity during the emergence and global spread of the Omicron variant [38]. Neither the extent nor timing of these factors were uniform across European countries covered by the Forecast Hub [39]. This meant that the performance of any single forecasting model depended partly on the ability, speed, and precision with which it could adapt to new conditions for each forecast target.

We observed a contrast between a more stable performance of forecasting deaths further into the future compared to forecasts of cases. Previous work has found rapidly declining performance for case forecasts with increasing horizon [31], [40], while death forecasts can perform well with up to six weeks lead time [41]. We can similarly link this to the specific epidemic dynamics in this study. COVID-19 has a typical serial interval of less than a week [42]. This implies that case forecasts of more than two weeks only remain valid if rates of transmission and detection remain stable over the entire forecast horizon. This is unlikely to have held given the rapid changes in epidemic dynamics across many countries in Europe. Meanwhile, we can interpret the higher reliability of death forecasts as due to the longer time lag between infection and death [43], and higher consistency of reporting deaths in surveillance data [44]. This allows forecasters to incorporate the effect of changes in transmission. Additionally, the performance of trend-based forecasts may have benefited from the slower changes to trends in incident deaths caused by gradually increasing vaccination rates.

We found the ensemble in this study continued to outperform both other models and the baseline at up to four weeks ahead. Our results support previous findings that ensemble forecasts are the best or nearly the best performing models with respect to absolute predictive performance and appropriate coverage of uncertainty [12], [14], [31]. While the ensemble was consistently high performing, it was not strictly dominant across all forecast targets, reflecting findings from previous comparable studies of COVID-19 forecasts [11], [45]. Our finding suggests the usefulness of an ensemble as a robust summary when forecasting across many spatiotemporal targets, without replacing the importance of communicating the full range of model predictions.

When exploring variations in ensemble methods, we found that the choice of median over means yielded the most consistent improvement in predictive performance, regardless of the method of weighting. Other work has supported the importance of the median in providing a stable forecast that better accounts for outlier forecasts than the mean [45], although though this finding may be dependent on the quality of the individual forecast submissions. In contrast, weighing models by past performance did not result in any consistent improvement in performance. This is in line with existing mixed evidence for any optimal ensemble method for combining short term probabilistic infectious disease forecasts. Many methods of combination have performed competitively in analyses of forecasts for COVID-19 in the US, including the simple mean and weighted approaches outperforming unweighted or median methods [33]. This contrasts with later analyses finding weighted methods to give similar performance to a median average [10], [45]. We can partly explain this inconsistency if performance of each method depends on the outcome being predicted (cases, deaths), its count (incident, cumulative) and absolute level, the changing disease dynamics, and the varying quality and quantity of forecasting teams over time.

We note several limitations in our approach to assessing the relative performance of an ensemble among forecast models. Our results are the outcome of evaluating forecasts against a specific performance metric and baseline, where multiple options for evaluation exist and the choice reflects the aim of the evaluation process. Further, our choice of baseline model affects the given performance scores in absolute terms, and more generally the choice of appropriate baseline for epidemic forecast models is not obvious when assessing infectious disease forecasts. The model used here is supported by previous work [31], yet previous evaluation in a similar context has suggested that choice of baseline affects relative performance in general [46], and future research should be done on the best choices of baseline models in the context of infectious disease epidemics.

Our assessment of forecast performance may further have been inaccurate due to limitations in the observed data against which we evaluated forecasts. We sourced data from a globally aggregated database to maintain compatibility across 32 countries [21]. However, this made it difficult to identify the origin of lags and inconsistencies between national data streams, and to what extent these could bias forecasts for different targets. In particular we saw some real time data revised retrospectively, introducing bias in either direction where the data used to create forecasts was not the same as that used to evaluate it. We attempted to mitigate this by using an automated process for determining data revisions, and excluding forecasts made at a time of missing, unreliable, or heavily revised data. More generally it is unclear if the expectation of observation revisions should be a feature built into forecasts. Further research is needed to understand the perspective of end-users of forecasts in order to assess this.

In an emergency setting, open access to visualised forecasts and underlying data is useful for researchers, policymakers, and the public. For forecast producers, an easily accessible comparison between results from different methods can highlight individual strengths and weaknesses and help prioritise new areas of work. For forecast users, probabilistic information about the future can influence decisions in the present that can then change epidemic dynamics [1].

Existing participatory modelling efforts for COVID-19 have been useful for policy communication [2], while multi-country efforts have included only single models adapted to country-specific parameters [47]–[49]. By expanding participation to many modelling teams, our work was able to create robust ensemble forecasts across Europe while allowing comparison across forecasts built with different interpretations of current data, on a like for like scale in real time. At the same time, collating time-stamped predictions ensures that we can test true out-of-sample performance of models and avoid retrospective claims of performance. Testing the limits of forecasting ability with these comparisons forms an important part of communicating any model-based prediction to decision makers.

This study raises many further questions which could inform epidemic forecast modellers and users. The dataset created by the European Forecast Hub is an openly accessible, standardised, and extensively documented catalogue of real time forecasting work from a range of teams and models across Europe [24], and we recommend its use for further research on forecast performance. In the code developed for this study we provide a worked example of downloading and using both the forecasts and their evaluation scores [27].

Future work could explore the impact on forecast models of changing epidemiology at a broad spatial scale by combining analyses of trends and turning points in cases and deaths with forecast performance, or extending to include data on vaccination, variant, or policy changes over time. There is also much scope for future research into methods for combining forecasts to improve performance of an ensemble. This includes altering the inclusion criteria of forecast models based on different thresholds of past performance, excluding or including only forecasts that predict the lowest and highest values (trimming) [33], or using alternative weighting methods such as quantile regression averaging [12]. Exploring these questions would add to our understanding of real time performance, supporting and improving future forecasting efforts.

We see additional scope to adapt the Hub format to the changing COVID-19 situation across Europe. We have extended the Forecast Hub infrastructure to include short term forecasts for hospitalisations with COVID-19, which is a challenging task due to limited data across the locations covered by the hub. As the policy focus shifts from immediate response to anticipating changes brought by vaccinations or the geographic spread of new variants [39], we are also separately investigating models for longer term scenarios in addition to the short term forecasts in a similar framework to existing scenario modelling work in the US [50].

In conclusion, we have shown that during a rapidly evolving epidemic spreading through multiple populations, an ensemble forecast performed highly consistently across a large matrix of forecast targets, typically outperforming the majority of its separate component models and a naive baseline model. In addition, we have linked issues with the predictability of short-term case forecasts to underlying COVID-19 epidemiology, and shown that ensemble methods based on past model performance were unable to reliably improve forecast performance. Our work constitutes a step towards both unifying COVID-19 forecasts and improving our understanding of them.

## Supporting information

Supplement: SI table 1, SI figure 1, SI figure 2 [[supplements/276024_file02.pdf]](pending:yes)

## Data Availability

All source data were openly available before the study, originally available at: [https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe](https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe). All data and code for this study are openly available on Github: covid19-forecast-hub-europe/euro-hub-ensemble.

[https://github.com/covid19-forecast-hub-europe/euro-hub-ensemble](https://github.com/covid19-forecast-hub-europe/euro-hub-ensemble) 

[https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe](https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe) 

*   Received June 16, 2022.
*   Revision received June 16, 2022.
*   Accepted June 16, 2022.


*   © 2022, Posted by Cold Spring Harbor Laboratory

This pre-print is available under a Creative Commons License (Attribution 4.0 International), CC BY 4.0, as described at [http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/)

## References

1.  [1]. P. van Basshuysen,  L. White,  D. Khosrowi, and  M. Frisch, “Three Ways in Which Pandemic Models May Perform a Pandemic,” Erasmus Journal for Philosophy and Economics, vol. 14, no. 1, 1, pp. 110-127-110-127, Jul. 2021, doi: 10.23941/ejpe.v14i1.582.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.23941/ejpe.v14i1.582&link_type=DOI) 

2.  [2].CDC, “Coronavirus Disease 2019 (COVID-19),” Feb. 11, 2020. [https://www.cdc.gov/coronavirus/2019-ncov/science/forecasting/forecasting.html](https://www.cdc.gov/coronavirus/2019-ncov/science/forecasting/forecasting.html) (accessed Jan. 09, 2022).
    
    

3.  [3].European Centre for Disease Prevention and Control, “Forecasting COVID-19 cases and deaths in Europe - new hub will support European pandemic planning,” Apr. 22, 2021. [https://www.ecdc.europa.eu/en/news-events/forecasting-covid-19-cases-and-deaths-europe-new-hub](https://www.ecdc.europa.eu/en/news-events/forecasting-covid-19-cases-and-deaths-europe-new-hub)
    
    

4.  [4]. J. Zelner,  J. Riou,  R. Etzioni, and  A. Gelman, “Accounting for uncertainty during a pandemic,” PATTER, vol. 2, no. 8, Aug. 2021, doi: 10.1016/j.patter.2021.100310.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.patter.2021.100310&link_type=DOI) 

5.  [5]. L. P. James,  J. A. Salomon,  C. O. Buckee, and  N. A. Menzies, “The Use and Misuse of Mathematical Modeling for Infectious Disease Policymaking: Lessons for the COVID-19 Pandemic,” Med Decis Making, vol. 41, no. 4, pp. 379–385, May 2021, doi: 10.1177/0272989X21990391.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/0272989X21990391&link_type=DOI) 

6.  [6]. N. G. Reich et al., “A collaborative multiyear, multimodel assessment of seasonal influenza fore-casting in the United States,” PNAS, vol. 116, no. 8, pp. 3146–3154, Feb. 2019, doi: 10.1073/pnas.1812594116.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoicG5hcyI7czo1OiJyZXNpZCI7czoxMDoiMTE2LzgvMzE0NiI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIyLzA2LzE2LzIwMjIuMDYuMTYuMjIyNzYwMjQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

7.  [7]. M. A. Johansson et al., “An open challenge to advance probabilistic forecasting for dengue epidemics,” PNAS, vol. 116, no. 48, pp. 24268–24274, Nov. 2019, doi: 10.1073/pnas.1909865116.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoicG5hcyI7czo1OiJyZXNpZCI7czoxMjoiMTE2LzQ4LzI0MjY4IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDYvMTYvMjAyMi4wNi4xNi4yMjI3NjAyNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

8.  [8]. N. G. Reich et al., “Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S,” PLoS Comput Biol, vol. 15, no. 11, p. e1007486, Nov. 2019, doi: 10.1371/journal.pcbi.1007486.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pcbi.1007486&link_type=DOI) 

9.  [9]. E. Y. Cramer et al., “The United States COVID-19 Forecast Hub dataset,” p. 2021.11.04.21265886, Nov. 2021, doi: 10.1101/2021.11.04.21265886.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoibWVkcnhpdiI7czo1OiJyZXNpZCI7czoyMToiMjAyMS4xMS4wNC4yMTI2NTg4NnYxIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDYvMTYvMjAyMi4wNi4xNi4yMjI3NjAyNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

10. [10]. E. L. Ray et al., “Ensemble Forecasts of Coronavirus Disease 2019 (COVID-19) in the U.S.” p. 2020.08.19.20177493, Aug. 2020, doi: 10.1101/2020.08.19.20177493.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoibWVkcnhpdiI7czo1OiJyZXNpZCI7czoyMToiMjAyMC4wOC4xOS4yMDE3NzQ5M3YxIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDYvMTYvMjAyMi4wNi4xNi4yMjI3NjAyNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

11. [11]. J. Bracher et al., “A pre-registered short-term forecasting study of COVID-19 in Germany and Poland during the second wave,” Nat Commun, vol. 12, no. 1, 1, p. 5173, Aug. 2021, doi: 10.1038/s41467-021-25207-0.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41467-021-25207-0&link_type=DOI) 

12. [12]. S. Funk et al., “Short-term forecasts to inform the response to the Covid-19 epidemic in the UK,” medRxiv, p. 2020.11.11.20220962, Nov. 2020, doi: 10.1101/2020.11.11.20220962.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoibWVkcnhpdiI7czo1OiJyZXNpZCI7czoyMToiMjAyMC4xMS4xMS4yMDIyMDk2MnYyIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDYvMTYvMjAyMi4wNi4xNi4yMjI3NjAyNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

13. [13]. M. Bicher et al., “Supporting COVID-19 Policy-Making with a Predictive Epidemiological Multi-Model Warning System,” medRxiv, p. 2020.10.18.20214767, Apr. 2021, doi: 10.1101/2020.10.18.20214767.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoibWVkcnhpdiI7czo1OiJyZXNpZCI7czoyMToiMjAyMC4xMC4xOC4yMDIxNDc2N3YzIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDYvMTYvMjAyMi4wNi4xNi4yMjI3NjAyNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

14. [14]. C. Viboud et al., “The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt,” Epidemics, vol. 22, pp. 13–21, Mar. 2018, doi: 10.1016/j.epidem.2017.08.002.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/J.EPIDEM.2017.08.002.&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=28958414&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F06%2F16%2F2022.06.16.22276024.atom) 

15. [15]. R. Buizza, “Introduction to the special issue on ‘25 years of ensemble forecasting’,” Quarterly Journal of the Royal Meteorological Society, vol. 145, no. S1, pp. 1–11, 2019, doi: 10.1002/qj.3370.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/qj.3370&link_type=DOI) 

16. [16]. K. R. Moran et al., “Epidemic Forecasting is Messier Than Weather Forecasting: The Role of Human Behavior and Internet Data Streams in Epidemic Forecast,” J Infect Dis, vol. 214, pp. S404–S408, Dec. 2016, doi: 10.1093/infdis/jiw375.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/infdis/jiw375&link_type=DOI) 

17. [17].European Covid-19 Forecast Hub, European COVID-19 Forecast Hub. covid19-forecast-hub-europe, 2021.Available: [https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe](https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe)
    
    

18. [18]. E. Cramer et al., “Reichlab/Covid19-forecast-hub: Release for Zenodo, 20210816,” Aug. 2021, doi: 10.5281/zenodo.5208210.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.5281/zenodo.5208210&link_type=DOI) 

19. [19]. S. Y. Wang et al., “Reichlab/covidHubUtils: Repository release for Zenodo,” Aug. 2021, doi: 10.5281/zenodo.5207940.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.5281/zenodo.5207940&link_type=DOI) 

20. [20]. J. Bracher et al., The German and Polish COVID-19 Forecast Hub. 2020.Available: [https://github.com/KITmetricslab/covid19-forecast-hub-de](https://github.com/KITmetricslab/covid19-forecast-hub-de)
    
    

21. [21]. E. Dong,  H. Du, and  L. Gardner, “An interactive web-based dashboard to track COVID-19 in real time,” The Lancet Infectious Diseases, vol. 20, no. 5, pp. 533–534, May 2020, doi: 10.1016/S1473-3099(20)30120-1.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S1473-3099(20)30120-1&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32087114&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F06%2F16%2F2022.06.16.22276024.atom) 

22. [22].European Covid-19 Forecast Hub, “Community.” [https://covid19forecasthub.eu/community.html](https://covid19forecasthub.eu/community.html)
    
    

23. [23].European Covid-19 Forecast Hub, “Covid19-forecast-hub-europe: Wiki.” [https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe](https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe)
    
    

24. [24].European Covid-19 Forecast Hub, “European Covid-19 Forecast Hub.” [https://covid19forecasthub.eu/index.html](https://covid19forecasthub.eu/index.html)
    
    

25. [25].EpiForecasts, “Project: ECDC European COVID-19 Forecast Hub - Zoltar,” 2021. [https://www.zoltardata.com/project/238](https://www.zoltardata.com/project/238)
    
    

26. [26]. N. G. Reich,  M. Cornell,  E. L. Ray,  K. House, and  K. Le, “The Zoltar forecast archive, a tool to standardize and store interdisciplinary prediction research,” Sci Data, vol. 8, no. 1, 1, p. 59, Feb. 2021, doi: 10.1038/s41597-021-00839-5.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41597-021-00839-5&link_type=DOI) 

27. [27].Predictive performance of multi-model ensemble forecasts of Covid-19 across European nations. covid19-forecast-hub-europe, 2022.Available: [https://github.com/covid19-forecast-hub-europe/euro-hub-ensemble](https://github.com/covid19-forecast-hub-europe/euro-hub-ensemble)
    
    

28. [28]. Nikos I Bosse,  Hugo Gruson,  Sebastian Funk,  Epi Forecasts, and  Sam Abbott, Scoringutils: Utilities for Scoring and Assessing Predictions. 2020.Available: [https://github.com/epiforecasts/scoringutils](https://github.com/epiforecasts/scoringutils)
    
    

29. [29]. J. Bracher,  E. L. Ray,  T. Gneiting, and  N. G. Reich, “Evaluating epidemic forecasts in an interval format,” PLOS Computational Biology, vol. 17, no. 2, p. e1008618, Feb. 2021, doi: 10.1371/jour-nal.pcbi.1008618.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/jour-nal.pcbi.1008618&link_type=DOI) 

30. [30]. T. Gneiting and  A. E. Raftery, “Strictly Proper Scoring Rules, Prediction, and Estimation,” Journal of the American Statistical Association, vol. 102, no. 477, pp. 359–378, Mar. 2007, doi: 10.1198/016214506000001437.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1198/016214506000001437&link_type=DOI) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000244361000032&link_type=ISI) 

31. [31]. E. Y. Cramer et al., “Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the US,” medRxiv, p. 2021.02.03.21250974, Jan. 2021, doi: 10.1101/2021.02.03.21250974.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoibWVkcnhpdiI7czo1OiJyZXNpZCI7czoyMToiMjAyMS4wMi4wMy4yMTI1MDk3NHYzIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDYvMTYvMjAyMi4wNi4xNi4yMjI3NjAyNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

32. [32]. C. Genest, “Vincentization Revisited,” The Annals of Statistics, vol. 20, no. 2, pp. 1137–1142, 1992,Available: [https://www.jstor.org/stable/2242003](https://www.jstor.org/stable/2242003)
    
    

33. [33]. J. W. Taylor and  K. S. Taylor, “Combining Probabilistic Forecasts of COVID-19 Mortality in the United States,” Eur J Oper Res, Jun. 2021, doi: 10.1016/j.ejor.2021.06.044.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.ejor.2021.06.044&link_type=DOI) 

34. [34]. E. L. Ray et al., “Comparing trained and untrained probabilistic ensemble forecasts of COVID-19 cases and deaths in the United States,” Jan. 28, 2022. Accessed: Mar. 30, 2022. [Online]. Available: [http://arxiv.org/abs/2201.12387](http://arxiv.org/abs/2201.12387)
    
    

35. [35]. F. E. Harrell and  C. E. Davis, “A new distribution-free quantile estimator,” Biometrika, vol. 69, no. 3, pp. 635–640, Dec. 1982, doi: 10.1093/biomet/69.3.635.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/biomet/69.3.635&link_type=DOI) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1982PR72700016&link_type=ISI) 

36. [36].European Centre for Disease Prevention and Control, “Interim guidance on the benefits of full vaccination against COVID-19 for transmission and implications for non-pharmaceutical interventions - 21 April 2021,” ECDC, Stockholm, 2021.Available: [https://www.ecdc.europa.eu/en/publications-data/interim-guidance-benefits-full-vaccination-against-covid-19-transmission](https://www.ecdc.europa.eu/en/publications-data/interim-guidance-benefits-full-vaccination-against-covid-19-transmission)
    
    

37. [37].European Centre for Disease Prevention and Control, “Threat Assessment Brief: Implications for the EU/EEA on the spread of the SARS-CoV-2 Delta (B.1.617.2) variant of concern,” ECDC, Stockholm, Jun. 2021.Available: [https://www.ecdc.europa.eu/en/publications-data/threat-assessment-emergence-and-impact-sars-cov-2-delta-variant](https://www.ecdc.europa.eu/en/publications-data/threat-assessment-emergence-and-impact-sars-cov-2-delta-variant)
    
    

38. [38].European Centre for Disease Prevention and Control, “Assessment of the further spread and potential impact of the SARS-CoV-2 Omicron variant of concern in the EU/EEA, 19th update,” Jan. 27, 2022. [https://www.ecdc.europa.eu/en/publications-data/covid-19-omicron-risk-assessment-further-emergence-and-potential-impact](https://www.ecdc.europa.eu/en/publications-data/covid-19-omicron-risk-assessment-further-emergence-and-potential-impact)
    
    

39. [39].European Centre for Disease Prevention and Control, “Overview of the implementation of COVID-19 vaccination strategies and deployment plans in the EU/EEA,” ECDC, Stockholm, Nov. 2021.Available: [https://www.ecdc.europa.eu/en/publications-data/overview-implementation-covid-19-vaccination-strategies-and-deployment-plans](https://www.ecdc.europa.eu/en/publications-data/overview-implementation-covid-19-vaccination-strategies-and-deployment-plans)
    
    

40. [40]. M. Castro,  S. Ares,  J. A. Cuesta, and  S. Manrubia, “The turning point and end of an expanding epidemic cannot be precisely forecast,” Proceedings of the National Academy of Sciences, vol. 117, no. 42, pp. 26190–26196, Oct. 2020, doi: 10.1073/pnas.2007868117.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoicG5hcyI7czo1OiJyZXNpZCI7czoxMjoiMTE3LzQyLzI2MTkwIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDYvMTYvMjAyMi4wNi4xNi4yMjI3NjAyNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

41. [41]. J. Friedman et al., “Predictive performance of international COVID-19 mortality forecasting models,” Nat Commun, vol. 12, no. 1, 1, p. 2609, May 2021, doi: 10.1038/s41467-021-22457-w.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41467-021-22457-w&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33972512&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F06%2F16%2F2022.06.16.22276024.atom) 

42. [42]. M. Alene,  L. Yismaw,  M. A. Assemie,  D. B. Ketema,  W. Gietaneh, and  T. Y. Birhan, “Serial interval and incubation period of COVID-19: A systematic review and meta-analysis,” BMC Infectious Diseases, vol. 21, no. 1, p. 257, Mar. 2021, doi: 10.1186/s12879-021-05950-x.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s12879-021-05950-x&link_type=DOI) 

43. [43]. R. Jin, “The lag between daily reported Covid-19 cases and deaths and its relationship to age,” J Public Health Res, vol. 10, no. 3, p. 2049, Mar. 2021, doi: 10.4081/jphr.2021.2049.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.4081/jphr.2021.2049&link_type=DOI) 

44. [44]. M. Català et al., “Robust estimation of diagnostic rate and real incidence of COVID-19 for European policymakers,” PLOS ONE, vol. 16, no. 1, p. e0243701, Jan. 2021, doi: 10.1371/jour-nal.pone.0243701.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/jour-nal.pone.0243701&link_type=DOI) 

45. [45]. L. Brooks, “Comparing ensemble approaches for short-term probabilistic COVID-19 forecasts in the U.S.” 2020. [https://forecasters.org/blog/2020/10/28/comparing-ensemble-approaches-for-short-term-probabilistic-covid-19-forecasts-in-the-u-s/](https://forecasters.org/blog/2020/10/28/comparing-ensemble-approaches-for-short-term-probabilistic-covid-19-forecasts-in-the-u-s/) (accessed Jul. 15, 2021).
    
    

46. [46]. J. Bracher et al., “National and subnational short-term forecasting of COVID-19 in Germany and Poland, early 2021,” p. 2021.11.05.21265810, Nov. 2021, doi: 10.1101/2021.11.05.21265810.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoibWVkcnhpdiI7czo1OiJyZXNpZCI7czoyMToiMjAyMS4xMS4wNS4yMTI2NTgxMHYxIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDYvMTYvMjAyMi4wNi4xNi4yMjI3NjAyNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

47. [47]. R. Aguas et al., “Modelling the COVID-19 pandemic in context: An international participatory approach,” BMJ Global Health, vol. 5, no. 12, p. e003126, Dec. 2020, doi: 10.1136/bmjgh-2020-003126.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiYm1qZ2giO3M6NToicmVzaWQiO3M6MTI6IjUvMTIvZTAwMzEyNiI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIyLzA2LzE2LzIwMjIuMDYuMTYuMjIyNzYwMjQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

48. [48]. K. Adib et al., “A participatory modelling approach for investigating the spread of COVID-19 in countries of the Eastern Mediterranean Region to support public health decision-making,” BMJ Global Health, vol. 6, no. 3, p. e005207, Mar. 2021, doi: 10.1136/bmjgh-2021-005207.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiYm1qZ2giO3M6NToicmVzaWQiO3M6MTE6IjYvMy9lMDA1MjA3IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDYvMTYvMjAyMi4wNi4xNi4yMjI3NjAyNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

49. [49]. A. Agosto,  A. Campmas,  P. Giudici, and  A. Renda, “Monitoring COVID-19 contagion growth,” Statistics in Medicine, vol. 40, no. 18, pp. 4150–4160, 2021, doi: 10.1002/sim.9020.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/sim.9020&link_type=DOI) 

50. [50]. R. K. Borchering, “Modeling of Future COVID-19 Cases, Hospitalizations, and Deaths, by Vaccination Rates and Nonpharmaceutical Intervention Scenarios — United States, April–September 2021,” MMWR Morb Mortal Wkly Rep, vol. 70, 2021, doi: 10.15585/mmwr.mm7019e3.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.15585/mmwr.mm7019e3&link_type=DOI)

 [1]: /embed/graphic-1.gif
 [2]: /embed/graphic-2.gif
 [3]: /embed/graphic-3.gif