Predictive Accuracy of a Hierarchical Logistic Model of Cumulative SARS-CoV-2 Case Growth

BACKGROUND

Mathematical and simulation models of infectious disease dynamics are essential for understanding and forecasting the development of epidemics.¹ As of June 2020, the ongoing severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic has called increased attention to epidemiological modeling both as a method of scientific inquiry and as a tool to inform political decision making.^2–5

Among epidemiological modeling methods, a distinction between mechanistic and phenomenological approaches is frequently made. While mechanistic approaches model the transmission dynamics based on substantial concepts from biology, virology, infectology, and related disciplines, phenomenological (sometimes termed ‘statistical’) models are looking for a mathematical function that fits observed data well without clear assumptions about the underlying processes.^1,2 Mechanistic models are usually used to compare possible scenarios and to estimate the relative effects of different interventions rather than to produce precise predictions. On the contrary, phenomenological models are commonly optimized for forecasting. From a broader perspective, mechanistic and phenomenological approaches can be considered as the epidemiological modeling representatives of the long-standing explanation-prediction controversy.⁶ It should be noted that although the distinction between these two model classes is instructive and one side usually predominates, most approaches have both mechanistic and phenomenological components, and some are explicitly balanced (so called ‘semi-mechanistic’ or ‘hybrid’ models).

Although the value of any predictive model is ultimately determined by whether it improved critical decision making,^7,8 a rigorous scientific appraisal should also include a comparison of what have been predicted to what have actually happened.^1,9,10 Unfortunately, infectious disease predictions models are rarely evaluated during or after outbreaks.^9,10 Notable exceptions include systematic evaluation of models about the epidemiology of severe acute respiratory syndrome (SARS),^11,12 influenza,^13,14 ebola,^7,9,15,16 dengue,^10,17 foot-and-mouth disease,⁸ and trachoma.¹⁸

The SARS-CoV-2 pandemic has prompted a large amount of epidemiological modeling efforts, including studies with primarily mechanistic (e.g., references^19–25), primarily phenomenological (e.g., references^26–29), and hybrid (e.g., reference³⁰) approaches. According to the knowledge of the author up to June 2020, none of the existing models was systematically evaluated with regard to the accuracy of their predictions. In order to start closing this gap, the objective of the present study was to evaluate the predictive accuracy of a phenomenologically oriented model that was trained on data up to the end of March 2020 for forecasting the development of the cumulative number of reported SARS-CoV- 2 cases in countries and administrative regions worldwide.³¹

METHODS

RESULTS

Individual estimates of predictive accuracy

The probability density function of the percentage error (PE) at the day of estimation as well at the forecasts after two weeks, one month, and two months, respectively, is displayed in Figure 1. At the day of estimation, the median relative error indicated an average underestimation of the cumulative case count by about one third across regions. The relative error distribution was rather narrow, with only a tenth of predictions showing an underestimation exceeding −62.8 percent and none of the predictions having more than 36.9 percent error. Across forecasts, the median percentage error was always less than twenty percent, although an overestimation by more than two hundred percent was observed in 7.2, 19.1, and 19.5 percent of the cases at the two weeks, one months, and two months forecasts, respectively. The proportion of regions with an underestimation exceeding minus two thirds (−66.6 percent) was 12.4, 19.5, and 28.7 percent at the two weeks, one months, and two months forecasts, respectively.

• Download figure
• Open in new tab

Figure 1. Probability density function of the percentage error at different forecast horizons

The solid line shows the median, the dashed lines show the first and third quartiles, and the dotted lines show the first and ninth deciles. The x-axis is trimmed at 2.5.

The calibration plots suggest an increasing number of regions for which case counts are substantially under- or overestimated with increasing length of the forecast period (Figure 2). Nevertheless, a strong positive association between predicted and observed case counts is apparent even after two months.

• Download figure
• Open in new tab

Figure 2. Calibration plots at different forecast horizons

Points refer to regions. The solid black line indicates no prediction error, the blue area indicates a prediction error by a factor of two or less, and the green area indicates a prediction error by a factor of ten or less. Both axes are log-transformed.

Factors associated with accuracy

Visual analysis suggests that a larger number of available data points at estimation (Figure 3) and a more extensive growth of the logarithmic case counts from the first reported case until estimation (Figure 4) are associated with a lower prediction error.

• Download figure
• Open in new tab

Figure 3. Association of the amount of available data at estimation and predictive accuracy (AEIL) at different forecast horizons

AEIL= absolute difference between logarithmic predicted and observed case counts. Points refer to regions. The grey line corresponds to a linear smoothing curve.

• Download figure
• Open in new tab

Figure 4. Association of growth in logarithmic case counts until estimation and predictive accuracy (AEIL) at different forecast horizons

AEIL= absolute difference between logarithmic predicted and observed case counts. Points refer to regions. The grey line corresponds to a linear smoothing curve.

This is confirmed by regression analyses indicating statistically significant associations that are becoming stronger with increasing forecast horizon (Table 2). These two factors have also a multiplicative effect, as indicated by the statistically significant interaction term.

View this table:

• View inline
• View popup
• Download powerpoint

Table 2. Regression coefficients for factors associated with prediction accuracy (AEIL)

View this table:

• View inline
• View popup
• Download powerpoint

Table 3. Most extreme under- or overestimation for regions with a minimum number of 10,000 cases

Strongly affected regions (a minimum of 10,000 cases) with extreme under- or overestimation of the cumulative case counts are presented in Table 4. Among the listed regions, the extent of underestimation was considerable (an EIL below −1.6, roughly corresponding to an underestimation by a factor of five) at the one and two months forecasts, with most regions being located in Asia. Among strongly affected regions, overestimation was rather moderate (an EIL below 0.7, roughly corresponding to an overestimation by a factor of two) in most cases. Substantial overestimation (an EIL between 0.7 and 1.6) was present in Austria and Switzerland at the one and two months forecasts and in the United States at the one month forecast. No strongly affected region with a considerable overestimation (EIL above 1.6) was identified.

DISCUSSION

In the present study, a hierarchical logistic model was used to predict cumulative counts of confirmed SARS-CoV-2 cases in 251 countries and administrative regions with two weeks, one month, and two months forecasting horizons. Several metrics were used to evaluate predictions visually and statistically. In summary, case counts could be predicted in the majority of the regions with a surprising accuracy. In spite of the facts that at the time of estimation (29 March 2020) only about one month’s data were available on average in each region, and that most regions were at the very beginning of the epidemic, a massive difference between forecast and observation was rather the exception than the rule. Summary metrics of predictive accuracy suggested very strong prognostic validity the model for a horizon of two weeks, substantial accuracy after one month, and still notable, although markedly lower, accuracy after two months. This is in good agreement with studies finding that the horizon for reasonable epidemiological predictions covers a few weeks at most.^9,17

Although most predictions were fairly accurate, some were still considerably off. They were most likely to be found in regions with a lower amount of available data at the date of estimation and/or with a more limited growth between the date of the first case and the date of estimation. In general, underestimation seems to be somewhat more pronounced than overestimation, particularly in strongly affected regions (i.e., with cumulative case counts above 10,000 at the point of validation). The strongly affected regions for which the model provided too low predictions included several countries in which mitigation strategies might have been less effective than in other regions, as suggested by the only slowly or not at all decelerating cumulative case growth curves at the beginning of June 2020 (e.g., India, Bangladesh, Qatar). On the other hand, the strongly affected regions with a substantial overestimation of cumulative case counts are characterized with an extremely successful mitigation of the epidemic (mainly Austria and Switzerland). Hence, predictive errors are likely to be closely related to one of the central assumptions of the model, i.e., that timing, extent, and effectiveness of control measures is comparable across regions. Obviously, the forecasts based on the presented model are likely to reach their limits in regions that deviate too strongly from the average case. As shifting individual estimates towards the group mean is also a statistical property of hierarchical models,³⁹ extreme cases are likely to fall outside the scope of validity of the presented approach.

A notable feature of the model that it provides predictions without any reference to measures taken to control the epidemic. This “ignorance” towards interventions, paired with fairly accurate predictions, may be misinterpreted as evidence of dispensability of the mitigation and containment measures implemented in most countries. However, it is far more likely that the key model assumption suggesting similarity of the course of the epidemic and of the control measures taken across regions holds to a substantial extent. In cases when it does not, model performance is very poor, as discussed above. Bringing these issue together, the hierarchical structure of the model appears to have both benefits and risks: sufficiently accurate predictions for a large number of regions even at a very early stage of the epidemic come with the price of considerably erroneous predictions for atypical regions. Consequently, if used with the aim of generating locally applicable predictions for a particular region, forecasts may be improved by using data from comparable regions with a higher probability than from rather dissimilar regions.⁴⁰

The presented evaluation study has several limitations. First, the case counts were not standardized in any form. Expressing them as cumulative incidence rates (e.g., per 100,000 persons) is likely to have increased homogeneity across regions and enhanced interpretability. As it has been shown in a specific analysis of the development of the SARS-CoV-2 epidemic in German federal states, standardization has rendered using log-transformation of case counts for homogenization superfluous and allowed estimating models with normally distributed errors.⁴¹ Second, in the present study uncertainty of the predictions remained unconsidered, although measures of uncertainty, such as reliability and sharpness, can be just as important for forecasting as bias.⁹ Third, predictions only at selected time points were analyzed, and it cannot be excluded that choosing other time points would have led to different results. Nevertheless, the general pattern of findings is unlikely to have changed substantially. The forecasting model itself has some weaknesses as well.^31,41 Most importantly, it models the reported rather than the true number of cases and therefore can be subject to different forms of testing and reporting bias. Considerable improvement regarding this point can realistically be expected first when regional findings form well-conducted epidemiological studies become available. Another major limitation of the model is that it works only as long as the conditions of the epidemic remain largely unchanged in each region, i.e., within a single epidemic wave with fairly constant testing and reporting practices and without serious disruptions. This issue could perhaps be addressed by using dynamic (time-dependent) rather than fixed (time-invariant) model parameters.⁴² Finally, the primarily phenomenological nature of the model calls for integration with mechanistic components, in order to create a hybrid approach that is capable of producing widely generalizable conclusions.⁴³

As stated by one of the most prominent epidemiologist of the SARS-CoV-2 pandemic, Neil Ferguson, models are “not crystal balls”.³ However, without rigorous scientific evaluation, they run the risk of becoming one, characterized not by correct predictions but by obscurity. Some state that epidemiological forecasting is “more challenging than weather forecasting”,⁴⁴ and complexity of modelling and reliance on assumptions make it difficult to assess the trustworthiness of models based solely on their inherent structure. Just like we trust weather forecasts that prove to be accurate by experience, empirical comparison of modeling predictions with actual observations should become an essential step of epidemiological model evaluation.

Conflict of interest

The author reports no conflict of interest.

Data availability

The analyzed data are publicly available.