A roadmap to account for reporting delays for public health situational awareness – a case study with COVID-19 and dengue in United States jurisdictions

Background Decision making in public health is limited by data availability where the most recent reports do not reflect the actual trajectory of an epidemic. Nowcasting is a modeling tool that can estimate eventual case counts by accounting for reporting delays. While these tools have generated reliable predictions when designed for specific use cases, several limitations exist when scaling the models to systems composed of multiple distinct surveillance systems. We seek to identify flexible application of nowcasting models to address these problems.

Methods We used a previously developed Bayesian nowcasting tool, which dynamically estimates delay probabilities up to a user-defined maximum delay using a user-defined training window. We tested automated approaches to select the maximum delay and training window, setting maximum delay values at the 90^th, 95^th, and 99^th quantile distribution of the most recently reported data and training windows to the maximum delay plus one week or multiplied by 1.5 or 2.0. We generated and evaluated nowcasts for 321 datasets reflecting COVID-19 cases and dengue cases in different United States jurisdictions. We assessed prediction error and precision via logarithmic scoring and coverage metrics for the most recent three weeks of predictions in each nowcast. We used these metrics to further assess why nowcasts may fail and to compare predictions generated from three different publicly available tools.

Results Using recent data to estimate dynamic delay and training window parameters resulted in nowcast with less error relative to nowcasts made with static parameters for long historic periods. Nowcasts likely to fail could be predicted a priori by the relative width of the prediction intervals and the permutation entropy of the epidemic trend. More complex models do not significantly improve nowcast performance compared to simple models.

Conclusions We tested multiple systems for scaling up nowcasts in a flexible framework. We recommend using dynamic parameter selection and creating a system to suppress nowcasts likely to fail. This requires collaboration with surveillance colleagues to implement data-driven choices to improve the utility of predictions for decision making.