A Multi-City COVID-19 Categorical Forecasting Model Utilizing Wastewater-Based Epidemiology

2.1 Input Variables

This study utilizes SARS-CoV-2 wastewater surveillance data alongside other epidemiological metrics such as hospitalizations, previous infections, vaccination coverage, and local vulnerability indicators to forecast weekly COVID-19 hospitalizations for six U.S. cities. City boundaries were delineated using USPS ZIP codes identified through the USPS ZIP code lookup tool (United States Postal Service, 2024). Detailed descriptions of each input variable are provided below.

2.2 Target Design: Hospitalization Categorization

This study aims to provide city-level decision-makers with interpretable and actionable forecasts for COVID-19 hospitalizations by enhancing robustness against reporting issues. Rather than relying on traditional numerical predictions of hospitalization counts, we forecast risk categories that are derived from the rate of COVID-19 hospitalizations per 100,000 people. We propose two distinct categorization models, each emphasizing different aspects of the future impact of COVID-19 on a city healthcare system: the Hospitalization Capacity Risk (HCR) Categorization, which predicts the burden on the healthcare system and the Hospitalization Rate Trend (HRT) Categorization, which predicts the trajectory of COVID-19 hospitalization trends.

Hospitalization Capacity Risk (HCR) Categorization

Hospital demand can be used to measure times of overburdened healthcare systems, indicating when personal risk of infection is at its highest. We develop the HCR as a 5-tier categorization model based on hospital demand, defined as the static ratio between observed hospital admission rate and average number of available hospital beds for COVID-19 forecasted in a city. The HCR categorization is defined as follows: Where is the Hospitalization Capacity Risk category for city i at week t, is the forecasted COVID-19 hospitalization rate per 100,000 people for city i at week t, and B_i is the average number of available hospital beds per 100,000 people for city i across the study period. The thresholds of this categorization reflect the static capacities of each specific city’s healthcare system.

The thresholds are designed after the City of Austin stage alert system (Yang et al., 2021). They created a static 4-tier alert system based on the percent of filled ICU beds and the 7-day average of COVID-19 hospitalizations at a city level from March 2020 to September 2021. We calculate the percentage of utilized COVID-19 beds for Austin equivalent to the threshold number of hospitalizations determining each stage threshold. Additionally, we introduce a fifth category of highest risk to further specify the intensity of burden during times of peak hospitalizations. This categorization provides a more detailed and city-specific understanding of risk than the CDC 3-tier risk metric that identifies high, medium, and low hospital admission rates per 100,000 population based on a single threshold for all US locations (Centers for Disease Control and Prevention, 2024b). By increasing the number of high-risk categories, we expand on the CDC 3-tier community risk system and the Austin 4-tier system while still maintaining thresholds that reflect predetermined foundations for risk levels.

Figure 1 illustrates the five risk categories of the HCR for the six cities, where the black line indicates the observed hospitalization rate per 100,000 population, and the colored bars indicate the risk category. Across cities, there are common times of higher risk, such as January 2022 during the peak of the Omicron wave, but the city-specific dynamics vary by risk level at other times. Table 2 presents the distribution of categories for the HCR across the study period and all locations. The majority of weeks are assigned as Low and Very Low risk, indicating that times of very high community risk are less prevalent across multiple locations over the study period.

View this table:

• View inline
• View popup
• Download powerpoint

Table 2: Thresholds for Hospitalization Rate Trend Categorization and distribution of categories

View this table:

• View inline
• View popup
• Download powerpoint

Table 3: Inputs and Targets of the model

• Download figure
• Open in new tab

Figure 1. Hospitalization Capacity Risk Categorization Visualization.

HCR category thresholds by color in Charlotte, NC; Denver, CO; Houston, TX; New York, NY; San Diego, CA; and San Francisco, CA. The background color indicates the risk category. The dotted white lines indicate the thresholds, and the black line is the true hospitalization rate.

Hospitalization Rate Trend (HRT) Categorization

Observing trends in the rate of change of COVID-19 hospitalizations provides public health decision makers with a dynamic understanding of the change in burden on a city’s healthcare system. We develop the HRT categorization to capture city-level COVID-19 hospitalization trends to support broader city-level decision-making to describe how quickly the number of hospitalizations is changing between weeks, providing a complementary metric to the static categorization of the HCR. This categorization is also derived from the weekly average COVID-19 hospitalization rate per 100,000 people (). This metric is useful to forecast periods of large changes for public planning purposes and is inspired by the FluSight forecasting target (FluSight-forecast-hub, 2024). We define the rate trend as the change in COVID-19 hospitalizations for a city i in a given week t relative to the average change of the prior three weeks. The rate change is calculated as: where is the hospitalizations per 100,00 at week t for city i, and the summation is the mean hospitalization rate for the previous 3 weeks in the same city. Using this growth rate, we define the HRT categorization as:

The thresholds of this categorization reflect the changing dynamics of the COVID-19 hospitalization trends.

Figure 2 illustrates the rate trend categorical assignment for all six cities. The black line indicates the change in hospitalizations compared to previous weeks. The rate trend differs between cities based on their individual dynamics, not an aggregated national trend, providing more specific and actionable information. Table 2 presents the distribution of categories for the HRT across the study period for all locations. Over our study period, most cities for most weeks are in the Large Increase or Large Decrease category, indicating the volatile trends of COVID-19 that have been historically difficult to predict.

• Download figure
• Open in new tab

Figure 2. Hospitalization Rate Trend Categorization Visualization.

HRT category thresholds by color in Charlotte, NC; Denver, CO; Houston, TX; New York, NY; San Diego, CA; and San Francisco, CA. The dotted white lines indicate the thresholds, the black line represents the observed rate change, and the background color indicates the category.

2.3 Model

We develop 1, 2, and 3-week out forecasts of weekly COVID-19 hospitalizations for the 82-week period from January 2021 to September 2022 using wastewater viral load, past infections, the full vaccination coverage, previous hospitalizations, and 7 social vulnerability indices, summarized in Table 3. These forecasts are created via Generalized Additive Models (GAMs), an additive model that can learn non-linear predictor variables by modeling the outcome as a sum of spline functions of each predictor. We fit a distinct GAM for each forecasting window (1-week, 2-week, and 3-week), each utilizing the same model formulation:

Where l is the forecasting window, taking values of 1, 2, or 3. Each f_i coefficient indicates the spline smooth function of the dynamic variables, α indicates the intercept and µ^l is the mean of the hospitalization distribution l weeks ahead. Each s_i represents the parametric coefficient of the static variables. In order to mimic real-time forecasting practices, we used an expanding training window. This method involves initially training the model on a small set of data and then gradually expanding the training set as new data becomes available. In this study, the model is first trained on data before February 26, 2021. As each new week of data is collected, it is added to the training set, thereby expanding the window of historical data used for model training. This analysis was done using the pyGAM package in Python 3.9.13.

The continuous predictions are then converted into discrete categories to generate categorical predictions for each forecasting window. To create a probabilistic distribution over these categories, we performed prediction interval sampling to generate multiple continuous outcomes. We then compute the frequency with which these interval sample falls into each category. Specifically, for each week and city, we utilize 100 H^t values representing the upper and lower bounds of 1% to 99% prediction intervals. Each interval value is then converted to a categorical label based on the procedure described in Section 2.2. The final forecast for each week and city is the most frequent label among 100 probabilistic samples, selecting the categorical prediction with the greatest overall agreement.

2.4 Evaluation

This section details the various ways in which we measure forecasting performance. We define and justify the error metrics used to determine categorical forecast performance, as well as describe a novel procedure to determine model accuracy at the critical points where the disease burden is increasing or decreasing.

3.1 Model Performance

Using the error metrics described in Section 2.4.1, we demonstrate the performance of the HCR and HRT categorization models for the 1, 2, and 3-week forecasting windows in Table 4. The Accuracy and BSS are error metrics where higher values indicate better model performance, whereas lower MSE, WMSE, and BS values indicate better model performance. The model performance reveals several key findings:

Both categorization models significantly outperform random guesses. A random guess model would yield an accuracy of 20% and a BSS of 0, whereas the HCR accuracy spans from 75 to 89% and the HRT accuracy spans from 46 to 70% across all forecasting windows. Notably, all BSS values remain above 0, further underscoring the predictive power of the models.

For both the HRT and HCR categorization models, performance worsens as the forecasting window increases, demonstrated by the decrease in accuracy and BSS and the increase in MSE, BS, and WMSE. The decrease in model performance over increasing forecasting windows mirrors the decrease in correlations between wastewater and future hospitalizations over time. This diminishing relationship reflects rapidly changing COVID-19 dynamics in a community which makes the further future more difficult to predict.

Across all error metrics, model performance when predicting capacity (HCR) is consistently better than when predicting trends (HRT). The fluctuating shifts in COVID-19 hospitalization trends tend to be more volatile than changes in capacity, making the HRT more difficult to accurately predict. We detail these specific categorization model performances in the following sections.

3.1.1 Hospitalization Capacity Risk Categorization Results

In this section, we provide a detailed description of the performance of the HCR categorization model for the 2-week forecasting window. The 1 and 3-week forecast results are provided in Figures S1 and S2 of Supplementary B. In Figure 3, we demonstrate the overall categorical assignment accuracy, as well as the city specific categorical assignments over the study period.

• Download figure
• Open in new tab

Figure 3. Summary of Model performance of Hospitalization Capacity Risk (HCR) Categorization, January 2021-September 2022.

(a) Confusion Matrix detailing the accuracy of categorization for the HCR. Darker colors indicate that more assigned labels match the true labels for that category. (b) Hospital Capacity Risk true and predicted label for 2-week forecasts by city for January 2021 to September 2022. The color indicates the predicted category. Grey squares indicate that a prediction was not made for that week due to a lack of available data.

The HCR model performance can be described with a confusion matrix, which visualizes the accuracy of the HCR model across all cities and the entire study period for the 2-week forecast. In the matrix, each row represents the true categorical label of a week, and each column is the predicted label of that week. Each matrix entry contains the number of weeks where the true label is categorized as that predicted label, with incorrect categorical assignments are indicated by dissimilar row and column labels. In the confusion matrix in Figure 3a, the concentration of values along the diagonal indicates that the model accurately labels most forecasts. Off-diagonal values are primarily adjacent to the correct category, indicating that even when the model is incorrect, the magnitude of inaccuracy is relatively small.

Figure 3b compares the weekly HCR predictions to the true HCR categorizations for each city at a 2-week forecasting window, where the first row of the city illustrates the true weekly category and the second row illustrates the predicted category. The color indicates the categorical assignment, with grey indicating that predictions were not made due to a lack of available data over the study period from January 2021 to September 2022. Although the HCR is dependent on specific city dynamics, there are similar trends across all cities. There is a similarly high risk for all cities in January 2022, during the Omicron wave. San Francisco maintains a relatively low risk throughout the study period due to the low rate of hospitalizations per 100,000 people, but the subsequent increase at this time is still present. When the model predictions do not match the true category, it tends to predict the correct category with a minor time lag.

3.1.2 Hospitalization Rate Trend Categorization Results

In this section, we focus on the performance of the HRT categorization model at the 2-week forecasting window. The 1 and 3-week forecast results are provided in Figures S3 and S4 in Supplementary B. In Figure 4, we demonstrate the overall categorical assignment accuracy with a confusion matrix, as well as the city specific categorical assignments with a time series plot equivalent to those shown in Figure 3.

• Download figure
• Open in new tab

Figure 4. Summary of Model performance of Hospitalization Rate Trend (HRT) Categorization, January 2021-September 2022.

(a) Confusion matrix detailing the accuracy of the categorization of the HRT. Darker colors indicate that more assigned labels match the true labels for that category. (b) City-specific Rate Trend target and 2-week prediction over time from January 2021 to September 2022. Grey squares indicate that a prediction was not made for that week due to a lack of available data.

The performance of the HRT categorization model for the 2-week forecasting window is illustrated in the confusion matrix in Figure 4a. The concentration of values at the top left and bottom right corners indicate that most COVID-19 hospitalization rate changes are large changes, and that the model can accurately predict Large Increase and Large Decrease growth rates. The concentration of values in the center of the confusion matrix indicates that weeks categorized as Stable, of which there are many, are also accurately predicted. The HRT model performance is worst during weeks categorized as Decrease and Increase, illustrated in their respective rows in the confusion matrix. In these rows, we can see that errors are nearly evenly split between overestimations and underestimations.

Figure 4b compares the weekly HRT predictions to the true HRT categorizations for each city at a 2-week forecasting window from January 2021 to September 2022 equivalently to Figure 3b. The top rows for each city illustrate the true HRT trends, revealing that periods of large increases are quickly followed by periods of large decreases, indicating the turbulent changes in COVID-19 hospitalizations over the study period. In the bottom row of each city we present the HRT predictions. Although the exact category is not always perfectly predicted, true increases generally yield forecasted increases and true decreases typically yield forecasted decreases. This phenomenon is expanded upon in Section 3.2.

3.2 Change Points Performance

In addition to building and evaluating interpretable city-level forecasting models, we also aim to develop a novel method to evaluate categorical forecasting model performance during periods of increases or decreases in reported hospitalization rates, denoted as change points. As outlined in the section 2.4.2, we measured model performance by determining model accuracy at all change points, as well as at upward and downward shifts specifically. We present the resulting WMSE scores in Table 5, and provide the MSE, Brier Score, and Brier Skill Score error metrics in Table T1 and T2 in Supplementary B. These results indicate that the HCR model consistently outperforms the HRT model, with the HCR predictions usually within a one category margin. Additionally, the error metrics generally worsen for farther ahead forecasts. Finally, we observe that both models perform worse at change points than in their overall evaluations.

3.3 Variable Importance Analysis

This work aims to demonstrate the value of wastewater based epidemiology for improving disease surveillance capabilities. The following analysis illustrates the importance of each variable in improving forecast performance. We measure variable importance as the difference in mean average percent error (MAPE) of the continuous hospitalization forecasts when each variable is removed. This variable importance is measured for the 1, 2, and 3-week forecasts. This approach allows us to evaluate variable contribution independently of any categorical threshold selection biases. In Figure 5, we demonstrate the changes in MAPE after the removal of each variable, namely wastewater viral load, full vaccination coverage, past infections, and CCVI. For the 1 and 2-week forecasting window, the removal of wastewater leads to the highest increase in MAPE, indicating its critical role in improving forecasting performance. For the 3-week forecast, the removal of the CCVI leads to the largest increase in MAPE, indicating that in addition to wastewater, the community vulnerability indices serve an important role in understanding community risk and creating useful predictions. Removing the full vaccination rate from the model does not generate significant differences in model performance. Additionally, the importance of past infections diminishes as the forecasting window extends.

• Download figure
• Open in new tab

Figure 5. Variable contribution in model performance for 1, 2, and 3 week out continuous hospitalization forecasts.

The black dotted line indicates the full model performance, and the colored bars indicate the MAPE with that variable removed. A larger increase in MAPE indicates a worse performance and a higher importance in model performance.

Integrating Wastewater Surveillance Data Enhances COVID-19 Hospitalization Forecasting

As traditional epidemiological datasets, such as reported cases, become less available and more subject to reporting issues over time, alternative data streams must be considered to inform short-term forecasting models. In this study, we built a comprehensive forecasting model to predict near term COVID-19 hospitalization risks in six US cities. This model incorporates novel data streams, including wastewater surveillance data and community vulnerability indices, alongside traditional epidemiological inputs such as previous infections and vaccinations. Our model performance confirms the effectiveness of this approach, particularly emphasizing the importance of wastewater surveillance data, as explored in Section 3.3.

Our variable importance analysis identified wastewater surveillance data as a critical factor in our models, particularly for forecasting city-level COVID-19 hospitalizations 1 and 2 weeks into the future (see Figure 5). Therefore, publicly available, reliable wastewater data is crucial for improving our capabilities to predict future hospitalization risks and inform both the public and public health practitioners. Wastewater disease surveillance systems, such as the CDC National Wastewater Surveillance System (NWSS), represent a centralized database that has the potential to inform disease risk assessment and bolster modeling capabilities. Thus, we encourage continued investment in and expansion of this critical surveillance tool, along with the centralized sharing of data in standardized formats.

Beyond the immediate utility of wastewater surveillance, our findings underscore the importance of local vulnerability indices on model performance. We determined that the CCVI is the most influential variable for our 3-week continuous forecasting model. This finding aligns with previous studies demonstrating the critical role of vulnerability data in predicting weekly COVID-19 hospital admissions at less-immediate forecasting windows, such as 3 and 4 weeks in the future (Li et al., 2023). The importance of CCVI in our analysis suggests that more severe health outcomes from COVID-19 infection may be amplified in vulnerable communities. As the sole static variable in the model, the community vulnerability index avoids any information decay that may arise with the temporally dependent variables, such as wastewater surveillance data. This enduring relevance underscores the need to incorporate data on both current disease dynamics, such as wastewater surveillance data, and persistent vulnerabilities, such as the CCVI, when forecasting near future infectious disease risk.

Categorical Forecasts Support Public Health Decision Making

The well-documented challenges of forecasting traditional continuous disease outcomes during the COVID-19 pandemic (Nixon et al., 2022) motivated this study’s focus on categorical targets for hospitalization forecasting. This approach, exemplified by the Hospitalization Capacity Risk (HCR) and Hospitalization Rate Trend (HRT) categorizations defined in this study, enhances model interpretability, actionability, and robustness.

The HCR model, derived from hospitalization rates, provides hospital administrators with an intuitive understanding of capacity risk, enabling proactive resource allocation during times of high risk. Our models achieve a 75% accuracy in predicting HCR three weeks in advance, proving consistently reliable prediction across cities and forecasting windows. Exceptionally, HCR forecasts for San Francisco resulted in a uniquely high accuracy due to the city’s uniquely low hospitalization rates. While the HRT model, based on the trends of hospitalization growth rates, is a less predictable outcome, the model successfully captures 64% of weeks classified as Large Increases and 67% of weeks classified as Large Decreases for the 2-week forecasting window (Figure 4a). The focus of the HRT on directional trends, when considered in conjunction with HCR, offers decision-makers a more comprehensive understanding of potential outbreak scenarios, facilitating timely and targeted interventions.

The inherent stability of the two categorical target variables defined in this study, which are less sensitive to minor data fluctuations than continuous outputs, further reinforces the reliability of our approach and provides a robust foundation for data-driven decision-making in the face of evolving public health challenges.

Evaluating Forecasting Performance at Change Points Illuminates Model Capabilities

A notable issue with COVID-19 forecasting during the pandemic was the inability to accurately predict inflection points in case and hospitalization trends (Lopez et al., 2024; Cramer et al., 2021). To assess the effectiveness of our categorical models in addressing this challenge, we evaluated performance at change points, examining the models’ ability to accurately account for shifts in the COVID-19 hospitalization category. As outlined in section 3.2, we evaluated model performance by determining accuracy at all change points, as well as at upward and downward shifts specifically.

The results outlined in section 3.2 presented several important insights that merit further discussion. First, we note that our HCR model generally outperforms the HRT model for all change point evaluation methods. This aligns with the overall model performance in section 3.1, where the HRT model’s weaker performance could be attributed to its prediction of highly variable hospitalization growth rates. Second, we highlight that error metrics at change points generally worsen as forecasting window increases. This is also consistent with the results in section 3.1, which suggests that this decline is likely due to the decreased correlations between wastewater and future hospitalizations over time. Third, we recognize that the performance at change points is weaker for both models compared to overall model accuracy, as per Table 5 the difference in WMSE is less than 0.3 across all models and forecast horizons. While there is still room for improvement, this indicates that the model is reasonably effective at predicting the correct category at change points. Nevertheless, this discrepancy in model performance at change points further highlights the need for a greater focus on improving forecasting accuracy during periods of rapid changes.

Our change point analysis methodology evaluates the utility of categorical forecasting models at critical points. Sudden shifts in population level disease dynamics are some of the most crucial trends to understand in order to better inform decision makers at times of great uncertainty. Our change point analysis enhances model utility by evaluating how well each categorization captures hospitalization dynamics during periods of rapid change. We believe that this straightforward yet comprehensive approach to assessing categorical model accuracy at change points is an effective and transferable method for evaluating model performance and developing timely policies to mitigate the effects of a virus.

City-Level COVID-19 Hospitalization Forecasting is Effective and Useful

We propose that achieving a balance between forecast accuracy and actionable spatial scales, such as cities, is crucial for providing key insights that can be translated into effective mitigation strategies for COVID-19 and other infectious diseases. COVID-19 forecasting using wastewater surveillance has been implemented at several different granularities, such as country, state, county, city, and sewershed level. (Schenk et al., 2023; Hill et al., 2023; Li et al., 2023; Galani et al., 2022). We contend that forecasting at a city-level, which has received the least attention by modelers and is well-aligned with the spatial boundaries of wastewater data, is particularly advantageous for decision-makers. Densely populated urban areas are often the epicenter of COVID-19 infections and have government structures that allow for more targeted public health interventions. Thus, accurate city-level forecasts can more effectively help inform local government policies to prevent harm to the population and minimize economic burden at these disease epicenters (Lopez et al., 2024). Despite this potential, city-level forecasting is an under-utilized tool, with most forecasts targeting larger, aggregated populations such as county-level, state-level, and national-level (Sen et al., 2023). This is often due to the lack of reliable city-level disease surveillance, a challenge addressed by the natural alignment between wastewater treatment plants and city boundaries as demonstrated in Figure S1 in Supplementary A. Additionally, many cities have multiple wastewater treatment facilities which can be aggregated to provide a smoother prediction (Medina et al., 2022). Multiple wastewater treatment facilities also enable higher modeling granularities within cities. Overall, WBE provides an opportunity to advance higher resolution infectious disease forecasting and permits local governments to implement mitigation strategies that best suits their communities.

We note that the quality of wastewater data significantly impacts model performance across cities. Houston and New York City consistently demonstrate superior performance due to their high-quality wastewater data, characterized by consistent reporting and a large number of sewersheds. Conversely, as shown in Figures 3b and 4b, Charlotte and Denver exhibit poorer model performance for both HCR and HRT due to lesser data quality, stemming from recorded lapses in reporting or changes in reporting styles (Keshaviah et al., 2023; Ghanbari et al., 2024). This finding underscores the importance of consistent and transparent WBE data for both understanding local trends and enhancing forecasting capabilities.

Limitations and Future Work

Wastewater disease surveillance for COVID-19 is a rapidly evolving field. Different cities and wastewater treatment plants have varying timelines for reporting and normalization techniques that make retrospective data difficult to compare to currently reported information. Despite the expansion of the NWSS to expand wastewater disease surveillance systems, a lack of data accessibility and inconsistencies in data collection, standardization and preprocessing can still impact data quality across locations. Consequently, the observed variation in model performance across cities can be attributed to these inherent data quality differences.

The models in this study are better equipped to capture downward shifts, which may be reflective of limited input data streams. Therefore, we propose including alternative disease prevalence information, such as circulating variants, which may improve model performance, as demonstrated in (Du et al., 2023). Wastewater-based epidemiology is a vehicle to understand community disease prevalence, and variant information can be easily recovered from it to strengthen model performance, something considered for future extensions of this work.

In this study, we focus on aggregating epidemiological data to a city level to create a more granular forecasting scale than the traditional state or county-level forecasts. By utilizing wastewater disease surveillance, we are able to provide an even more specific geographical scale at which to understand disease dynamics. In future work, we aim to expand our modeling capabilities to understand wastewater signals at the sewershed level as an early warning signal for smaller spatial scales.