Using machine learning to forecast peak health care service demand in real-time during the 2022-23 winter season: a pilot in England, UK

Data selection

The UKHSA coordinates a programme of real-time syndromic surveillance that supports and augments other UKHSA health surveillance programmes.⁶ For this pilot study, we used two syndromic indicators that are routinely part of the ongoing UKHSA daily syndromic surveillance service; NHS 111 telehealth calls for ‘cough’ and emergency department (ED) attendances for ‘acute bronchiolitis’. Both syndromic indicators were restricted to children aged five years or under because they are known to be sensitive to seasonal outbreaks of RSV.^7-9 Using established indicators that are well understood aids interpretation and enables comparison with previous years.

Data cleaning and formatting

Firstly, data were smoothed to remove day of the week effects caused by weekends and public holidays.¹⁰ Secondly data were normalised, so that variables were in the range zero to one. Finally, derived variables were created that were used to create the forecast models (although noting that not all the derived variables were used in all the models tested; Table 1).¹¹

Training models

Rather than restrict forecasts to a single methodology, for both indicators we tested a wide range of alternate models, using the data to select the best method for each indicator ¹¹.

Firstly, we choose seven alternate supervised machine learning methods; linear regression, generalised linear models with elastic net regularization (with and without internal optimisation of parameter lambda), k-Nearest-Neighbour regression, random forest regression, support vector machine for regression, and eXtreme Gradient Boosting regression. For each of these seven different methods we included the following options:

1. No seasonality or including month of year or with Fourier transformation

2. No secular trend, a linear trend or a quadratic trend

3. With or without a quadratic term for current activity

4. With or without using a three-day moving average for activity, slope and change of slope to avoid undue influence of single-day spikes in activity.

The combination of seven methods and the options above gave 252 alternate model specifications to be tested. Furthermore, for each model specification 28 models were created and trained to forecast from 1 to 28 days ahead respectively. Models were trained using historical data prior to October 2022, the supervised learning using actual data from either 1 to 28-days ahead of daily forecasts as the ‘labels’ for the target forecast. Historical data were split randomly into training and test data sets, 80% of the historical data being used for training.

We used anonymised health service data that is routinely used by UKHSA for public health surveillance of respiratory illnesses, including RSV. This study was part of ongoing work to improve the capabilities of UKHSA surveillance systems. As such, no specific approvals were required to use the anonymised data included this study.

Selection of ensemble model

For each of the 252 alternate model specifications an ensemble forecast was created which forecast when and how high activity was going to peak over the next 28 days. The ‘forecast peak’ for each model specification was defined as the highest value forecast by the individual 1 to 28-day ahead forecast models. Using the test data set, the forecast peaks were compared with the actual highest value or ‘peak’ that occurred in the 28 days following the forecast. An ‘intensity error’ was calculated as the difference between the height of the forecast peak and the actual peak. Similarly, a ‘timing error’ was calculated which was the difference in days between the day when activity peaked and the date when activity was forecast to peak. The intensity and timing errors were combined to give a single ‘forecast peak error’. The forecast peak error includes weighting to emphasise the importance of accurately forecasting peaks when activity is high.¹¹ The model specification which resulted in the smallest mean forecast peak error was selected for daily forecasts.

Model validation

Once the best model specification has been selected based on the training data, forecasts were retrained using all the available historical data. The historical intensity and timing errors were calculated for forecast peaks and used to estimate forecast uncertainty. To allow for variation in the standard deviation of errors as activity approaches a peak, a gamlss model was used to estimate standard deviation variation against the level of current activity. Thus, we can create uncertainty intervals which vary as activity approaches a seasonal peak.

Creating daily reports

To produce daily forecasts, recent data is extracted and formatted using the same data processes as for training models. The validated model for each indicator is used to produce 1 to 28-day ahead forecasts based on the latest data available. These forecasts are used to create daily reports which predict when activity will peak in the next 28 days and at what level, including the uncertainty intervals.

Model selection

The model specifications with the lowest forecast peak errors for both indicators used a random forest learning method, with seasonality modelled by Fourier transformations. However, the other specification options differed between the two indicators. The lowest forecast peak errors for the NHS 111 cough calls data included a quadratic term for activity, a quadratic secular trend and averaging over three consecutive days’ data points. Whilst for ED acute bronchiolitis attendances the lowest errors involved a linear trend and no quadratic term for activity or averaging over consecutive days. Supplementary table 1 shows the forecast peak errors for each model specification. In general, errors were lower for NHS 111 calls than for ED bronchiolitis attendances, with 52 NHS 111 model specifications performing better than the best ED specification. Overall, including seasonality improved peak forecasts, with Fourier transformations performing better than seasonality using months. The learning method with the lowest mean errors was random forest, followed by k-nearest neighbour regression (Table 2).

Model validation

The gamlss models show that the variation in intensity errors increase as actual counts increase (Table 3). By contrast, the variation in timing errors decrease as actual counts increase.

The gamlss model coefficients were used to create confidence intervals around the timing and intensity errors, which varied depending on the number of actual counts at the time of the forecast (Figures 2 and 3). Intensity errors are not symmetric because forecast peaks cannot be negative, thus when actual counts are low a forecast peak can over-estimate by more than it can under-estimate.

• Download figure
• Open in new tab

Figure 2:

Intensity errors during training period for A) emergency department acute bronchiolitis attendances and B) NHS 111 cough calls. Lines show 50% and 95% confidence intervals.

• Download figure
• Open in new tab

Figure 3:

Timing errors during training period for A) emergency department acute bronchiolitis attendances and B) NHS 111 cough calls. Lines show 50% and 95% confidence intervals.

Pilot season 2022-23

During October 2022, ED acute bronchiolitis attendances in children aged under 5 years reporting to EDSSS increased until a peak of 220.0 attendances on 31 October. Subsequently, there was a decrease until 5 November before attendances increased again reaching a seasonal high of 311.4 attendances on 29 November. Similarly, NHS 111 calls for cough in children under 5 years rose to a peak of 991.6 calls on 22 October, decreased until 2 November and then started rising. However, whilst the increase in NHS 111 calls slowed prior to 30 November it was then followed a sharp increase in calls, reaching a seasonal high of 1,842.9 calls on 6 December 2022.

The seasonal peak in ED attendances at the end of November coincided with the usual timing of peak RSV activity seen in previous years (as monitored by laboratory reporting).¹² The additional increase in NHS 111 calls after 30 November 2022 was unprecedented, being 39.3% higher than the previous highest winter peak, 1,323.4 on 7 December 2019. Consequently, the level of activity was outside the range of anything seen in the training data.

The pilot forecast made on 26 November 2022, forecast that ED attendances for acute bronchiolitis would peak at 340.1 attendances on 29 November (Figure 4). The same day forecast for NHS 111 calls predicted that they had already peaked. The timing for the ED forecast was correct but the level of the peak was an over-estimate of 29.3 (9.4%) attendances.

• Download figure
• Open in new tab

Figure 4:

28 day forecast for peak emergency department acute bronchiolitis attendances in children aged <5 years. Red squares are 28 day forecast, blue lines show 50% (dark blue) and 95% (light blue) data intervals around the peak forecast.

The NHS 111 forecasts failed to predict the unprecedented rise in NHS 111 cough calls in children in December 2022 until the rise had started. However, a forecast using a linear regression learning method proved to be adaptive, forecasting a later and higher seasonal peak once activity began to rise sharply at the start of December 2022 (Figure 5).

• Download figure
• Open in new tab

Figure 5:

28 day forecast for peak NHS 111 cough calls in children aged <5 years. Red squares are 28 day forecast, blue lines show 50% (dark blue) and 95% (light blue) data intervals around the peak forecast.

Key findings

Machine learning pipelines can be used to train, select models and create daily forecast reports that predict the peak in demand for RSV activity over the following 28 days. During 2022 our pilot forecasts were able correctly predict the peak in ED acute bronchiolitis attendances in children under 5 years. During November 2022, our forecasts for NHS 111 cough calls in children aged under 5 years, predicted a similar peak was going to occur as in previous pre-pandemic years. However, as cough calls began to increase sharply at the start of December, our forecasts also began to change, predicting a later peak in December that was higher than previous years.

What was known before

Prior to the COVID-19 pandemic, seasonal activity for RSV was predictable in its seasonality, peaking in England at the end of November to beginning of December.^13-15 However, during 2020 and 2021, the seasonality of RSV was disrupted with no winter peak in 2020 and a deferred peak occurring in summer 2021.⁷ Traditional surveillance methods based on historical data and recurring seasonality¹⁶ continued to predict syndromic indicators would rise in winter 2020 due to RSV. Whilst short-term forecasts based on recent trends are more adaptive, these too would not perform well during atypical seasons, unless seasonality was excluded from model variables.¹¹

Interpretation of findings

The unexpected dramatic rise in NHS 111 cough calls in December 2022 coincided with media reporting about unusual increases in invasive Group A streptococcal (GAS) infections in children.^17,18 Extensive news coverage reported children with GAS becoming severely ill very quickly with symptoms including sore throat, fever and cough. Over the days following the release of the GAS news coverage, the NHS 111 telehealth service saw an unprecedented increase in calls relating to children. However, ED attendances were less affected by changes in patient presenting behaviour, and consequently ED syndromic indicators did not have an additional large peak. The differences between syndromic systems illustrate some of the strengths and weaknesses of syndromic surveillance. Syndromic indicators cannot identify specific causal pathogens, thus NHS 111 cough calls although sensitive to RSV are not specific enough to exclude other causal factors. Thus, NHS 111 cough calls were not a reliable indicator for assessing the total burden of health care demand attributable to RSV. However, if policy and decision-makers need to understand the current pressures on health services from all causes then syndromic indicators are more sensitive than pathogen-specific surveillance such as laboratory reporting. Importantly, a syndromic surveillance service that comprises a range of data sources across the spectrum of health services is better able to distinguish between pressures due to changes in underlying disease incidence and those due to changing patient behaviour.

Limitations

Inevitably, forecasts trained on historical data will perform better when current data is within the range and seasonality seen previously. Previously, we have shown that a model that is not trained to expect a recurring seasonal pattern performs better during atypical seasons ¹¹.

However, the unexpected peak in December 2022 NHS 111 calls was not out of season and so the accuracy of forecasts was not due to inclusion of seasonality variables. In this case, we found that some of the regression methods which performed best using our forecast peak error measure, e.g. random forest, generated forecasts that assumed activity had already peaked. By contrast, using simple linear regression generated forecasts that correctly predicted activity was going to continue to increase in line with current trends. Therefore, the simpler method outperformed the method automatically selected by our machine learning algorithm.

The unprecedented increase in NHS 111 cough calls in children during December 2022 revealed a limitation in the use of this indicator for forecasting peak pressures due to RSV. The exceptional additional winter pressures were not due to RSV but due to reaction to iGAS media reports. Thus, without additional intelligence, a report intended to show pressures due to RSV could have misinterpreted as showing that RSV activity was exceptionally high.

Public health implications

Short-term forecasts can provide additional information compared to existing surveillance baselines based on previous years because they are more adaptive to recent changes in trends. Our automated pipeline for creating short-term forecasts of seasonal peaks is useful in identifying the timing and intensity during typical seasons. Furthermore, the process of using machine learning methods to produce a reproducible automated pipeline means new indicators can easily and quickly be added to syndromic reports. However, automated reporting of trends and forecasts for syndromic data should always be accompanied with expert interpretations which can warn of emerging events. For instance, where a real-time change in patient behaviour means one or more indicators is no longer comparable with previous years. Improved automation and real-time interpretation is important as we may need to create forecasts quickly when notified that there is an increase in disease incidence. Also, the same pipeline can be used to assess other causal pathogens, including influenza and SARS-Cov2. Similarly, it may be possible to model non-infectious diseases such as allergic rhinitis (hay fever) where the historical data includes recurring seasonal peaks.

Recommendations and future work

Importantly, when current data is outside the range of training data, or seasonality does not match the training data forecasts should be interpreted in caution. We’d recommend that any forecasts used for routine surveillance include tests for data that is outside the range of testing data. Also, we suggest that when increases start to occur out-of-season forecast models are selected that do not include seasonality variables.

In future, it may be possible to provide better forecasts during atypical seasons by weighting the training data to give more emphasis to the rare events. Also, synthetic data could be incorporated in training data to allow for plausible events that have not yet occurred in the training data, e.g. out-of-season outbreaks, or more virulent pathogens.