Quantifying the impact of hospital catchment area definitions on hospital admissions forecasts: COVID-19 in England, September 2020 - April 2021

Trust-level COVID-19 hospital admissions

A confirmed COVID-19 hospital patient is any patient admitted who has recently (in the last 14 days) tested positive for COVID-19 following a polymerase chain reaction (PCR) test, including both new admissions with a known test result and inpatient tests. Data on daily Trust-level COVID-19 hospital activity, including COVID-19 hospital admissions, COVID-19 and non-COVID-19 bed occupancy, are published weekly by NHS England and were accessed via the covid19.nhs.data R package [15].

LTLA-level COVID-19 cases

A confirmed COVID-19 case in England is defined as an individual with at least one confirmed positive test from a PCR, rapid lateral flow tests or loop-mediated isothermal amplification (LAMP) test. Positive rapid lateral flow test results can be confirmed with PCR tests taken within 72 hours; if the PCR test results are negative, these are not reported as cases. Aggregated data by lower-tier local authority (LTLA) are published daily on the UK Government dashboard [16] and reported totals include both pillar 1 (tests in healthcare settings and for health and care workers) and pillar 2 (community) tests. We use data published by specimen date, the date on which the test is taken.

Hospital catchment area definitions

Since COVID-19 cases and hospital admissions are reported on different scales (local authority and NHS Trust, respectively), we need to define each Trust’s catchment area to estimate COVID-19 cases “local” to each Trust.

We considered six definitions of hospital catchment areas for Acute NHS Trusts in England. The catchment area definitions are based on heuristics about where people may seek hospital care, or derived from data on historical hospital admissions. We defined a Trust’s catchment area with respect to local authority (UTLA or LTLA) boundaries, and for each local authority-Trust pair assigned a weight, p, that represented the expected proportion of people/admissions from that local authority that would seek hospital care at the given Trust. Catchment area definitions are defined to ensure that each local authority is matched to at least one Trust, and vice versa; consequently, the catchment area boundaries are overlapping, since each local authority can be mapped to more than one Trust.

The six catchment area definitions are as follows:

Descriptive statistics

We summarised and compared the following Trust-level properties of the hospital catchment area definitions. First, the distribution of catchment area weights. Second, the number of local authorities under each catchment area definition for which the assigned weight is more than x%, that is, for which at least 10% of the local authority population are assigned to that Trust; we consider x = 0%, 1%, and 10%. Third, the median distance from each Trust to any local authority for which the assigned weight is more than x%; again, we consider x = 0%, 1%, and 10%. Finally, we calculate the proportion of a catchment area’s total weight that is from either the nearest, or nearby (< 40 km), local authority or authorities.

Quantifying the similarity of catchment area definitions

To summarise and quantify the similarity between pairs of catchment area definitions for a given Trust, we defined an overlap-similarity metric. The asymmetric overlap-similarity between catchment area definitions X and Y relative to definition X is defined as the proportion of the definition X that is contained in the overlap with Y. The overlap-similarity between X and Y is then simply the minimum of the asymmetric overlap-similarity relative to each of X and Y. See the Supplementary Information Section 3.2.2, for details of these definitions.

By construction, the overlap-similarity is equal to zero if, and only if, the two catchment area definitions do not include any of the same local authorities. The asymmetric overlap-similarity between X and Y, relative to X, is equal to 1 if, and only if, the weights assigned by X are less than or equal to the weights assigned by Y for all local authorities. The overlap-similarity metric is equal to 1 if, and only if, the two definitions assign exactly equal weights to all local authorities. A higher overlap-similarity score indicates that X and Y are more similar, that is, the weights assigned to each local authority are similar for definitions X and Y.

Quantifying the similarity of COVID-19 case time series

We expect that any difference between forecasts made using different catchment area definitions will require some heterogeneity in the trends of reported COVID-19 cases in the local authorities in the catchment area - if nationwide trends are aligned, the definition of catchment areas should not matter. To this end, we used the Pearson correlation coefficient to quantify the similarity between COVID-19 case time series between local authorities. First, for each forecast date and pair of local authorities, we calculated the Pearson correlation between the time series of COVID-19 cases in each local authority from 21 days before until 14 days after the forecast date. Then for each forecast date we calculated two values. First for each local authority the median correlation coefficient with all other local authorities. Second for each Trust the median correlation coefficient between all local authorities for which any catchment area definition assigns a weight of more than 10%. The threshold value of 10% is arbitrarily chosen, but such that only the main local authorities in each catchment area were included.

Calibration

Calibration measures models’ ability to correctly quantify predictive uncertainty. We assessed calibration by calculating the empirical coverage: for a given forecast horizon and prediction interval width, the empirical coverage of a model is the proportion of forecast targets (here, across all dates and locations) for which the prediction interval contains the true value. We calculated the empirical coverage for the 50% and 90% prediction intervals. A perfectly-calibrated model has empirical coverage equal to the width of the nominal prediction interval.

Probabilistic forecast error

We measured probabilistic forecast accuracy with the weighted interval score (WIS), a proper scoring rule, a rule for which a forecaster is incentivised to give their honest forecast to obtain the best score [24]. The WIS comprises a weighted sum of interval scores for quantile forecasts of increasing widths, measuring both the sharpness (the width of the forecast interval) and accuracy (whether the interval contains the true value) of each quantile forecast. By combining these interval scores, the probabilistic forecast accuracy of the full forecast distribution is summarised in a single value. The relative weighted interval score (rWIS) is an adjusted WIS value that allows us to compare the performance of different definitions over forecast dates or between Trusts, both of which vary in the number of admissions.

Catchment area definition descriptive statistics and overlap-similarity

A visual inspection reveals some clear differences between the hospital catchment area definitions (Figure 1); these differences were more evident from the descriptive summary statistics (supplementary Figure S3 - 5). The marginal distribution definition is clearly very different to all other definitions, since it includes all local authorities in the catchment area for every Trust; we do not discuss it further here. The other major differences were between the “nearest” and “nearby” heuristic definitions, and between the heuristic definitions and the three data-derived definitions.

• Download figure
• Open in new tab

Figure 1: Comparison of six LTLA-level hospital catchment area definitions for University Hospitals Bristol And Weston NHS Foundation Trust.

The hospital catchment areas are defined as follows: by the marginal distribution of hospital admissions to Trusts June 2020 - May 2021 (“marginal”); the nearest Trust for each LTLA (“nearest”); any Trust within 40 km of the LTLA (“nearby”); by the distribution of emergency, or elective, hospital admissions in 2019 (“emergency” and “elective”, respectively); and by the distribution of COVID-19 hospital admissions June 2020 - May 2021 (“covid”). In each panel, the colour denotes the proportion of all patients from that LTLA that are admitted to University Hospitals Bristol And Weston: darker colours indicate a higher proportion, and white indicates zero admissions. The Trust’s main site is marked by a red cross.

First, we compared the nearest and nearby heuristics. The characteristics of the nearby heuristic were very different for Trusts inside versus outside the London NHS region. In London, the distribution of weights was very homogeneous (median 0.037, IQR [0.037, 0.04]), each Trust had on average 38 UTLAs in its catchment area (IQR [38, 38]), and the median distance between Trusts and UTLAs was 14.4 km (IQR [13.4, 17.9]). This is because local authorities in London (boroughs) are small (all less than 10 km in diameter) and so the 40 km radius includes most other boroughs as well as other nearby local authorities. Outside of London, the distribution of weights was more heterogeneous, as was the median number of UTLAs per catchment area (median 7, IQR [3, 11]) and the median distance was higher (median 23.1 km, IQR [18.8, 29.5]). In comparison to the “nearby” heuristic, the nearest heuristic was very homogeneous and, by construction, constrained to a much smaller geographic area. The majority of Trusts (98/138) had a single UTLA in its catchment area (median 1, IQR [1, 2]). The median distance between Trusts and their nearest UTLA is very small (median 5.4 km and IQR [2.8, 14.7]; supplementary Figure S5A).

In contrast to the simple heuristics, the distributions of weights for the three definitions derived from admissions data were more heterogeneous (Figure 1 and supplementary Figure S3). More specifically, we found that the weight assigned to local authorities for a given Trust decreased as the distance between local authority and Trust increased. Compared to the heuristic definitions, the emergency, elective and COVID-19 admissions definitions shared many similarities. The number of local authorities in a Trust’s catchment area was comparable across the three definitions (median of 4, 5 and 3 UTLAs for the emergency, elective, and COVID-19 admissions data definitions, respectively, for a threshold of x = 1%; supplementary Figure S4, second column), as was the average distance (median 17.0, 18.2 and 14.3 km for the emergency, elective, and COVID-19 admissions data definitions, respectively, using a weight threshold of x = 1%; supplementary Figure S5A, second column). A large proportion of Trusts’ emergency, elective and COVID-19 admissions catchment areas are from the nearest local authority (median 76.5%, 72.4% and 83.1%, respectively; supplementary Figure S5B). Furthermore, virtually all of Trusts emergency, elective and COVID-19 catchment areas were from nearby local authorities (within 40 km) (median 99.3%, 97.8% and 100%, respectively; supplementary Figure S5C). By contrast, only 15.3% of the “nearby” heuristic definition was from the nearest local authority, since all local authorities within a Trust’s 40 km radius were assigned the same weight, irrespective of distance.

According to the overlap-similarity metric, the emergency and elective catchment area definitions were most similar (median overlap-similarity 0.84; supplementary Figure S6A), and both were also similar to the COVID-19 definition (0.74 and 0.77 median overlap-similarity with the emergency and elective definitions, respectively; supplementary Figure S6A). Moreover, the median asymmetric overlap-similarity relative to the COVID-19 definition was 1 and 0.99 for the emergency and elective admissions definitions, respectively (supplementary Figure S6B), that is, for more than half the Trusts all local authority weights assigned by the COVID-19 definition were less than or equal to the weights assigned by either the emergency or elective definitions.

As expected, the marginal distribution definition was very dissimilar to all other definitions, with a median overlap-similarity ≤ 0.05 with all other definitions (supplementary Figure S6A), and the majority of asymmetric overlap-similarity values < 20% (supplementary Figure S6C). The nearby heuristic was also dissimilar to the other definitions on average (median overlap-similarity < 0.3; supplementary Figure S6A), although individual asymmetric overlap-similarity values varied considerably from one Trust to another for all definitions except the marginal distribution (supplementary Figure S6C).

Similarity of COVID-19 case time series within catchment areas

Although trends in local COVID-19 cases often varied across England, cases in neighbouring local authorities were generally strongly correlated with each other (supplementary Figure S7). The median pairwise correlation by LTLA (averaged across the correlation with all other LTLAs in England) varied substantially, with some notable dates and locations where the median value was negative (supplementary Figure S7A). For example, the median correlation in Liverpool in mid-October and early-November 2020 was negative: while cases in most local authorities were rising, they were decreasing in Liverpool and nearby local authorities due to local restrictions on social distancing [27]. Another example: the median correlation in Medway (a mainly rural local authority in South East England) in the second half of November and early December 2020 was negative: cases in Medway were rising as the Alpha variant emerged, while cases were stable or declining in most local authorities following the second national lockdown (05 November - 02 December 2020) and additional earlier restrictions on social distancing.

In contrast, the number of cases reported by local authorities within the same catchment area were usually strongly correlated with each other (median correlation coefficient > 0.5; Figure S7B), especially October 2020 through February 2021. For example, in Liverpool NHS Foundation Trust, the median correlation between the main LTLAs in its catchment area (Liverpool, Knowsley, Sefton and West Lancashire) was above 0.7 throughout October and November 2020, despite the negative correlation nationally. Similarly, the correlation between the main LTLAs in the catchment area of Medway NHS Foundation Trust (Medway and Swale) was above 0.5.

Calibration

All catchment area definitions resulted in forecasts that overestimated the uncertainty (Figure S9): for nominal coverage of 50%, the empirical coverage of all definitions was 60 - 70%, and for nominal coverage of 90% the empirical coverage was 90 - 95%. The difference between nominal and empirical coverage decreased, albeit not substantially, at longer forecast horizons. For example, for a nominal coverage of 50%, empirical coverage of all definitions was in the range 68 - 70% and 61 - 65% at a 1- and 14-day forecast horizon, respectively (supplementary Figure S9, second row).

There was little difference between calibration of the forecasts using the different catchment area definitions, especially compared to the difference between nominal and empirical coverage. The COVID-19 definition was the definition with the smallest overestimate of uncertainty for nominal coverage of 20% (Figure S9, first row), but this difference disappeared at higher nominal coverage values. There was also no difference between calibration at spatial scales (upper- vs. lower-tier local authority; Figure S9, first and second columns) or for future observed vs. future forecast cases (Figure S9, first and third column).

Probabilistic forecast accuracy

By forecast horizon, the forecasts made using the marginal distribution definition are consistently the least accurate (highest rWIS values); on the other hand the “nearby” heuristic and COVID-19 data definitions were generally the most accurate, with the other definitions (“nearest” heuristic, and emergency and elective data) falling in the mid-ranks (Figure 3A). There was only a small difference between definitions’ absolute rWIS values, which could suggest only small differences in probabilistic forecast accuracy, or no consistent trend in performance across forecast dates and/or locations. Finally, the average accuracy of forecasts made with the emergency and elective data definitions were very similar at all forecast horizons.

• Download figure
• Open in new tab

Figure 3: Forecasting performance under different hospital catchment area definitions (by UTLA) using future observed cases.

(A) Median interval score (taken over all forecast dates and locations) for each forecast horizon, with the values highlighted for a 7-day forecast horizon. (B) Median interval score for each forecast date; 7-day forecast horizon. (C) Median interval score for the top 40 acute NHS Trusts (by total COVID-19 admissions); 7-day forecast horizon.

There was no clear best catchment area definition when we evaluated forecasts by forecast date (Figure 3B), with almost all definitions being either first- and last-ranked by rWIS for at least one forecast date (only the nearest heuristic was never first-ranked, and the elective admissions data definition was never last-ranked). The COVID-19 admissions data definition was first- or second-ranked by rWIS for the majority (9/15) of forecast dates at both a 7- and 14-day forecast horizon (supplementary Figure S10). At the same time, the definition was only last-ranked once at a 7-day horizon, and was never last-ranked at a 14-day horizon. The “nearby” heuristic performed comparably to the COVID-19 definition, and especially at a 7-day forecast horizon there was little to differentiate them. Forecasts made with the marginal distribution definition had the most variable accuracy, but were especially poor during December 2020. As noted previously, it was during this period that local COVID-19 case trends were more heterogeneous as a result of the emergence of Alpha and local social distancing regulations. These results therefore suggest that it is more important to use a local catchment area definition where there is heterogeneity in local case trends. Finally, we saw again that the emergency and elective admissions data definitions performed similarly across all forecast dates.

Again, there was no clear best definition when we evaluated forecasts by location, but there was more variation between definitions (Figure 3C). The COVID-19 definition was the first- or second-ranked definition more frequently than other definitions: (approx. 45% and 50% for a 7- and 14-day horizon, respectively; supplementary Figure S10). Again, the “nearby” heuristic also performed well (first- or second-ranked for approximately 35% and 40% of Trusts for a 7- and 14-day forecast horizon, respectively). However, the COVID-19 was more consistent than the nearby heuristic: while the COVID-19 definition was last-ranked for only 5% of Trusts, the nearby heuristic was last-ranked for 20%. Interestingly, the marginal distribution baseline definition was first-ranked, that is, it resulted in more accurate forecasts than other definitions for approximately 30% of locations, yet it also resulted in the least accurate forecasts in approximately 40% of locations (supplementary Figure S10B).

We found no change in relative forecast accuracy for any of the catchment area definitions when using LTLA-level catchment area definitions compared to UTLA-level definitions (Figure S11). Forecasts either performed comparatively or there was no clear pattern to differences when forecasts were evaluated by forecast horizon (Figure S11A), forecast date (Figure S11B) or location (Figure S11C).

When using forecasts of future cases instead of the retrospectively known case trajectories to make forecasts, the choice of catchment area definition had the biggest effect on probabilistic forecast accuracy at a 14-day forecast horizon (Figure 4). The marginal distribution definition had the largest rWIS value (rWIS = 1.53; Figure 4A). This poor performance was linked to a few forecast dates (4 and 18 October, 29 November and 13 December 2020; Figure 4B); for forecast dates in January 2021 onwards, the relative accuracy of all definitions was comparable. The nearest hospital heuristic resulted in the most accurate forecasts for a 14-day horizon (rWIS = 0.84; Figure 4A), largely due to particularly good relative forecast performance on 13 and 27 December 2020 (Figure 4B). All other definitions had rWIS values in the range 0.92 - 0.96 (Figure 4A). At shorter forecast horizons, the relative performance of all definitions was comparable, although the marginal distribution definition was consistently one of the worst-performing definitions.

• Download figure
• Open in new tab

Figure 4: Forecasting performance under different hospital catchment area definitions using future forecast cases.

(A) Median interval score (taken over all forecast dates and locations) for each forecast horizon, with the values highlighted for a 14-day forecast horizon. (B) Median interval score for each forecast date; 14-day forecast horizon. (C) Median interval score for the top 40 acute NHS Trusts (by total COVID-19 admissions); 14-day forecast horizon.

When considering the rWIS rankings by forecast date and location, the COVID-19 data definition stood out (Figure S11): it was first- or second-ranked for 7/14 forecast dates for both a 7- and 14-day forecast horizon, and for approximately 35% and 50% of locations for a 7- and 14-day forecast horizon. Although the nearby hospitals heuristic performed comparably to the COVID-19 definition as measured by top rWIS rankings, it was less consistent when evaluated by location: it was ranked in the bottom-two for 40% of locations, compared to only 21% and 12% for the COVID-19 definition for 7- and 14-day horizons, respectively.