Analyses of academician cohorts generate biased pandemic excess death estimates

INTRODUCTION

Estimating pandemic excess death burden is difficult, even in countries with excellent death registration systems, because estimates depend on many assumptions.^1,2 In countries with deficient death registration, one option is to make inferences from cohorts where death data are hopefully complete. One approach is to use cohorts of members of prestigious academies, where birth and death dates are recorded for their members. This approach was widely popularized when in 2023 a highly-read New York Times article³ noticed that deaths of Chinese Academy of Engineering (CAE) and Chinese Academy of Sciences (CAS) members peaked during December 2022 and January 2023. Linked to explosive spread of COVID-19 in Mainland China after removal of restrictive measures, deaths of academicians were believed to reflect the magnitude of excess death toll across Mainland China. This concept adopted in a meticulous epidemiological study concluded from CAE death data that the urban-dwelling population of Mainland China had 917,000 excess deaths during these two months.⁴ The current analysis dissects the validity of this approach using data from different academies and from Nobel laureates.

METHODS

RESULTS

DISCUSSION

Cohorts of academicians are, by definition, subject to extreme selection bias and cover a tiny proportion of the general population age structure. Extrapolating to the whole general population, in particular younger strata, can thus be highly misleading. Empirical data from BA and Nobel laureates also show that trends for both increases (BA) and decreases (Nobel laureates and their US subset) were seen during the pandemic in the death rates of these prestigious people. These point estimates were far from the annual excess deaths in the general population of the respective country. BA Fellows’ data point estimate would suggest roughly 5-fold (or more) higher excess deaths than what really happened in the general population, while inspecting Nobel laureate data naively would give the totally false impression that a peak, if any, might had occurred in 2018, i.e. before the pandemic. Given that all these cohorts have limited membership, the sample size is small and thus 95% CIs are very wide. Given the potential biases involved, uncertainty is even larger than what the already impressive 95% CIs would suggest. The bias magnitude may vary across countries, settings, and academician cohorts. Thus, using such data to infer excess deaths in countries where excess deaths cannot be otherwise calculated in the population directly from reliable death registration, could lead to severe miscalculations.

Several hundreds of honorific academies exist. The major focus in public media on the Chinese academies may herald selective reporting bias. Differences seemed too stark even to non-scientist journalists³ and thus worth writing a story about them. Modest or no differences in death rates (or even lower death rates during the pandemic versus pre-pandemic years, as among Nobel laureates) would not have attracted journalistic interest. Interestingly, selective analysis reporting bias was apparently worsened in the scientific peer-reviewed literature versus the original journalistic investigation media coverage. E.g. the analysis of ref. 4 included only CAE data, while the New York Times article³ that inspired that analysis (as the authors admit)⁴ had presented data on both CAE and CAS. Excess death estimates using CAS might have been only about half those calculated using CAE data.

The combined effect of diverse biases cannot be generalized across all academy cohorts. In the specific example where excess deaths in Mainland China were estimated,⁴ the most likely scenario is large over-estimation in the published calculations. Comparing cohorts of academicians to the urban-dwelling subset of the general population (as in ref. 4), diminishes this bias modestly. E.g. in Mainland China, there is a rural-urban life expectancy gap; however, this gap is comparatively modest (1-2 years),¹⁹ thus most of the bias remains.

Excess death inferences using academician cohorts can be further markedly distorted, if the academies’ datasets do not capture all deaths and if birth/death dates have errors. While the latter error is hopefully uncommon for cohorts of famous people, there is no guarantee that academies can keep perfect track of all their members’ deaths. The very low death rates described in the Chinese and British academies analyzed here may be partly explained also by unreported deaths. No overt cases of missed deaths were documented for CAE and BA, at least in the fashion that missingness could be overtly documented for Royal Society and Academy of Athens datasets. However, it is tantalizing that death rates in CAE and BA were substantially lower than those of Nobel laureates who are likely to be at least as protected and privileged healthcare-wise. This may also reflect missing deaths in CAE and BA.

Extrapolating from very elderly cohorts poses further challenges. E.g., excess death calculations for all Mainland China based on CAE⁴ used death data on those ≥80 years old and extrapolated to younger populations using also data from Hong Kong where death registration is apparently more complete. In Mainland China, people ≥80 years old represent only 2.5% of the population.²⁰ In fact, recorded deaths in academicians aged 80-84 years were too few; unsurprisingly the wide CI of the excess deaths’ estimate in that age stratum included even a possibly large death deficit rather than excess.⁴ Academician data offer mostly information on deaths in the stratum of ≥85 years old, but only ∼1% of the general population in China is ≥85 years old.²⁰ Different countries where death registration data are reliably complete have shown variable rates of excess deaths across age strata.¹⁷ On average, excess death (expressed in percentages above expected deaths, p%) have been higher in the elderly than in non-elderly strata; particularly in France, Slovenia, and Poland, these differences have been stark.¹⁷ Conversely, a few countries (USA, Canada, UK and Chile) have shown higher p% in the nonelderly, probably boosted by non-COVID-19 causes during the pandemic (e.g. increased overdose deaths or deaths from poor access to care).^1,21,22 Also during the pandemic there were no excess deaths but consistently reduced mortality for children and adolescents.^17,23 Inappropriate extrapolation of the mortality experience from elderly frail populations to non-frail and younger populations was a major misunderstanding perpetuated in various forms throughout the pandemic.^24–27

Given the selective privileged status, cohorts of academicians probably include several people with extreme frailty, who are nevertheless maintained alive because of access to excellent care. Peak death burden during pandemic waves may be modulated from mortality displacement (harvesting) effects, if previous years have seen lower or higher than average death rates. Mortality displacement has been documented at a seasonal timeframe (e.g. inverse coupling of summer heat-related deaths and winter deaths among frail individuals)^28,29 or longer timeframes, as described previously specifically also with COVID-19.³⁰ However, in contrast to other frail populations (e.g. cohorts of long-term care residents) where mortality displacement could be readily documented within months,³¹ academician cohorts are very heterogeneous. Heterogeneity cannot be characterized without concomitant health data on the members. Moreover, lacking such data, one cannot adjust for comorbidities and frailty.

Besides data from academicians, other cohorts related to universities have been used for excess death calculations. Xiao et al.³² used obituaries from 3 Chinese universities. Populations of university professors and staff are also select groups with higher than average socioeconomic status. Thus they may still be privileged in life expectancy versus the general population, but the advantage is probably less pronounced than for prestigious academy members. Therefore, similar considerations, albeit to a smaller extent, apply to such cohorts. Moreover, missingness may be more prominent in university cohorts, since it is very unlikely that all deaths are captured through obituary collections. Worse, missingness may be uneven over time, with heightened interest in obituaries (and thus more obituaries written and captured) during a crisis or perceived crisis – as during active pandemic waves. Using obituaries from a wider population^32,33 makes the sample less selective, but missingness (and uneven missingness over time) may become even more prominent. Reliability may be higher in sharply defined populations with focused, exclusive venues for publishing obituaries, e.g. Amish and Mennonites.^34,35 Using information from online searches for funeral services and related terms^32,33 expands further the data collection, but the representativeness of collected samples is difficult to gauge. Moreover, online searches are probably affected by both real and perceived crises and it is difficult to separate the two. E.g. if there is a perception (reinforced by media, social media, and or public health announcements) of a major lethal problem, more people may search online with terms related to that problem. Thus these methods would tend to generate inflated estimates of excess deaths in the general population. These estimates are accordingly higher than those obtained by standardizing population structure and infection fatality rates against those of populations with accurate death registration.^36,37

Overall, proper death registration is indispensable for estimating excess deaths. In its absence, well-characterized cohorts may be meaningfully used, provided their participants are representative of the general population - or can be properly adjusted to be representative thereof - and information on deaths is complete. One should avoid using cohorts that may suffer from major biases.