Reviewer #2 (Public Review):
The authors set out to estimate excess mortality in a large set of countries globally, and this has generated a unique impression of the mortality impact of this pandemics that were in some countries missed in the official counts. In the process they have generated a central, frequently updated repository of the all-cause mortality data across countries that is a wonderful tool for all epidemiologists to follow the development in near real time. Such data have long been available in Europe (EuroMoMo) but worldwide the publication of weekly or monthly allcause mortality data have been scarce. So all in all, this work is incredibly important and rather extraordinary. A great research tool for researchers in the field. They truly fill a gap with their collection of weekly, monthly, or quarterly all-cause mortality data from 89 countries and territories, which are openly available and will be regularly-updated: the World Mortality Data. And for this reason the paper is both original and of great importance to understand the COVID-19 crisis at a global level, and should be published as soon as possible. The database is already in use by Our World in Data, the Economist and the Financial Times.
The strength of the paper is the demonstration of very substantial excess mortality in several world countries like Peru, Russia, Brazil, Bolivia, and Bulgaria. This was missed so far at the country level, although such reports had been seen from select cities like Manaus, Brazil. Also, it provides several interesting metrics, such as incidence of excess deaths, and elevation above a baseline of expected deaths, and finally the uncercount ratio of these estimates compared to official data. That the top countries underreport by a factor 10 to 100 is nicely documented. Finally, it is commendable that the authors in figure 4 demonstrates the time series coincidence of reported and excess deaths.
Also, the authors discuss the finding of undercount ratios of as low as 0,5 in some countries such as France. The interesting discussion that ensues about the meaning of excess mortality estimates when both reductions and increases may be expected due to lockdowns (fewer accidents, suicides) and due to large epidemic sizes (poor care due to overfilled hospitals), and also other effects such as heat waves and disappeared influenza epidemics. I think the authors should discuss their thinking by also looking at what IHME has put out in this regard very recently, see here:
IHME on Excess Mortality http://www.healthdata.org/special-analysis/estimation-excess-mortality-due-covid-19-and-scalars-reported-covid-19-deaths
A few critical points about the methodology for assessing and reporting excess mortality from these data. The conclusion reached in the paper is nevertheless solid: some countries like Peru, Russia and Brazil have gone through a particularly deadly experience with COVID-19 so that as many as 0,5% of their entire population have died over a couple of pandemic waves. And much of this mortality is not always reflected in the official reports: the true death toll may be 1.6x greater than the reported numbers of death. And in some countries the mortality reporting only captures about 1/10 of excess mortality. Unfortunately, many countries do not have national vital statistics data with week, month or quarterly detail, and are not represented in the mortality database.
Now to the criticism:
-
Work is not connected to the vast literature on the topic. The authors are out-of-field statisticians and seem unaware of the literature in this domain. They had generate a baseline of expected mortality based on past years time series data, as one would do when estimating excess mortality for influenza. In this way their approach is a bit similar to that used by Murray et al (Murray, Lancet 2006) to estimate the 1918 pandemic excess mortality above an annual baseline of surrounding years for a number of countries. The authors should consider at least including a reference for excess mortality estimation for each of the past influenza virus pandemics, and ponder whether it is possible to do the same that was done in these analyses to create a baseline of expected deaths that did NOT include winter-seasonal epidemic diseases like influenza (see the collected works of Olson et al, Viboud et al, Chowell et al, Olson et al, Simonsen et al, for the pandemics of 1918, 1957, 1968 and 2009). See also the latest thinking on the problem of sorting out true excess deaths from the disappeared traffic accidents, increased mental health deaths, and other complications by IHME (see link below).
-
No attempt to correct baselines for seasonal influenza. The authors use past years and generate a baseline that includes mid-winter seasonal influenza mortality. By doing so, the excess mortality estimates in the present manuscript represent excess above what is normal in a season. Thus, as the authors comment on, the excess mortality estimates are affected by the too high baseline which includes mortality due to influenza, RSV and other respiratory viruses that are now largely not circulating during the COVID-19 pandemic. Particularly, the "disappeared" influenza burden in 2020-2021 results in a meaningful underestimation of the true COVID-19 excess mortality. This problem of removing seasonal influenza from the baseline has actually been worked out by epidemiologists using various statistical approaches (sometimes harmonic terms, sometimes using influenza virus data from the WHO as predictors) in the field of epidemiology the literature mentioned above, but the entire literature of excess mortality estimation is missing from the reference list. One that I am very familiar with (!) is Simonsen et al, Plos Med 2014 - but there are many many more similar published papers computing excess mortality for seasonal and recent pandemic influenza out there (look for Viboud, Chowell, Goldstein, Paget, Olson.....). I suggest you simply discuss this situation, and makee reference to this - plus suggest others to work out ways to remove influenza from the baseline, for example incorporate WHOs seasonal influenza timeseries database data (FluNet.org) in the excess mortality regression models (to identify and remove excess mortality during influenza periods).
-
Varying COVID-19 study time for different countries. Another problem with the way they report the excess mortality is in the difference in follow-up time. Some countries have data up to March 2021, while others only until last summer. This should be dealt with in the estimates, for example by comparing countries with complete year 2000 data. It probably cannot be helped that some countries publish their data late, but the authors should highlight these issues of comparison between countries in the text.
-
About the finding of a 1.6x higher excess mortality than reported deaths. It seems important to say that this is a finding for countries with national vital statistics in near-real time, so things may be very different in countries where such data to not exist.
-
Figure 4. Can you explain the time shift between the reported and excess deaths in the United States? Must be a data issue. Also, would be better to chose line colors or width so that one can distinguish the two in black and white.