RT Journal Article
SR Electronic
T1 Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the US
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2021.02.03.21250974
DO 10.1101/2021.02.03.21250974
A1 Cramer, Estee Y
A1 Ray, Evan L
A1 Lopez, Velma K
A1 Bracher, Johannes
A1 Brennen, Andrea
A1 Castro Rivadeneira, Alvaro J
A1 Gerding, Aaron
A1 Gneiting, Tilmann
A1 House, Katie H
A1 Huang, Yuxin
A1 Jayawardena, Dasuni
A1 Kanji, Abdul H
A1 Khandelwal, Ayush
A1 Le, Khoa
A1 Mühlemann, Anja
A1 Niemi, Jarad
A1 Shah, Apurv
A1 Stark, Ariane
A1 Wang, Yijin
A1 Wattanachit, Nutcha
A1 Zorn, Martha W
A1 Gu, Youyang
A1 Jain, Sansiddh
A1 Bannur, Nayana
A1 Deva, Ayush
A1 Kulkarni, Mihir
A1 Merugu, Srujana
A1 Raval, Alpan
A1 Shingi, Siddhant
A1 Tiwari, Avtansh
A1 White, Jerome
A1 Abernethy, Neil F
A1 Woody, Spencer
A1 Dahan, Maytal
A1 Fox, Spencer
A1 Gaither, Kelly
A1 Lachmann, Michael
A1 Meyers, Lauren Ancel
A1 Scott, James G
A1 Tec, Mauricio
A1 Srivastava, Ajitesh
A1 George, Glover E
A1 Cegan, Jeffrey C
A1 Dettwiller, Ian D
A1 England, William P
A1 Farthing, Matthew W
A1 Hunter, Robert H
A1 Lafferty, Brandon
A1 Linkov, Igor
A1 Mayo, Michael L
A1 Parno, Matthew D
A1 Rowland, Michael A
A1 Trump, Benjamin D
A1 Zhang-James, Yanli
A1 Chen, Samuel
A1 Faraone, Stephen V
A1 Hess, Jonathan
A1 Morley, Christopher P
A1 Salekin, Asif
A1 Wang, Dongliang
A1 Corsetti, Sabrina M
A1 Baer, Thomas M
A1 Eisenberg, Marisa C
A1 Falb, Karl
A1 Huang, Yitao
A1 Martin, Emily T
A1 McCauley, Ella
A1 Myers, Robert L
A1 Schwarz, Tom
A1 Sheldon, Daniel
A1 Gibson, Graham Casey
A1 Yu, Rose
A1 Gao, Liyao
A1 Ma, Yian
A1 Wu, Dongxia
A1 Yan, Xifeng
A1 Jin, Xiaoyong
A1 Wang, Yu-Xiang
A1 Chen, YangQuan
A1 Guo, Lihong
A1 Zhao, Yanting
A1 Gu, Quanquan
A1 Chen, Jinghui
A1 Wang, Lingxiao
A1 Xu, Pan
A1 Zhang, Weitong
A1 Zou, Difan
A1 Biegel, Hannah
A1 Lega, Joceline
A1 McConnell, Steve
A1 Nagraj, VP
A1 Guertin, Stephanie L
A1 Hulme-Lowe, Christopher
A1 Turner, Stephen D
A1 Shi, Yunfeng
A1 Ban, Xuegang
A1 Walraven, Robert
A1 Hong, Qi-Jun
A1 Kong, Stanley
A1 van de Walle, Axel
A1 Turtle, James A
A1 Ben-Nun, Michal
A1 Riley, Steven
A1 Riley, Pete
A1 Koyluoglu, Ugur
A1 DesRoches, David
A1 Forli, Pedro
A1 Hamory, Bruce
A1 Kyriakides, Christina
A1 Leis, Helen
A1 Milliken, John
A1 Moloney, Michael
A1 Morgan, James
A1 Nirgudkar, Ninad
A1 Ozcan, Gokce
A1 Piwonka, Noah
A1 Ravi, Matt
A1 Schrader, Chris
A1 Shakhnovich, Elizabeth
A1 Siegel, Daniel
A1 Spatz, Ryan
A1 Stiefeling, Chris
A1 Wilkinson, Barrie
A1 Wong, Alexander
A1 Cavany, Sean
A1 España, Guido
A1 Moore, Sean
A1 Oidtman, Rachel
A1 Perkins, Alex
A1 Kraus, David
A1 Kraus, Andrea
A1 Gao, Zhifeng
A1 Bian, Jiang
A1 Cao, Wei
A1 Ferres, Juan Lavista
A1 Li, Chaozhuo
A1 Liu, Tie-Yan
A1 Xie, Xing
A1 Zhang, Shun
A1 Zheng, Shun
A1 Vespignani, Alessandro
A1 Chinazzi, Matteo
A1 Davis, Jessica T
A1 Mu, Kunpeng
A1 y Piontti, Ana Pastore
A1 Xiong, Xinyue
A1 Zheng, Andrew
A1 Baek, Jackie
A1 Farias, Vivek
A1 Georgescu, Andreea
A1 Levi, Retsef
A1 Sinha, Deeksha
A1 Wilde, Joshua
A1 Perakis, Georgia
A1 Bennouna, Mohammed Amine
A1 Nze-Ndong, David
A1 Singhvi, Divya
A1 Spantidakis, Ioannis
A1 Thayaparan, Leann
A1 Tsiourvas, Asterios
A1 Sarker, Arnab
A1 Jadbabaie, Ali
A1 Shah, Devavrat
A1 Penna, Nicolas Della
A1 Celi, Leo A
A1 Sundar, Saketh
A1 Wolfinger, Russ
A1 Osthus, Dave
A1 Castro, Lauren
A1 Fairchild, Geoffrey
A1 Michaud, Isaac
A1 Karlen, Dean
A1 Kinsey, Matt
A1 Mullany, Luke C.
A1 Rainwater-Lovett, Kaitlin
A1 Shin, Lauren
A1 Tallaksen, Katharine
A1 Wilson, Shelby
A1 Lee, Elizabeth C
A1 Dent, Juan
A1 Grantz, Kyra H
A1 Hill, Alison L
A1 Kaminsky, Joshua
A1 Kaminsky, Kathryn
A1 Keegan, Lindsay T
A1 Lauer, Stephen A
A1 Lemaitre, Joseph C
A1 Lessler, Justin
A1 Meredith, Hannah R
A1 Perez-Saez, Javier
A1 Shah, Sam
A1 Smith, Claire P
A1 Truelove, Shaun A
A1 Wills, Josh
A1 Marshall, Maximilian
A1 Gardner, Lauren
A1 Nixon, Kristen
A1 Burant, John C.
A1 Wang, Lily
A1 Gao, Lei
A1 Gu, Zhiling
A1 Kim, Myungjin
A1 Li, Xinyi
A1 Wang, Guannan
A1 Wang, Yueying
A1 Yu, Shan
A1 Reiner, Robert C
A1 Barber, Ryan
A1 Gakidou, Emmanuela
A1 Hay, Simon I.
A1 Lim, Steve
A1 Murray, Chris J.L.
A1 Pigott, David
A1 Gurung, Heidi L
A1 Baccam, Prasith
A1 Stage, Steven A
A1 Suchoski, Bradley T
A1 Prakash, B. Aditya
A1 Adhikari, Bijaya
A1 Cui, Jiaming
A1 Rodríguez, Alexander
A1 Tabassum, Anika
A1 Xie, Jiajia
A1 Keskinocak, Pinar
A1 Asplund, John
A1 Baxter, Arden
A1 Oruc, Buse Eylul
A1 Serban, Nicoleta
A1 Arik, Sercan O
A1 Dusenberry, Mike
A1 Epshteyn, Arkady
A1 Kanal, Elli
A1 Le, Long T
A1 Li, Chun-Liang
A1 Pfister, Tomas
A1 Sava, Dario
A1 Sinha, Rajarishi
A1 Tsai, Thomas
A1 Yoder, Nate
A1 Yoon, Jinsung
A1 Zhang, Leyou
A1 Abbott, Sam
A1 Bosse, Nikos I
A1 Funk, Sebastian
A1 Hellewell, Joel
A1 Meakin, Sophie R
A1 Sherratt, Katharine
A1 Zhou, Mingyuan
A1 Kalantari, Rahi
A1 Yamana, Teresa K
A1 Pei, Sen
A1 Shaman, Jeffrey
A1 Li, Michael L
A1 Bertsimas, Dimitris
A1 Lami, Omar Skali
A1 Soni, Saksham
A1 Bouardi, Hamza Tazi
A1 Ayer, Turgay
A1 Adee, Madeline
A1 Chhatwal, Jagpreet
A1 Dalgic, Ozden O
A1 Ladd, Mary A
A1 Linas, Benjamin P
A1 Mueller, Peter
A1 Xiao, Jade
A1 Wang, Yuanjia
A1 Wang, Qinxia
A1 Xie, Shanghong
A1 Zeng, Donglin
A1 Green, Alden
A1 Bien, Jacob
A1 Brooks, Logan
A1 Hu, Addison J
A1 Jahja, Maria
A1 McDonald, Daniel
A1 Narasimhan, Balasubramanian
A1 Politsch, Collin
A1 Rajanala, Samyak
A1 Rumack, Aaron
A1 Simon, Noah
A1 Tibshirani, Ryan J
A1 Tibshirani, Rob
A1 Ventura, Valerie
A1 Wasserman, Larry
A1 O’Dea, Eamon B
A1 Drake, John M
A1 Pagano, Robert
A1 Tran, Quoc T
A1 Tung Ho, Lam Si
A1 Huynh, Huong
A1 Walker, Jo W
A1 Slayton, Rachel B
A1 Johansson, Michael A
A1 Biggerstaff, Matthew
A1 Reich, Nicholas G
YR 2021
UL http://medrxiv.org/content/early/2021/12/10/2021.02.03.21250974.abstract
AB Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multi-model ensemble forecast that combined predictions from dozens of different research groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-week horizon 3-5 times larger than when predicting at a 1-week horizon. This project underscores the role that collaboration and active coordination between governmental public health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks.Significance Statement This paper compares the probabilistic accuracy of short-term forecasts of reported deaths due to COVID-19 during the first year and a half of the pandemic in the US. Results show high variation in accuracy between and within stand-alone models, and more consistent accuracy from an ensemble model that combined forecasts from all eligible models. This demonstrates that an ensemble model provided a reliable and comparatively accurate means of forecasting deaths during the COVID-19 pandemic that exceeded the performance of all of the models that contributed to it. This work strengthens the evidence base for synthesizing multiple models to support public health action.Competing Interest StatementAV, MC, and APP report grants from Metabiota Inc outside the submitted work.Funding StatementFor teams that reported receiving funding for their work, we report the sources and disclosures below. The content is solely the responsibility of the authors and does not necessarily represent the official views of any of the funding agencies. CMU-TimeSeries: CDC Center of Excellence, gifts from Google and Facebook. CU-select: NSF DMS-2027369 and a gift from the Morris-Singer Foundation. COVIDhub: US CDC (1U01IP001122); NIGMS (R35GM119582). Helmholtz Foundation (SIMCARD Information & Data Science Pilot Project). Klaus Tschira Foundation. Columbia_UNC-SurvCon: GM124104 DDS-NBDS: NSF III-1812699. EPIFORECASTS-ENSEMBLE1: Wellcome Trust (210758/Z/18/Z) GT_CHHS-COVID19: William W. George Endowment, Virginia C. and Joseph C. Mello Endowments, NSF DGE-1650044, NSF MRI 1828187, CDC and CSTE NU38OT000297, PACE at GATech. Andrea Laliberte, Joseph C. Mello, Richard (Rick) E. & Charlene Zalesky, and Claudia & Paul Raines. GT-DeepCOVID: CDC MInD-Healthcare U01CK000531-Supplement. NSF (Expeditions CCF-1918770, CAREER IIS-2028586, RAPID IIS-2027862, Medium IIS-1955883, NRT DGE-1545362), CDC MInD program, ORNL and funds/computing resources from Georgia Tech and GTRI. IHME: The Bill & Melinda Gates Foundation; the state of Washington and NSF (FAIN: 2031096). IowaStateLW-STEM: Iowa State University Plant Sciences Institute Scholars Program, NSF DMS-1916204, NSF CCF-1934884, Laurence H. Baker Center for Bioinformatics and Biological Statistics. JHU CSSE: NSF RAPID (2108526, 2028604) JHU_IDD-CovidSP: State of California, US HHS, US DHS, US Office of Foreign Disaster Assistance, Johns Hopkins Health System, Office of the Dean JHBSPH, Johns Hopkins University Modeling and Policy Hub, CDC (5U01CK000538-03), University of Utah Immunology, Inflammation, & Infectious Disease Initiative (26798 Seed Grant). LANL-GrowthRate: LANL LDRD 20200700ER. MOBS-GLEAM_COVID: COVID Supplement CDC-HHS-6U01IP001137-01; CSTE Cooperative Agreement no. NU38OT000297. NotreDame-mobility and NotreDame-FRED: NSF RAPID DEB 2027718 PSI-DRAFT: NSF RAPID Grant # 2031536. UA-EpiCovDA: NSF RAPID DMS 2028401. UCSB-ACTS: NSF RAPID IIS 2029626. UCSD-NEU: Google Faculty Award, DARPA W31P4Q-21-C-0014, COVID Supplement CDC-HHS-6U01IP001137-01. UMass-MechBayes: NIGMS R35GM119582, NSF 1749854. UMich-RidgeTfReg: UMich Physics Department and Office of Research.  Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:UMass-Amherst IRBI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesAll data and code referred to in the manuscript are publicly available. https://github.com/reichlab/covid19-forecast-hub/ https://github.com/reichlab/covidEnsembles https://zoltardata.com/project/44 https://github.com/reichlab/covid19-forecast-evals/blob/main/figures/data_revisions.pdf