RT Journal Article
SR Electronic
T1 Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the US
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2021.02.03.21250974
DO 10.1101/2021.02.03.21250974
A1 Cramer, Estee Y
A1 Ray, Evan L
A1 Lopez, Velma K
A1 Bracher, Johannes
A1 Brennen, Andrea
A1 Rivadeneira, Alvaro J Castro
A1 Gerding, Aaron
A1 Gneiting, Tilmann
A1 House, Katie H
A1 Huang, Yuxin
A1 Jayawardena, Dasuni
A1 Kanji, Abdul H
A1 Khandelwal, Ayush
A1 Le, Khoa
A1 Mühlemann, Anja
A1 Niemi, Jarad
A1 Shah, Apurv
A1 Stark, Ariane
A1 Wang, Yijin
A1 Wattanachit, Nutcha
A1 Zorn, Martha W
A1 Gu, Youyang
A1 Jain, Sansiddh
A1 Bannur, Nayana
A1 Deva, Ayush
A1 Kulkarni, Mihir
A1 Merugu, Srujana
A1 Raval, Alpan
A1 Shingi, Siddhant
A1 Tiwari, Avtansh
A1 White, Jerome
A1 Woody, Spencer
A1 Dahan, Maytal
A1 Fox, Spencer
A1 Gaither, Kelly
A1 Lachmann, Michael
A1 Meyers, Lauren Ancel
A1 Scott, James G
A1 Tec, Mauricio
A1 Srivastava, Ajitesh
A1 George, Glover E
A1 Cegan, Jeffrey C
A1 Dettwiller, Ian D
A1 England, William P
A1 Farthing, Matthew W
A1 Hunter, Robert H
A1 Lafferty, Brandon
A1 Linkov, Igor
A1 Mayo, Michael L
A1 Parno, Matthew D
A1 Rowland, Michael A
A1 Trump, Benjamin D
A1 Corsetti, Sabrina M
A1 Baer, Thomas M
A1 Eisenberg, Marisa C
A1 Falb, Karl
A1 Huang, Yitao
A1 Martin, Emily T
A1 McCauley, Ella
A1 Myers, Robert L
A1 Schwarz, Tom
A1 Sheldon, Daniel
A1 Gibson, Graham Casey
A1 Yu, Rose
A1 Gao, Liyao
A1 Ma, Yian
A1 Wu, Dongxia
A1 Yan, Xifeng
A1 Jin, Xiaoyong
A1 Wang, Yu-Xiang
A1 Chen, YangQuan
A1 Guo, Lihong
A1 Zhao, Yanting
A1 Gu, Quanquan
A1 Chen, Jinghui
A1 Wang, Lingxiao
A1 Xu, Pan
A1 Zhang, Weitong
A1 Zou, Difan
A1 Biegel, Hannah
A1 Lega, Joceline
A1 Snyder, Timothy L
A1 Wilson, Davison D
A1 McConnell, Steve
A1 Walraven, Robert
A1 Shi, Yunfeng
A1 Ban, Xuegang
A1 Hong, Qi-Jun
A1 Kong, Stanley
A1 Turtle, James A
A1 Ben-Nun, Michal
A1 Riley, Pete
A1 Riley, Steven
A1 Koyluoglu, Ugur
A1 DesRoches, David
A1 Hamory, Bruce
A1 Kyriakides, Christina
A1 Leis, Helen
A1 Milliken, John
A1 Moloney, Michael
A1 Morgan, James
A1 Ozcan, Gokce
A1 Schrader, Chris
A1 Shakhnovich, Elizabeth
A1 Siegel, Daniel
A1 Spatz, Ryan
A1 Stiefeling, Chris
A1 Wilkinson, Barrie
A1 Wong, Alexander
A1 Gao, Zhifeng
A1 Bian, Jiang
A1 Cao, Wei
A1 Ferres, Juan Lavista
A1 Li, Chaozhuo
A1 Liu, Tie-Yan
A1 Xie, Xing
A1 Zhang, Shun
A1 Zheng, Shun
A1 Vespignani, Alessandro
A1 Chinazzi, Matteo
A1 Davis, Jessica T
A1 Mu, Kunpeng
A1 Piontti, Ana Pastore y
A1 Xiong, Xinyue
A1 Zheng, Andrew
A1 Baek, Jackie
A1 Farias, Vivek
A1 Georgescu, Andreea
A1 Levi, Retsef
A1 Sinha, Deeksha
A1 Wilde, Joshua
A1 Penna, Nicolas D
A1 Celi, Leo A
A1 Sundar, Saketh
A1 Cavany, Sean
A1 España, Guido
A1 Moore, Sean
A1 Oidtman, Rachel
A1 Perkins, Alex
A1 Osthus, Dave
A1 Castro, Lauren
A1 Fairchild, Geoffrey
A1 Michaud, Isaac
A1 Karlen, Dean
A1 Lee, Elizabeth C
A1 Dent, Juan
A1 Grantz, Kyra H
A1 Kaminsky, Joshua
A1 Kaminsky, Kathryn
A1 Keegan, Lindsay T
A1 Lauer, Stephen A
A1 Lemaitre, Joseph C
A1 Lessler, Justin
A1 Meredith, Hannah R
A1 Perez-Saez, Javier
A1 Shah, Sam
A1 Smith, Claire P
A1 Truelove, Shaun A
A1 Wills, Josh
A1 Kinsey, Matt
A1 Obrecht, RF
A1 Tallaksen, Katharine
A1 Burant, John C
A1 Wang, Lily
A1 Gao, Lei
A1 Gu, Zhiling
A1 Kim, Myungjin
A1 Li, Xinyi
A1 Wang, Guannan
A1 Wang, Yueying
A1 Yu, Shan
A1 Reiner, Robert C
A1 Barber, Ryan
A1 Gaikedu, Emmanuela
A1 Hay, Simon
A1 Lim, Steve
A1 Murray, Chris
A1 Pigott, David
A1 Prakash, B Aditya
A1 Adhikari, Bijaya
A1 Cui, Jiaming
A1 Rodríguez, Alexander
A1 Tabassum, Anika
A1 Xie, Jiajia
A1 Keskinocak, Pinar
A1 Asplund, John
A1 Baxter, Arden
A1 Oruc, Buse Eylul
A1 Serban, Nicoleta
A1 Arik, Sercan O
A1 Dusenberry, Mike
A1 Epshteyn, Arkady
A1 Kanal, Elli
A1 Le, Long T
A1 Li, Chun-Liang
A1 Pfister, Tomas
A1 Sava, Dario
A1 Sinha, Rajarishi
A1 Tsai, Thomas
A1 Yoder, Nate
A1 Yoon, Jinsung
A1 Zhang, Leyou
A1 Abbott, Sam
A1 Bosse, Nikos I
A1 Funk, Sebastian
A1 Hellewel, Joel
A1 Meakin, Sophie R
A1 Munday, James D
A1 Sherratt, Katherine
A1 Zhou, Mingyuan
A1 Kalantari, Rahi
A1 Yamana, Teresa K
A1 Pei, Sen
A1 Shaman, Jeffrey
A1 Ayer, Turgay
A1 Adee, Madeline
A1 Chhatwal, Jagpreet
A1 Dalgic, Ozden O
A1 Ladd, Mary A
A1 Linas, Benjamin P
A1 Mueller, Peter
A1 Xiao, Jade
A1 Li, Michael L
A1 Bertsimas, Dimitris
A1 Lami, Omar Skali
A1 Soni, Saksham
A1 Bouardi, Hamza Tazi
A1 Wang, Yuanjia
A1 Wang, Qinxia
A1 Xie, Shanghong
A1 Zeng, Donglin
A1 Green, Alden
A1 Bien, Jacob
A1 Hu, Addison J
A1 Jahja, Maria
A1 Narasimhan, Balasubramanian
A1 Rajanala, Samyak
A1 Rumack, Aaron
A1 Simon, Noah
A1 Tibshirani, Ryan
A1 Tibshirani, Rob
A1 Ventura, Valerie
A1 Wasserman, Larry
A1 O’Dea, Eamon B
A1 Drake, John M
A1 Pagano, Robert
A1 Walker, Jo W
A1 Slayton, Rachel B
A1 Johansson, Michael
A1 Biggerstaff, Matthew
A1 Reich, Nicholas G
YR 2021
UL http://medrxiv.org/content/early/2021/02/05/2021.02.03.21250974.abstract
AB Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. In 2020, the COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized hundreds of thousands of specific predictions from more than 50 different academic, industry, and independent research groups. This manuscript systematically evaluates 23 models that regularly submitted forecasts of reported weekly incident COVID-19 mortality counts in the US at the state and national level. One of these models was a multi-model ensemble that combined all available forecasts each week. The performance of individual models showed high variability across time, geospatial units, and forecast horizons. Half of the models evaluated showed better accuracy than a naïve baseline model. In combining the forecasts from all teams, the ensemble showed the best overall probabilistic accuracy of any model. Forecast accuracy degraded as models made predictions farther into the future, with probabilistic accuracy at a 20-week horizon more than 5 times worse than when predicting at a 1-week horizon. This project underscores the role that collaboration and active coordination between governmental public health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks.Competing Interest StatementAV, MC, and APP report grants from Metabiota Inc outside the submitted work.Funding StatementFor teams that reported receiving funding for their work, we report the sources and disclosures below. CMU-TimeSeries: CDC Center of Excellence, gifts from Google and Facebook. CU-select: NSF DMS-2027369 and a gift from the Morris-Singer Foundation. COVIDhub: This work has been supported by the US Centers for Disease Control and Prevention (1U01IP001122) and the National Institutes of General Medical Sciences (R35GM119582). The content is solely the responsibility of the authors and does not necessarily represent the official views of CDC, NIGMS or the National Institutes of Health. Johannes Bracher was supported by the Helmholtz Foundation via the SIMCARD Information& Data Science Pilot Project. Tilmann Gneiting gratefully acknowledges support by the Klaus Tschira Foundation. DDS-NBDS: NSF III-1812699. EPIFORECASTS-ENSEMBLE1: Wellcome Trust (210758/Z/18/Z) GT_CHHS-COVID19: William W. George Endowment, Virginia C. and Joseph C. Mello Endowments, NSF DGE-1650044, NSF MRI 1828187, research cyberinfrastructure resources and services provided by the Partnership for an Advanced Computing Environment (PACE) at Georgia Tech, and the following benefactors at Georgia Tech: Andrea Laliberte, Joseph C. Mello, Richard Rick E. & Charlene Zalesky, and Claudia & Paul Raines GT-DeepCOVID: CDC MInD-Healthcare U01CK000531-Supplement. NSF (Expeditions CCF-1918770, CAREER IIS-2028586, RAPID IIS-2027862, Medium IIS-1955883, NRT DGE-1545362), CDC MInD program, ORNL and funds/computing resources from Georgia Tech and GTRI. IHME: This work was supported by the Bill & Melinda Gates Foundation, as well as funding from the state of Washington and the National Science Foundation (award no. FAIN: 2031096). IowaStateLW-STEM: Iowa State University Plant Sciences Institute Scholars Program, NSF DMS-1916204, NSF CCF-1934884, Laurence H. Baker Center for Bioinformatics and Biological Statistics. JHU_IDD-CovidSP: State of California, US Dept of Health and Human Services, US Dept of Homeland Security, US Office of Foreign Disaster Assistance, Johns Hopkins Health System, Office of the Dean at Johns Hopkins Bloomberg School of Public Health, Johns Hopkins University Modeling and Policy Hub, Centers for Disease Control and Prevention (5U01CK000538-03), University of Utah Immunology, Inflammation, & Infectious Disease Initiative (26798 Seed Grant). LANL-GrowthRate: LANL LDRD 20200700ER. MOBS-GLEAM_COVID: COVID Supplement CDC-HHS-6U01IP001137-01. NotreDame-mobility and NotreDame-FRED: NSF RAPID DEB 2027718 UA-EpiCovDA: NSF RAPID Grant # 2028401. UCSB-ACTS: NSF RAPID IIS 2029626. UCSD-NEU: Google Faculty Award, DARPA W31P4Q-21-C-0014, COVID Supplement CDC-HHS-6U01IP001137-01. UMass-MechBayes: NIGMS R35GM119582, NSF 1749854. UMich-RidgeTfReg: The University of Michigan Physics Department and the University of Michigan Office of Research.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:UMass-Amherst IRBAll necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesAll data and code referred to in the manuscript are publicly available. https://github.com/reichlab/covid19-forecast-hub/ https://github.com/reichlab/covidEnsembles https://zoltardata.com/project/44