RT Journal Article
SR Electronic
T1 Challenges of COVID-19 Case Forecasting in the US, 2020-2021
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2023.05.30.23290732
DO 10.1101/2023.05.30.23290732
A1 Lopez, Velma K
A1 Cramer, Estee Y
A1 Pagano, Robert
A1 Drake, John M
A1 O’Dea, Eamon B
A1 Adee, Madeline
A1 Ayer, Turgay
A1 Chhatwal, Jagpreet
A1 Dalgic, Ozden O
A1 Ladd, Mary A
A1 Linas, Benjamin P
A1 Mueller, Peter P
A1 Xiao, Jade
A1 Bracher, Johannes
A1 Rivadeneira, Alvaro J Castro
A1 Gerding, Aaron
A1 Gneiting, Tilmann
A1 Huang, Yuxin
A1 Jayawardena, Dasuni
A1 Kanji, Abdul H
A1 Le, Khoa
A1 Mühlemann, Anja
A1 Niemi, Jarad
A1 Ray, Evan L
A1 Stark, Ariane
A1 Wang, Yijin
A1 Wattanachit, Nutcha
A1 Zorn, Martha W
A1 Pei, Sen
A1 Shaman, Jeffrey
A1 Yamana, Teresa K
A1 Tarasewicz, Samuel R
A1 Wilson, Daniel J
A1 Baccam, Sid
A1 Gurung, Heidi
A1 Stage, Steve
A1 Suchoski, Brad
A1 Gao, Lei
A1 Gu, Zhiling
A1 Kim, Myungjin
A1 Li, Xinyi
A1 Wang, Guannan
A1 Wang, Lily
A1 Wang, Yueying
A1 Yu, Shan
A1 Gardner, Lauren
A1 Jindal, Sonia
A1 Marshall, Maximilian
A1 Nixon, Kristen
A1 Dent, Juan
A1 Hill, Alison L
A1 Kaminsky, Joshua
A1 Lee, Elizabeth C
A1 Lemaitre, Joseph C
A1 Lessler, Justin
A1 Smith, Claire P
A1 Truelove, Shaun
A1 Kinsey, Matt
A1 Mullany, Luke C.
A1 Rainwater-Lovett, Kaitlin
A1 Shin, Lauren
A1 Tallaksen, Katharine
A1 Wilson, Shelby
A1 Karlen, Dean
A1 Castro, Lauren
A1 Fairchild, Geoffrey
A1 Michaud, Isaac
A1 Osthus, Dave
A1 Bian, Jiang
A1 Cao, Wei
A1 Gao, Zhifeng
A1 Ferres, Juan Lavista
A1 Li, Chaozhuo
A1 Liu, Tie-Yan
A1 Xie, Xing
A1 Zhang, Shun
A1 Zheng, Shun
A1 Chinazzi, Matteo
A1 Davis, Jessica T
A1 Mu, Kunpeng
A1 Piontti, Ana Pastore y
A1 Vespignani, Alessandro
A1 Xiong, Xinyue
A1 Walraven, Robert
A1 Chen, Jinghui
A1 Gu, Quanquan
A1 Wang, Lingxiao
A1 Xu, Pan
A1 Zhang, Weitong
A1 Zou, Difan
A1 Gibson, Graham Casey
A1 Sheldon, Daniel
A1 Srivastava, Ajitesh
A1 Adiga, Aniruddha
A1 Hurt, Benjamin
A1 Kaur, Gursharn
A1 Lewis, Bryan
A1 Marathe, Madhav
A1 Peddireddy, Akhil Sai
A1 Porebski, Przemyslaw
A1 Venkatramanan, Srinivasan
A1 Wang, Lijing
A1 Prasad, Pragati V
A1 Walker, Jo W
A1 Webber, Alexander E
A1 Slayton, Rachel B
A1 Biggerstaff, Matthew
A1 Reich, Nicholas G
A1 Johansson, Michael
YR 2023
UL http://medrxiv.org/content/early/2023/06/04/2023.05.30.23290732.abstract
AB During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making.Author Summary As SARS-CoV-2 began to spread throughout the world in early 2020, modelers played a critical role in predicting how the epidemic could take shape. Short-term forecasts of epidemic outcomes (for example, infections, cases, hospitalizations, or deaths) provided useful information to support pandemic planning, resource allocation, and intervention. Yet, infectious disease forecasting is still a nascent science, and the reliability of different types of forecasts is unclear. We retrospectively evaluated COVID-19 case forecasts, which were often unreliable. For example, forecasts did not anticipate the speed of increase in cases in early winter 2020. This analysis provides insights on specific problems that could be addressed in future research to improve forecasts and their use. Identifying the strengths and weaknesses of forecasts is critical to improving forecasting for current and future public health responses.Competing Interest StatementAPP report grants from Metabiota Inc outside the submitted work. No other authors have competing interests to declare.Funding StatementCEID-Walk National Science Foundation (NSF) DEB-2027786 Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.YesThe forecasts from models used in this paper are available from the COVID-19 Forecast Hub GitHub repository (https://github.com/reichlab/covid19-forecast-hub) and the Zoltar forecast archive (https://zoltardata.com/project/44). The code used to generate all figures and tables in the manuscript is available in a public repository (https://github.com/cdcepi/Evaluation-of-case-forecasts-submitted-to-COVID19-Forecast-Hub).