An Open Repository of Real-Time COVID-19 Indicators =================================================== * Alex Reinhart * Logan Brooks * Maria Jahja * Aaron Rumack * Jingjing Tang * Wael Al Saeed * Taylor Arnold * Amartya Basu * Jacob Bien * Ángel A. Cabrera * Andrew Chin * Eu Jing Chua * Brian Clark * Nat DeFries * Jodi Forlizzi * Samuel Gratzl * Alden Green * George Haff * Robin Han * Addison J. Hu * Sangwon Hyun * Ananya Joshi * Jimi Kim * Andrew Kuznetsov * Wichada La Motte-Kerr * Yeon Jin Lee * Kenneth Lee * Zachary C. Lipton * Michael X. Liu * Lester Mackey * Kathryn Mazaitis * Daniel J. McDonald * Balasubramanian Narasimhan * Natalia L. Oliveira * Pratik Patil * Adam Perer * Collin A. Politsch * Samyak Rajanala * Dawn Rucker * Nigam H. Shah * Vishnu Shankar * James Sharpnack * Dmitry Shemetov * Noah Simon * Vishakha Srivastava * Shuyi Tan * Robert Tibshirani * Elena Tuzhilina * Ana Karina Van Nortwick * Valérie Ventura * Larry Wasserman * Jeremy C. Weiss * Kristin Williams * Roni Rosenfeld * Ryan J. Tibshirani ## Abstract The COVID-19 pandemic presented enormous data challenges in the United States. Policy makers, epidemiological modelers, and health researchers all require up-to-date data on the pandemic and relevant public behavior, ideally at fine spatial and temporal resolution. The COVIDcast API is our attempt to fill this need: operational since April 2020, it provides open access to both traditional public health surveillance signals (cases, deaths, and hospitalizations) and many auxiliary indicators of COVID-19 activity, such as signals extracted from de-identified medical claims data, massive online surveys, cell phone mobility data, and internet search trends. These are available at a fine geographic resolution (mostly at the county level) and are updated daily. The COVIDcast API also tracks all revisions to historical data, allowing modelers to account for the frequent revisions and backfill that are common for many public health data sources. All of the data is available in a common format through the API and accompanying R and Python software packages. This paper describes the data sources and signals, and provides examples demonstrating that the auxiliary signals in the COVIDcast API present information relevant to tracking COVID activity, augmenting traditional public health reporting and empowering research and decision-making. Public health decision makers, healthcare providers, epidemiological researchers, employers, institutions, and the general public benefit from promptly and readily accessible data regarding COVID-19 activity levels, countermeasures, and pandemic impact. Realtime indicators of COVID-19 activity levels, such as statistics on cases, deaths, test positivity, and hospitalizations, enable reports and interactive dashboard applications for situational awareness [1–3], and are essential for most analyses of the pandemic. These data are available for locations across the United States from a number of offcial sources and independent aggregators in varied and inconsistent formats. Different data types and sources vary in timeliness, based on when measured events occur in the progression of the disease, testing capabilities, the reporting pipeline, and their publication schedules. Additional, auxiliary data sources can improve on the timeliness, scope, and utility of the “topline” indicators (cases, test positivity, hospitalizations, deaths) coming from the public health reporting system. For example, in the context of other infectious diseases: syndromic surveillance in ambulatory clinics and emergency rooms improves the accuracy of outbreak detection for emerging pathogens such as H1N1 [4]; and digital surveillance (based on, e.g., search and social media trends) enables more accurate “nowcasts” and forecasts of traditional disease surveillance streams such as the CDC’s ILINet [5, 6], as do publication formats providing access to historical versions of a given data set [7, 8]. Several other examples exist that span a wide variety of data platforms and diseases [9–12]. During the COVID-19 pandemic, digital data streams have permitted faster prediction of case increases [13, 14], while enabling analyses of the impact of public health policies on public behavior, the economy, and disease spread [15–18]. The Delphi Group works with partner organizations and public data sets to build a massive database of indicators tracking COVID-19 activity and other relevant phenomena in the United States, which has been publicly available and continuously updated since April 2020. Alongside public data on reported cases and deaths, this database includes several unique data streams, including indicators extracted from de-identified medical claims data, antigen test results from a major testing manufacturer, large-scale public surveys that measure symptoms and public behavior, and indicators based on particular Google search queries. (We use the terms “indicator” and “signal” interchangeably.) We make aggregate signals publicly available, generally at the county level, via the COVIDcast API [19]. We store and provide access to all previous (historical) versions of the signals, a key feature that exposes the effects of data revisions. Lastly, we provide R [20] and Python [21] packages to facilitate interaction with the API, and an online dashboard to visualize the data [22]. In a companion paper, we analyze the utility provided by a core set of the indicators in COVID-19 forecasting and hotspot prediction models. In another companion paper, we elaborate on our research group’s (Delphi’s) large-scale public surveys, run in partnership with Facebook and available in aggregate form in the COVIDcast API. ## 1 Methods ### 1.1 Data Collection We receive data daily from healthcare partners, technology companies, and from surveys conducted daily by Delphi in partnership with Facebook. These data sources provide information not available from standard public health reporting or other common sources, such as: #### Health Insurance Claims Based on de-identified medical insurance claims from Change Healthcare and other health system partners, we release indicators on the estimated per-centage of covered outpatient visits and hospitalizations that involved COVID diagnoses or symptoms. #### Internet-Based Surveys Conducted in partnership with Facebook, Delphi’s COVID-19 Trends and Impact Survey receives an average of 50,000 responses daily, and has received over 20 million responses since April 2020 [23, 24]. From the surveys, we construct indicators on symptoms, social distancing, vaccination, and other attitudes and behaviors related to COVID. #### COVID Antigen Tests Based on data from Quidel, a manufacturer of COVID antigen tests in the United States, we calculate and release (Quidel-specific) test volumes and positivity rates. #### Search Trends Based on Google’s COVID-19 Search Trends data set [25], we provide indicators reflecting COVID-related search activity. #### Mobility Data SafeGraph, a company that collects geospatial data from smartphone apps, calculates COVID-related mobility signals [26, 27] and makes them available to researchers under a data use agreement; we aggregate (some of) these signals to the county level and make them publicly available. We also scrape data accessible from other public sources, such as cases and deaths data aggregated from public reporting by JHU CSSE [1] and by USAFacts [3], so that we can track revisions and updates to this data (see Section 1.3). Altogether, we collect 110 signals from 12 distinct sources, aggregate and process them, and provide them in a common format for public access. This unifies both unique (not available anywhere else) and standard COVID data streams into a single common format, enabling efficient comparison and modeling. A summary of the data sources and signals in the API is in Table 1, and detailed documentation is available at [https://cmu-delphi.github.io/delphi-epidata/api/covidcast_signals.html](https://cmu-delphi.github.io/delphi-epidata/api/covidcast_signals.html). View this table: [Table 1:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/T1) Table 1: Data sources available in Delphi’s COVIDcast API [19], as of date of publication. The first group of data sources are produced by Delphi from data not otherwise available publicly (or only available in limited form); the second group is mirrored from public sources. ### 1.2 Signal Processing Because each data source reports data in different formats, we must convert each source to a common format. In this format, each record represents an observation of one quantity at one time point in one location. Locations are coded consistently using standard identifiers such as FIPS codes; the sample size and standard error for each observation is also reported when applicable. Each signal is reported at the finest geographic resolution its source supports (such as county or state) and also aggregated to metropolitan statistical areas, Health and Human Services regions, and hospital referral regions. National averages are also provided. Crucially, each record is tagged with an *issue date* referring to when the value was first issued, as described below. This allows tracking of revisions made to individual observations, as each revision is tagged with its own issue date. When appropriate, additional post-processing (often nontrivial) is applied to the data. For example, data on visits to doctors’ offices is subject to strong day-of-week effects, and so regression is used to adjust for these effects. Other indicators are available in raw versions and versions smoothed with a 7-day trailing average. All processing is done using open-source code written primarily in Python and R, and available publicly at [https://github.com/cmu-delphi/covidcast-indicators/](https://github.com/cmu-delphi/covidcast-indicators/). ### 1.3 Revision Tracking Many data sources that are useful for epidemic tracking are subject to revision after their initial publication. For example, aggregated medical claims data may be initially published after several days, but additional claims and corrections may take days to weeks to be discovered, processed, and aggregated. Medical testing data are also often subject to backlogs and reporting delays, and estimates for any particular date are revised over time as errors are found or additional data becomes available. This revision process is generally referred to as *backfill*. For this reason, the COVIDcast API annotates every observation with two dates: the *time value*, the date the underlying events (such as tests or doctor’s visits) occurred, and the *issue date* when Delphi aggregated and reported the data for that time value. Importantly, there can be multiple observations for a single time value with different issue dates, for example if data is revised or claims records arrive late. Delphi tracks revisions to all data sources we ingest, including external data sources (such as sources tracking cases and deaths). Many external sources do not keep a public or conveniently accessible record of revisions of their data. For many purposes it is sufficient to use the most recently issued observation at a given time value, and the COVIDcast API returns the most recent issue as its default. However, for some applications it is crucial to know what was known *as of* a specific date. For example, an epidemic forecasting model will be called upon to make its forecasts based on preliminary data about recent trends, so when it is trained using historical data, it should be trained using the initial versions of that data, not updates that would have been received later. Moreover, these revision records allow models to be modified to account for noise and bias in early data versions, or to exclude data that is too new to be considered stable, and to “rewind” time and simulate how these revised models would have performed using only the versions of data available *as of* those times. Research on data revisions in the context of influenza-like illness has shown that backfill can significantly alter forecast performance [7, 29], and that careful training on preliminary data can reduce this influence [8]. Recent research has shown similar results for COVID-19 forecasts [30]. We also examine this in our companion paper on forecasting, where we observe that training and validating models on finalized data yields overly optimistic estimates of true test-time performance. ### 1.4 Public API The data described above is publicly available through the Delphi COVIDcast API [19]. By making HTTP requests specifying the data source, signal, geographic level, and time period desired, users can receive data in JSON or CSV form. For added convenience, we have written covidcast R [20] and Python [21] packages with functions to request data, format it as a data frame, plot and map it, and combine it with data from other sources. The R and Python package software is public and open-source, at [https://github.com/cmu-delphi/covidcast/](https://github.com/cmu-delphi/covidcast/). The API server software is itself also public and open-source, at [https://github.com/cmu-delphi/delphi-epidata/](https://github.com/cmu-delphi/delphi-epidata/). Lastly, most data sources are provided under the Creative Commons Attribution license, and a small number have additional restrictions imposed by the data source; see [https://cmu-delphi.github.io/delphi-epidata/api/covidcast_licensing.html](https://cmu-delphi.github.io/delphi-epidata/api/covidcast_licensing.html). ### 1.5 Interactive Visualization Since April 2020, Delphi has been maintaining and continually improving various online visualization tools for the COVIDcast indicators [22]. These tools fetch data directly from the API, and allow for exploration of both temporal (e.g., time series graphs) and spatial (e.g., choropleth maps) trends in the signals, as well as many other aspects, such as correlations, anomalies, and backfill. There is also a dedicated dashboard for exploring results from the COVID-19 Trends and Impact Survey. The visualizations have been continually improved as new sources of data arrive, and in response to interviews with users and health experts, usage analytics from the site, and user surveys. Key design goals for visualizations displaying COVIDcast data have included representing uncertainty and missing data, making features consistent and discoverable despite a wide variety of signal types, and smoothly switching between different geographic aggregations and time scales. ### 1.6 COVID Forecasting Since July 2020, Delphi has been regularly submitting short-term forecasts of COVID-19 case and death incidence, at the state and county levels, to the COVID-19 Forecast Hub [31], with “CMU-TimeSeries” as the team-model name. The process of building, training, and deploying our forecasting models leverages much of the infrastructure described in this paper (such as the COVIDcast API’s *as of* feature), and some of our forecasting systems rely on auxiliary indicators (such as survey-based and claims-based COVID-like illness signals, which are described below). ## 2 Results The indicators that are available in the COVIDcast API have been used in dashboards produced by COVID Act Now [32], COVID Exit Strategy [33], and others; to inform the Delphi, DeepCOVID [34], and the Institute for Health Metrics and Evaluation (IHME) [35] COVID forecasting models; in various federal and state government reports and analyses; and in a range of news stories. Aside from operational use in decision-making and forecasting, they have also facilitated numerous analyses studying the impacts of COVID-19 on the public, the effectiveness of policy interventions, and factors that influenced the spread of the pandemic [17, 18, 36–38]. The API currently serves hundreds of thousands of requests to thousands of users every day. In what follows, we present examples of the usefulness of some of the novel signals available in the API. These examples demonstrate that such indicators are meaningfully related to COVID activity, that they provide alternate views on pandemic activity that are not subject to the same reporting glitches and delays as traditional public health surveillance streams, and that they provide information about public behavior and attitudes that are not available from any other source. Code to reproduce all examples (which uses the covidcast R package and fetches data from the API) can be found at [https://github.com/cmu-delphi/covidcast-pnas/tree/main/indicators/code/](https://github.com/cmu-delphi/covidcast-pnas/tree/main/indicators/code/). ### 2.1 Tracking Trends Many of the indicators in the COVIDcast API are intended to track COVID activity. Five indicators in particular have the closest connections to confirmed cases: * Change Healthcare COVID-like illness (CHNG-CLI): The percentage of outpatient visits that are primarily about COVID-related symptoms, based on de-identified Change Healthcare claims data. * Change Healthcare COVID (CHNG-COVID): The percentage of outpatient visits with confirmed COVID-19, based on the same claims data. * COVID-19 Trends and Impact Survey CLI (CTIS-CLI): The estimated percentage of the population with COVID-like illness based on Delphi’s surveys of Facebook users. * COVID-19 Trends and Impact Survey CLI in the community (CTIS-CLI-in-community): The estimated percentage of the population who know someone in their local community who is sick, based on the same surveys. * Quidel test positivity rate (Quidel-TPR): The percentage of positive results among Quidel COVID antigen tests. Figure 1 compares the first three of these signals to COVID cases in the United States (from JHU CSSE, smoothed with a 7-day trailing average) over a year of the pandemic (April 15, 2020 to April 15, 2021), illustrating how they track national trends quite well. Importantly, this same relationship persists across multiple resolutions of the data, down to smaller geographic regions such as states and counties, as shown in the supplement. This will also be illustrated in a more detailed correlation analysis in the next subsection. ![Figure 1:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F1.medium.gif) [Figure 1:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F1) Figure 1: National trends, from April 2020 to April 2021, of four signals in the COVIDcast API. The auxiliary signals, based on medical claims data and massive surveys, track changes in offcially reported cases quite well. (They have all been placed on the same scale as reported cases per 100,000 people.) Besides tracking contemporaneous COVID activity, these and other indicators can be used to improve forecasts of future COVID case trends, as investigated in a companion paper. ### 2.2 Correlation Analyses To quantify the ability of the signals described above to track trends in COVID cases, we use the Spearman (rank) correlation and analyze two key correlation patterns, between each signal and confirmed COVID case rates (cases per 100,000 people): 1. *Geo-wise correlations* (i.e., on a specific date, do values of the signal correlate with case rates across locations?): Formally, let *Xt* and *Yt* be vectors of values of a signal and case rates, over all locations, on date *t*. The geo-wise correlation at time *t* is defined as cor(*Xt, Yt*) (where here and throughout cor(·, ·) denotes Spearman correlation). This examines whether a signal has the capability to help spot locations with high case rates at any given time. 2. *Time-wise correlations* (i.e., at a specific location, do values of the signal correlate with case rates across time?): Let *Xl* and *Yl* be vectors of values of a signal and case rates, over all times, at location *l*. The time-wise correlation at location *l* is defined as cor(*Xl, Yl*). This examines whether changes in a signal over time correspond to changes in reported cases at the same location. Figure 2 shows the geo-wise correlations achieved by the five signals and COVID case rates (from JHU CSSE, smoothed using a 7-day trailing average), from April 15, 2020 to April 15, 2021. This calculation is performed over all counties with at least 500 cumulative cases by the end of this period, and at which all indicators are available (956 counties in total). The large positive correlations suggest that these signals could be useful in hotspot detection (identifying counties that have relatively high COVID activity, at a given time). Somewhat surprisingly, the survey-based CLI-in-community signal shows the strongest correlations for much of the time period. This clearly demonstrates the value of a large-scale survey such as CTIS for tracking symptoms and case trends, especially when other data is unavailable. ![Figure 2:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F2.medium.gif) [Figure 2:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F2) Figure 2: Geo-wise correlations with case rates, from April 15, 2020 to April 15, 2021, calculated over all counties for which all signals were available and which had at least 500 cumulative cases by the end of this period. Figure 3 summarizes time-wise correlations from these five signals over the same time period, and for the same set of counties. For each signal, we display the set of correlations that it achieves in histogram form (more precisely, using a kernel density estimate). All signals produce positive correlations in the majority of counties considered (with very little mass in each estimated density being to the left of zero). The largest correlations, in bulk, are achieved by the CHNG-COVID signal; the CTIS-CLI-in-community signal is a close second, and the CHNG-CLI signal is third. There are two noteworthy points: ![Figure 3:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F3.medium.gif) [Figure 3:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F3) Figure 3: Time-wise correlations with case rates, from April 15, 2020 to April 15, 2021, calculated over all counties for which all signals were available and which had at least 500 cumulative cases by the end of this period. * This is different from what is observed in Figure 2, where the CTIS-CLI-in-community signal achieves clearly the highest correlations for most of the time period. However, it is worth emphasizing that time-wise and geo-wise correlations are truly measuring different properties of a signal; and the claims signals (CHNG-COVID and CHNG-CLI) seem more appropriate for temporal—rather than spatial—comparisons. We revisit this point in the discussion. * It is still quite impressive (and surprising) that the CTIS-CLI-in-community signal, based on people reporting on the symptoms of others around them, can achieve nearly as strong time-wise correlations to confirmed cases as can a signal that is based on picking up the occurrence of a confirmed case passing through the outpatient system. ### 2.3 Helping Robustness Public health reporting of COVID tests, cases, deaths, and hospitalizations is subject to a number of possible delays and problems. For example, COVID testing data is reported inconsistently by different states using different definitions and inclusion criteria, and differences in reporting processes mean state data often does not match data reported to the federal government [39]. Case and death data is frequently backlogged and corrected, resulting in artificial spikes and drops [40, 41]. As an example, looking back at Figure 1, we can see clear dips in the confirmed COVID case curve that occur around the Thanksgiving and New Year’s holidays. This is artificial, and due to the fact that public health departments usually close over holiday periods, which delays case and death reporting (for this reason, the artificial dips persist at the stateand county-level as well). The CLI signal from the survey, on the other hand, displays no such dips. The claims signals actually display holiday effects going in the other direction: they exhibit *spikes* around Thanksgiving and New Year’s. This is because they measure the fraction of all outpatient visits with a certain condition, and the denominator here (total outpatient visits) drops disproportionately during holiday periods, as people are likely less willing to go to the doctor for more routine issues. Fortunately, in principle, the holiday effects in claims signals should be correctable: they are mainly due to *overall* changes in medical seeking behavior during holiday, periods, and we can estimate such effects using historical claims data. As a further example, Figure 4 displays data from Bexar County, Texas (which contains San Antonio) during July 2020. On July 16, 2020, San Antonio reported 4,810 backlogged cases after reporting problems prevented them from being reported over the past two weeks [42], resulting in a clearly visible spike in the left-hand panel of the figure (case data from JHU CSSE, smoothed using a 7-day trailing average). Meanwhile, Delphi’s COVID Trends and Impact Survey averaged around 350 responses per day in Bexar County over the same time period, and was able to estimate the fraction of the population who know someone in their local community with COVID-Like Illness (CLI). As we can see in the right-hand panel of the figure, this signal was not affected by Bexar County’s reporting problems and, as shown in the last subsection, it is (in general) highly correlated with case rates, providing an alternate stream of data about COVID activity unaffected by backlogs. Similar reporting problems have occurred in many jurisdictions across the United States, making it valuable to cross-check against external sources not part of the same reporting systems. ![Figure 4:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F4.medium.gif) [Figure 4:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F4) Figure 4: Reported cases per day in Bexar County, Texas during the summer of 2020. On July 16, 4,810 backlogged cases were reported, though they actually occurred over the preceding two weeks (this shows up as a prolonged spike in the left panel due to the 7-day trailing averaging applied to the case counts). Daily CTIS estimates of CLI-in-community showed more stable underlying trends. ### 2.4 Revisions Matter The revision tracking feature in the API assists in model-building and evaluation. Figure 5 illustrates how one COVIDcast medical claims signal evolved as it was revised across multiple issue dates, in four different states, between June 1 and August 1, 2020. In each plot, the rightmost ends of the lines correspond to estimates for the last day that data are available for each issue date, which are generally the most tentative estimates, and appear to be significantly biased upward in Arizona in June 2020, and significantly biased downward in New York throughout June and July 2020. ![Figure 5:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F5.medium.gif) [Figure 5:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F5) Figure 5: Estimated percentage of doctor’s visits due to COVID-like illness displayed across multiple issue dates, with later issue dates adding additional data and revising past data from prior issue dates. Claims-based signals typically undergo heavy backfill as additional claims are processed and errors are corrected; the median relative error between initial reports and final values is over 20% for such data, and only after roughly 35 days do estimates typically match finalized values within 5%. However, the systematic nature of this backfill, as illustrated in Figure 5, suggests that statistical models could be fit (potentially separately for each location) to estimate the final values from preliminary reports. On the other hand, offcial public health reporting of COVID cases and deaths can be subject to revision as death certificates are audited and backlogs cleared, resulting in thousands of cases and deaths being added or removed. This process is much more difficult to predict, and thus claims data and other sources may be a useful stand-in while public health reports are aggregated and corrected. To reiterate a previous point, when training and validating forecast models (on historical data), users will want to use data that was known *as of* the forecast date, not revised versions that only became available much later. The COVIDcast API makes all historical versions available and easily accessible for this purpose; and this feature plays a prominent role in our own analysis of forecasting and hotspot prediction models appearing in a companion paper. ### 2.5 New Perspectives Auxiliary signals (outside of the standard public health reporting streams) can serve as indicators of COVID activity, but they can also illustrate the effect of mitigating actions (such as shelter-in-place orders) and can guide resource allocation for fighting the pandemic. For example, medical claims data reflects healthcare-seeking behavior; measures of mobility reflect adherence to public health recommendations; and measures of COVID vaccine acceptance can guide outreach efforts. As an illustration, Figure 6 maps the sharp increase in rates of people staying at home between March 1 and April 15, 2020. Using SafeGraph data on the fraction of mobile devices included in SafeGraph’s panel that did not leave the immediate area of their home, it illustrates the sharp drop in travel and work outside the home that occurred in the early stages of the pandemic. It also shows that this drop was much more pronounced in some states than others, enabling analysis of policy impacts and disease spread. Similar maps can be quickly constructed for any signal in the COVIDcast API. ![Figure 6:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F6.medium.gif) [Figure 6:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F6) Figure 6: Percentage point increase in SafeGraph estimates of the percentage of people staying completely at home, based on aggregated mobile device data, between March 1, 2020 and April 15, 2020. ## 3. Discussion The COVIDcast API provides open access to real-time and geographically-detailed indica-tors of COVID activity in the United States, which supports and enhances standard public health reporting streams in several ways. First, several signals in the API closely track COVID activity (over both time and space); yet they are derived from different data streams (such as surveys, medical insurance claims, and medical devices), and are thus not subject to the same sources of error as public health reporting streams. This can be important both for robustness and situational awareness, allowing decision-makers to diagnose potential anomalies in standard surveillance streams, and for modeling tasks such as forecasting and nowcasting. Our companion paper on forecasting discusses this in more detail. Second, the API features many other signals that are relevant to understanding aspects of the pandemic and its effects on the United States population that are not found in traditional public health streams, such as data on mobility patterns, internet search trends, mask wearing, and vaccine hesitancy, to name just a few. (The latter two signals are derived from the COVID-19 Trends and Impact Survey; our companion paper on this survey gives a more detailed view of its features and capabilities.) These signals have already supported pandemic research and policy-making. Third, the underlying database tracks all revisions made to the data, allowing us to query the API to learn “what was known when,” which is critical for understanding the behavior (and potential pitfalls) of real-time surveillance signals. Such revision data is rarely available in standardized format from other sources. Finally, we emphasize that unifying many relevant signals into a single common format, with comprehensive revision tracking, is an important goal in and of itself. The ability to combine public health reporting data, syndromic surveillance data, and digital measures of mobility and behaviors goes beyond providing traditional situational awareness. Convenient and real-time access to this data enables continuous telemetry summarizing how things are, how they are expected to change, which areas need additional resources to be allocated in response, and how effective public communication is. There are a number of open questions, and challenges that remain. Several signals are subject to biases, such as survey sampling and nonresponse biases, geographic differences in market share for medical claims data, or biases in the population represented in appbased mobility data. Claims data tends also to be subject to biases during major national holidays and other events that change health-seeking behavior. Characterizing these biases will be important for future research and operational systems that use these signals. Several data sources are also subject to extensive revision and backfill, which must be studied and modeled to enable effective real-time use of these sources in forecasting and nowcasting systems. The breadth and unique features of the COVIDcast API will help facilitate this and other related work, which will be vital to advancing pandemic modeling and preparedness. ## Data Availability All data used in the article is freely available in the public COVIDcast API. [https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html) [https://github.com/cmu-delphi/covidcast-pnas/tree/main/indicators/code](https://github.com/cmu-delphi/covidcast-pnas/tree/main/indicators/code) ## Supplemental figures ![Figure 7:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F7.medium.gif) [Figure 7:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F7) Figure 7: The epidemiological “severity pyramid” represents the progression of cases from the public, through infection, through increasingly severe stages of disease. The annotations here represent the data sources collected by Delphi’s COVIDcast Epidata API. ![Figure 8:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F8.medium.gif) [Figure 8:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F8) Figure 8: Trends of cases, CHNG-CLI, CHNG-COVID, and CTIS-CLI-in-community for states across the United States. Cases are reported cases per 100,000 population. Other signals are scaled to have the same global maximum across all counties and times, so they can be presented in the same range. (part 1 of 4) ![Figure 9:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F9.medium.gif) [Figure 9:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F9) Figure 9: Trends of cases, CHNG-CLI, CHNG-COVID, and CTIS-CLI-in-community for states across the United States. (part 2 of 4) ![Figure 10:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F10.medium.gif) [Figure 10:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F10) Figure 10: Trends of cases, CHNG-CLI, CHNG-COVID, and CTIS-CLI-in-community for states across the United States. (part 3 of 4) ![Figure 11:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F11.medium.gif) [Figure 11:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F11) Figure 11: Trends of cases, CHNG-CLI, CHNG-COVID, and CTIS-CLI-in-community for states across the United States. (part 4 of 4) ![Figure 12:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F12.medium.gif) [Figure 12:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F12) Figure 12: Trends of cases, CHNG-CLI, CHNG-COVID, and CTIS-CLI-in-community for the 50 most populous counties in the United States. Cases are reported cases per 100,000 population. Other signals are scaled to have the same global maximum across all counties and times, so they can be presented in the s2a7me range. (part 1 of 4) ![Figure 13:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F13.medium.gif) [Figure 13:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F13) Figure 13: Trends of cases, CHNG-CLI, CHNG-COVID, and CTIS-CLI-in-community for the 50 most populous counties in the United States. (part 2 of 4) ![Figure 14:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F14.medium.gif) [Figure 14:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F14) Figure 14: Trends of cases, CHNG-CLI, CHNG-COVID, and CTIS-CLI-in-community for the 50 most populous counties in the United States. (part 3 of 4) ![Figure 15:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F15.medium.gif) [Figure 15:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F15) Figure 15: Trends of cases, CHNG-CLI, CHNG-COVID, and CTIS-CLI-in-community for the 50 most populous counties in the United States. (part 4 of 4) ![Figure 16:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F16.medium.gif) [Figure 16:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F16) Figure 16: National trends, from April 2020 to April 2021, of HHS-reported confirmed COVID-19 hospital admissions, plus another hospitalization signal in the COVIDcast API. Here, the HHS data has been limited to values from 2020-08-01 onward due to changes in reporting behavior, and processed into a 7-day trailing average rate per 100,000 resident population using Census Bureau estimates for 2019. HSP-Hosp is the percentage of new hospital admissions with COVID-associated diagnoses, based on claims data from health system partners, smoothed in time and adjusted for systematic day-of-week effects (HSP-Hosp is available from an earlier date when working at finer geographical resolutions). HSP-Hosp has been scaled to have the same maximum value as the HHS data. ![Figure 17:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F17.medium.gif) [Figure 17:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F17) Figure 17: Geo-wise correlations with hospitalization rates derived from HHS data, from April 15, 2020 to April 15, 2021, calculated for all times with sufficient available data within this period, over all state-like jurisdictions for which each signal was reported on at least 50 days during this period, limited to state-day combinations for which both signals are available. ![Figure 18:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/07/16/2021.07.12.21259660/F18.medium.gif) [Figure 18:](http://medrxiv.org/content/early/2021/07/16/2021.07.12.21259660/F18) Figure 18: Time-wise correlations with hospitalization rates derived from HHS data, from April 15, 2020 to April 15, 2021, calculated over all state-like jurisdictions for which each signal was reported on at least 50 days during this period, limited to state-day combinations for which both signals are available. ## Acknowledgments We thank Carrie Reed, Matt Biggersta?, Michael Johansson, Rachel Slayton, Velma Lopez, Jo Walker and others on the CDC COVID-19 Modeling Team; Hal Varian, Brett Slatkin, and others on Google’s Surveys team; Erin Hattersley, Rasmi Elasmar, and others at Google.org; Evgeniy Gabrilovich and others on Google’s Health team; Curtiss Cobb and others on Facebook’s Demography and Survey Science, Data for Good and Health teams; Alex Smola and others at Amazon Web Services; Tim Suther and others at Change Healthcare; Paul Nielsen and others at Optum; John Tamerius and others at Quidel; and Ross Epstein and others at SafeGraph. This material is based on work supported by gifts from Facebook, Google.org, the McCune Foundation, and Optum; Centers for Disease Control and Prevention (CDC) grant U01IP001121; National Science Foundation Graduate Research Fellowship Program (NSF GRFP) award DGE1745016; and the Center for Machine Learning and Health (CMLH) at Carnegie Mellon. * Received July 12, 2021. * Revision received July 12, 2021. * Accepted July 16, 2021. * © 2021, Posted by Cold Spring Harbor Laboratory The copyright holder for this pre-print is the author. All rights reserved. The material may not be redistributed, re-used or adapted without the author's permission. ## References 1. Ensheng Dong, Hongru Du, and Lauren Gardner. An interactive web-based dashboard to track COVID-19 in real time. The Lancet Infectious Diseases, 20(5):533–534, 2020. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S1473-3099(20)30120-1&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F16%2F2021.07.12.21259660.atom) 2. New York Times. Coronavirus in the U.S.: Latest map and case count. [https://www.nytimes.com/interactive/2021/us/covid-cases.html](https://www.nytimes.com/interactive/2021/us/covid-cases.html), 2020. 3. USAFacts. US COVID-19 cases and deaths by state. [https://usafacts.org/visualizations/coronavirus-covid-19-spread-map/](https://usafacts.org/visualizations/coronavirus-covid-19-spread-map/), 2020. 4. Taha A Kass-Hout, Zhiheng Xu, Paul McMurray, Soyoun Park, David L Buckeridge, John S Brownstein, Lyn Finelli, and Samuel L Groseclose. Application of change point analysis to daily influenza-like illness emergency department visits. Journal of the American Medical Informatics Association, 19(6):1075–1081, 2012. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1136/amiajnl-2011-000793&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22759619&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F16%2F2021.07.12.21259660.atom) 5. Mauricio Santillana, André T Nguyen, Mark Dredze, Michael J Paul, Elaine O Nsoesie, and John S Brownstein. Combining search, social media, and traditional data sources to improve influenza surveillance. PLOS Computational Biology, 11(10):e1004513, 2015. 6. David C Farrow. Modeling the Past, Present, and Future of Influenza. PhD thesis, Carnegie Mellon University, 2016. 7. Logan C Brooks, David C Farrow, Sangwon Hyun, Ryan J Tibshirani, and Roni Rosen-feld. Nonmechanistic forecasts of seasonal influenza with iterative one-week-ahead distributions. PLOS Computational Biology, 14(6):e1006134, 2018. 8. Logan C Brooks. Pancasting: forecasting epidemics from provisional data. PhD thesis, Carnegie Mellon University, 2020. 9. John S Brownstein, Clark C Freifeld, and Lawrence C Madoff. Digital disease detection — harnessing the web for public health surveillance. New England Journal of Medicine, 360(21):2153–2157, 2009. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMp0900702&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19423867&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F16%2F2021.07.12.21259660.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000266206000001&link_type=ISI) 10. Taha A Kass-Hout and Xiaohui Zhang. Biosurveillance: Methods and Case Studies. CRC Press, 2011. 11. Marcel Salathé, Linus Bengtsson, Todd J Bodnar, Devon D Brewer, John S Brownstein, Caroline Buckee, Ellsworth M Campbell, Ciro Cattuto, Shashank Khandelwal, Patricia L Mabry, and Alessandro Vespignani. Digital epidemiology. PLOS Computational Biology, 8(7):1–3, 2012. 12. Taha A Kass-Hout and Hend Alhinnawi. Social media in public health. British Medical Bulletin, 108(1):5–24, 2013. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/bmb/ldt028&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24103335&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F07%2F16%2F2021.07.12.21259660.atom) 13. Imama Ahmad, Ryan Flanagan, and Kyle Staller. Increased internet search interest for GI symptoms may predict COVID-19 cases in US hotspots. Clinical Gastroenterology and Hepatology, 18(12):2833–2834.e3, 2020. 14. Nicole E Kogan, Leonardo Clemente, Parker Liautaud, Justin Kaashoek, Nicholas B Link, Andre T Nguyen, Fred S Lu, Peter Huybers, Bernd Resch, Clemens Havas, Andreas Petutschnig, Jessica Davis, Matteo Chinazzi, Backtosch Mustafa, William P Hanage, Alessandro Vespignani, and Mauricio Santillana. An early warning approach to monitor COVID-19 activity with multiple digital traces in near real time. Science Advances, 7(10):eabd6989, 2021. [FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6MzoiUERGIjtzOjExOiJqb3VybmFsQ29kZSI7czo4OiJhZHZhbmNlcyI7czo1OiJyZXNpZCI7czoxMzoiNy8xMC9lYWJkNjk4OSI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIxLzA3LzE2LzIwMjEuMDcuMTIuMjEyNTk2NjAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 15. Giovanni Bonaccorsi, Francesco Pierri, Matteo Cinelli, Andrea Flori, Alessandro Galeazzi, Francesco Porcelli, Ana Lucia Schmidt, Carlo Michele Valensise, Antonio Scala, Walter Quattrociocchi, and Fabio Pammolli. Economic and social consequences of human mobility restrictions under COVID-19. Proceedings of the National Academy of Sciences, 117(27):15530–15535, 2020. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoicG5hcyI7czo1OiJyZXNpZCI7czoxMjoiMTE3LzI3LzE1NTMwIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjEvMDcvMTYvMjAyMS4wNy4xMi4yMTI1OTY2MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 16. Pierre Nouvellet, Sangeeta Bhatia, Anne Cori, Kylie E. C. Ainslie, Marc Baguelin, Samir Bhatt, Adhiratha Boonyasiri, Nicholas F. Brazeau, Lorenzo Cattarino, Laura V. Cooper, Helen Coupland, Zulma M. Cucunuba, Gina Cuomo-Dannenburg, Amy Dighe, Bimandra A. Djaafara, Ilaria Dorigatti, Oliver D. Eales, Sabine L. van Elsland, Fabricia F. Nascimento, Richard G. FitzJohn, Katy A. M. Gaythorpe, Lily Geidelberg, William D. Green, Arran Hamlet, Katharina Hauck, Wes Hinsley, Natsuko Imai, Benjamin Jeffrey, Edward Knock, Daniel J. Laydon, John A. Lees, Tara Mangal, Thomas A. Mellan, Gemma Nedjati-Gilani, Kris V. Parag, Margarita Pons-Salort, Manon Ragonnet-Cronin, Steven Riley, H. Juliette T. Unwin, Robert Verity, Michaela A. C. Vollmer, Erik Volz, Patrick G. T. Walker, Caroline E. Walters, Haowei Wang, Oliver J. Watson, Charles Whittaker, Lilith K. Whittles, Xiaoyue Xi, Neil M. Ferguson, and Christl A. Donnelly. Reduction in mobility and COVID-19 transmission. Nature Communications, 12(1):1090, 2021. 17. Dhaval Adjodah, Karthik Dinakar, Matteo Chinazzi, Samuel P Fraiberger, Alex Pentland, Samantha Bates, Kyle Staller, Alessandro Vespignani, and Deepak L Bhatt. Association between COVID-19 outcomes and mask mandates, adherence, and attitudes, 2021. 18. Sean Jewell, Joseph Futoma, Lauren Hannah, Andrew C Miller, Nicholas J Foti, and Emily B Fox. It’s complicated: characterizing the time-varying relationship between cell phone mobility and COVID-19 spread in the US, 2021. 19. Delphi Research Group. COVIDcast Epidata API. [https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html), 2020. 20. Delphi Research Group. covidcast: R Client for Delphi’s COVIDcast Epidata API. [https://cmu-delphi.github.io/covidcast/covidcastR](https://cmu-delphi.github.io/covidcast/covidcastR), 2020. 21. Delphi Research Group. COVIDcast Python API client. [https://cmu-delphi.github.io/covidcast/covidcast-py/html/](https://cmu-delphi.github.io/covidcast/covidcast-py/html/), 2020. 22. Delphi Research Group. Welcome to COVIDcast. [https://delphi.cmu.edu/covidcast/](https://delphi.cmu.edu/covidcast/), 2020. 23. Delphi Research Group. COVID symptom survey. [https://cmu-delphi.github.io/delphi-epidata/symptom-survey/](https://cmu-delphi.github.io/delphi-epidata/symptom-survey/), 2020. 24. Frauke Kreuter, Neta Barkay, Alyssa Bilinski, Adrianne Bradford, Samantha Chiu, Roee Eliat, Junchuan Fan, Tal Galili, Daniel Haimovich, Brian Kim, Sarah LaRocca, Yao Li, Katherine Morris, Stanley Presser, Tal Sarig, Joshua A Salomon, Kathleen Stewart, Elizabeth A Stuart, and Ryan J Tibshirani. Partnering with a global platform to inform research and public policy making. Survey Research Methods, 14(2):159–163, 2020. 25. Google. COVID-19 search trends symptoms dataset. [http://goo.gle/covid19symptomdataset](http://goo.gle/covid19symptomdataset), 2020. 26. SafeGraph. Social distancing metrics. [https://docs.safegraph.com/docs/social-distancing-metrics](https://docs.safegraph.com/docs/social-distancing-metrics), 2020. 27. SafeGraph. Weekly patterns. [https://docs.safegraph.com/docs/weekly-patterns](https://docs.safegraph.com/docs/weekly-patterns), 2020. 28. National Center for Health Statistics. Provisional death counts for coronavirus disease 2019 (COVID-19). [https://www.cdc.gov/nchs/nvss/vsrr/COVID19/index.htm](https://www.cdc.gov/nchs/nvss/vsrr/COVID19/index.htm), 2021. 29. Nicholas G Reich, Logan C Brooks, Spencer J Fox, Sasikiran Kandula, Craig J Mc-Gowan, Evan Moore, Dave Osthus, Evan L Ray, Abhinav Tushar, Teresa K Yamana, Matthew Biggerstaff, Michael A Johansson, Roni Rosenfeld, and Jeffrey Shaman. A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States. Proceedings of the National Academies of Sciences, 116(8):3146–3154, 2019. 30. Harshavardhan Kamarthi, Alexander Rodríguez, and B. Aditya Prakash. Back2Future: Leveraging backfill dynamics for improving real-time predictions in future. arxiv:2106.04420, 2021. 31. Reich Lab. The COVID-19 Forecast Hub. [https://covid19forecasthub.org](https://covid19forecasthub.org), 2020. 32. COVID Act Now. COVID risk & vaccine tracker. [https://covidactnow.org](https://covidactnow.org), 2020. 33. COVID Exit Strategy. Tracking our COVID-19 response. [https://www.covidexitstrategy.org](https://www.covidexitstrategy.org), 2020. 34. Alexander Rodríguez, Anik Tabassum, Jiaming Cui, Jiajia Xie, Javen Ho, Pulak Agarwal, Bijaya Adhikari, and B Aditya Prakash. DeepCOVID: An operational deep learning-driven framework for explainable real-time COVID-19 forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 15393–15400, 2021. 35. Institute for Health Metrics and Evaluation. COVID-19 projections. [https://covid19.healthdata.org](https://covid19.healthdata.org), 2021. 36. Francesco Pierri, Brea Perry, Matthew R DeVerna, Kai-Cheng Yang, Alessandro Flammini, Filippo Menczer, and John Bryden. The impact of online misinformation on U.S. COVID-19 vaccinations. arxiv:2104.10635, 2021. 37. Rajashri Chakrabarti, Lindsay Meyerson, William Nober, and Maxim Pinkovskiy. The Affordable Care Act and the COVID-19 pandemic: A regression discontinuity analysis, 2020. 38. Alexa J Doerr. Locked (down) and loaded (language): Effect of policy and speech on COVID-19 outcomes. Journal of Leadership & Organizational Studies, 1, 2021. 39. Kara W Schechtman. Federal testing data’s last mile. COVID Tracking Project, [https://covidtracking.com/analysis-updates/federal-testing-datas-last-mile](https://covidtracking.com/analysis-updates/federal-testing-datas-last-mile), 2021. 40. Sara Simon. Inconsistent reporting practices hampered our ability to analyze COVID-19 data. Here are three common problems we identified. COVID Tracking Project, [https://covidtracking.com/analysis-updates/three-covid-19-data-problems](https://covidtracking.com/analysis-updates/three-covid-19-data-problems), 2021. 41. Simone Arvisais-Anhalt, Christoph U Lehmann, Jason Y Park, Ellen Araj, Michael Holcomb, Andrew R Jamieson, Samuel McDonald, Richard J Medford, Trish M Perl, Seth M Toomay, Amy E Hughes, Melissa L McPheeters, and Mujeeb Basit. What the coronavirus disease 2019 (COVID-19) pandemic has reinforced: The need for accurate data. Clinical Infectious Diseases, 72(6):920–923, 11 2021. ISSN 1058-4838. 42. Joey Palacios. ’It’s frustrating’: Bexar County adds 5,000 COVID-19 cases from backlog as Texas disagrees on data. Texas Public Radio, [https://www.tpr.org/news/2020-07-16/its-frustrating-bexar-county-adds-5-000-covid-19-cases-from-backlog-as-texas-disagrees-on-data](https://www.tpr.org/news/2020-07-16/its-frustrating-bexar-county-adds-5-000-covid-19-cases-from-backlog-as-texas-disagrees-on-data), 2020.