Model training periods impact estimation of COVID-19 incidence from wastewater viral loads ========================================================================================== * Maria L. Daza–Torres * J. Cricelio Montesinos-López * Minji Kim * Rachel Olson * C. Winston Bess * Lezlie Rueda * Mirjana Susa * Linnea Tucker * Yury E. García * Alec J. Schmidt * Colleen Naughton * Brad H. Pollock * Karen Shapiro * Miriam Nuño * Heather N. Bischel ## Abstract **Background** Wastewater-based epidemiology (WBE) has been deployed broadly as an early warning tool for emerging COVID-19 outbreaks. WBE can inform targeted interventions and identify communities with high transmission, enabling quick and effective response. As wastewater becomes an increasingly important indicator for COVID-19 transmission, more robust methods and metrics are needed to guide public health decision making. **Objectives** The aim of this research was to develop and implement a mathematical framework to infer incident cases of COVID-19 from SARS-CoV-2 levels measured in wastewater. We propose a classification scheme to assess the adequacy of model training periods based on clinical testing rates and assess the sensitivity of model predictions to training periods. **Methods** We present a Bayesian deconvolution method and linear regression to estimate COVID-19 cases from wastewater data. We described an approach to characterize adequacy in testing during specific time periods and provided evidence to highlight the importance of model training periods on the projection of cases. We estimated the effective reproductive number (*R**e*) directly from observed cases and from the reconstructed incidence of cases from wastewater. The proposed modeling framework was applied to three Northern California communities served by distinct wastewater treatment plants. **Results** Both deconvolution and linear regression models consistently projected robust estimates of prevalent cases and *R**e* from wastewater influent samples when assuming training periods with adequate testing. Case estimates from models that used poorer-quality training periods consistently underestimated observed cases. **Discussion** Wastewater surveillance data requires robust statistical modeling methods to provide actionable insight for public health decision-making. We propose and validate a modeling framework that can provide estimates of COVID-19 cases and *R**e* from wastewater data that can be used as tool for disease surveillance including quality assessment for potential training data. ## 1 Introduction During the early phases of the COVID-19 pandemic, caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), the World Health Organization (WHO) recommended implementing mass testing programs as a containment measure. Individual diagnostic testing informs contact tracing and medical interventions, ideally cutting chains of transmission short and containing outbreaks. Mass clinical screening programs can also provide valuable data on community-level health trends, but maintaining mass testing programs for the purpose of community-level monitoring is expensive and requires robust infrastructure with consistent availability of testing supplies and human resources. 1 Moreover, diagnostic tests validated in low-throughput clinical settings (like nucleic acid amplification tests or NAATs) are not necessarily efficient platforms for constructing community screening programs. 2 Design-wise, employing such tests for large-scale screening requires extensive logistical coordination over large geographic areas. This becomes especially complicated when the options for diagnostic tests are myriad, lack standardization, and depend heavily on local social landscapes. Small biases in the tests may be inflated when deployed broadly, leading to large spurious associations at the population level. 3,4 Public health authorities are turning to wastewater-based epidemiology (WBE) as an alternative strategy for unbiased population-level surveillance of COVID-19. WBE uses biomarkers in wastewater to monitor trends in community-level health indices. WBE methods have been used to detect changes in drug consumption, 5,6 dietary patterns, 7 and the circulation of pathogens like poliovirus and norovirus. 8 Measurements of SARS-CoV-2 RNA in wastewater correlate strongly with changes in COVID-19 prevalence in the associated communities. 9–12 Since the onset of the pandemic, WBE of SARS-CoV-2 has been implemented in over 67 countries and 279 universities. 13 In some places, WBE programs have detected changes in SARS-CoV-2 RNA levels in wastewater prior to changes in local COVID-19 hospitalization activity and spikes in NAAT screening cases. 14–19 Others have used WBE to assess the effectiveness of public health interventions, 20 and recently, WBE was used to predict hospitalizations and ICU admissions. 21 In addition to monitoring trends, WBE can provide estimates of critical disease transmission parameters in the community without the biases associated with test-seeking behavior or poor access to testing programs. An ongoing challenge for WBE is developing robust data collection and interpretation methods that are comparable across time and geography. Variation in sampling design and sample processing methods, natural variability in viral shedding rates in feces, variability in wastewater flow volume, population fluctuations, and location-specific characteristics of wastewater management are all factors that make inference of new COVID-19 cases from wastewater data challenging. 22–25 Such factors will ultimately affect uncertainty estimates when modeling disease incidence and other public health indicators. An ideal WBE program would implement a generalized approach that provides consistent estimates of disease burden in a targeted population, yielding pubic health metrics like the disease incidence, disease prevalence and/or the effective reproductive number (*R**e*). Previous studies that approach this problem include: simple algebraic adjustments with environmental constants 26; estimating the total number of cases with a susceptible-exposed-infectious-recovered (SEIR) model informed by wastewater results11; using regression analysis to estimate the number of infected people 9; and making near real-time estimates of *R**e*. 10,27 We propose and compare two modeling approaches: a simple linear model and a Bayesian deconvolution approach to estimate COVID-19 incident cases from wastewater viral loads. Both models rely on short training periods to calibrate wastewater measurements using clinical testing data from a community screening program. We evaluate the impact of different training periods on model predictions, hypothesizing that relative rates of change in clinical testing and reported cases can be used to identify appropriate model training periods. We then apply the framework to estimate incident cases and *R**e* from wastewater influent data generated for three communities in Northern California. The methodology we describe can be generalized to other WBE systems to track the evolution and assess the magnitude of COVID-19 fluctuations and outbreaks in a manner that is comparable across programs, locations, and time. ## 2 Methods The analytical framework was developed using data from the City of Davis (Davis) and replicated for the City of Woodland and the University of California Davis. Analysis includes case and wastewater (WW) data from December 1, 2021 to March 31, 2022. Results for the City of Woodland and UC Davis are presented in the Supplementary Materials section. ### 2.1 Wastewater sample collection Staff from three Northern California wastewater treatment facilities (Davis, Woodland, and UC Davis) provided 24-hour composite wastewater samples 5-7 days per week. Samples were acquired using Teledyne ISCO 5800 refrigerated autosamplers in Davis and Woodland and a Hach Sigma 900 autosampler for UC Davis. The autosampler in Davis was programmed to collect 400 mL of influent every 15 “pulses”, where one pulse was set at 10,000 gallons. An average of 24 pulses was expected per day based on an average daily influent flow of 3.6 million gallons per day (MGD). The autosampler in Woodland was programmed to acquire 100 mL of influent every 15 min over a 24-hours period. The autosampler for UC Davis was programmed to acquire approximately 200 mL of influent every 20 min over a 24-hours period. The reported sample collection date corresponds to the date when an autosampler program was completed. Davis and UC Davis provided 12 ml samples in new 15-ml polypropylene centrifuge tubes. Woodland provided 1 L samples in Nalgene bottles that were washed, sterilized, and reused over the duration of sampling. Samples were stored at 4°C and transported weekly in coolers on ice to the analytical lab at UC Davis. For biosafety compliance, samples were placed in a water bath set at 60°C for 30 minutes and returned to 4°C prior to sample processing. Concentration and extraction were performed in a biosafety level 2 (BSL2)-certified laboratory. ### 2.2 Sample concentration and extraction The sample concentration and extraction protocol were adapted from Karthikeyan et al. 28 using 4.875 mL instead of 10 mL starting sample volume. Each wastewater sample was deposited into a separate well of a KingFisher 24 deep-well plate (Thermo Fisher). An extraction control blank (nuclease-free water) was included in 90% of the deep-well plates to assess potential contamination during concentration and extraction. Each well was spiked with 50 *µ*L of Nanotrap® Enhancement Reagent 1 (Ceres Nanosciences product ER1 SKU # 10111-10, 10111-30) and 5 *µ*L of a stock of vaccine-strain Bovine Coronavirus (BCoV, Bovilis® Coronavirus vaccine) containing an estimated 1.3*x*108 gc/mL as measured by ddPCR. 500 *µ*L aliquots of the initial BCoV vaccine stock, prepared from the suspension of lyophilized BCoV vaccine in 20 mL buffer provided with the kit, were stored at −80°*C* prior to use. Each spiked sample was manually agitated by pipetting up and down at least three times using a 5 mL pipette. Samples were then incubated for 30 minutes at room temperature. Following incubation, concentration was carried out using 75 *µ*L Nanotrap® Magnetic Virus Particles (Ceres Nanosciences) on a KingFisher Apex robot (Thermo Scientific). Concentrated viruses were eluted from the Nanotrap® beads into 400 mL of lysis buffer per sample from the MagMAX Microbiome Ultra Nucleic Acid Isolation Kit (Thermo Fisher). Concentrated samples were extracted per the MagMAX kit manufacturer instructions in 96 deep-well plates on the KingFisher Apex. Samples were eluted in 100 *µ*L of MagMAX Elution Solution. Extracts were typically stored on ice and immediately subjected to same-day analysis. When same-day analysis was not possible, extracts were immediately stored at −80°*C* until analysis. ### 2.3 Extract analysis by ddPCR Sample extracts were analyzed by digital droplet polymerase chain reaction (ddPCR) for four targets: N1 and N2 targeting regions of the nucleocapsid (N) gene of SARS-CoV-2, and Bovine Coronavirus (BCoV) and pepper mild mottle virus (PMMoV) for normalization of the SARS-CoV-2 results. N1/N2 and BCoV/PMMoV were quantified in separate duplex assays. Due to high levels of PMMoV, the sample for the PMMoV/BCoV duplex was diluted 40x prior to loading. The duplex ddPCR amplifications were performed in 20 *µ*L reactions on a QX ONE ddPCR System (Bio-Rad). Each reaction contained the following components: 1x Supermix, 20 *U/µ*L Reverse transcriptase, 15 mM Dithiothreitol from the One-Step RT-ddPCR Advanced Kit for Probe (Bio-Rad), 900 nM of each primer, 250 nM of each probe, and 5 *µ*L of sample extract or control. The one-step ddPCR reaction consisted of 3 min plate equilibrium at 25°*C*, 60 min reverse transcription at 50°*C*, 10 min enzyme activation at 95°*C*, followed by 40 cycles of 30 s denaturation at 94°*C* and 1 min annealing/extension at 58°*C*, and then 10 min enzyme deactivation at 98°*C* and 1 min droplet stabilization at 25°*C*. Preparation and plating of ddPCR master mix were carried out in a separate location from sample loading to avoid contamination. Sample loading was performed using an epMotion ® 5075 (Eppendorf) liquid handler. Each ddPCR plate included duplicate positive controls (stock mixture of synthesized gene fragments containing for the four target regions) for each target and duplicated no-template controls (nuclease free water). Additional information on the ddPCR assay designs is available in Tables S1–S4. Table S1 summarizes primers, probes for ddPCR assays performed as part of this work. Table S2 and S3 provide the ddPCR reaction and 20X primer/probe mix recipes. From October 21 to December 21, Cy5 and Cy5.5 were used in place of FAM and HEX as the fluorophores for PMMoV and BCoV, respectively. Table S4 lists details for the positive controls. Prior to 12/21/22 the annealing temperature was 60°*C*. The selection of positive and negative droplet clusters in samples and controls was conducted manually based on visual inspection of clusters. Results were considered invalid if the distribution of positive or negative droplets appeared abnormal in shape or if the total number of droplets generated fell below a threshold of 10,000 droplets in a single well. Additional information on data processing and quality control is provided in the Supplemental Material. We utilize N/PMMoV (the average SARS-CoV-2 RNA concentration (N) divided by the concentration of PMMoV) as the resulting WW signal for subsequent model development. ### 2.4 COVID-19 case data Healthy Davis Together (HDT) and Healthy Yolo Together (HYT) provided daily COVID-19 cases and total tests performed during the study period for the City of Davis, UC Davis, and Woodland,from the community screening program. 29,30 Data were smoothed for the implementation of the linear model using a 7-day moving average (the mean of the current and the previous six days). This approach improves harmonization between the current WW concentration and observed cases (Figure 1B). The 7-day moving average of cases for the linear model is similar to a deconvolution model with equal weights (uniform shedding load distribution) and a shedding time of 7 days (Section 2.6.1). Rate of change for tests administered and positive cases were calculated from a weekly aggregation of daily test counts and positive cases identified. Changes in test and case rates were then used to determine training periods with adequate testing. ![Figure 1:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/07/18/2022.07.16.22276772/F1.medium.gif) [Figure 1:](http://medrxiv.org/content/early/2022/07/18/2022.07.16.22276772/F1) Figure 1: (A) Wastewater data (N/PMMoV) and 10-day moving average (Smoothed N/PMMoV) from December 1, 2021 to March 31, 2022. (B) Cases, 7-day moving average smoothed cases, and 10-day moving average of smoothed WW data. ### 2.5 Smoothed wastewater signal To reduce uncertainty and to minimize daily fluctuations of cases observed, we applied a 10-day moving average for daily influent WW data (Figure 1A). We use the resulting smoothed influent WW data to correlate with smoothed cases (Figure 1B). ### 2.6 Models We present two models to estimate COVID-19 cases from SARS-CoV-2 RNA in the WW. The first model was adapted from Huisman et al.’s approach 10 and relates past infections with WW signal through the convolution described in Equation (3). The number of daily cases is modeled with a negative binomial (NB) distribution through the deconvolution (the inverse operation of convolution) noted in Equation (3). The second approach uses a simple linear regression to estimate cases (dependent variable) from WW data (independent variable). We also propose a strategy for selecting model training periods with adequate clinical testing to estimate parameters and improve estimation. #### 2.6.1 Deconvolution model Viral RNA concentrations measured in wastewater (*C**i*’s) are related to the number of new infections per day (*I**i*’s) through the profile of SARS-CoV-2 RNA shedding in the wastewater by an infected individual days after infection or symptom onset. 10 The measurement *C**i* of WW on day *i* is related to infections *I**j* on prior day *j* through the following convolution: ![Formula][1] where *w**j*, *j* = 1, …, *m* (sums to 1) is the shedding load distribution describing the temporal dynamics of shedding and *m* is the duration of viral shedding or shedding time. The normalization factor *N* represents the total virus shed by an infected individual during the infection period. *M* is a constant that depends on the sewer system, wastewater treatment plant, and processing pipeline. The measurement of viral RNA in wastewater *C**i* on the day *i* is used to estimate COVID-19 cases from wastewater concentration data via convolution. As noted by Huisman et al., 10 normalization factors *N* and *M* are difficult to measure, and they assume *B* = *N* · *M* as the lowest concentration of the viral load or concentration from a single infection.10 The weights for shedding load distribution (*w**j*) can be estimated using individual-pooled analysis of SARS-CoV-2 viral loads. 31–33 Instead, we estimate *B* and corresponding weights using measured WW data and cases within a specified period (training period) by directly modeling the deconvolution process through a Bayesian approach. We model *w**j* as follows: ![Formula][2] where *f**b* (*k*) is the probability density function (pdf) of a random variable *X* with exponential distribution of rate parameter *b*. Hereafter, notation ![Graphic][3] will be used instead of *w**j* emphasizing that weights depends on parameter *b*. Notice that, if *X* ∼ *Exp*(*b*) then its pdf is *f**b* (*k*) = *be**−b·k*, thus: ![Formula][4] Equation (1) is rewritten using Equation (2) as follows: ![Formula][5] The deconvolution of the Equation (3) will be denoted as *dec*(**C**, *B, b*), where **C** = (*C*1, *C*2, … *C**n*) represents the vector of WW data and parameters *B* and *b* are described above. **I** = (*I*1, *I*2, …, *I**n*) correspond to daily cases counts and its theoretical expectation is given by (𝔼(**I**) = ***µ***) estimated in terms of the deconvolution model as ***µ*** = (*µ*1, *µ*2, …, *µ**n*) = *dec*(**C**, *B, b*). The deconvolution is approximated using the Richardson–Lucy algorithm. 34 ##### Observational model We estimate the number of COVID-19 cases per day (*I**i*’s) using the Negative Binomial (NB) model, which is most relevant for overdispersed count data. In this situation, the variance exceeds the mean. The NB distribution describes a sequence of independent and identically distributed Bernoulli trials with a probability of success *p* before a specified (non-random) number of successes (*r*) occurs. Assuming a similar approach as in Lindén et al., 35 we reparametrized the NB distribution in terms of its mean *µ* and “overdispersion” parameters *ω* and *α*, with ![Graphic][6] and ![Graphic][7] in the usual NB parametrization. We assume that *I**i* follows a NB distribution. Denoting the mean and variance as *µ**i* and ![Graphic][8], respectively, and requiring that ![Graphic][9], we enforce overdispersion for suitable chosen parameters *ω* and *α*. The index of dispersion is ![Graphic][10]. Overdispersion concerning the Poisson distribution is achieved when *ω >* 1 and the index of dispersion increases with size if *α* ≠ 0; adding variability as counts increase. We found good performance fixing *ω* = 2 and *α* = 0.05, implying higher variability for the later. Using the deconvolution model and parameter as described above, we obtain the following likelihood function with the assumed NB model: ![Formula][11] where ![Graphic][12] is the number of successes, ![Graphic][13] is the probability of a single success, and (*µ*1, *µ*2, …, *µ**n*) = *dec*(**C**, *B, b*). We estimate *θ* = (*B, b*) from measurements of WW data **C** = (*C*1, *C*2, … *C**n*) and COVID-19 cases **I** = (*I*1, *I*2, … *I**n*). We adopt a Bayesian statistical approach, which is well suited to model multiple sources of uncertainty and allows the incorporation of background knowledge on the model’s parameters. In this framework, a prior distribution, *π*Θ(*θ*), is required to account for unknown parameter *θ* in order to obtain the posterior distribution. For *b*, we assumed a Gamma distribution with shape and scale parameters *v**b* = 2 and *S**b* = 1, respectively; this assumption is based on published data on viral shedding duration in gastrointestinal samples. 36 For *B*, we assumed a Gamma distribution with shape and scale parameters *v**M* = 2 and *S**M* = 2*/*1*e**−*4, respectively; based on the lowest viral RNA concentrations observed. Having specified the likelihood and the prior, we use Bayes’ rule to calculate the posterior distribution, ![Formula][14] where *Z*(**I**) = ∫ *π*Θ(*θ*)*L*(*θ*|**C, I**)*dθ* is the normalization constant. The posterior distribution is simulated using a t-walk Markov chain Monte Carlo (MCMC) algorithm. 37 ##### Duration of viral shedding The deviance information criterion (DIC) was used to select the shedding time (*m*). DIC is a Bayesian generalization of the Akaike information criterion (AIC) for model selection in a finite set of models, with preference given to models with lower DIC. The DIC is preferred in settings with Bayesian model selection problems where the model’s posterior distributions are obtained by MCMC simulation. 38 We selected the appropriate shedding time by computing DIC in a grid search along the parameter space *m* : {6, …, 10}. ##### Simple linear regression model We assume the following noise model, ![Formula][15] where *I**i* is the number of new infections on day *i, C**i* is the measurement of viral RNA in the wastewater on the *i*th day, and *ϵ**i* is a random residual associated with day *i* which is assumed to be distributed as *N* (0, *σ*2), with *σ*2 as the residual variance. This inference problem aims to estimate *θ* = (*β, σ*) from WW data and cases. A log-linear model is assumed to address positively skewed data and prevent negative fitted values. ### 2.7 Selection of model training periods We describe whether or not testing is adequate in a particular period of observed cases by calculating the rate of change in tests conducted and new cases within a specific time period. We define the rate of change of both tests conducted ![Graphic][16] and confirmed positive cases ![Graphic][17] during period *i* as ![Graphic][18] and ![Graphic][19], respectively, where *T**i−*1, *T**i* denote the number of tests carried out in two consecutive periods, and *c**i−*1, *c**i* denote the number of positive cases detected in these periods. We classify a testing period as adequate when the rate of change in testing is greater than the rate of change in cases; otherwise, if the rate of change in testing is lower/equal to the rate of change in cases, we conclude that testing is inadequate. We summarize various scenarios of testing adequacy in Table 1. Our determination of testing adequacy, and thus suitability for model training for both linear and deconvolution models, assumes that observed cases would be sufficiently close to actual cases when testing rates are high compared to case rates and test positivity remains low as determined through the community screening programs. View this table: [Table 1:](http://medrxiv.org/content/early/2022/07/18/2022.07.16.22276772/T1) Table 1: Test and case scenarios to assess adequacy in testing for training periods. ### 2.8 Effective reproductive number The number of people in a population who are susceptible to infection by an infected individual at any particular time is denoted by *R**e*, the effective reproductive number. This dimensionless quantity is sensitive to time-dependent variation due to reductions in susceptible individuals, changes in population immunity, and other factors. *R**e* can be estimated by the ratio of the number of new infections (*I**t*) generated at time *t*, to the total infectious individuals at time *t*, given by ![Graphic][20], the sum of infection incidence up to time step *t* − 1, weighted by the infectivity function *w**s*. We implemented Cori et al.’s approach 39 to estimate *R**e* directly from observed cases and from cases that were estimated from the WW data. ## 3 Results ### 3.1 Identification of adequate training periods We computed the rate of change in the number of tests and cases by week for the City of Davis between December 1, 2021, and March 31, 2022 (Figure 2 and Table S6). Each week was compared with a previous week and classified as adequate whenever the rate of change in tests was greater than the rate of change in cases and as not adequate otherwise. ![Figure 2:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/07/18/2022.07.16.22276772/F2.medium.gif) [Figure 2:](http://medrxiv.org/content/early/2022/07/18/2022.07.16.22276772/F2) Figure 2: Number of tests administered in the City of Davis (cyan line) and cases (black line) by week, on a log-scale, from December 1, 2021 to March 31, 2022. The week-to-week rate of change in cases and tests are displayed; green numbers indicate the test rate is greater than the case rate, and red numbers are the opposite. The green and red shaded region correspond training periods with Adequate (*T**A*) and Not Adequate (*T**NA*) testing, respectively. Figure 2 illustrates two specific training periods assumed for the analysis of the City of Davis. The first training period includes data from December 12, 2021 to January 8, 2022 (red region, denoted by *T**NA*), and the second training period assumes data from January 9 to February 2, 2022 (green region, *T**A*). A training period designated by *T**NA* (Not Adequate) corresponds to a scenario where test rate is consistently lower than the rate of new cases. Similarly, a training period denoted by *T**A* corresponds to a scenario in which the testing rate exceeds the rate of new cases. We assess testing adequacy for Woodland and UC Davis (Figures S1 and S3), and similarly identified a period of inadequate testing prior to an observed surge in infections.. ### 3.2 Comparison of models to estimate public health metrics from wastewater data We applied both a deconvolution technique and a linear regression to reconstruct incident cases of COVID-19 from the WW data, assuming model training periods according to the adequacy of clinical testing efforts. We found that the magnitude of case projections and trends was sensitive to the assumption of the model training period for both model constructs (Figure 3), but the timing of peaks in cases predicted were independent of the training period. Case predictions from the models that assumed a training period with inadequate testing (*T**NA*) were consistently lower than projections from the models that assumed a training period with adequate testing (*T**A*). These results suggest that models using *T**NA* systematically underestimated true case counts, a finding consistent with our expectations since fewer cases are detected during periods of inadequate clinical testing than when testing is adequate. Projection of cases from the models using *T**A* aligned more consistently with observed cases in periods where testing was deemed adequate. The difference in predictions in cases from the two training periods was particularly evident in January 2021, during the onset of the Omicron variant surge in Davis. ![Figure 3:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/07/18/2022.07.16.22276772/F3.medium.gif) [Figure 3:](http://medrxiv.org/content/early/2022/07/18/2022.07.16.22276772/F3) Figure 3: Predicted cases assuming (A) the deconvolution and (B) the linear regression models from WW data between December 1, 2021, and March 31, 2022. The estimated cases are displayed in green(blue) for the deconvolution(linear) model, when the models were trained on the period classified as adequate (*T**A*), and in red when both models were trained on the period classified as inadequate (*T**NA*). Solid lines represent median estimates of cases, and 95% prediction intervals are depicted in shaded regions. Incident case projections from the linear model that assumed *T**A* was able to capture the peak of the curve more closely than the deconvolution model, although with greater uncertainty. It is worth noting that results from the deconvolution and the linear models are similar because the linear model is fitted with the 7-day moving-average of case data. Data smoothing of this kind corresponds to a convolution with equal daily weights. The estimation of cases from the linear regression model that *T**NA* was similar to the results of the deconvolution model, Figure 3B. *R**e* monitors changes in disease transmission over time, assesses the effectiveness of interventions, and can be useful to guide policy decision making. Estimates of *R**e* from the median of the predicted cases using the deconvolution model and the linear model are similar. In most of the time period assessed, *R**e* determined from WW results are quite similar in magnitude and follow the trends for *R**e* calculated using observed cases (Figure 4B). A notable difference between the *R**e* estimated with the observed cases and that obtained with the WW data with both the linear model and the deconvolution model occurs at the end of March. At this time, the median of *R**e* for cases is below 1, and the median of *R**e* with WW data for the linear and deconvolution model are above 1. A value of *R**e* lower than 1 indicates that overall transmission is declining, while thresholds higher than 1 suggest that an outbreak is expected to continue. 39 The posterior analysis using the estimated *R**e* from WW data thus indicates that an outbreak may have occurred in Davis that was not detected through clinical cases Figure S5. ![Figure 4:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/07/18/2022.07.16.22276772/F4.medium.gif) [Figure 4:](http://medrxiv.org/content/early/2022/07/18/2022.07.16.22276772/F4) Figure 4: (A) Cases predicted with the linear regression model (blue) and the deconvolution model (green) using WW data between December 1, 2021, and March 31, 2022. Both models were trained in the period classified as appropriate (*T**A*). (B) Effective *R**e* of city of Davis computed with the median of the cases estimated for the deconvolution model (blue) and the linear regression (green). We demonstrate the adaptability of our methodology using data for Woodland and UC Davis and present results in the Supplemental Material (Figures S1 and S3). The trends of the observed cases is recovered with both models (Figures S2 and S4), yielding results consistent with those obtained for Davis. The code implemented for the study are available in the github repository **??**. Analyses were carried out using Python version 3. ## 4 Discussion Community-wide testing has played a critical role in mitigating the COVID-19 pandemic. However, large-scale testing has been limited and falls further behind during surges of infections. We developed criteria to classify the adequacy of clinical testing in a community through time, and we applied the classification scheme to three Northern California communities. As was observed in many other communities at the time, we found that clinical testing was inadequate at the front end of the wave of Omicron infections that occurred during our study period. Inadequate clinical testing during surges of infection makes it particularly challenging to discern true levels of SARS-CoV-2 infections in a population. WBE can fill data gaps caused by inadequate testing programs. As clinical testing transitions further towards at-home self-testing, measurements of SARS-CoV-2 RNA in wastewater can serve as an increasingly important indicator for COVID-19 transmission. Myriad sources of variability and uncertainty in WW data can nevertheless impact the accuracy of estimates of COVID-19 cases or other public health metrics derived from WW data. 40–43 Statistically representative samples can also be difficult to obtain because of the complexity of wastewater collection systems and the physical challenge of ensuring consistency in sample acquisition and processing. 44 Such challenges can limit comparability of wastewater results across different WBE programs. The modeling framework we described to estimate COVID-19 cases from WW data accounts for uncertainty and relies on short training periods using clinical testing data to calibrate wastewater measurements to local conditions. We showed that case projections reconstructed from either the Bayesian deconvolution or the simple linear model were generally higher than cases observed through clinical testing, particularly during periods with sub-optimal testing. These results are not surprising, as we expected that the WW models would yield case estimates higher than cases observed through screening given that WW is not subject to the same selection biases as testing. While both the deconvolution and linear regression models captured the overall trends in observed cases overall, qualitative differences were evident between the approaches, particularly when testing was limited. Both models identified steep upward trends in cases during the surge in mid-January and at the onset of the Omicron surge Figure 4A. The classification approach that we developed to assess the adequacy of model training periods was essential to providing robust estimates of case projections. Training periods that satisfied the proposed characteristics (i.e., adequate testing) resulted in similar estimates from each of the models, and yielded trends that were consistent with observed cases. Case projections that assumed training periods with poor testing generally underestimated cases compared to projections from adequate training periods. While the proposed models do not seek to recover the curve of reported cases (an underestimate of actual cases specially in a limited testing scenario), use of adequate training periods for the WW models enabled us to capture trends in case counts much more closely. It is evident that training periods with inadequate testing introduce a downward bias into the model. WBE has substantially lower resource requirements than mass diagnostic testing, and WW data lack bias from care- and test-seeking behavior in the catchment population. WBE programs that determine COVID-19 public health metrics at the community level can work as a powerful and cost-effective complement to other, more traditional intervention methods. The analytic methods presented here can inform local public health policy and community-level interventions, for instance by helping to assess when initiation of clinical screening programs and non-pharmaceutical interventions are needed. The model can be especially valuable to fill data gaps during surges of infection when clinical testing is inadequate and could be used to assess when estimates of case rates exceed certain thresholds. WBE does come with the inherent challenge of determining the populations being monitored, the effect of which is exacerbated if the population served is highly mobile (e.g., a university campus). In other words, WBE methods for tracking COVID-19 are inherently location-specific, whereas public screening programs are tied to the people they serve. Calibration of wastewater models using clinical data will be most robust in places with minimal mobility or within populations that are adequately described and understood. With screening programs winding down across the United States, finding training periods with adequate testing rates may not be feasible. In such cases, application of the deconvolution model for WBE can still highlight important trends. Where tests are only administered for clinical diagnostics, a periodic sentinel system could be employed to produce sufficient prevalence estimates for training periods. Such a system would recruit a representative sample of the population for repeated testing during a training period to establish a baseline, which then enables the wastewater deconvolution model to track incidence for an extended period of time. The same sentinel group could be called back later when the model needs to be updated to retrain for new situations. WBE surveillance systems should be cognizant that they are not unduly targeting and stigmatizing vulnerable communities. WBE is much less invasive than diagnostic testing and protects individual identities, thereby avoiding the stigmatization of individuals and not requiring individual consent.45 Yet focusing too heavily on public surveillance efforts can negatively influence public perception of those being monitored. 46 Mathematical models that employ machine learning, such as the deconvolution model described herein, must be trained with data sets that are not sampled by biased collection methods, else they may inadvertently reintroduce social biases into the results and contribute to larger inequities in public health. ## Data Availability All data produced in the present study are available upon reasonable request to the authors. ## Human Participant Protection This study was determined to be exempt from institutional review board review by the UC Davis Office of Research. ## Supplementary Material ### Results for City of Woodland Figure S1A illustrates data from Woodland that was used to reconstruct cases from WW. Figure S1B described the smoothed cases and N/PMMoV signals used to provide prediction of cases. Figure S1C describes the training periods used for the analysis; one with adequate testing (shaded in green, denoted by *T**A*) and one in which testing was not adequate (shaded in red, denoted by *T**NA*). ![Figure S1:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/07/18/2022.07.16.22276772/F5.medium.gif) [Figure S1:](http://medrxiv.org/content/early/2022/07/18/2022.07.16.22276772/F5) Figure S1: City of Woodland: (A) Wastewater data (N/PMMoV) and 10-day trimmed average (Smoothed N/PMMov) from December 1, 2021 to March 31, 2022; (B) COVID-19 cases, 7-day moving average (Smoothed cases) for cases, and 10-day trimmed average for WW data (Smoothed N/PMMov); (C) Number of tests administered and cases by week, on a log-scale, from December 1, 2021 to February 1, 2022. The week-to-week rate of change in cases and tests are printed with green labels (shaded-region) corresponding to period of adequate testing and red labels (shaded region) for periods in which testing is not adequate. ![Figure S2:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/07/18/2022.07.16.22276772/F6.medium.gif) [Figure S2:](http://medrxiv.org/content/early/2022/07/18/2022.07.16.22276772/F6) Figure S2: City of Woodland. Reconstruction of incident cases from the deconvolution model and (B) linear regression model using WW data from December 1, 2021 to March 31, 2022. The estimated cases are display (*T**A*: January 13 - February 2, 2022), and red when the models are trained on the period classified as inadequate (*T**NA*: December 11-30, 2021). Solid lines represent median estimates of cases, and 95% prediction intervals are depicted in shaded regions. (C) Predicted cases using the linear regression model (blue) and the deconvolution (green) trained in the period selected as appropriate (*T**A*). (D) Calculation of *R**e* from case data (black) and from cases reconstructed from WW data using the deconvolution model (green) and the linear regression (blue). ### Results for the UC Davis Campus Figure S3A illustrates data from UC Davis campus that was used to reconstruct cases from WW. Figure S3B described the smoothed cases and N/PMMoV signals used to provide prediction of cases. Figure S3C describes the training periods used for the analysis; one with adequate testing (shaded in green, denoted by *T**A*) and one in which testing was not adequate (shaded in red, denoted by *T**NA*). ![Figure S3:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/07/18/2022.07.16.22276772/F7.medium.gif) [Figure S3:](http://medrxiv.org/content/early/2022/07/18/2022.07.16.22276772/F7) Figure S3: UC Davis Campus. (A) Wastewater data (N/PMMoV) and 10-day trimmed average (Smoothed N/PMMov) from December 1, 2021 to March 31, 2022. (B) COVID-19 cases, 7-day moving average (Smoothed cases) for cases, and 10-day trimmed average for WW data (Smoothed N/PMMov). (C) Number of tests administered and cases by week, on a log-scale, from December 1, 2021 to February 1, 2022. The week-to-week rate of change in cases and tests are printed, with green labels (and shaded-region) corresponding to period of adequate testing and red labels (and shaded region) for inadequate testing. ![Figure S4:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/07/18/2022.07.16.22276772/F8.medium.gif) [Figure S4:](http://medrxiv.org/content/early/2022/07/18/2022.07.16.22276772/F8) Figure S4: UC Davis Campus. Reconstruction of incident cases from the deconvolution (A) and (B) linear regression models from WW data between December 1, 2021, and March 31, 2022. The estimated cases are display (*T**A*: January 12 - February 3, 2022), and red when the models are trained on the period classified as inadequate (*T**NA*: December 12, 2021 - January 3, 2022). Solid lines represent median estimates of cases, and 95% prediction intervals are depicted in shaded regions. (C) Predicted cases using the linear regression model (blue) and the deconvolution (green) trained in the period selected as appropriate (*T**A*). (d) Effective Re computed with the median of the cases estimated for the deconvolution model (blue) and the linear regression (green). ![Figure S5:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2022/07/18/2022.07.16.22276772/F9.medium.gif) [Figure S5:](http://medrxiv.org/content/early/2022/07/18/2022.07.16.22276772/F9) Figure S5: City of Davis data. COVID-19 cases, 7-day moving average (Smoothed cases) for cases, and number of tests administered from December 1, 2021 to June 1, 2022. ### Laboratory quality control and data processing The sensitivity of the analytical assay was assessed by determining a limit of detection (LOD) and a limit of blank (LOB) following protocols recommended by the ddPCR manufacturer (Bio-Rad Laboratories, 2020, A practical guide for evaluating detection capability using ddPCR.). Fifteen wastewater samples that were initially screened as negative for SARS-CoV-2 in routine wastewater ddPCR monitoring (i.e., extracts had less than 4 positive droplets in merged wells from duplicate analysis) were used to determine the lowest detectable concentrations in ostensibly blank wastewater samples. Selection of these extracts provided a conservative approach to determining the LOB. The selected extracts were re-analyzed by ddPCR to obtain data for four additional replicates for each sample. A non-parametric (rank order) method was then used to select the LOB, since results from the blank were not normally distributed. The ddPCR number of droplets from individual wells were tabulated from lowest to highest. The LOB was set at the value of the concentration measurement for the rank position corresponding to the 95th percentile, calculated as follows: *Rank* = 0.5 + 0.95∗(number of measurements). Since the calculated rank position was a non-integer value, the rank position was rounded up to provide a more conservative LOB. The theoretical LOD was set as LOB plus two times the standard deviation of all replicate results (Biorad, 2020). The LOD and LOB are reported in Table S5. In terms of droplet numbers in the blank samples, the highest numbers of positive droplets in the merged wells (four replicates) amongst the fifteen blank samples were 6 (N1) and 8 (N2). Since routine wastewater samples were analyzed in duplicate, 3 (N1) and 4 (N2) droplets were set as the cutoff to mark samples below the droplet threshold. Samples were also considered below the droplet threshold if there were fewer N1 and N2 droplets twice the number of droplets in the extraction control blank analyzed on the same day. Runs with an extraction control blank that had > 15 positive droplets in either N1 or N2 were considered contaminated and extracts were re-processed. If samples passed all checks, the relative concentration of N gene was calculated as follows. Duplicate results for each target were merged, and the concentration of each target in the ddPCR reaction was calculated assuming a Poisson distribution using the QXOne Software 1.1.1 Standard Addition (Bio-Rad). The average SARS-CoV-2 RNA concentration in the initial wastewater sample was calculated from the average of the N1 and N2, corrected for sample and reagent volumes used, and reported as genome copies (gc) per mL wastewater. BCoV was detected in 100% of spiked samples, and concentrations of targets were not corrected for BCoV recovery efficiency. The average SARS-CoV-2 RNA concentration (*N*), was divided by the concentration of PMMoV. If N1 or N2 merged droplet counts were below the minimum droplet threshold, the target was excluded from the average concentration. If both N1 and N2 targets were below the droplet threshold, the concentration was reported as 0. View this table: [Table S1:](http://medrxiv.org/content/early/2022/07/18/2022.07.16.22276772/T2) Table S1: RT-ddPCR primers and probes used in this study. View this table: [Table S2:](http://medrxiv.org/content/early/2022/07/18/2022.07.16.22276772/T3) Table S2: Preparation of the duplex One-Step RT-ddPCR reaction. View this table: [Table S3:](http://medrxiv.org/content/early/2022/07/18/2022.07.16.22276772/T4) Table S3: Preparation of 20X primer/probe Mix (p/p Mix). View this table: [Table S4:](http://medrxiv.org/content/early/2022/07/18/2022.07.16.22276772/T5) Table S4: Synthesized gene fragments used for positive controls in ddPCR. View this table: [Table S5:](http://medrxiv.org/content/early/2022/07/18/2022.07.16.22276772/T6) Table S5: Limit of Blank and Limit of Detection. View this table: [Table S6:](http://medrxiv.org/content/early/2022/07/18/2022.07.16.22276772/T7) Table S6: Weekly cases for Davis between October 21, 2021 and March 31. ## Acknowledments M.L.D.T and J.C.M.L., literature review, model development, implementation, wrote overall first draft of paper, paper revisions. M.K., analytical assay development and quality control, writing methods. R.O., laboratory process management, quality control, data collection, writing methods. L.R. and C.W.B.: laboratory processing and wastwater data collection, writing methods. L.T. literature review and initial model evaluation. L.T. and M.S., partner engagement and project management. Y.E.G. and A.S., literature review, paper writing and editing. K.S. and C.N., wastewater monitoring project supervision. Pollock B., Healthy Davis Together project PI, paper editing/revision. M.N., model development, research oversight/supervision, paper editing/revision. H.N.B., Healthy Central Valley Together project PI; project conception, funding, research oversight and collaborator coordination, paper editing/revision. This research was supported by the Healthy Central Valley Together (HCVT) and Healthy Davis Together (HDT) programs at the University of California, Davis. This research was also supported in part by the NIH Rapid Acceleration of Diagnostics (RADxSM) initiative with federal funds from the National Institute of Biomedical Imaging and Bioengineering, National Institutes of Health. The current contract is funded from the Public Health and Social Services Emergency Fund through the Biomedical Advanced Research and Development Authority, HHS Office of the Assistant Secretary for Preparedness and Response, Department of Health and Human Services, under Contract No. 75N92021C00012. All authors reviewed and approved the final manuscript. ## Footnotes * + co-first author * Received July 16, 2022. * Revision received July 16, 2022. * Accepted July 18, 2022. * © 2022, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at [http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/) ## References 1. [1]. Olivier Vandenberg, Delphine Martiny, Olivier Rochas, Alex van Belkum, and Zisis Kozlakidis. Considerations for diagnostic covid-19 tests. Nature Reviews Microbiology, 19(3):171–183, 2021. doi:10.1038/s41579-020-00461-z. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41579-020-00461-z&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33057203&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F07%2F18%2F2022.07.16.22276772.atom) 2. [2]. Angela E Raffle, Allyson M Pollock, and Louisa Harding-Edgar. Covid-19 mass testing programmes. BMJ, 370, 2020. doi:10.1136/bmj.m3262. [FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNzAvYXVnMjBfNS9tMzI2MiI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIyLzA3LzE4LzIwMjIuMDcuMTYuMjIyNzY3NzIuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 3. [3]. Tim R Mercer and Marc Salit. Testing at scale during the covid-19 pandemic. Nature Reviews Genetics, 22:415–426, 2021. doi:10.1038/s41576-021-00360-w. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41576-021-00360-w&link_type=DOI) 4. [4]. Yan Mardian, Herman Kosasih, Muhammad Karyana, Aaron Neal, and Chuen-Yen Lau. Review of current covid-19 diagnostics and opportunities for further development. Frontiers in Medicine, 8, 2021. doi:10.3389/fmed.2021.615099. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3389/fmed.2021.615099&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=34026773&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F07%2F18%2F2022.07.16.22276772.atom) 5. [5]. Ettore Zuccato, Chiara Chiabrando, Sara Castiglioni, Davide Calamari, Renzo Bagnati, Silvia Schiarea, and Roberto Fanelli. Cocaine in surface waters: a new evidence-based tool to monitor community drug abuse. Environmental Health, 4(1):1–7, 2005. doi:10.1186/1476-069X-4-14. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1476-069X-4-14&link_type=DOI) 6. [6]. Sara Castiglioni, Kevin V Thomas, Barbara Kasprzyk-Hordern, Liesbeth Vandam, and Paul Griffiths. Testing wastewater to detect illicit drugs: state of the art, potential and research needs. Science of the Total Environment, 487:613–620, 2014. doi:10.1016/j.scitotenv.2013.10.034. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.scitotenv.2013.10.034&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24963530&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F07%2F18%2F2022.07.16.22276772.atom) 7. [7]. Phil M Choi, Benjamin Tscharke, Saer Samanipour, Wayne D Hall, Coral E Gartner, Jochen F Mueller, Kevin V Thomas, and Jake W O’Brien. Social, demographic, and economic correlates of food and chemical consumption measured by wastewater-based epidemiology. Proceedings of the National Academy of Sciences, 116(43):21864–21873, 2019. doi:10.1073/pnas.1910242116. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoicG5hcyI7czo1OiJyZXNpZCI7czoxMjoiMTE2LzQzLzIxODY0IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDcvMTgvMjAyMi4wNy4xNi4yMjI3Njc3Mi5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 8. [8]. Humayun Asghar, Ousmane M Diop, Goitom Weldegebriel, Farzana Malik, Sushmitha Shetty, Laila El Bassioni, Adefunke O Akande, Eman Al Maamoun, Sohail Zaidi, Adekunle J Adeniji, et al. Environmental surveillance for polioviruses in the global polio eradication initiative. The Journal of Infectious Diseases, 210(suppl 1):S294–S303, 2014. doi:10.1093/infdis/jiu384. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/infdis/jiu384&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25316848&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F07%2F18%2F2022.07.16.22276772.atom) 9. [9]. Juan A. Vallejo, Noelia Trigo-Tasende, Soraya Rumbo-Feal, Kelly Conde-Pérez, Ángel LópezOriona, Inés Barbeito, Manuel Vaamonde, Javier Tarrío-Saavedra, Rubén Reif, Susana Ladra, Bruno K. Rodiño-Janeiro, Mohammed Nasser-Ali, Ángeles Cid, María Veiga, Antón Acevedo, Carlos Lamora, Germán Bou, Ricardo Cao, and Margarita Poza. Modeling the number of people infected with sars-cov-2 from wastewater viral load in northwest spain. Science of The Total Environment, 811:152334, 2022. doi:[https://doi.org/10.1016/j.scitotenv.2021.152334](https://doi.org/10.1016/j.scitotenv.2021.152334). 10. [10]. Jana S. Huisman, Jérémie Scire, Lea Caduff, Xavier Fernandez-Cassi, Pravin Ganesanandamoorthy, Anina Kull, Andreas Scheidegger, Elyse Stachler, Alexandria B. Boehm, Bridgette Hughes, Alisha Knudson, Aaron Topol, Krista R. Wigginton, Marlene K. Wolfe, Tamar Kohn, Christoph Ort, Tanja Stadler, and Timothy R. Julian. Wastewater-based estimation of the effective reproductive number of sars-cov-2. medRxiv, 2022. doi:10.1101/2021.04.29.21255961. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoibWVkcnhpdiI7czo1OiJyZXNpZCI7czoyMToiMjAyMS4wNC4yOS4yMTI1NTk2MXYyIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDcvMTgvMjAyMi4wNy4xNi4yMjI3Njc3Mi5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 11. [11]. Christopher S McMahan, Stella Self, Lior Rennert, Corey Kalbaugh, David Kriebel, Duane Graves, Cameron Colby, Jessica A Deaver, Sudeep C Popat, Tanju Karanfil, and David L Freedman. Covid-19 wastewater epidemiology: a model to estimate infected populations. The Lancet Planetary Health, 5(12):e874–e881, 2021. doi:10.1016/S2542-5196(21)00230-8. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S2542-5196(21)00230-8&link_type=DOI) 12. [12]. Marlene K. Wolfe, Aaron Topol, Alisha Knudson, Adrian Simpson, Bradley White, Duc J. Vugia, Alexander T. Yu, Linlin Li, Michael Balliet, Pamela Stoddard, George S. Han, Krista R. Wigginton, Alexandria B. Boehm, and Charles R. Langelier. High-frequency, high-throughput quantification of sars-cov-2 rna in wastewater settled solids at eight publicly owned treatment works in northern california shows strong association with covid-19 incidence. mSystems, 6(5):e00829–21, 2021. doi:10.1128/mSystems.00829-21. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1128/mSystems.00829-21&link_type=DOI) 13. [13].Summary of global sars-cov-2 wastewater monitoring efforts by uc merced researchers. [https://www.arcgis.com/apps/dashboards/c778145ea5bb4daeb58d31afee389082](https://www.arcgis.com/apps/dashboards/c778145ea5bb4daeb58d31afee389082). (Accessed on 06/14/2022). 14. [14]. Smriti Mallapaty et al. How sewage could reveal true scale of coronavirus outbreak. Nature, 580(7802):176–177, 2020. doi:10.1038/d41586-020-00973-x. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/d41586-020-00973-x&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32246117&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F07%2F18%2F2022.07.16.22276772.atom) 15. [15]. Jordan Peccia, Alessandro Zulli, Doug E Brackney, Nathan D Grubaugh, Edward H Kaplan, Arnau Casanovas-Massana, Albert I Ko, Amyn A Malik, Dennis Wang, Mike Wang, et al. Measurement of sars-cov-2 rna in wastewater tracks community infection dynamics. Nature Biotechnology, 38(10):1164–1167, 2020. doi:10.1038/s41587-020-0684-z. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41587-020-0684-z&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32948856&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F07%2F18%2F2022.07.16.22276772.atom) 16. [16]. Gertjan Medema, Leo Heijnen, Goffe Elsinga, Ronald Italiaander, and Anke Brouwer. Presence of sars-coronavirus-2 rna in sewage and correlation with reported covid-19 prevalence in the early stage of the epidemic in the netherlands. Environmental Science & Technology Letters, 7(7):511–516, 2020. doi:10.1021/acs.estlett.0c00357. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1021/acs.estlett.0c00357&link_type=DOI) 17. [17]. S Wurtzer, V Marechal, JM Mouchel, Y Maday, R Teyssou, E Richard, JL Almayrac, and L Moulin. Evaluation of lockdown impact on sars-cov-2 dynamics through viral genome quantification in paris wastewaters. medRxiv, 2020. doi:10.1101/2020.04.12.20062679. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoibWVkcnhpdiI7czo1OiJyZXNpZCI7czoyMToiMjAyMC4wNC4xMi4yMDA2MjY3OXYyIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDcvMTgvMjAyMi4wNy4xNi4yMjI3Njc3Mi5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 18. [18]. Shimoni Shah, Sylvia Xiao Wei Gwee, Jamie Qiao Xin Ng, Nicholas Lau, Jiayun Koh, and Junxiong Pang. Wastewater surveillance to infer covid-19 transmission: A systematic review. Science of The Total Environment, 804:150060, 2022. doi:10.1016/j.scitotenv.2021.150060. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.scitotenv.2021.150060&link_type=DOI) 19. [19]. Amy E Kirby, Rory M Welsh, Zachary A Marsh, Alexander T Yu, Duc J Vugia, Alexandria B Boehm, Marlene K Wolfe, Bradley J White, Shannon R Matzinger, Allison Wheeler, et al. Notes from the field: Early evidence of the sars-cov-2 b. 1.1. 529 (omicron) variant in community wastewater—united states, november–december 2021. Morbidity and Mortality Weekly Report, 71(3):103–105, 2022. doi:10.15585/mmwr.mm7103a5. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.15585/mmwr.mm7103a5&link_type=DOI) 20. [20]. Leanne Pillay, Isaac Dennis Amoah, Nashia Deepnarain, Kriveshin Pillay, Oluyemi Olatunji Awolusi, Sheena Kumari, and Faizal Bux. Monitoring changes in covid-19 infection using wastewater-based epidemiology: A south african perspective. Science of The Total Environment, 786:147273, 2021. doi:[https://doi.org/10.1016/j.scitotenv.2021.147273](https://doi.org/10.1016/j.scitotenv.2021.147273). 21. [21]. Aikaterini Galani, Reza Aalizadeh, Marios Kostakis, Athina Markou, Nikiforos Alygizakis, Theodore Lytras, Panagiotis G Adamopoulos, Jordan Peccia, David C Thompson, Aikaterini Kontou, et al. Sars-cov-2 wastewater surveillance data can predict hospitalizations and icu admissions. Science of The Total Environment, 804:150151, 2022. doi:10.1016/j.scitotenv.2021.150151. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.scitotenv.2021.150151&link_type=DOI) 22. [22]. Warish Ahmed, Aaron Bivins, Paul M Bertsch, Kyle Bibby, Phil M Choi, Kata Farkas, Pradip Gyawali, Kerry A Hamilton, Eiji Haramoto, Masaaki Kitajima, et al. Surveillance of sars-cov-2 rna in wastewater: Methods optimization and quality control are crucial for generating reliable public health information. Current Opinion in Environmental Science & Health, 17:82–93, 2020. doi:10.1016/j.coesh.2020.09.003. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.coesh.2020.09.003&link_type=DOI) 23. [23]. Gertjan Medema, Frederic Been, Leo Heijnen, and Susan Petterson. Implementation of environmental surveillance for sars-cov-2 virus to support public health decisions: Opportunities and challenges. Current Opinion in Environmental Science & Health, 17:49–71, 2020. doi:10.1016/j.coesh.2020.09.006. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.coesh.2020.09.006&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33024908&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F07%2F18%2F2022.07.16.22276772.atom) 24. [24]. Kelly Hill, Arash Zamyadi, Dan Deere, Peter A. Vanrolleghem, and Nicholas D. Crosbie. SARS-CoV-2 known and unknowns, implications for the water sector and wastewater-based epidemiology to support national responses worldwide: early review of global experiences with the COVID-19 pandemic. Water Quality Research Journal, 56(2):57–67, 05 2020. doi:10.2166/wqrj.2020.100. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2166/wqrj.2020.100&link_type=DOI) 25. [25]. Yifan Zhu, Wakana Oishi, Chikako Maruo, Mayuko Saito, Rong Chen, Masaaki Kitajima, and Daisuke Sano. Early warning of covid-19 via wastewater-based epidemiology: potential and bottlenecks. Science of The Total Environment, 767:145124, 2021. doi:10.1016/j.scitotenv.2021.145124. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.scitotenv.2021.145124&link_type=DOI) 26. [26]. Sílvia Monteiro, Daniela Rente, Mónica V. Cunha, Manuel Carmo Gomes, Tiago A. Marques, Artur B. Lourenço, Eugénia Cardoso, Pedro Álvaro, Marco Silva, Norberta Coelho, João Vilaça, Fátima Meireles, Nuno Brôco, Marta Carvalho, and Ricardo Santos. A wastewater-based epidemiology tool for covid-19 surveillance in portugal. Science of The Total Environment, 804:150264, 2022. doi:10.1016/j.scitotenv.2021.150264. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.scitotenv.2021.150264&link_type=DOI) 27. [27]. Mary E Schoen, Marlene K Wolfe, Linlin Li, Dorothea Duong, Bradley J White, Bridgette Hughes, and Alexandria B Boehm. Sars-cov-2 rna wastewater settled solids surveillance frequency and impact on predicted covid-19 incidence using a distributed lag model. ACS ES&T Water, 2022. doi:10.1021/acsestwater.2c00074. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1021/acsestwater.2c00074&link_type=DOI) 28. [28]. Smruthi Karthikeyan, Nancy Ronquillo, Pedro Belda-Ferre, Destiny Alvarado, Tara Javidi, Christopher A. Longhurst, Rob Knight, and Ileana M. Cristea. High-throughput wastewater sars-cov-2 detection enables forecasting of community infection dynamics in san diego county. mSystems, 6(2):e00045–21, 2021. doi:10.1128/mSystems.00045-21. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1128/mSystems.00045-21&link_type=DOI) 29. [29].Healthy Davis Together. Testing. [https://healthydavistogether.org/testing-data/](https://healthydavistogether.org/testing-data/). (Accessed on 05/23/2022). 30. [30].Healthy yolo together. [https://healthydavistogether.org/](https://healthydavistogether.org/). (Accessed on 05/23/2022). 31. [31]. Anne Weiss, Mads Jellingsø, and Morten Otto Alexander Sommer. Spatial and temporal dynamics of sars-cov-2 in covid-19 patients: A systematic review and meta-analysis. EBioMedicine, 58:102916, 2020. doi:10.1016/j.ebiom.2020.102916. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.ebiom.2020.102916&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32711256&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F07%2F18%2F2022.07.16.22276772.atom) 32. [32]. Muge Cevik, Matthew Tate, Ollie Lloyd, Alberto Enrico Maraolo, Jenna Schafers, and Antonia Ho. Sars-cov-2, sars-cov-1 and mers-cov viral load dynamics, duration of viral shedding and infectiousness: a living systematic review and meta-analysis. SARS-CoV-1 and MERS-CoV Viral Load Dynamics, Duration of Viral Shedding and Infectiousness: A Living Systematic Review and Meta-Analysis, 2020. doi:10.2139/ssrn.3677918. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2139/ssrn.3677918&link_type=DOI) 33. [33]. Cecilia LH Xu, Manjri Raval, Jesse A Schnall, Jason C Kwong, and Natasha E Holmes. Duration of respiratory and gastrointestinal viral shedding in children with sars-cov-2: a systematic review and synthesis of data. The Pediatric Infectious Disease Journal, 39(9):e249–e256, 2020. doi:10.1097/INF.0000000000002814. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/INF.0000000000002814&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F07%2F18%2F2022.07.16.22276772.atom) 34. [34]. Edward Goldstein, Jonathan Dushoff, Junling Ma, Joshua B Plotkin, David JD Earn, and Marc Lipsitch. Reconstructing influenza incidence by deconvolution of daily mortality time series. Proceedings of the National Academy of Sciences, 106(51):21825–21829, 2009. doi:10.1073/pnas.0902958106. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoicG5hcyI7czo1OiJyZXNpZCI7czoxMjoiMTA2LzUxLzIxODI1IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDcvMTgvMjAyMi4wNy4xNi4yMjI3Njc3Mi5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 35. [35]. Andreas Lindén and Samu Mäntyniemi. Using the negative binomial distribution to model overdispersion in ecological count data. Ecology, 92(7):1414–1421, 2011. doi:10.1890/10-1831.1. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1890/10-1831.1&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21870615&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F07%2F18%2F2022.07.16.22276772.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000292814300005&link_type=ISI) 36. [36]. Amy E Benefield, Laura A Skrip, Andrea Clement, Rachel A Althouse, Stewart Chang, and Benjamin M Althouse. Sars-cov-2 viral load peaks prior to symptom onset: a systematic review and individual-pooled analysis of coronavirus viral load from 66 studies. medRxiv, 2020. doi:10.1101/2020.09.28.20202028. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoibWVkcnhpdiI7czo1OiJyZXNpZCI7czoyMToiMjAyMC4wOS4yOC4yMDIwMjAyOHYxIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDcvMTgvMjAyMi4wNy4xNi4yMjI3Njc3Mi5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 37. [37]. J Andrés Christen, Colin Fox, et al. A general purpose sampling algorithm for continuous distributions (the t-walk). Bayesian Analysis, 5(2):263–281, 2010. doi:10.1214/10-BA603. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1214/10-BA603&link_type=DOI) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000278452200006&link_type=ISI) 38. [38]. David J Spiegelhalter, Nicola G Best, Bradley P Carlin, and Angelika Van Der Linde. Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series b (Statistical Methodology), 64(4):583–639, 2002. doi:10.1111/1467-9868.00353. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/1467-9868.00353&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=617541238146&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F07%2F18%2F2022.07.16.22276772.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000179221100001&link_type=ISI) 39. [39]. Anne Cori, Neil M Ferguson, Christophe Fraser, and Simon Cauchemez. A new framework and software to estimate time-varying reproduction numbers during epidemics. American Journal of Epidemiology, 178(9):1505–1512, 2013. doi:10.1093/aje/kwt133. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/aje/kwt133&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24043437&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F07%2F18%2F2022.07.16.22276772.atom) 40. [40]. Xuan Li, Shuxin Zhang, Jiahua Shi, Stephen P Luby, and Guangming Jiang. Uncertainties in estimating sars-cov-2 prevalence by wastewater-based epidemiology. Chemical Engineering Journal, 415:129039, 2021. doi:10.1016/j.cej.2021.129039. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.cej.2021.129039&link_type=DOI) 41. [41]. Rezgar Arabzadeh, Daniel Martin Grünbacher, Heribert Insam, Norbert Kreuzinger, Rudolf Markt, and Wolfgang Rauch. Data filtering methods for sars-cov-2 wastewater surveillance. Water Science and Technology, 84(6):1324–1339, 2021. doi:10.2166/wst.2021.343. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2166/wst.2021.343&link_type=DOI) 42. [42]. E Belia, Y Amerlinck, Lorenzo Benedetti, B Johnson, Gürkan Sin, Peter A Vanrolleghem, KV Gernaey, S Gillot, MB Neumann, L Rieger, et al. Wastewater treatment modelling: dealing with uncertainties. Water Science and Technology, 60(8):1929–1941, 2009. doi:10.2166/wst.2009.225. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoicHBpd2F3c3QiO3M6NToicmVzaWQiO3M6OToiNjAvOC8xOTI5IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDcvMTgvMjAyMi4wNy4xNi4yMjI3Njc3Mi5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 43. [43]. Marie Courbariaux, Nicolas Cluzel, Siyun Wang, Vincent Maréchal, Laurent Moulin, Sébastien Wurtzer, Obépine consortium, Jean-Marie Mouchel, Yvon Maday, and Grégory Nuel. A flexible smoother adapted to censored data with outliers and its application to sars-cov-2 monitoring in wastewater. arXiv, 2022. URL: [https://arxiv.org/abs/2108.02115v3](https://arxiv.org/abs/2108.02115v3). 44. [44]. Deepak Panchal, Om Prakash, Prakash Bobde, and Sukdeb Pal. Sars-cov-2: sewage surveillance as an early warning system and challenges in developing countries. Environmental Science and Pollution Research, 28(18):22221–22240, 2021. doi:[doi.org/10.1007/s11356-021-13170-8](http://doi.org/10.1007/s11356-021-13170-8). 45. [45]. Michio Murakami, Akihiko Hata, Ryo Honda, and Toru Watanabe. Letter to the editor: Wastewater-based epidemiology can overcome representativeness and stigma issues related to covid-19. Environmental Science & Technology, 54(9):5311–5311, 2020. PMID: 32323978. doi:10.1021/acs.est.0c02172. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1021/acs.est.0c02172&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32323978&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F07%2F18%2F2022.07.16.22276772.atom) 46. [46].Natalie Sims and Barbara Kasprzyk-Hordern. Future perspectives of wastewater-based epidemiology: Monitoring infectious disease spread and resistance to the community level. Environment International, 139:105689, 2020. doi:10.1016/j.envint.2020.105689. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.envint.2020.105689&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32283358&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F07%2F18%2F2022.07.16.22276772.atom) 47. [47].Cdc 2019-novel coronavirus (2019-ncov) real-time rt-pcr diagnostic panel. [https://www.fda.gov/media/134922/download](https://www.fda.gov/media/134922/download). 48. [48]. Nicola Decaro, Gabriella Elia, Marco Campolo, Costantina Desario, Viviana Mari, Arianna Radogna, Maria Loredana Colaianni, Francesco Cirone, Maria Tempesta, and Canio Buonavoglia. Detection of bovine coronavirus using a taqman-based real-time rt-pcr assay. Journal of Virological Methods, 151(2):167–171, 2008. doi:10.1016/j.jviromet.2008.05.016. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jviromet.2008.05.016&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=18579223&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F07%2F18%2F2022.07.16.22276772.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000258307200001&link_type=ISI) 49. [49]. Eiji Haramoto, Masaaki Kitajima, Naohiro Kishida, Yoshiaki Konno, Hiroyuki Katayama, Mari Asami, and Michihiro Akiba. Occurrence of pepper mild mottle virus in drinking water sources in japan. Applied and Environmental Microbiology, 79(23):7413–7418, 2013. doi:10.1128/AEM.02354-13. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYWVtIjtzOjU6InJlc2lkIjtzOjEwOiI3OS8yMy83NDEzIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDcvMTgvMjAyMi4wNy4xNi4yMjI3Njc3Mi5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) [1]: /embed/graphic-2.gif [2]: /embed/graphic-3.gif [3]: /embed/inline-graphic-1.gif [4]: /embed/graphic-4.gif [5]: /embed/graphic-5.gif [6]: /embed/inline-graphic-2.gif [7]: /embed/inline-graphic-3.gif [8]: /embed/inline-graphic-4.gif [9]: /embed/inline-graphic-5.gif [10]: /embed/inline-graphic-6.gif [11]: /embed/graphic-6.gif [12]: /embed/inline-graphic-7.gif [13]: /embed/inline-graphic-8.gif [14]: /embed/graphic-7.gif [15]: /embed/graphic-8.gif [16]: /embed/inline-graphic-9.gif [17]: /embed/inline-graphic-10.gif [18]: /embed/inline-graphic-11.gif [19]: /embed/inline-graphic-12.gif [20]: /embed/inline-graphic-13.gif