Mixed methods approach to examining the implementation experience of a phone-based health research survey investigating risk factors for SARS-CoV-2 infection in California =========================================================================================================================================================================== * Nozomi Fukui * Sophia S. Li * Jennifer DeGuzman * Jennifer F. Myers * John Openshaw * Anjali Sharma * James Watt * Joseph A. Lewnard * Seema Jain * Kristin L. Andrejko * Jake M. Pry * the California COVID-19 Case-Control Study Team ## Abstract **Objective** To describe the implementation of a test-negative design case-control study in California during the Coronavirus Disease 2019 (COVID-19) pandemic. **Methods** Between February 24, 2021 - February 24, 2022, 34 interviewers called 38,470 SARS-CoV-2-tested Californians to enroll 1,885 cases and 1,871 controls in a 20-minute telephone survey. We estimated adjusted odds ratios for answering the phone and consenting to participate using mixed effects logistic regression. We used a web-based anonymous survey to compile interviewer experiences. **Results** Cases had 1.29-fold (95% CI: 1.24–1.35) higher adjusted odds of answering the phone and 1.69-fold (1.56–1.83) higher adjusted odds of consenting to participate compared to controls. Calls placed from 4pm to 6pm had the highest adjusted odds of being answered. Interviewers who faced participants with dire need for social services or harassment experienced poor mental health. **Conclusions** We suggest calling during afternoons and allocating more effort towards enrolling controls when designing a case-control study. Remaining adaptive to the dynamic needs of the team is critical to a successful study, especially in a pandemic setting. ## INTRODUCTION The Coronavirus Disease 2019 (COVID-19) pandemic called for a rapid mobilization of public health research to inform public health policy.1 Observational studies have played critical roles in defining COVID-19 epidemiology by identifying risk factors for infection and estimating the effectiveness of vaccines and other mitigation strategies.2–7 Many observational studies conducted during the COVID-19 pandemic utilized remote technologies, such as phone surveys, to safely enroll participants, however, unknown willingness to participate on this platform may pose unique challenges.8–12 Understanding participation patterns in phone surveys conducted during the pandemic may help optimize the implementation of future epidemiologic studies by informing sampling design strategies. Prior to the COVID-19 pandemic, determinants of participation in phone surveys varied by disease, age, and time of day.11,13–15 Individuals who have or individuals adjacent to person(s) who have a history of disease are more likely to participate than individuals who are unaffected.11 Younger people may be more willing to answer an unknown caller, but less willing to participate in a survey with a public health official that involves disclosing sensitive information such as their recent contacts.14 Additionally, the time of day that individuals are called may also influence participation in the survey.15 The polarization of public health throughout the pandemic, including increasingly negative attitudes towards contact tracing, may limit willingness to participate in phone-based COVID-19 research.16–19 In the novel, dynamic context of the COVID-19 pandemic, identification of predictors of participation in observational studies using remote technologies are limited. Additionally, public health professionals report experiencing harassment, substantial mental health burdens, and burnout during the pandemic.20 Details regarding the toll of emotionally demanding research on researchers, especially novice researchers, is scant.21–23 We describe the implementation of a phone-based, test-negative design SARS-CoV-2 case-control study in California during the COVID-19 pandemic. We estimate predictors of answering the phone, subsequently enrolling in the study, and identify reasons for declining survey participation. Furthermore, we provide qualitative description of interviewer experiences to identify successes and gaps in staff support systems. These components are critical to successful and ethical study implementation and can inform the design of future epidemiologic studies conducted throughout the COVID-19 pandemic or similar pandemic settings. ## METHODS ### Study design and enrollment process We reviewed data collected from February 24, 2021 to February 24, 2022 by the California Department of Public Health (CDPH) test-negative design case-control study that evaluated risk factors for SARS-CoV-2 infection (**Figure S1**).6,7 Potential case and control participants were defined as individuals with a positive and negative laboratory-confirmed SARS-CoV-2 test result, respectively. Cases and controls were individually matched by age group, birth sex, multi-county region, and SARS-CoV-2 test result window (≤7-day difference). Throughout the study period, trained interviewers used soft-phone technology with a California area code to call a list of individuals and facilitate a 20-minute survey in English or Spanish (**Figure S2**). A guiding script accompanied the semi-structured electronic survey instrument to standardize the experience of study participants (**Item S1**). Before procuring consent, interviewers alerted potential participants of the estimated 20 minutes to complete the survey. Individuals were eligible to participate if they could provide informed consent and reported no clinical diagnosis of COVID-19 or positive test result for SARS-CoV-2 infection prior to their most recent test result. From January 6, 2022, as at-home test use increased, those with a previous (within 2 days) positive result using an at-home test became eligible. If not capable of answering questions, enrollment proceeded if a proxy respondent was available, and the potential participant gave informed consent both to participation and to have the proxy respondent answer on their behalf. For minors (<18 years old), the parent/guardian provided informed consent. Interviewers were encouraged to enroll a case, followed by 30 or more calls to matched controls, in a repeated case-control pair format. If unsuccessful in completing a survey with a matched control within their shift, interviewers posted a request in an instant messaging platform for other interviewers to attempt enrollment in subsequent shifts. To limit recall bias, cases were excluded in the primary analysis if not matched by the 7th day. Interviewers documented the outcome (no answer, no consent, partial survey, completed survey) of each call and noted reasons for refusing participation or ending the call early. The State of California, Health and Human Services Agency, Committee for the Protection of Human Subjects (Project Number: 2021-034) approved the study protocol. ### Implementation infrastructure Interviewers collected data daily (excluding holidays) for an average of 10 hours per week. Research associates, promoted from interviewer roles, helped fulfill administrative and research support tasks such as maintaining databases, managing interviewer training, assigning call lists, facilitating weekly meetings, monitoring enrollment, and cultivating community. A communication platform provided live support to interviewers who encountered questions during surveys and served as an option for private communication, group communication, and team-sourced support. Supervisors monitored the platform daily to ensure that all questions received timely responses. The online messaging platform allowed for consolidation of high volume and frequency of communication and positioned the team to easily deliver critical updates, solicit feedback on survey implementation, and detect any issues quickly. The team met weekly to discuss study enrollment progress, check in on wellbeing, highlight interviewer accomplishments, and announce protocol or survey updates. Supervisors offered professional development opportunities during these meetings such as presentations from various public health professionals and workshops covering relevant skills and topics. Interviewers intermittently encountered difficult conversations with participants. Team-wide, small-group, and 1-on-1 discussions about wellbeing recurred throughout the year to debrief difficult experiences, and mental wellness resources, including counseling and general support conferences across CDPH COVID-19 response sections, were advertised and encouraged. ### Interviewer recruitment and training Interviewers were recruited from undergraduate and graduate institutions with pay (**Table S1**). Successful candidates demonstrated strong empathy, patience, good communication, interest in public health or a related field, and had prior customer service, data collection, or healthcare experience. Interviewers were required to complete a rigorous training program to ensure that they were well prepared for challenging interviews and collecting high-quality data (**Figure 1**). Due to high interviewer turnover in the first three months of the study, multiple hiring sessions occurred. With successive rounds of interviewer on-boarding we implemented a train-the-trainer approach, empowering experienced interviewers to mentor newer interviewers and respond to questions in communal message channels. ![FIGURE 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/01/2023.02.27.23286454/F1.medium.gif) [FIGURE 1.](http://medrxiv.org/content/early/2023/03/01/2023.02.27.23286454/F1) FIGURE 1. Process diagram for recruitment, onboarding, and training interviewers ### Quantitative methods We define three cohorts representing different call outcomes: 1) individuals who answered the phone, 2) eligible individuals who consented, and 3) eligible individuals who refused to participate in the study. To estimate determinants of participation, we estimated the adjusted odds ratio of answering the phone, consenting to participate, and citing time as a reason for not participating using mixed effects logistic regression. Models included age group, birth sex, region, SARS-CoV-2 infection status, month, time of day and time of week contacted as fixed effects and allowed random effects at the interviewer level. Additionally, we assessed interaction effects between predictors by SARS-CoV-2 infection status and between time of day and time of week. Model fitting was performed to compare fit with and without inclusion of interaction terms using the Bayesian Information Criterion (**Table S2**). All analyses were conducted with R software (version 4.1.3; R Foundation for Statistical computing). We used lme4 packages for analyses. ### Qualitative methods From June 29 through July 12, 2022, we used an anonymous, self-administered, web-based survey to contextualize quantitative results with interviewer experiences (**Item S2**). All interviewers involved with the study were invited to participate. We compensated active interviewers for the time spent on their responses. We also reviewed weekly meeting notes for identification of themes. ## RESULTS During the study period, we placed 38,470 calls including 15,154 (39.4%) to cases and 23,316 (60.6%) to controls (**Table 1, Figure S3**). Among the cases and controls called, 35.5% (5,383/15,154) and 31.3% (7,289/23,316) answered the phone, respectively. Of those who answered the phone, 37.2% (2,004/5,383) and 27.3% (1,991/7,289) consented to participate. Ultimately, 1,885 cases and 1,871 controls completed the survey. Over time, survey completion declined for both cases and controls despite change in calling rate (**Figure 2**). View this table: [Table 1.](http://medrxiv.org/content/early/2023/03/01/2023.02.27.23286454/T1) Table 1. Characteristics of SARS-CoV-2 test-seekers in California who were called, answered the phone, consented to participate, and completed the telephone survey. See Table S8 for comparison to 2020 U.S. Census Bureau American Community Survey demographics ![FIGURE 2.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/01/2023.02.27.23286454/F2.medium.gif) [FIGURE 2.](http://medrxiv.org/content/early/2023/03/01/2023.02.27.23286454/F2) FIGURE 2. Study timeline mapped against weekly enrollment trends by case-control status On average, interviewers placed 8 calls per case and 13 calls per control to complete a survey. Parents or guardians of children aged 0 to 4 years required fewer calls to complete a survey compared to participants in other age categories (7 calls per case and 10 calls per control) (**Figure S4**). The greatest number of calls to complete a survey for a potential participant occurred between 8am to 11am (10 calls per case and 15 calls per control). On average, the weekly calls per completed survey for a control increased over time while remaining relatively steady for cases (**Figure S5**). During the study, there were three hiring rounds and a total of 34 interviewers. Interviewers were, on average, active for 23 weeks. 17.6% (6/34) remained active for 34-52 weeks. 32.4% (11/34) of interviewers responded to the anonymous qualitative experience survey. ### Predictors of answering the phone and consenting to participate We found SARS-CoV-2 infection status, age, region, time of day called, and time of week called were significantly associated with answering the phone. Cases were more likely (aOR: 1.29 [95% CI: 1.24–1.35]) to answer the phone than controls (**Figure 3**). The likelihood of answering the phone was lowest among older individuals. Individuals from Central Coast (aOR: 0.77 [95% CI: 0.70, 0.85]), San Joaquin Valley (0.84 [0.77, 0.92]), and Northwestern California (0.84 [0.76, 0.93]) had the lowest adjusted odds of answering when compared to individuals in the Bay Area (**Table S3**). Calls placed after 6pm (aOR: 0.79 [0.68, 0.90]) and between 8 to 11am (0.84 [0.79–0.90]) were associated with the lowest adjusted odds of answering the phone when compared to calls placed between 4 to 6pm. ![FIGURE 3.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/01/2023.02.27.23286454/F3.medium.gif) [FIGURE 3.](http://medrxiv.org/content/early/2023/03/01/2023.02.27.23286454/F3) FIGURE 3. Predictors of participants answering the telephone and consenting to participate in the California COVID-19 Case Control study. We did not observe significant interaction between SARS-CoV-2 infection status and the adjusted odds of consenting to participate by region or sex. Estimates for cases and controls are not pictured for these two predictors of consenting to participate. We also evaluated predictors of consenting to participate and found significant associations with SARS-CoV-2 infection status, age group, birth sex, and region. Cases had 1.69-fold higher adjusted odds of consenting compared to controls ([95%CI: 1.56–1.83], **Figure 3**). Women had 1.13-fold (1.04–1.22) higher adjusted odds of consenting than men. Parents or guardians of minors aged 0 to 4 were 1.53-times (1.18–1.98) more likely to consent than those aged 23 to 29. When compared with individuals from the Bay Area, individuals from San Joaquin Valley (aOR: 0.83 [0.70–0.97]) and Greater Los Angeles Area (0.73 [0.62–0.86]) were significantly less likely to consent. Some motivations for participants to consent, as cited among interviewer reflections, were desire to contribute to public health research, relieve boredom, and express perspectives about the pandemic: > *Participants genuinely believe their answers will help end the pandemic in some way*. > > *I think for the people who were inclined to not participate because of beliefs, some changed their minds when it was reframed as us wanting to make sure everyone is represented, that their voice matters, and this is a chance for them to be heard*. We identified differences in the likelihood of answering the phone within SARS-CoV-2 infection status strata among regions (aOR for cases 0.73 [95% CI: 0.63,0.85] versus aOR for controls 0.94 [95% CI: 0.83,1.07] in the Sierras) (**Table S4**). Differences in likelihood of consenting to participate occurred within SARS-CoV-2 infection status strata among age groups (aOR for cases 0.77 [95% CI: 0.62, 0.95] versus aOR for controls 1.14 [95% CI: 0.94,1.39] aged 60 and older). ### Reasons for refusing to participate We identified differences in the reasons for refusing to participate among 8,015 eligible individuals who answered the phone. The majority (90.9%; 7285/8015) cited insufficient time as the reason for refusing participation, with the proportion citing this reason increasing over time (**Figure S6**). Others cited language barriers (2.6%; 206/8015), lack of interest (2.3%; 184/8015), call fatigue (0.45%; 36/8015), and/or being unwell or grieving (0.45%; 36/8015) as refusal reasons (**Table S5**). We assessed determinants of indicating insufficient time as a reason for refusing study participation. Cases were associated with a 0.44-fold (95% CI: 0.37–0.53) lower adjusted odds of citing insufficient time compared to controls (**Table S6**). Individuals aged 23 to 29 were most likely to cite insufficient time compared to all other age categories. We did not find evidence of significant associations between the time of day or the time of week the individual was called and citing time as a reason for declining participation. Although interviewers observed that individuals often refused based on timing, they identified additional reasons, including personal beliefs, distrust, illness, and stress: > *Some people that were politically against the public health response often declined to get interviewed or were outright confrontational*. > > *It is stressful for COVID-19 positive and negative cases to follow through with the interview due to sickness, worry, or even suspicions of the intents and validity of our study*. ### Sample diversity Participants completing the survey were similar to the SARS-CoV-2 test seeking population in California across birth sex and in age groups 0-4, 18-22, 40-49, and 60+ (**Table 1, Table S7**). By design, participants were enrolled equally across each study region. The composition of study participants was roughly proportional to the state by household income and race/ethnicity (**Figure S7**). Pandemic sentiments and behaviors, as self-reported by participants, were diverse and changed over time. Agreement with social distancing and face mask recommendations generally remained constant throughout the study period (**Figure S8**), however, anxiety about the pandemic fluctuated between 67.7% in February 2021 (44/65 participants) and 29.6% in July 2021 (84/284 participants). Participants reporting visiting two or more public indoor settings within the two weeks prior to getting tested increased from 58.5% (38/65) in February 2021 to 85.1% (126/148) by February 2022. Attendance to each type of indoor setting remained constant - except for a decrease in grocery store visits and increase in school visits (**Figure S9**). The proportion of individuals ineligible for enrollment due to previously being infected with SARS-CoV-2 increased throughout the study period (**Figure S10**). Emotional states among participants, as encountered by interviewers, were also variable. The range of pandemic-related emotions that participants expressed included resilience, weariness, loneliness, and anger: > *A lot of people stated that they were so tired of living through a pandemic*. > > *Some people angrily shared experiences about being knowingly exposed to COVID-19 by their bosses or clients during work*. Some interviewers reported occurrences of previously vaccine-opposed participants expressing willingness to seek COVID-19 vaccination after testing positive. > *A participant who was firmly against believing in covid ended up changing his mind after testing positive. He told me that he would get the vaccine when he could*. ### Impact on interviewer wellbeing Of the 11 interviewers who responded to the interviewer experience survey, 63.6% (7) stated they occasionally encountered scenarios where they were compelled to search for or connect participants to social services and 18.3% (2) stated they encountered this need often (**Figure S11**). Resources pertaining to COVID-19 healthcare access, general healthcare access, housing security, food security, and financial relief were the most frequently requested. Most interviewers reported encountering grief or anger occasionally (81.8%, 9/11). 18.2% (2) reported encountering anger often. Interviewers expressed instances of poor mental health and lingering feelings after difficult calls. Interviews where participants discussed socioeconomic and demographic burdens, pandemic hardship, grief, suffering, inequitable conditions, or acted with hostility and bullying left a particularly strong impact on interviewers. Interviewers described feelings of stress, especially when participants compelled them to fulfill social service or counselor roles. > *Calls that I had where people shared their anxiety, confusion, fear, anger, and sadness fed into my own anxiety and negative feelings*. Interviewers also described many encounters which instilled a sense of purpose, pride, spurred personal growth, cultivated a sense of community, expanded empathy, and uplifted moods. Notably, encounters when participants expressed appreciation, gratitude, humor, or warmth despite hardships had resounding effects on interviewers. > *A lot of people said, “thank you for what you’re doing*.*”*… *That made me proud to be part of such an important research group*. Interviewers also mentioned how the study provided remote career growth and employment during a time of scarce opportunities. ### Structural successes and adaptations Feedback was frequently solicited to identify improvement opportunities. When mental health concerns surfaced, quick action was taken to strengthen structural support, community engagement, and resources. Research associates, with experience as interviewers, developed and led a robust training process that emphasized mental wellbeing and methods to navigate difficult conversations. They also compiled information on frequently requested social services and expanded on the standard operating procedure with scenario-specific protocols and responses to demands beyond interviewer duties. Active efforts to sustain a work environment that felt safe, supportive, and caring were made to better protect the mental health of interviewers. > *We were allowed to give feedback (and our feedback was valuable and used in survey changes), and supervisors cared about our mental health over collecting data*. Interviewers reported certain structural components as being particularly beneficial: self-assigned scheduling of shift times and weekly meetings. Self-assigned shifts allowed interviewers affected by difficult conversations to take breaks. Meeting weekly helped boost team morale, relieve isolation, and created bonding between team members: > *The team was very supportive when I was sharing my experience, which helped show me that it is normal to feel the impact the participants may have on us whether they are at their highest or their lowest*. > > *Seeing the data I had helped collect be used in real time to help improve understandings of COVID inspired me to keep calling people, even when I would reach voicemail after voicemail*. > > *Because it was a remote job, I sometimes felt as though I was working alone, but weekly meetings helped provide that sense of teamwork*. ## DISCUSSION Over a one-year period, during the COVID-19 pandemic, 9.8% of 38,470 individuals invited to our COVID-19 phone-based questionnaire consented to participate. Because the study was conducted across an evolving landscape of COVID-19 epidemiology and public health recommendations, flexibility to adapt protocols, exclusion criteria, and survey questions so that they remained meaningful was necessary. Results were consistent with prior research demonstrating that individuals who have a history of disease are more willing to participate in a health study than those naïve to the disease.11 The overall likelihood of an individual answering the phone decreased with age. Older individuals may experience more severe health burdens or reside in institutions unreachable by direct calls.24,18 The time of day that a potential participant was called influenced the likelihood of answering the phone, but not of consenting to participate. Results confirmed literature reporting that morning calls yield lower enrollment, indicating that strategically timing calls is crucial in maximizing enrollment efficiency.15 This study was successful in representing the population seeking SARS-CoV-2 testing in California, with a recruitment effort of almost 40,000 calls and a well-powered size of nearly 4,000 participants within the first year. The quality of the data allowed for identification of reasons for unsuccessful enrollment and determinants of participation. The infrastructure of the study, particularly the weekly meetings, detailed standard operating procedure documentation, and use of a messaging platform enabled quick identification of obstacles and adaptations in implementation. Enrollment—especially of controls—became more difficult throughout the study. This difficulty may be explained by the increase in previously positive individuals but also by potentially diminished interest or perceived risk regarding the pandemic. Insufficient time was the predominant reason for refusing participation. We recommend shortening survey length or offering call-backs. Interviewers highlighted key themes unique to the experience of remote phone-based data collection during the COVID-19 pandemic. Notably, instances of strong participant emotion and harassment especially burdened the mental health of student researchers. We believe this unforeseen consequence is novel in remote, phone-based quantitative health research and may be unique to the national context of polarized attitudes towards the COVID-19 pandemic.25 Proactively adapting to emerging obstacles was critical to the success of the study. Designing trainings that simulated various realistic scenarios and specific, detailed protocols for difficult encounters resulted in considerable improvement for subsequent interviewer cohorts. We suggest implementation of frequent proactive mental health check-ins, continual collection of anonymous feedback, and an exit survey for interviewers. There are several limitations to this analysis. First, due to data constraints, we were unable to examine how socioeconomic status, race, education, occupation, and setting, such as housing, may influence the likelihood of answering the phone and consenting to participate. Second, results may not be generalizable to the broader California population, as individuals who did not seek laboratory-confirmed SARS-CoV-2 testing are excluded by design. Third, severely ill SARS-CoV-2 positive individuals, unwell individuals with comorbidities, those without stable phone service, and those cautious about phone solicitations might not be well represented in our study. ## CONCLUSIONS We recommend placing calls during the afternoon and evening, allocating more efforts towards enrolling controls, and restricting survey length if possible. It is also imperative to consistently monitor unsuccessful recruitment efforts to detect unforeseen barriers to data collection and to ensure timely adjustments to the study protocol. Actively evaluating study implementation is vital to adapting data collection strategies when conducting a health study in a dynamic context such as a pandemic. In summary, these findings allow researchers to strategize recruitment for future phone-based observational studies conducted amidst an evolving pandemic. We hope that our results can improve the design and implementation of future phone-based observational research. ## Supporting information Supplemental tables and figures [[supplements/286454_file03.docx]](pending:yes) ## Data Availability All data produced in the present study are available upon reasonable request to the authors. ## DISCLAIMER The findings and conclusions in this article are those of the author(s) and do not necessarily represent the views or opinions of the California Department of Public Health or the California Health and Human Services Agency. ## CONTRIBUTIONS NF: Formal Analysis, Writing – Original Draft Preparation, Project Administration, Methodology, Visualization, Data Curation, Conceptualization SSL: Writing – Original Draft Preparation, Visualization, Validation, Methodology JD: Visualization, Writing – Original Draft Preparation, Conceptualization, Methodology JM: Methodology, Conceptualization, Writing – Review & Editing JO: Methodology, Conceptualization, Writing – Review & Editing JW: Methodology, Conceptualization, Writing – Review & Editing, Funding Acquisition JAL: Supervision, Methodology, Conceptualization, Writing – Review & Editing, Funding Acquisition SJ: Methodology, Conceptualization, Writing – Review & Editing, Funding Acquisition KLA: Supervision, Project Administration, Methodology, Writing – Review & Editing, Data Curation, Conceptualization JMP: Supervision, Project Administration, Methodology, Writing – Review & Editing, Data Curation, Conceptualization, Funding Acquisition AS: Writing – Review & Editing, Methodology ## FUNDING Centers for Disease Control and Prevention, Epidemiology and Laboratory Capacity (ELC) grant 0187.0150. ## ACKNOWLEDGEMENTS We would like to thank all participants that dedicated time to complete our survey, making the findings stemming from this research possible. ## Footnotes * § Members of the California COVID-19 Case-Control Study Team include: Adrian Cornejo, Amanda Lam, Amanda Moe, Amandeep Kaur, Anna Fang, Ashly Dyke, Camilla Barbaduomo, Christine Wan, Diana Nicole Morales Felipe, Diana Poindexter, Erin Xavier, Hyemin Park, Helia Samani, Jessica Ni, Julia Cheunkarndee, Mahsa Javadi, Maya Spencer, Michelle Spinosa, Miriam Bermejo, Monique Miller, Najla Dabbagh, Natalie Dassian, Nikolina Walas, Paulina Frost, Savannah Corredor, Shrey Saretha, Timothy Ho, Vivian Tran, Yang Zhou, Yasmine Abdulrahim, Zheng Dong * Received February 27, 2023. * Revision received February 27, 2023. * Accepted March 1, 2023. * © 2023, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at [http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/) ## REFERENCES 1. 1.Capano G, Howlett M, Jarvis DSL, Ramesh M, Goyal N. Mobilizing Policy (In)Capacity to Fight COVID-19: Understanding Variations in State Responses. Policy and Society. 2020;39(3):285–308. doi:10.1080/14494035.2020.1787628 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/14494035.2020.1787628&link_type=DOI) 2. 2.Lipsitch M, Swerdlow DL, Finelli L. Defining the Epidemiology of Covid-19 — Studies Needed. New England Journal of Medicine. 2020;382(13):1194–1196. doi:10.1056/NEJMp2002125 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMp2002125&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32074416&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F01%2F2023.02.27.23286454.atom) 3. 3.Dean NE, Hogan JW, Schnitzer ME. Covid-19 Vaccine Effectiveness and the Test-Negative Design. New England Journal of Medicine. 2021;385(15):1431–1433. doi:10.1056/NEJMe2113151 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMe2113151&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=34496195&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F01%2F2023.02.27.23286454.atom) 4. 4.Andrejko KL, Pry J, Myers JF, et al. Predictors of Severe Acute Respiratory Syndrome Coronavirus 2 Infection Following High-Risk Exposure. Clinical Infectious Diseases. December 2021:ciab1040. doi:10.1093/cid/ciab1040 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/cid/ciab1040&link_type=DOI) 5. 5.Tenforde MW, Fisher KA, Patel MM. Identifying COVID-19 Risk Through Observational Studies to Inform Control Measures. JAMA. 2021;325(14):1464–1465. doi:10.1001/jama.2021.1995 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jama.2021.1995&link_type=DOI) 6. 6.Andrejko KL, Pry J, Myers JF, et al. Prevention of Coronavirus Disease 2019 (COVID-19) by mRNA-Based Vaccines Within the General Population of California. Clinical Infectious Diseases. July 2021:ciab640. doi:10.1093/cid/ciab640 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/cid/ciab640&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=34282839&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F01%2F2023.02.27.23286454.atom) 7. 7.Andrejko KL. Effectiveness of Face Mask or Respirator Use in Indoor Public Settings for Prevention of SARS-CoV-2 Infection — California, February–December 2021. MMWR Morb Mortal Wkly Rep. 2022;71. doi:10.15585/mmwr.mm7106e1 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.15585/mmwr.mm7106e1&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=35143470&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F01%2F2023.02.27.23286454.atom) 8. 8.Gourlay S, Kilic T, Martuscelli A, Wollburg P, Zezza A. Viewpoint: High-frequency phone surveys on COVID-19: Good practices, open questions. Food Policy. 2021;105:102153. doi:10.1016/j.foodpol.2021.102153 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.foodpol.2021.102153&link_type=DOI) 9. 9.Ridolfo H, Boone J, Dickey N, eds. Will They Answer the Phone If They Know It’s Us? Using Caller ID to Improve Response Rates.; 2013. doi:10.22004/ag.econ.234302 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.22004/ag.econ.234302&link_type=DOI) 10. 10.Brick JM, Williams D. Explaining Rising Nonresponse Rates in Cross-Sectional Surveys. The ANNALS of the American Academy of Political and Social Science. 2013;645(1):36–59. doi:10.1177/0002716212456834 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/0002716212456834&link_type=DOI) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000311519200003&link_type=ISI) 11. 11.Glass D, Kelsall H, Slegers C, et al. A telephone survey of factors affecting willingness to participate in health research surveys. BMC Public Health. 2015;15(1):1017. doi:10.1186/s12889-015-2350-9 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s12889-015-2350-9&link_type=DOI) 12. 12.Ravanam MS, Skalland B, Zhao Z, Yankey D, Smith C. An Evaluation of the Impact of Using an Alternate Caller ID Display in the National Immunization Survey. Proc Am Stat Assoc. 2018;73:AAPOR2018. [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7182364/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7182364/). Accessed April 6, 2022. 13. 13.Vicente P, Lopes I. When Should I Call You? An Analysis of Differences in Demographics and Responses According to Respondents’ Location in a Mobile CATI Survey. Social Science Computer Review. 2015;33(6):766–778. doi:10.1177/0894439314560696 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/0894439314560696&link_type=DOI) 14. 14.Mcclain C, Rainie L. The Challenges of Contact Tracing as U.S. Battles COVID-19. Pew Research Center: Internet, Science & Tech. October 2020. [https://www.pewresearch.org/internet/2020/10/30/the-challenges-of-contact-tracing-as-u-s-battles-covid-19/](https://www.pewresearch.org/internet/2020/10/30/the-challenges-of-contact-tracing-as-u-s-battles-covid-19/). Accessed April 6, 2022. 15. 15.Shino E, McCarty C. Telephone Survey Calling Patterns, Productivity, Survey Responses, and Their Effect on Measuring Public Opinion. Field Methods. 2020;32(3):291–308. doi:10.1177/1525822X20908281 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/1525822X20908281&link_type=DOI) 16. 16. S.V. P, Ittamalla R. General public’s attitude toward governments implementing digital contact tracing to curb COVID-19 – a study based on natural language processing. International Journal of Pervasive Computing and Communications. 2020;ahead-of-print(ahead-of-print). doi:10.1108/IJPCC-09-2020-0121 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1108/IJPCC-09-2020-0121&link_type=DOI) 17. 17. Ali Ggmn, Rahman MM, Hossain MA, et al. Public Perceptions of COVID-19 Vaccines: Policy Implications from US Spatiotemporal Sentiment Analytics. Healthcare. 2021;9(9):1110. doi:10.3390/healthcare9091110 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3390/healthcare9091110&link_type=DOI) 18. 18.Druckman JN, Klar S, Krupnikov Y, Levendusky M, Ryan JB. Affective polarization, local contexts and public opinion in America. Nat Hum Behav. 2021;5(1):28–38. doi:10.1038/s41562-020-01012-5 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41562-020-01012-5&link_type=DOI) 19. 19.Goldstein DAN, Wiedemann J. Who Do You Trust? The Consequences of Partisanship and Trust for Public Responsiveness to COVID-19 Orders. Perspectives on Politics. 2022;20(2):412–438. doi:10.1017/S1537592721000049 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1017/S1537592721000049&link_type=DOI) 20. 20.Koné A. Symptoms of Mental Health Conditions and Suicidal Ideation Among State, Tribal, Local, and Territorial Public Health Workers — United States, March 14–25, 2022. MMWR Morb Mortal Wkly Rep. 2022;71. doi:10.15585/mmwr.mm7129a4 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.15585/mmwr.mm7129a4&link_type=DOI) 21. 21.Bluvstein I, Ifrah K, Lifshitz R, Markovitz N, Shmotkin D. Vulnerability and Resilience in Sensitive Research: The Case of the Quantitative Researcher. J Empir Res Hum Res Ethics. 2021;16(4):396–402. doi:10.1177/15562646211027418 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/15562646211027418&link_type=DOI) 22. 22.Kumar S, Cavallaro L. Researcher Self-Care in Emotionally Demanding Research: A Proposed Conceptual Framework. Qual Health Res. 2018;28(4):648–658. doi:10.1177/1049732317746377 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/1049732317746377&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F01%2F2023.02.27.23286454.atom) 23. 23.Fried AL, Fisher CB. Moral stress and job burnout among frontline staff conducting clinical research on affective and anxiety disorders. Professional Psychology: Research and Practice. 2016;47(3):171–180. doi:10.1037/pro0000060 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1037/pro0000060&link_type=DOI) 24. 24.Miller EA. Protecting and Improving the Lives of Older Adults in the COVID-19 Era. Journal of Aging & Social Policy. 2020;32(4-5):297–309. doi:10.1080/08959420.2020.1780104 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/08959420.2020.1780104&link_type=DOI) 25. 25.Hart PS, Chinn S, Soroka S. Politicization and Polarization in COVID-19 News Coverage. Sci Commun. 2020;42(5):679–697. doi:10.1177/1075547020950735 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/1075547020950735&link_type=DOI)