SARS-CoV-2 Serosurveys: How antigen, isotype and threshold choices affect the outcome

Study participant recruitment and enrollment

Study participants were enrolled at the University of Massachusetts Chan Medical School (UMass Chan) and the University of Massachusetts Memorial Medical Center (UMass Memorial) in Worcester between April and August of 2020 under the Institutional Review Board (IRB) approved Consolidated COVID-19 Clinical and Observational Pathogenesis and Epidemiology (COVID-COPE) Study Protocol (H00020145), see Supplemental Methods for more details. Briefly, once consent was obtained, participants were asked to complete a survey capturing demographic and symptomatic information, and blood was collected. Health care workers (HCW Group) consisted of front-line workers from the UMass Memorial Emergency Department (ED) and their family members (HCW Family Group). Return to work employees (RTW Group), consisted of UMass Chan employees who returned to campus in August after the state mandated stay-at-home order. RTW, HCW and HCW Family members received antibody results (Supplemental Figure 1) with the caveat that this was a research assay and not a diagnostic test. Additionally, patients with COVID-19-like symptoms that were being evaluated at the UMass Memorial ED were approached for study participation (CHC Group). Once consented, the patients or their healthcare proxy completed a survey, and blood was collected. COVID-19 related clinical information, including symptoms, were extracted from the participants’ medical charts. Finally, four sets of banked de-identified banked blood samples (collected between October of 2003 and January of 2020), were sourced from the Moormann Laboratory (n=17) or kindly provided by Dr. Larry Stern (n=13), Dr. Liisa Seline and Dr. Anna Gil from UMass Chan (n=34). Dr. Christopher King from Case Western Reserve University, Cleveland, Ohio (SeroNet U01 CA260539) kindly provided de-identified pre-screened SARS-CoV-2 antibody negative sera samples (Luminex screening for anti-S, N and RB SARS-CoV-2 Wuhan antibodies) collected during a community blood drive in July of 2020 (n=32). Both the pre-pandemic and pre-screened negative samples served as negative controls (total n=96).

SARS-CoV-2 Enzyme-Linked Immunosorbent Assay (ELISA)

The SARS-CoV-2 ELISA assay developed at the Ragon Institute was implemented at UMass Chan [4], using receptor-binding domain (RBD), S Trimer (S; truncated to 1208 amino acids, deleted transmembrane domain and C-terminal domain), and nucleocapsid protein (N; full-length) SARS-CoV-2 antigens provided by MassBiologics of the UMass Chan, Mattapan, MA [5]. We also used SARS-CoV2 RBD antigen provided by the Ragon Institute, Cambridge, MA. See Supplemental Methods for more details and validation.

Statistical analysis

Statistical calculations and graphs were done in Prism v8.2.1, R v3.6.1/v4.1.1 and STATA v17. The heatmap was generated by the pheatmap package in R v4.1.1. Three different thresholds were evaluated, see Supplemental Methods for more details. Briefly, we applied the threshold formula “mean + 3 standard deviations of negative controls” (3SD threshold) [6]. Next, a maximum specificity threshold was applied by setting the cutoff at 100% specificity for each antigen/isotype combination (i.e., the cutoff value is the highest OD value among the negative control group of each antigen/isotype combination; Max Spec threshold). Finally, a threshold based on the maximal Youden index (sensitivity + specificity – 1) on the ROC curve (Supplemental Figure 2) was determined, which balances sensitivity and specificity for each antigen/isotype combination (Youden threshold). Experimental samples with OD values equal to or higher than the determined threshold value for each antigen/isotype combination were considered serologically positive (Supplemental Table 1).

Study participant recruitment

A total of 212 suspected COVID-19 patients (CHCs) were approached for study participation when they were being evaluated in the ED and 134 (63.2%) were subsequently enrolled. Of those, 86 (64.2%) were PCR-confirmed COVID-19 cases (Table 1). The majority of the CHCs were white (60.4%, n=81) and non-Hispanic (65.7%, n=88), aged 60 years or older (59.7%, n=80; average age 67 years) and male (47.8%, n=64). The latter two determinants being common COVID-19 risk factors and consistent with the demographics of those severely affected during the first wave in Massachusetts [7]. A total of 253 HCWs volunteered to participate in the study, and 299 of their family members were also enrolled. Based on self-reported data, the majority of HCWs were white (76.3%, n=193) and non-Hispanic (80.2%, n=203). The average age of the HCWs was 41 years (range: 21 to 68 years), and 49.8% (n=126) were female. Among the HCW family members who reported demographic information, 54.2% (n=162) were white, 57.5% (n=172) non-Hispanic, and 33.4% (n=100) female. A total of 327 RTW volunteered for the study, of whom the majority were white (70.0%, n=229) and non-Hispanic (93.0%, n=304). About half of the RTWs were female (56.3%, n=184), and the average age was 40 years (range: 22 to 73 years). Hence, a total of 1,013 study participants were successfully enrolled, demographic and symptomatic information recorded, blood samples collected, and sera or plasma screened for anti-SARS-CoV-2 antibodies. Among these study participants, the missing data ranged from 0.3% to 40.8% (Table 1), depending on the demographic category or study population subgroup. Note, this does not include the COVID-19 symptom reports as RTWs, HCWs and HCW family members were only asked to report COVID-19 symptoms if they had a positive SARS-COV-2 PCR test or had reason to believe they were infected. HCW family members were the least likely to answer demographic questions and therefore had the most missing data, despite repeated contact attempts. RTWs were most likely to fill out all the online questionnaires. Part of the draw to the study was the distribution of SARS-CoV-2 antibody reports in the absence of FDA approved tests in the beginning of the pandemic (Supplemental Figure 1).

Threshold methods and sensitivity/specificity

Three different threshold methods were evaluated to convert the quantitative OD values into qualitative positive/negative results and subsequent sensitivity and specificity: the 3 SD, Max Spec and Youden threshold. PCR-confirmed SARS-CoV-2 positive CHCs (n=86) and negative samples (n=96) were screened for anti-RBD, anti-S and anti-N SARS-CoV-2 IgA and IgG antibodies by ELISA. We further evaluated (i) combining the three antigen/isotype combinations (RBD IgG, N IgG, and RBD IgA) with the largest area under the ROC curve (i.e., with maximum sensitivity and specificity; Top Three); (ii) combining all IgA isotypes (N, RBD and S IgA; All IgA); and (iii) combining all IgG isotypes (N, RBD and S IgG; All IgG) as serological outcome measures (Figure 1). We used isotype-based compound measurements since a heatmap of CHC, HCW and RTW OD values with at least one positive antigen/isotype combination clustered by isotype, rather than antigen (Figure 2). The clustering was confirmed with a correlogram comparing antigen/isotype combinations. Again, when analyzing PCR-confirmed CHCs or all seropositive participants, the strongest positive correlation was between the same isotypes rather than the same antigens (Supplemental Figure 3). This clustering of IgG and IgA responses to SARS-CoV-2 infection has been described by others [8, 9], and is likely a function of time since infection [10] and protective class-switching [8].

• Download figure
• Open in new tab

Figure 1. Optical density value distribution among each measured subgroup.

Optical density values (450-570nm) were plotted according to each antigen/isotype combination and subgroup (y-axis in log10 scale). The red line indicates the cut off for the 3 standard deviation above the mean threshold method (3 SD). The blue line indicates the cut off when the threshold was chosen at the highest value of the negative controls (Max Spec). The black line indicates the cut off when the threshold was chosen based on the Youden threshold (Youden). RBD; SARS-CoV-2 Receptor-Binding Domain. S Trimer; SARS-CoV-2 Spike Trimer. N; SARS-CoV-2 Nucleocapsid Protein. CHC; Hospitalized COVID-19 patients. HCW; Health care workers. HCW family member; Family members of health care worker listed under HCW. RTW; Return to work employees who had been working remotely from March to August of 2020, during the first COVID-19 wave in Worcester, Massachusetts.

• Download figure
• Open in new tab

Figure 2. Heatmap of optical density values.

A heatmap was built based on unbiased clustering of the optical density (OD) values of the experimental subgroups with at least one positive antigen/isotype combination. The results clustered by isotype, rather than antigen (see top branching into IgG and IgA from left to the right). RBD; SARS-CoV-2 Receptor-Binding Domain. S Trimer; SARS-CoV-2 Spike Trimer. N; SARS-CoV-2 Nucleocapsid Protein. CHC; Hospitalized COVID-19 patients. HCW; Health care workers and their family members. RTW; Return to work employees who had been working remotely from March to August of 2020, during the first COVID-19 wave in Worcester, Massachusetts.

Applying the 3 SD threshold resulted in 100% specificity for all isotype/antigen combinations across IgG and IgA (Table 2). Conversely, sensitivity was low and ranged from 25.6% to 76.7%. Sensitivities for RBD IgA and IgG, 25.6% and 55.8%, were particularly low. Sensitivities for IgA and IgG S Trimer and IgA and IgG N were similar and ranged from 72.1% to 76.7%. The RBD IgM antigen/isotype combination resulted in relatively low sensitivity (67.4%) and specificity (98.5%), similar to previously published results [4, 11]. Given the quick resolution of IgM, while appearing concomitantly with IgG after SARS-CoV-2 infection [3], IgM was not deemed a useful serosurvey tool and was omitted from the rest of the analyses.

Based on the Max Spec threshold (fixing the specificity at 100%), sensitivity for RBD IgA and IgG increased compared to the 3 SD threshold but remained relatively low, 74.4% and 64.0%, respectively. Again, sensitivities for IgA and IgG S Trimer and IgA and IgG N were similar and ranged from 81.4% to 84.9%

Based on the Youden threshold, robust sensitivity was obtained across all antigen/isotype combinations, ranging from 86.0% to 97.7%. Sensitivity for RBD IgA and IgG were 96.5% and 97.7%, respectively. Sensitivities for IgA and IgG S Trimer and IgA and IgG N ranged from 86.0% to 95.3%. Conversely, specificity was slightly lower compared to the other two methods and ranged from 92.7% to 99.0%, see Table 2.

HCW, HCW Family and RTW serological results

Next, we screened HCW (n=253), HCW Family members (n=299), and RTWs (n=327) for anti-RBD, anti-S and anti-N SARS-CoV-2 IgA and IgG antibodies by ELISA. The overall OD value spread for the HCW, HCW Family and RTW subgroups were similar (Figure 1). However, the overlap between the PCR-positive CHCs (positive controls) and negative control values varied notably depending on the antigen/isotype combination, affecting the cutoffs for each method and therefore the subsequently determined seroprevalence for each subgroup. Furthermore, most negative control OD values were notably lower than the experimental subgroups (HCW, HCW Family, and RTW) for RBD IgG, S IgG, and N IgG, but not so much for the IgA equivalents. Hence, when applying the three different thresholds to the experimental subgroups the seroprevalence outcomes varied widely, not just across threshold methods (Youden, 3 SD, and Max Spec), but also across antigen/isotype combinations (Figure 1, Figure 3 and Supplemental Figure 4).

• Download figure
• Open in new tab

Figure 3. Seropositivity of each subgroup, according to the antigen/isotype combination.

The red bars graph the seroprevalence based on the cut off for the 3 standard deviation above the mean threshold method (3 SD). The blue bars graph the seroprevalence based on the maximum specificity threshold (Max Spec). The black bars graph the seroprevalence based on the Youden threshold (Youden). RBD; SARS-CoV-2 Receptor-Binding Domain. S Trimer; SARS-CoV-2 Spike Trimer. N; SARS-CoV-2 Nucleocapsid Protein. HCW; Health care workers. HCW family member; Family members of health care worker listed under HCW. RTW; Return to work employees who had been working remotely from March to August of 2020, during the first COVID-19 wave in Worcester, Massachusetts.

When comparing the overall seroprevalence for each antigen/isotype combination, RBD IgA, RBD IgG and the Top Three compound estimate (RBD IgG, RBD IgA, and N IgG) yielded the highest seroprevalence, especially when applying the Youden threshold, ranging from 29.7% to 85.4% (Supplemental Table 2, Figure 3). The S IgA- and All IgA-based seroprevalence ranged from 12.5% to 38.6% based on the Youden threshold (Supplemental Table 2). Overall, N IgA-, N IgG- and S IgG-based seroprevalence was lowest, ranging from 0.0% to 20.6%, independent of applied threshold; again, noting the wide range of determined seroprevalence (Supplemental Table 2).

When comparing the three participant sub-groups based on the Youden threshold, the HCWs had the highest seroprevalence followed by the HCW Family and RTWs groups when measuring RBD IgA (51.4%, 34.1%, and 29.7%, respectively) and RBD IgG (85.4%, 62.2%, and 52.9%, respectively; Supplemental Table 2 and Figure 3). For the other antigen/isotype combinations, the order was less pronounced, except for S IgA where the RTW subgroup had the highest seroprevalence based on the Youden and Max Spec thresholds, compared to the HCWs and HCW Family subgroups.

When comparing the threshold methods, applying the Youden cutoff, which was the most sensitive one, yielded the highest seroprevalence across all antigen/isotype combinations even though with a wide range (6.0% to 85.4%; Supplemental Table 2, Figure 3). Further, while the RBD IgA-, RBD IgG- and the Top Three compound-based seroprevalence estimates were highest when applying the Youden threshold method, they also represented the most notable discrepancy when comparing the Youden to the Max Spec and 3 SD thresholds. Based on the Youden threshold method, the RBD IgG-based seroprevalence among all three subgroups ranged from 52.9% to 85.4%, as compared to the 3 SD and Max Spec thresholds where the seroprevalence ranged from 0.0% to 2.0%.

Given the notable variation in OD spread among the antigen/isotype combinations and the resulting wide range of seroprevalence estimates (including the overinflated seroprevalence based on RBD IgA/IgG when applying the Youden threshold, Figure 3), maximizing the specificity and including more than one antigen (compound measurements) for the overall serosurvey measurement was prioritized.

Being compound measurements with the least antigen/isotype-dependent variation, the Top Three and the All IgG compound estimates, based on the Max Spec threshold, were considered the most reliable outcome measurements to determine seroprevalence among our study population. Based on the Top Three compound estimate and the Max Spec threshold, we estimated that the seroprevalence among the HCW, HCW Family, and RTW study participants was 14.7%, 10.1%, and 9.3%, respectively (HCWs ranking highest; Supplemental Table 2, Figure 3). Based on the All IgG compound estimate and the Max Spec threshold, we estimated that the seroprevalence among the HCW, HCW Family, and RTW study participants was 15.9%, 11.5%, and 25.9%, respectively (RTWs ranking highest). Hence, based on these two compound estimates and the Max Spec threshold, we estimate that the anti-SARS-CoV-2 antibody seroprevalence among the HCW, HCW Family, and RTW study participants ranged from 9.3% to 25.9% during the first COVID-19 wave in the Spring of 2020, while the ranking was unclear due to relatively low overall seroprevalence. Note that at the time, no COVID-19 vaccines were available and therefore detecting any SARS-CoV-2 antibodies was considered an indication of infection.

Finally, participant COVID-19 symptoms and test reports were analyzed in order to validate the serological results. As part of the participant surveys, HCWs and HCW Family members, and RTWs were asked to report COVID-19-like symptoms if they either had a positive SARS-CoV-2 PCR test or if they thought they had been infected (Table 1). Among HCWs, HCW family members and RTW study participants, HCWs reported the most positive SARS-CoV-2 PCR tests (33.2%, n=84) and COVID-19 symptoms (34.4%, n=87), followed by HCW family members and RTWs (Table 1). Thus, confirming a potential HCW, HCW Family and RTW ranking. Interestingly, among the RTWs, HCWs, and HCW family members who reported SARS-CoV-2 PCR positive tests only 54.8% (n=69) reported COVID-19 symptoms (32.5%, n=41 reported no symptoms and 12.7%, n=16 did not answer that question). Note, molecular SARS-CoV-2 tests were not widely available outside of hospital settings during the study period.