Abstract
Objectives We evaluated widely-used SARS-CoV-2 serological tests and their potential association with virus neutralization test (VNT) in a cohort of mild COVID-19 patients.
Methods A total of 439 specimens were longitudinally collected from 76 healthcare workers with RT-PCR-confirmed mild COVID-19. Nine serological assays developed by leading global companies (Abbott, DiaSorin, Siemens, Bio-Rad, Wantai, bioMérieux, Euroimmun) were assessed. For each test the sensitivity to detect SARS-CoV-2 antibodies was determined weekly after symptom onset. Correlation and concordance were assessed using the Spearman and Cohen’s Kappa coefficients, respectively. Positive percent agreement and negative percent agreement (NPA) with the VNT were also determined.
Results The Wantai Total Ab assay targeting the receptor binding domain (RBD) within the S protein presented the best sensitivity at different times during the course of disease. The best correlation between antibody level and neutralizing antibody titer was found with the Euroimmun S1-based IgA assay (Spearman coefficient [95%CI]: 0.71 [0.61-0.79]). A moderate concordance (Kappa [95%CI]: 0.43[0.23-0.63]) as well as the lowest NPA (33%) was found between the Wantai Total Ab assay and the VNT. Compared to the Wantai Total Ab assay, other total Ab or IgG assays targeting the S or the RBD (bioMérieux, DiaSorin, Siemens,) were more concordant with the VNT (Kappa>0.7 for the three tests) and had a higher NPA (range: 90% to 97%).
Conclusions Although some assays presented a better concordance with VNT than others, the present findings emphasize that commercialized serological tests including those targeting the RBD cannot substitute VNT for the assessment of functional antibody response.
Introduction
The evaluation of the humoral immune response to SARS-CoV-2 with serological tests is crucial to further manage the coronavirus disease 2019 (COVID-19) pandemic. Serological testing represents an easy to implement and cost-effective method allowing to rapidly identify individuals exposed to the virus [1,2]. Over the last few months, a large number of SARS-CoV-2 commercial assays have been evaluated for their ability to detect specific antibodies [3–8]. However, the detection of specific SARS-CoV-2 antibodies does not indicate whether or not the antibodies are functional for neutralizing the virus. In association with the evaluation of other immune responses, such as cellular immunity, the determination of neutralizing antibody titer is important to evaluate the protective immunity to SARS-CoV-2 after infection and therefore the risk of reinfection [9–11]. While the comparison of sensitivity and specificity of serological tests has been increasingly studied, the association between the results obtained with commercial tests and the virus neutralization test (VNT) has been explored in only a few studies, and mostly among severe COVID-19 patients [12,13]. VNT is considered as the reference to assess the functional ability of antibodies to block the entry of the virus into human cells [14]. However, such an assay requires living virus manipulated in a biosafety level 3 (BSL3) facility that needs trained staff and specific equipment, and which is a tedious and time-consuming method. The first study exploring the association of commercial serological assays and VNT claimed that the Wantai Total Ab assay detecting total antibodies directed against the SARS-CoV-2 receptor binding domain (RBD) had the best characteristics to detect functional antibodies at different stages and severity of disease [12]. The RBD, within the sub-unit S1 of the spike protein, enables the viral entry into human cells by fixing to the angiotensin-converting enzyme 2 (ACE2) receptor [15]. As emphasized by the authors [12], there is an urgent need for further studies addressing the performance of alternative high-throughput assays in correlation with neutralization among persons with mild COVID-19.
Thus, the aim of the present study was to evaluate widely-used, high-throughputs tests in a longitudinal cohort of mild COVID-19 patients by including the comparison with a VNT.
Methods
Study design and sample collection
A prospective longitudinal cohort study was conducted at the laboratory associated with the National reference center for respiratory viruses (University Hospital of Lyon, France)[16]. Healthcare workers (HCW) with symptoms suggesting a SARS-CoV-2 infection requiring a reverse transcriptase (RT)-PCR test were included. Patients with a positive RT-PCR result at inclusion (V1) returned weekly for 6 additional visits (V2-V7) for serum samples. Written informed consent was obtained from all participants; ethics approval was obtained from the national review board for biomedical research in April 2020 (Comité de Protection des Personnes Sud Méditerranée I, Marseille, France; ID RCB 2020-A00932-37), and the study was registered on ClinicalTrials.gov (NCT04341142). A total of 439 serum specimens were longitudinally collected from 76 HCW; with the exception of one patient who required hospitalization (not in intensive care unit), all of them developed mild forms of COVID-19. Among the 439 collected samples, 170 of them taken at V2, V4, V7 from 56 patients were tested by VNT.
Virological investigation
COVID-19 diagnosis for inclusion was performed by RT-PCR on nasopharyngeal swab using the cobas® SARS-CoV-2 assay (Roche, Basel, Switzerland).
A total of 9 serological assays developed by leading global companies in the field (Abbott, DiaSorin, Siemens, Bio-Rad, Wantai, bioMérieux, Euroimmun) were investigated according to the protocol recommended by each manufacturer (characteristics are summarized in Table 1). Positivity was established according to threshold value recommended by each manufacturer. As previously suggested, we also evaluated a cut-off (OD ratio ≥ 10) to indicate the presence of protective antibodies for the Wantai Total Ab assay [12].
Sensitivity of the serological assays was determined by comparing the outcome to positive SARS-CoV-2 PCR. Positivity was established according to threshold value recommended by each manufacturer. Ab: antibodies, Ig: immunoglobulin, ELISA: enzyme-linked immunosorbent assay, CMIA: chemiluminescence microparticule immune assay CLIA: chemiluminescence immune assay, ELFA: enzyme-linked fluorescent assay, n: number of samples, CI: confidence interval.
The VNT used for the detection and titration of neutralizing antibodies was performed as previously described [17]. Briefly, a ten-fold dilution of each serum specimen in culture medium (Dulbecco’s Modified Eagle Medium containing antibiotics and 2% foetal calf serum) was first heated for 30 min at 56°C to avoid complement-linked reduction of the viral activity. Serial two-fold dilutions (tested in duplicate) of the serum specimens in culture medium were mixed at equal volume with the live SARS-CoV2 virus. After gentle shaking and a contact of 30 minutes at room temperature in plastic microplates, 150 µL of the mix was transferred into 96-well microplates covered with Vero E6 cells. The plates were incubated at 37°C in a 5% CO2 atmosphere. The reading was evaluated microscopically 5 to 6 days later when the cytopathic effect of the virus control reached 100 TCID50/150 µL. Neutralization was recorded if more than 50% of the cells present in the well were preserved. The neutralizing titer was expressed as the inverse of the higher serum dilution that exhibited neutralizing activity; a threshold of 20 was used. All experiments were performed in a BSL3 laboratory. The comparison of this VNT with a standardized assay using retroviruses pseudo-typed with the SARS-CoV-2 S viral surface protein found a high correlation and concordance [17].
Statistical analyses
For each test the clinical sensitivity was determined weekly after symptom onset. The correlation and concordance with the VNT were assessed using the Spearman and Cohen’s Kappa coefficients, respectively. The concordance was classified as slight (Cohen’s Kappa coefficient, [0-0.2]), fair [0.21-0.4], moderate [0.41-0.6], substantial [0.61-0.8], and almost perfect [0.81-1] according to Landis and Koch criteria. The positive and negative percentage agreements (PPA, NPA) were also determined. The estimation of the correlation coefficient was not performed due to an upper limit of signal to cut-off ratio for the Siemens and Bio-Rad assays. Specificity was assessed with 30 pre-pandemic serum specimens collected from healthy donors in 2019. The estimates are given with their bilateral 95% confidence interval (CI) calculated using the Wilson method. The 95% CI for Cohen’s Kappa coefficient was calculated using the bootstrap percentile method. The paired comparison of sensitivity between two assays was performed with the non-parametric McNemar test. A p-value < 0.05 was considered as statistical significant.
Results
Sensitivity and specificity
During the first week after the onset of symptoms the sensitivity for the detection of antibodies ranged from 6.6% (DiaSorin, Liaison) to 25.0% (Euroimmun). The second week the sensitivity was greater than 70% for three tests including Bio-Rad, Wantai Total Ab, and Euroimmun IgA assays (74.2%, 79.0% and 72.6%, respectively). The highest of sensitivity was found at week # 3 for Bio-Rad (96.6%), Wantai Total Ab (100%), Wantai IgM (94.9%), bioMérieux IgM (78.0%) and Euroimmun (96.6%), at week # 4 for Abbott (93.2%), and at week # 6 for Diasorin (93.2%), Siemens (98.3%) and bioMérieux IgG (94.9%). After this point, a decrease of sensitivity was noted for all assays except for the Wantai Total Ab which remained steady at 100% over the course of the disease (Table 1). Compared to the Wantai Total Ab assay, the differences were significant before 14 days post-symptom onset with all other assays, except with the Euroimmun and Bio-Rad assays; after 14 days post-symptom onset, the differences were significant with all other assays.
Regarding specificity, no false positive result was found using 30 pre-pandemic sera, although 3 samples gave a borderline ratio (between 0.8 and 1.1) with the Euroimmun IgA assay (supplementary table 1).
Kinetics of neutralizing antibody titers
The neutralizing capacity of antibodies was determined at three time points for 56 patients (n=170 samples). No neutralizing antibody was detected in 42.0% (21/50), 5.8% (3/51), and 8.7% (6/69) of samples collected between, respectively, 1-14, 15-28, and more than 28 days after symptom onset. For the samples with a detection of neutralizing antibody, the median [IQR] titer was 60[40-100] between 1-14 days post symptom, reached 80[60-120] between 15-28 days post symptom and decreased in samples collected after more than 28 days (median: 60[40-120]).
Comparison of results between commercial kits and VNT
The best correlation between commercial kits and VNA was found with the Euroimmun S1-based IgA assay (Spearman coefficient [95%CI]: 0.71 [0.61-0.79]) while the Abbott N-based assay presented the lowest correlation (0.46 [0.32-0.59]; Figure 1, Table 2).
Comparison between serological assays and virus neutralization test. Ab: antibodies, Ig: immunoglobulin, ELISA: enzyme-linked immunosorbent assay, CMIA: chemiluminescence microparticule immune assay CLIA: chemiluminescence immune assay, ELFA: enzyme-linked fluorescent assay, n: number of samples, CI: confidence interval, dps: days post onset of symptoms, test. VNT: Virus neutralization test. The estimation of the correlation coefficient was not performed due to an upper limit of signal to cut-off ratio for the Siemens and Bio-Rad assays. The Cohen’s Kappa coefficient after 14 days post symptom onset cannot be interpreted for the Wantai Total Ab assay because the sensitivity of this test was 100%.
Correlation of SARS-CoV-2 neutralizing antibody titers tested by a virus neutralization test to antibodies level measured by selected assays. (A) bioMérieux IgM (B) Wantai IgM (C) Euroimmun IgA (D) Wantai total Ab (E) bioMérieux IgG (F) Abbott (G) DiaSorin IgG. Magenta dots indicate patient specimen collected ≤14 days post onset of symptoms (dps), orange dots indicate samples collected from 14-28 dps, green dots indicate specimen collected more than 28 dps. Dotted lines indicate the cut-off for positivity of each assay, as indicated by the manufacturer: Wantai, Abbott, bioMérieux, Euroimmun, OD ratio > 1; DiaSorin >12 AU/ml. OD: optical density.
A slight and fair concordance with VNT were noticed for the 2 IgM assays evaluated herein (Kappa [95%CI]: 0.24 [0.14-0.36] for bioMérieux IgM and 0.40 [0.21-0.58] for the Wantai IgM assays). Regarding total Ab or IgG assays targeting the S protein, three had substantial concordance with VNT (Kappa [95%CI]: 0.71 for bioMérieux [0.57-0.84], 0.70 [0.56-0.83] for DiaSorin, and 0.72 [0.55-0.85] for Siemens assays) while the concordance with the Wantai Total Ab assay was moderate (0.43[0.23-0.63]; Table 2).
For the Wantai Total Ab assay, 20/30 samples with no neutralizing antibody had a positive Wantai Total Ab; 18 these had an OD ratio < 10. Of note, 9/30 samples were collected more than 14 days post symptom onset. Regarding the samples with an OD ratio < 10 with the Wantai Total ab assay (39/140, 28%), all had a low level of neutralizing antibodies (range: 20-50; Figure 1).
The NPA ranged from 33.3% [21.1-48.3] for the Wantai Total Ab assay to 96.8% for the DiaSorin and was < 90% for 7/9 assays. The PPA was > 90% for all tests except the DiaSorin and the two IgM based assays (Wantai and bioMérieux) (Table 2).
Discussion
In a longitudinal study of 76 HCW with RT-PCR-confirmed COVID-19, we found that the Wantai Total Ab assay had the best sensitivity over the course of the disease. In particular, the sensitivity reached and remained at 100% as soon as week # 3 post symptom onset. This finding observed in mild COVID-19 patients is consistent with previous reports of excellent sensitivity of this test notably in severe patients [3,12].
In addition to sensitivity, the ability of a commercial test to evaluate the protective immunity needs to be assessed. With this aim, Tang et al. compared three commercial assays (Roche Total Ab, Abbott IgG, both tests targeting the N protein, and Euroimmun IgG assays targeting the S protein) to VNT on 67 specimens [13]. The NPA of these tests was poor, ranging from 56% for Roche to 81% for Euroimmun, making them imperfect proxies for neutralization. These findings are highly consistent with those of the present study that found a NPA below 90% for all tests except for bioMérieux IgG and DiaSorin. In contrast, it is interesting to note that the Wantai Total Ab had the lowest NPA (33%). Furthermore, the concordance between VNT and the Wantai Total Ab assay was only moderate while the concordance was substantial with bioMérieux IgG, DiaSorin, Siemens, Abbott, Euroimmun and Bio-Rad. The low NPA and moderate concordance noticed for the Wantai Total Ab might be partially explained by the excellent ability of this test to detect RBD-specific antibodies at the very early phase of infection, irrespective of their neutralizing properties in line with the delay required for antibody maturation [18]. In the first study comparing VNT with commercialized tests, the authors found that the Wantai Total Ab assay had the best characteristics to detect functional antibodies in different stages and severity of disease [12]. However the median interval between the onset of symptoms and sample collection was 43 days for mild patients (n=71 samples) and thus the antibodies could be detected with both the Wantai Total Ab and the VNT assay at this time. As the authors used VNT as the gold-standard for sensitivity assessment, this explains the difference in findings with the present study [12].
Importantly, as previously reported by others [18,19], not all RBD-binding antibodies have neutralizing properties which is consistent with that reported herein regarding the RBD-based assays that do not have perfect concordance with VNT. Conversely, serological assays targeting a region other than the S protein may be associated with functional information, as previously reported [14,20–22]. In the present study, the Abbott and Bio-Rad assays directed against the N protein presented a substantial concordance with VNT as N-directed and RBD-neutralizing antibodies can be produced concomitantly over the course of the disease.
In addition to the different targeted antigens, the heterogeneity in assay performance found herein could also be explained by various factors including the detected isotypes. Moreover, antibody levels may also be very different according to the time since symptom onset and according to clinical severity of the disease [23]. Herein, serum samples were collected longitudinally from disease diagnosis enabling to explore the early phase of the antibody response in a cohort of HCW with mild symptoms, which constitutes one of the main strength of the present study.
The present study does, however, have certain limitations. For instance, specificity was not been extensively studied; yet the Euroimmun IgA assay seemed to have the worst specificity, which is consistent with previous studies reporting a lack of specificity for this assay [5,6,12]. In addition, the performance of other notable commercial assays such as Euroimmun IgG or Roche Ig Total were not assessed. Second, not all the samples were systematically tested by VNT, in-line with the labor-intensive nature of this method. Finally, the size of the tested population remains small, which limits the extrapolation of the results, although the present study represents the largest one comparing VNT to other serological tests through a longitudinal design.
The results presented herein obtained from mild COVID-19 patients confirm that, for exposure assessment, the Wantai Total Ab assay should be preferred to other commercial kits due to a very high sensitivity. For evaluating protective immunity, the Wantai Total Ab assay with an optimized cut-off or other tests targeting the S protein as Euroimmun, DiaSorin or bioMérieux IgG could be more useful, notably to screen serum specimens candidate for the presence of neutralizing antibodies. However, these tests or others cannot substitute a VNT for assessing functional antibody response; neutralizing assays remain the gold standard and easy-to-use tests, such as those based on pseudoviruses [6,17,24], should be developed and standardized. Furthermore, the recent development of surrogate virus neutralization tests based on antibody-mediated blockage of the interaction between ACE-2 receptor and the RBD is very promising as they were designed in an ELISA format enabling high-throughput testing [19,25].
In conclusion, the present study provides original data concerning the performance of widely-used serological tests, which could help diagnostic laboratories in the choice of a particular assay according to the intended use.
COVID-SER study group
Adnot Jérôme, Alfaiate Dulce, Bal Antonin, Bergeret Alain, Boibieux André, Bonnet Florent, Bourgeois Gaëlle, Brunel-Dalmas Florence, Caire Eurydice, Charbotel Barbara, Chiarello Pierre, Cotte Laurent, d’Aubarede Constance, Durupt François, Escuret Vanessa, Fascia Pascal, Fassier Jean-Baptiste, Fontaine Juliette, Gaillot-Durand Lucie, Gaymard Alexandre, Gillet Myriam, Godinot Matthieu, Gueyffier François, Guibert Nicolas, Josset Laurence, Lahousse Matthieu, Lina Bruno, Lozano Hélène, Makhloufi Djamila, Massardier-Pilonchéry Amélie, Milon Marie-Paule, Moll Frédéric, Morfin Florence, Narbey David, Nazare Julie-Anne, Oria Fatima, Paul Adèle, Perry Marielle, Pitiot Virginie, Prudent Mélanie, Rabilloud Muriel, Samperiz Audrey, Schlienger Isabelle, Simon Chantal, Trabaud Mary-Anne, Trouillet-Assant Sophie
Data Availability
All data are available in the manuscript
Author contributor’s statement
All authors were involved in the analysis and interpretation of data as well as drafting the manuscript or revising it critically for important intellectual content. AB, BP, VP, FG, JBF and STA made substantial contributions to the conception and design of the study and designed the experiments. BP performed VNT. MAT and VE performed the serological assay experiments. NG, AP, CA, AMP, AB, and JBF were involved in patient care, VP performed the data collection, STA, AB, MA and BP performed the data analysis. MR and CLJ performed the statistical analysis. AB and STA wrote the paper, BL, BP, JBF, AP, VE and MAT revised the manuscript content. All authors read and approved the final manuscript.
Conflict interests statement
Antonin Bal has received grant from bioMérieux and has served as consultant for bioMérieux for work and research not related to this manuscript. Sophie Trouillet-Assant has received research grant from bioMérieux concerning previous works not related to this manuscript. The other authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript.
Funding
This research is being supported by Hospices Civils de Lyon and by Fondation des Hospices Civils de Lyon.
Acknowledgements
We thank all the personnel of the occupational health and medicine department of Hospices Civils de Lyon who contributed to the samples collection. We thank Lucie Charreton and Khadija Sfouli for their excellent work concerning serological testing, all the technicians from the virology laboratory whose work made it possible to obtain all these data, as well as Amira Lachekhab and Naima Rolnin for their technical assistance. We thank Karima Brahima and all members of the clinical research and innovation department for their reactivity (DRCI, Hospices Civils de Lyon). We thank Philip Robinson (DRCI, Hospices Civils de Lyon) for his help in manuscript preparation.
References
