Clinical assessment of the Roche SARS-CoV-2 rapid antigen test

Introduction

The ongoing coronavirus disease 2019 (COVID-19) pandemic outbreak is causing dramatic clinical, societal, and economic consequences all around the world. Despite the many public health strategies that have been implemented to face this challenge, including lockdowns, social distancing, widespread use of face masks and hand hygiene [1], data suggests that positive case identification, isolation and contact tracing are the most important factors for preventing further spread of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) [2]. While this strategy has been endorsed by many international and national organizations, evidence suggests that the current testing policy for diagnosing SARS-CoV-2 infections, mostly based on nucleic acid amplification tests (NAATs), appears to be failing [3], [4], [5], [6]. It is not readily apparent that the ideal SARS-CoV-2 testing volume cannot be reached in most countries worldwide, including the most industrialized nations such as the United Kingdom [3]. This generates a kaleidoscope of unfavourable consequences, including delayed diagnosis and late clinical management, which would then be associated with unfavourable disease progression [4], insufficient contact tracing and delayed isolation of SARS-CoV-2 positive cases, which in turn further contributes to propagate onward viral transmission [5]. Moreover, insufficient testing of individuals with high-risk exposures, most importantly, front-line healthcare workers, can lead to prolonged self-quarantine, jeopardizing health care delivery in conventional, urgent, and intensive care settings [6].

Due to the practical limitations of NAATs, encompassing a long turnaround time, low throughput and the need for specific instrumentation and skilled employees to conduct the test, rapid antigen immunoassay are emerging as a viable prospective for mass testing, as recently endorsed by the World Health Organization (WHO) [7]. Despite a reportedly lower diagnostic sensitivity compared to molecular testing, which would hence limit infection identification to subsets of patients with higher SARS-CoV-2 viral load, their widespread usage under well-defined circumstances may enable ample population screenings [8]. Nonetheless, analytical and clinical validation of these rapid immunoassays is a necessary preamble before their implementation within locally defined diagnostic algorithms. To this end, the purpose of this study was the clinical assessment of the new Roche SARS-CoV-2 Rapid Antigen Test.

Materials and methods

The Roche SARS-CoV-2 Rapid Antigen Test is an immunochromatographic assay for rapid qualitative detection of SARS-CoV-2 infection in nose and/or throat swabs. Briefly, the clinical specimen is collected and then deposited with twisting motion in a pre-filled extraction buffer container. After removing the swab, three drops of sample material are applied to the reagent tray (Figure 1). The presence of viral antigens in sufficient concentration enables their binding to specific mouse monoclonal anti-SARS-CoV-2 antibodies, then reflected by the appearance of a visual indication (i.e., a coloured line) in the lower section of the result window of the test strip, along with another “control” coloured line, which appears in the top section of the result window, when the device has been properly employed (Figure 1). The entire procedure can be completed within 15–30 min and does not require a dedicated environment (e.g., clinical laboratories), nor highly trained personnel to perform the test. According to manufacturer’s information, the limit of detection (LoD) per viral strain and the lowest concentration with uniform positivity per parameter are both 3.12×10^2.2 TCID50/mL, whilst the diagnostic sensitivity and specificity in nasopharyngeal swabs collected from both symptomatic and asymptomatic patients are reported to be 96.5 and 99.7%, respectively.

Figure 1:

Brief description of performance of Roche SARS-CoV-2 rapid antigen test.

The study population consisted of all consecutive patients referred for SARS-CoV-2 diagnostic testing to the Pederzoli Hospital (Peschiera del Garda, Verona, Italy), over a 2-week period (16–30 November, 2020). Upper respiratory specimens were collected in agreement with WHO recommendations [9]. A single swab (Virus swab UTM™, Copan, Brescia, Italy) was collected from each patient and concomitantly used for both Roche SARS-CoV-2 Rapid Antigen testing and molecular testing, which was performed using a commercial revere-transcription polymerase chain reaction (RT-PCR) assay (Seegene AllplexTM2019-nCoV Assay, Seegene, Seoul, South Korea). This method uses a volume of 350 µL and enables SARS-CoV-2 RNA identification by targeting three viral genes (N, E and RdRP), thus fulfilling internationally validated testing protocols [10]. Real-time PCR was interpreted using Seegene’s Viewer software. The viral load was finally expressed as cycle threshold (Ct), and test results with Ct values <37 for all three SARS-CoV-2 gene targets were considered “reactive” for SARS-CoV-2 RNA, thus specifically aimed at increasing the specificity of RNA viral detection, in accordance with current recommendations [11], and avoiding potential false negative test results due to emerging variants such as the VUI-202012/01 recently identified in the UK.

Quantitative and qualitative test results were presented as median with interquartile range (IQR) or percentage, respectively. Correlations between the Ct values of each SARS-CoV-2 gene were compared using Spearman’s rank correlation coefficient. The diagnostic efficiency of Roche SARS-CoV-2 Rapid Antigen Test for diagnosing SARS-CoV-2 infection was assessed by calculating the diagnostic accuracy, sensitivity and specificity, both cumulative and stratified according to NAAT Ct values. The statistical analysis was carried out using Analyse-it (Analyse-it Software Ltd, Leeds, UK) and MedCalc (MedCalc Software Ltd, Ostend, Belgium). The investigation was based on pre-existing specimens, already collected for routine SARS-CoV-2 diagnostic testing in the local facility, and thereby no patient’s informed consent, nor Ethical Committee approval were necessary. This study was conducted in accordance with the Declaration of Helsinki, under the terms of relevant local legislation.

Results

The final study population consisted of 321 consecutive patients (mean age, 46 years and IQR, 32–56 years; 181 women, 56.4%), who underwent SARS-CoV-2 diagnostic testing at the Pederzoli Hospital of Peschiera del Garda, Verona. Overall, 149/321 (46.4%) samples tested positive for SARS-CoV-2 RNA with the Seegene AllplexTM2019-nCoV Assay and 109/321 (34.0%) samples tested positive with the Roche SARS-CoV-2 Rapid Antigen Test, respectively. The median Ct values of positive samples were 25 (IQR, 21–32) for the E gene, 26 (IQR, 21–31) for the RdRP gene and 22 (IQR, 18–29) for the N gene, respectively. A high Spearman’s correlation was found between the Ct values of three gene targets, as follows: E vs. RdRP gene, r=0.98 (95% CI, 0.97–0.98; p<0.001); E vs. N gene, r=0.98 (95% CI, 0.97–0.98; p<0.001); RdRP vs. N gene, r=0.97 (95% CI, 0.96–0.97; p<0.001).

The distribution of Ct values is summarized in Table 1 and Figure 2. The overall accuracy of the Roche SARS-CoV-2 Rapid Antigen Test was 86.9%, with 72.5% sensitivity and 99.4% specificity. The best performance was found by comparison antigen test results with RdRP gene (86.0% accuracy, 71.0% sensitivity), followed by comparison with E gene (84.4% accuracy, 68.8% sensitivity) and N gene (80.7% accuracy, 63.9% sensitivity). A progressive decline in performance could be observed as the Ct values for the different SARS-CoV-2 gene targets increased. The sensitivity was found to range between 97–100% in clinical samples with Ct values <25, between 50–81% in those with Ct values between 25 and <30, but was as low as 12–18% in samples with Ct values between 30 and <37 (Table 1).

Figure 2:

Cycle threshold values of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) E, RdRP and N gene targets in samples with negative (Ag Neg) vs. positive (Ag pos) Roche SARS-CoV-2 rapid antigen test results.

Discussion

Although rapid, point-of-care antigen-based tests present a promising opportunity for mass (population) screening, and thereby for rapidly identifying, isolating and/or treating patients with SARS-CoV-2 infection, the reported analytical and diagnostic performance for many of the commercially available assays varies widely, as recently highlighted by the Cochrane COVID-19 Diagnostic Test Accuracy Group [12]. Although the cumulative specificity was found to be excellent, between 90–100%, the diagnostic sensitivity ranged between as low as 0%, to over 90%. As predictable, the diagnostic sensitivity was highly dependent on viral load, whereby ranged between 21–100% in samples with Ct≤25, but then decreased to 8–72% in those with higher Ct values. It is thus vital that thoughtful local assessment of analytical and clinical performance be performed before implementing any rapid antigen SARS-CoV-2 assay in routine COVID-19 diagnostics [13], [14]. In fact, only by precisely acknowledging the local diagnostic characteristics of the test, assessed on a target population within an environment with a specific population and prevalence of disease, would enable a suitable and reliable implementation of the test within specific protocols for SASR-CoV-2 diagnostics.

To the best of our knowledge, a previous study has assessed the clinical performance of the Roche SARS-CoV-2 Rapid Antigen Test. Briefly, Krüttgen et al. compared the qualitative test results of Roche SARS-CoV-2 Rapid Antigen Test with those obtained using the Real Star Sars-CoV-2 RT PCR Kit (Altona, Germany) on nasopharyngeal swabs collected from 150 patients admitted to the RWTH Aachen University (Germany), 75/150 (50%) of whom had a positive NAAT results [15]. The cumulative sensitivity and specificity of Roche SARS-CoV-2 Rapid Antigen Test were found to be 70.7% and 96.0%, respectively though, as observed in our results, the assay sensitivity was found to be highly dependent on the viral load present in the specimen. More specifically, Krüttgen et al. reported that the sensitivity was 95% or higher in samples with Ct <30, but then decreased to 45% and 22% in those with Ct between 30–35 and >35, respectively [15], thus comparing well with our findings on an almost double-sized population. This same rapid antigen test was also evaluated in a subsequent study, which analyzed its performance with either professional-collected or self-collected nasopharyngeal swabs [16]. Interestingly, the cumulative diagnostic sensitivity was confirmed to be between 70–80%, gradually decreasing in parallel with the reduction of viral load, but not differing significantly between the two collection modalities.

Taken together, the results of our study thus attest that the clinical performance of Roche SARS-CoV-2 Rapid Antigen Test are excellent in nasopharyngeal swabs with Ct values <25, which makes it a reliable screening test in patients with high viral load. Notably, the current recommendations from the World Health Organization (WHO) concerning the minimum antigen-detection performance are that these kits should have diagnostic a sensitivity and specificity of ≥80% and ≥97%, respectively [17]. Although the Roche SARS-CoV-2 Rapid Antigen Test does not completely match the WHO desired clinical performance, its cumulative sensitivity of 72%, which increases to 97–100% in specimens with Ct values <25, makes it one of the best commercially available assays for rapid SARS-CoV-2 antigen detection, equalling or even outperforming several other rapid SARS-Cov-2 antigen kits that are currently available on the market [18], [19], [20], and displaying diagnostic performance that are globally comparable to that of a rapid SARS-CoV-2 antigen chemiluminescence immunoassay [21].

Although the relationship between SARS-CoV-2 viral load and infectivity remains a matter of open debate, convincing evidence attests that SARS-CoV-2 positive subjects with high Ct values, especially when asymptomatic or if symptoms have disappeared by over 10–14 days, have relatively lower transmission potential [22]. Arons et al. demonstrated that the likelihood of obtaining positive SARS-CoV-2 cultures strictly depends on viral load, irrespective of the presence of clinical symptoms, whereby the positive rate was found to decline below 15% with Ct values ≥28 in pre-symptomatic patients who later developed a suggestive clinical picture [23]. Similar evidence has been reported in several other studies, such as those published by Basile et al. [24], La Scola et al. [25] and Jaafar et al. [26], who also concluded that the rate of positive SARS-CoV-2 cultures is inversely associated with viral load, achieving a virtually zero likelihood of a positive culture in clinical specimens with Ct values ≥33–34. Overall, a recent meta-analysis of published studies revealed a high heterogeneity in such Ct thresholds, with no growth ranging between Ct of 24 and 32 [27], thus paving the way to additional investigations aimed to more precisely define the correlation between NAAT results and viral culture of SARS-CoV-2.

In conclusion, the results of our clinical evaluation of Roche SARS-CoV-2 Rapid Antigen Test demonstrates that this method seems a suitable approach for rapid nasopharyngeal swab screening of patients with high SARS-CoV-2 viral load, especially those with symptoms or in the pre-symptomatic phase. However, mass population screening would still require the use of more sensitive techniques. A strategy based on more frequent testing shall also be envisaged when using rapid antigen testing rather than molecular assays, as recently endorsed by Mina et al. [28].