Abstract
The COVID-19 pandemic has increased the use of rapid antigen tests such as the Abbott BinaxNOW™ COVID-19 Antigen Self-Test. In winter of 2021-2022, the omicron variant surge made it quickly apparent that although rapid diagnostic tests (RDTs) are less sensitive than qRT-PCR, the accessibility, ease of use, and rapid read-outs of RDTs made them a sought after and often sold-out item at local suppliers. Here, we sought to qualify the BinaxNOW™ test for use in our university testing program as a method to rule in positive or rule out negative individuals quickly when they sought care at our priority qRT-PCR testing site. To perform this qualification study, we collected matched additional swabs from individuals attending this test site for standard of care qRT-PCR testing. All matched swabs were tested using the BinaxNOW™ RDT. 48 of 110 samples tested positive for SARS-CoV-2. All samples for which genome sequence could be collected were omicron (41). We observed a calculated sensitivity of 52.1%, specificity of 100%, a positive predictive value (PPV) of 100%, and a negative predictive value (NPV) of 72.9% for the BinaxNOW™ tests (n=110). Sensitivity was improved (75.7%) by changing the qRT-PCR positivity threshold from a CT of 40 to a CT of 30. The ROC curve shows that for qRT-PCR positive CT values between 23-40, the BinaxNOW™ test is of limited value diagnostically. Our results suggest that RDT tests could be used to confirm SARS-CoV-2 infection in individuals with substantial viral load, but that a significant fraction of infected individuals would be missed if we used RDT tests exclusively at this time.
Introduction
With the rising number of COVID-19 cases and the emergence of new SARS-CoV-2 variants, there has been an increased use of rapid antigen tests due to the longer turnaround time and lack of immediate availability of reverse transcriptase polymerase chain reaction (qRT-PCR) tests. Experience in the winter of 2020 suggested that 2021 fall/winter holiday travel would also lead to an increase in positivity rates (1). This prediction was confirmed when holiday travel coupled with the emergence of the omicron variant (B.1.1.529) initiated unprecedented levels of infection beginning in December of 2021. Omicron accounted for the majority of cases in the US a few weeks after it was first detected in the US on December 1, 2021 (2–4). The omicron surge has overwhelmed many existing qRT-PCR diagnostic sites, driving an increased use of rapid diagnostic tests (RDTs). These tests rely on viral antigen detection and were developed to recognize SARS-CoV-2 variants that existed before the appearance of the highly mutated omicron lineage.
The Abbott BinaxNOW™ COVID-19 Antigen Self-Test (BinaxNOW™, Abbott, Des Plaines, IL) has been at the forefront of rapid diagnostic testing throughout the pandemic. However, debate about the effectiveness of these tests in different use cases has led to efforts to track their specificity and sensitivity as the pandemic evolves (5–10). There has been a need for more studies to understand if variants, such as omicron, are being effectively detected using BinaxNOW™. At the time of this study, there has been an omicron limit of detection (LoD) dilution study (11) and a selective study focused on reported lower limits of detection (LLOD) CT value ranges for BinaxNOW™ (12). These studies suggested that there was analytical strength in the RDT, but also suggested a limited range of viral loads in which the assays consistently returned positive results from PCR positive samples.
Boston University Clinical Testing Laboratory (BU CTL) tests all members of the BU campus community at least once a week using qRT-PCR as reported previously (13,14). Symptomatic individuals can elect to get a test an any time and are directed to a special priority testing site at the BU Health Services Annex. As a continuing improvement exercise aimed at reducing cost and speeding turnaround time, we investigated whether the BinaxNOW™ test could be used effectively at this site. The investigation focused on whether RDT use was able to deliver results to patients attending our priority testing clinic t provided a rapid rule-in, rule-out of SARS-CoV-2 infection.
To assess the differential sensitivity for SARS-CoV-2 detection in qRT-PCR and BinaxNOW™ RDTs, we collected an additional swab from a series of individuals attending our test site for symptomatic individuals and those identified as close contacts by the clinical team. These extra swabs were tested in the lab using a BinaxNOW™ test. Results of these matched swabs were compared to qRT-PCR results. All positive samples were sequenced to determine the SARS-CoV-2 variant.
Methods
Sample Collection
Qualification testing of the BinaxNOW™ RDT was conducted on January 10th and January 12th, 2022. Participants in the qRT-PCR testing program were asked to give a second swab in a sequential manner as they arrived at the BU Health Services Annex for their scheduled appointment. Each participant provided one additional anterior nares (AN) swab immediately after they provided their initial swab for the routine qRT-PCR test. Individuals swabbed themselves while observed by the on-site testing personnel.
Individuals who test at this location can be symptomatic, a close contact of a positive individual, or a known positive individual coming for a confirmatory test. At this site, the qRT-PCR sample is collected using an ORAcollect•RNA swab (DNA Genotek, Inc, Ontario, Canada). The additional matched AN swab was taken using a Puritan Sterile Foam Tipped Applicator (Puritan, Guilford, Maine) swab that was placed into a dry 15 ml conical tube. A total of 220 paired AN swabs were collected from 106 individuals (4 individuals tested on both dates) as observed self-collections (110 ORAcollect•RNA for PCR and 110 Puritan for BinaxNOW™ tests). Samples taken for the RDT were stored at the end of each collection day at 4°C until being tested on January 13th with the BinaxNOW™ test. Since this was a qualification study, we prioritized and preferentially tested the qRT-PCR positive samples and 5 select negatives on January 13th and moved the qRT-PCR negative additional swabs to –80°C until they were processed on January 23rd. After the qualification study was finished, we received clearance from the BU Charles River Campus IRB to publish the results, as the work was ruled not human subjects research because we were evaluating the performance of the test and not accessing personal health information from the individuals (BU CRC IRB exemption #6402X).
qRT-PCR Testing
ORAcollect•RNA samples were processed, extracted, and tested by qRT-PCR at the BU CTL as detailed in Landaverde, et al. (13). The BU CTL test detects the N1 and N2 targets and uses RnaseP as a human RNA control. All of the qRT-PCR tests for this work were performed individually and none were pooled.
Abbott BinaxNOW™ COVID-19 Antigen Test
The additional matched AN swabs were tested according to the manufacturer instructions. Briefly, swabs were inserted into the BinaxNOW™ card, 6 drops of the provided buffer were added, the swab was rotated 3 times clockwise, and the card was sealed with the integrated adhesive strip. After 15 minutes, the test was read from the results window, and a photograph of the result was taken. A positive control provided in the kit was run on a separate test card to confirm the validity of the test kit.
SARS-CoV-2 Sequencing
Whole genome sequencing was performed on RNA extracted from all qRT-PCR positive samples using the excess discarded ORAcollect•RNA solution. Sequencing was performed using the Illumina COVIDSeq Assay (Illumina, San Diego, California) and sequenced on an Illumina NextSeq 500 (15). Full length genomes for each amplified sample were then assembled through alignment to the Wuhan-Hu-1 reference sequence (NC_045512.2) (16) using Bowtie2(17). Nucleotide substitutions, insertions, and deletions were identified with LoFreq (18) Lineage assignment for each genome was carried out using Pangolin(19).
Data Analysis
Data analysis was performed on MATLAB (MathWorks, Natick, Massachusetts). Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were determined for the BinaxNOW™ RDT.
Results and Discussion
Over two days of collection, 106 individuals (4 individuals tested twice) provided two swabs that were independently assessed by qRT-PCR or by RDT. A total of 48/110 matched swabs were positive for COVID-19 by qRT-PCR test. The CT values for the positive samples were between 11.7-38.8 for N1 and 11.7-38.8 for N2. The unused portions of these 48 positive samples were extracted and sequenced. Out of the 48 positive samples, 41 were successfully sequenced and 100% of the sequences were the omicron variant. Using the BU CTL qRT-PCR test as the gold standard, the BinaxNOW™ tests detected 25/48 positive results, and 62/62 of the negative results (Table 1, Supplemental Figure 1). Leading to a calculated sensitivity of 52.1%, specificity of 100%, a positive predictive value (PPV) of 100%, and a negative predictive value (NPV) of 72.9%.
Calculations of compared sensitivity here depend heavily on established criteria for positivity. For Boston University, the cut off for a positive result for the BU CTL qRT-PCR test is a CT ≤40 for one or both N1 and N2 targets. Other studies have suggested using a modified CT value that takes into account that individuals who have higher CT values may not transmit virus at that threshold (20,21).For example, Pilarowski, et al. use a CT cutoff of 30 for positivity (Supplemental Table 1), reasoning that people CT values above 30 are unlikely to transmit virus (5) Using that cutoff, our sensitivity would be 75.7%, specificity 100%, PPV 100%, and NPV 72.9%. In either case, the sensitivity is lower than the published 93.3% for BinaxNOW™ (5).
The BinaxNOW™ RDT returned variable test results at CT values between 23-26 for N1 and 24-26 for N2 (Figure 1). The mean CT value is 17.7 with a standard deviation of 4.0 for N1 and mean CT value is 18.2 with a standard deviation of 3.9 for N2 when both qRT-PCR and BinaxNOW™ are positive (Table 2). From the data, samples with a CT lower than 23 are highly likely to test positive on the BinaxNOW™ RDT. Our data is consistent with a smaller study on an omicron outbreak that paired saliva samples tested with RT-PCR and nasal rapid antigen tests (Quidel QuickVue™ At-Home OTC COVID-19 Test and Abbott BinaxNOW™ COVID-19 Antigen Self-Test)(22).
The BinaxNOW™ RDTs were positive for CT values from 11-26 for N1 and 11-26 for N2. The BinaxNOW™ RDT negative tests had CT values from 23-40 for N1 and 24-39 for N2. The average CT is 24 for N1 with a standard deviation of 8. The average CT is 23 for N2 with a standard deviation of 9. Within each qRT-PCR target N1 and N2, data is summarized in Table 2.
Figure 2 shows an ROC curve constructed from our dual testing data (Figure 2). The maximum Youden index for the ROC curve identifies where both sensitivity and specificity are at maximum (23,24). For this data set, a Youden Index of 0.98 corresponds to a CT of 23 for the qRT-PCR cycle cutoff. This result provided important information regarding our ability to implement BinaxNOW™, antigen based testing, as a surrogate for qRT-PCR testing at this time. A cutoff value of 23 for qRT-PCR is below the averaged CT value reported for both Delta and Omicron variants, suggesting that RDT testing alone would be insufficient to maintain control of spread in our community (Supplementary Table 2). Thus, the conclusion of our qualification study was that it is not possible to use the BinaxNOW™ test to replace surveillance qRT-PCR currently in either our symptomatic clinic or asymptomatic testing sites. In fact, we specifically chose our priority test site instead of one of our asymptomatic surveillance test sites to ensure enough high viral load swabs for qualification.
It was already known that RDTs, like BinaxNOW™, have a lower sensitivity compared to qRT-PCR for detecting SARS-CoV-2. The data presented here extends this conclusion to include the new omicron variant. As more data emerges linking transmission rates with viral loads, more solid conclusions will be made about the best use cases for RDTs. Until that time, it is prudent to be cautious about a negative BinaxNOW™ test, especially when symptoms are present.
Limitations of this study include the relatively small sample size. The disease prevalence at our clinic as measured by qRT-PCR on the two days of testing was approaching 50%, thus we are confident that the conclusions drawn from our sample size of 110 swabs are sound. It is also important to note that we asked individuals to swab each nostril twice, and the swab for the BinaxNOWTM test was always taken second. It is possible that less material was present on the swabs taken for the RDT, which could account for some of the lower sensitivity. Finally, we would like to note that four individuals tested on both days, and thus, our dataset includes 110 matched samples from 106 individuals. We do know that the 4 individuals who tested twice tested negative both times on both tests.
To be sure, tests like BinaxNOW™ are still valuable tools, as they provide immediate results, require no additional instrumentation, and are effective at rule-in diagnosis. However, there is still an unmet need for more sensitive rapid diagnostics for SARS-CoV-2 that could augment qRT-PCR testing at times of high-demand. Alternative approaches could include increased investment to increase the availability of qRT-PCR testing facilities.
Data Availability
All data produced in the present study are available upon reasonable request to the authors
Supplementary Information
Photographs of all BinaxNOW™ test cards. None of the images have been altered or adjusted, since these were taken only as a record of each test. They were taken with an iPhone 15 minutes after the test was started.
References
